Added infograph model finetuning support

[arunppsg]

Description

Added support for infograph model finetuning task.

Type of change

Please check the option that is related to your PR.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • In this case, we recommend to discuss your modification on GitHub issues before creating the PR
• Documentations (modification for documents)

Checklist

• My code follows the style guidelines of this project
  • Run yapf -i <modified file> and check no errors (yapf version must be 0.32.0)
  • Run mypy -p deepchem and check no errors
  • Run flake8 <modified file> --count and check no errors
  • Run python -m doctest <modified file> and check no errors
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New unit tests pass locally with my changes
• I have checked my code and corrected any misspellings

[arunppsg]

@tonydavis629 , just want to check in - does it make sense to add finetuning support for InfoGraph model?

From my understanding, the infograph paper performed two kind of experiments:

• exp 1: combining the supervised objective (finetuning) with the unsupervised pretraining objective (InfoGraph)
• exp 2: unsupervised infograph pretraining with semi-supervised InfoGraphStar for finetuning.

Am I right in the understanding here?

With the current setup, we can do exp 2 and I believe this code adds support for exp 1. But is there a better way to do it?

[tonydavis629]

@arunppsg I believe this change is redundant, the functionality to do pretraining and finetuning is already there in InfoGraphModel and InfoGraphStarModel. The difference being that this code uses 1 combined InfoGraph model for finetuning and pretraining, while the current implementation uses InfoGraph for pretraining and InfoGraphStar for finetuning.

The 2 experiments done in the paper are 1. unsupervised mutual information maximization between a global and local encoded graph representation (with no finetuning or supervised objective) and 2. layer by layer mutual information maximization between a trained encoder and untrained encoder plus a supervised loss (unsupervised objective + supervised objective being referred to in the paper as semi-supervised).

So the InfoGraphModel is reserved for this #1 unsupervised task to train the encoder. InfoGraphStarModel is to be used for #2 by loading the weights of pretrained InfoGraph into the encoder. Your InfoGraphFinetune functionality is very similar to InfoGraphStarModel. This pretrain + finetune regime is implemented in test_infograph_pretrain_overfit test, and is again similar to test_infograph_pretrain_finetune.

[arunppsg]

I agree that InfoGraph* does a semi-supervised finetuning and infograph doing a unsupervised learning to train an encoder. From the table 2 in the paper, they have two results: results from InfoGraph model finetuned via supervised setting and results from InfoGraph* finetuned via semi-supervised learning approach. In this pull request, I am proposing an approach for finetuning the encoder in a supervised setting. It it will be useful, we can merge it in else we can close it.

[tonydavis629]

Figure 2 shows only InfoGraph*, while figure 1 shows InfoGraph. Those 2 approaches were implemented with InfoGraphStarModel and InfoGraphModel. Finetuning the encoder with a supervised dataset is already possible with InfoGraphStarModel. The difference I see in your implementation is that you've combined both into a single model, which may be more convenient, but otherwise I believe it is the same functionality.

[arunppsg]

I leave it to @rbharath for final call on whether it will be useful or not to users.

[rbharath]

This is redundant with InfographStar, but I think it could be nice for users since our other models allow for pretraining/finetuning in the same model and it's convenient to have the same for infograph.

@arunppsg Let me know once all tests are passing and this is ready for my full review

[arunppsg]

This is ready for review and tests are passing.

[rbharath]

LGTM

[rbharath]

Can you remove this before merging in? Looks like cruft

[arunppsg]

sure, forgot to remove. removed it.

[rbharath]

These logger changes are also reflected in your other PR. I'm fine merging them in as part of this PR since they are small