Compatibility issues

[ziw-liu]

The models we report in the preprints were trained with different versions of the codebase:

• VSCyto3D and the model used in the Mantis preprint was trained with v0.1.0a0.
  • Old config files are not compatible with the current HEAD of main (d7d1200)
  • Weights can be loaded through Python API.
• VSNeuromast was trained with the current HEAD of main (after v0.1.0a1?)
• VSCyto2D was trained with 2D FCMAE #71.
  • Weights are not compatible with main due to architectural changes.

The VSCyto3D weights can be provided with an updated config file, while #71 will need to add API changes to separate 2D model building and also change config files accordingly before merging.

[ziw-liu]

@edyoshikun can you confirm the details for the neuromast model and comment on compatibility requirements of the mantis revision?

[edyoshikun]

Correct! Neuromast model was trained with the current HEAD of the main. I confirm.

[mattersoflight]

Hi @ziw-liu We'd fine-tune the VSCyto3D model with data acquired on Mantis as we discussed at the lab meeting yesterday. Please start a PR with changes that enable training of both VSCyto3D and VSNeuromast models with the main. This PR should be a priority since we need it to iterate on the models for mantis Fig. 1 next.

It seems like 2D FCMAE branch will require development such that the weights are compatible with the main. I don't follow this point, can you point to a few lines of code to illustrate? This can be the second PR.

Thanks for iterating quickly on these.

[ziw-liu]

We'd fine-tune the VSCyto3D model with data acquired on Mantis as we discussed at the lab meeting yesterday.

I think @edyoshikun is already training these models without needing code changes. (Recall that VSCyto3D and VSNeuromast are the same architecture trained on different data.) The incompatibility I mentioned in OP for VSCyto3D is between the old config files we published (e.g. on zenodo for the mantis preprint) and the current code base. This is a minor issue since that zenodo record points to a pinned version of VisCy that works properly with the files.

[ziw-liu]

It seems like 2D FCMAE branch will require development such that the weights are compatible with the main. I don't follow this point, can you point to a few lines of code to illustrate?

The network module changed, among other things. For example:
https://github.com/mehta-lab/VisCy/pull/71/files#diff-9af609b0fdd9a3358d7099a616e29f1fefdab379a92388bca856ab99944fc67c

[mattersoflight]

@ziw-liu How about the following order of operation?

• Create a release candidate 0.2.0 that works with current weights of VSCyto3D, VSNeuromast, and VSCyto2D models. My reading of the code suggests that all of these can be instantiated by passing parameters to UNet21d. You can rename this model UNeXt2 to match the nomenclature in the preprint.
• Update FullyConvolutionalMAE model such that the weights can be transferred from FCAMAE to VSCyto2D models.

[ziw-liu]

My reading of the code suggests that all of these can be instantiated by passing parameters to UNet21d.

The VSCyto2D models are made from the FCMAE class for both pre-training and fine-tuning.

My suggestion is:

1. Merge 2D FCMAE #71
2. Merge Rename UNeXt2 #84
3. Merge Test on Python 3.12 #88
4. Tag v0.1.0rc0
5. Prepare v0.1.0 release artifacts (weights and config files) based on the new code base

[mattersoflight]

@ziw-liu. This order of operation makes sense to me. Are you renaming what we thought of release 0.2.0 as 0.1.0? Totally fine to call the next release 0.1.0. Please update the milestone for clarity if you mean something else.

[edyoshikun]

Let's also update the Readme.md and add the preprint link here as well.

[ziw-liu]

Let's also update the Readme.md and add the preprint link here as well.

#85