multimodal data modeling in MIL

[Meijian]

Feature

Not sure if it's already in the plan but it'd be great if we can incorporate multimodal data in MIL just like the regular train() dose.

Pitch

Incorporating multimodal data in image processing is increasingly important as clinical studies routinely collect multiple data types. It should help to improve the model's performance.

Alternatives

Additional context

[jamesdolezal]

Thank you, this is a great idea. I will add it to the feature list. It will get included in 2.1 or 2.2.

[Meijian]

Perfect! Thanks for considering it.

[jamesdolezal]

So we're not quite ready for multimodal integration with clinical variables, but in version 2.2 we did just release a multimodal MIL strategy that allows you to train models at multiple magnification levels simultaneously (see here). This has laid the groundwork for future expansion into other types of mode integration (such as clinical variables, gene expression etc). Multimodal integration with non-pathology modalities is our next priority - stay tuned!

[Meijian]

It's nice to see the progress. Thank you for your efforts, James!