General thoughts - Models, Loss functions, Applications

[tylerstahl]

A few general thoughts. The benefits and tradeoffs of differentiable relaxations probably depend on a number of factors - model, loss function, specific goals, etc.. The loss function is always a huge barrier in biology; there is rarely a single objective with a source of ground truth. It is only through human interpretation of the data that such labels of truth are often applied/verified. This is in essence what bioinformatics is - turning a readout into a human interpretable format. So creating a differentiable bioinformatics pipeline becomes a bit of a catch 22.

We have seen "pockets" of differentiation in the transformation process where ground truth is easily defined. For example, Dorado is a neural network that takes electrochemical signal from Nanopore Sequencing and converts the signal to basecalls (A,C,G,or T). Here, you can construct a synthetic molecule with a known sequence to train the neural network.

Google's DeepVariant did something similar training a CNN on alignment (bam) files with known genotypes.

On the margin, it is uncertain how much end-to-end differentiation improves model performance. While some cases it might make sense to train on purely the raw data, maybe it can act as some form of regularization?

Then there is also always the considerations of compute time/cost/memory.

On a related note, I came across this paper using backpropagation for differential data cleaning and feature selection. Given the heterogeneity in biological data, this might be interesting angle as well. I previously wrote a short blurb on this here. Alternatively, this article does a really great job of breaking down bio AI models - including context scarcity, noise, and diversity.

Curious any additional thoughts. Thinking through use cases.

[mahdi-shafiei]

check out https://github.com/avitai/DiffBio/tree/main/paper for a first attempt. This is very very shallow at the moment and I am hoping to find more use-cases that are closer to the raw data and undergo a complex pre-processing pipeline with a lot of parameters that needs careful specification and are currently hand-tuned.