2024-04-07_Ufert_Homework_Session_8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use cases for LLMs in Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LLMs are designed for dealing with sequential data. Scientific examples of sequential data are DNA sequences or data series. Furthermore, LLMs generate output by chosing the most-likely token to extend the sequence it works on, among all tokens that are part of its vocabulary. In doing so, LLMs up to to date have no capability to assess or even only estimate the correctness of the output they generates. Both constraints the realm of scientific applications LLMs can meaningfully be deployed for. Another important factor is how close the final use case will be to the data the model has seen during training, or, in other words, is the model supposed to interpolate or extrapolate. <br>\n",
    "\n",
    "Based on these considerations, I believe there are two broad categories of tasks LLMs are well-suited for in science: Filling in missing parts in data series, and for creative tasks upon which's accuracy is not relied. For example, there might be analytical techniques in medicine or biology that are difficult, expensive, or which are error-prone or associated with high uncertainties. In that case and if good training data is available which spans the domain the model should be applied to, an LLM that ompletes/corrects the sequence to what it most likely is could be very beneficial. Examples for creative tasks are the prediction of possible mutations of a virus, or prediction of candidates for new antibiotics to counter microbial resistance. Another example is the generation of initial guesses for a simulation run that then uses first-principles based models to predict how a time series evolves. Of course, LLMs can also be useful for any applications for which no first-principles based model exists yet, or is simply not tractable with today's computational capacity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Evaluate LLMs in these Use Cases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of LLMs designed to fill in or correct missing or corrupted data can be tested using high quality data that is artificially augmented or part of which are purposefully omitted. The LLM would then be tasked to reconstruct the data, and the model prediction can be compared to the orignial data. By evaluating the model accuracy for different degrees of data augmentation or corruption, it may also be possible to identify under what conditions the LLM performs well, and in which cases it fails. For creative tasks, several factors may be evaluated. For new antibiotics, for example, the creativity/variety in the compounds it generates may be of interest. Also, other properties such as toxicity could be evaluated, or if the compounds can be synthesized. For the generation of initial guesses, it would be useful to see how close the generated initial guess is to the actual solution once the first-principles based model converged, or by how much these initial guesses can speed up convergence of the rigorous model. In cases for which no rigorous model exists yet, an expert would have to evaluate the prediciton of the model for plausibility."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "datascience/conda-2023-01-10",
   "language": "python",
   "name": "conda-2023-01-10"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}