2024-07-31-overfit [drill that arms drill]

Release of overfit model - all modalities.

Jirka: This model was trained with all modalities and exclusively on the video attached below, so that it would overfit and predict what's expected of it for that one video. In this model, the text-embedding layer overfitted so much, that the video is translated roughly properly even without any prompt and also asking any other prompt (e.g. "What is you purpose?") will not even repeat the system prompt. The LLM itself is broken. But it does translate the video somehow (coz it overfitted).

Installation into Demo

To install this checkpoint into a set-up demo backend, just go to the demo/backend folder and run:

rm -rf models/Sign_LLaVA
git clone git@github.com:JSALT2024/Sign_LLaVA.git models/Sign_LLaVA
(cd models/Sign_LLaVA && git reset --hard 658c608105337d9691a459b68f220bb3175d7a0b)
.venv/bin/python3 -m pip install --no-deps --editable ./models/Sign_LLaVA
rm -rf checkpoints/Sign_LLaVA
mkdir -p checkpoints/Sign_LLaVA
(cd checkpoints/Sign_LLaVA && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-overfit/overfit.zip")
(cd checkpoints/Sign_LLaVA && unzip overfit.zip)

Then run the model test:

.venv/bin/python3 -m app.debug.test_sign_llava

And the test should end with:

...
The LLM says: 'drill that arms drill that drill that will get drill that will get drill that will get'
The result form the LLM seems ok.