context2 [How to Make a S'mores]

Release of a model trained with context length = 2

Full Changelog: checkpoint-all_tasks...checkpoint-context

Jirka: This model is trained on MAE modality ONLY!, passing in other modalities does nothing. It was trained on translation with context from previous 2 clips.

Jirka: I needed to patch the sign_public_api.py file to make it run. It's attached under this release.

This model behaves very similar to the first [Would you like some water?] model from Xuan. LLM without visuals works ok, with visuals tends to repeat one sentence indefinitely. The sentence usually does not resemble the embeddings. Embeddings are collapsed into 1-2 clusters kannst and _ptr.

Installation into Demo

To install this checkpoint into a set-up demo backend, just go to the demo/backend folder and run:

rm -rf models/Sign_LLaVA
git clone git@github.com:JSALT2024/Sign_LLaVA.git models/Sign_LLaVA
(cd models/Sign_LLaVA && git reset --hard 8734172efc7a27209fff6252675cdcc35dfb8277)
.venv/bin/python3 -m pip install --no-deps --editable ./models/Sign_LLaVA
rm models/Sign_LLaVA/llava/sign_public_api.py
(cd models/Sign_LLaVA/llava && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-context/sign_public_api.py")
rm -rf checkpoints/Sign_LLaVA
mkdir -p checkpoints/Sign_LLaVA
(cd checkpoints/Sign_LLaVA && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-context/context2.zip")
(cd checkpoints/Sign_LLaVA && unzip context2.zip)
mv checkpoints/Sign_LLaVA/context2/* checkpoints/Sign_LLaVA/

Then run the model test:

.venv/bin/python3 -m app.debug.test_sign_llava

And the test should end with:

...
The LLM says: "Here is the translation of the given ASL video into English:\n\nVideo Title: How to Make a S'mores\n\nHost: Hi, I'm here to show you how to make a s'mores.\n\nStep 1: First, you're going to need some graham crackers, some chocolate bars, and some marshmallows.\n\nStep 2: Next, you're going to need a fire. You can use a fire pit, a campfire, or even a fire in your backyard.\n\nStep 3: Once you have your fire going, you're going to need to toast your marshmallow. You can use a marshmallow roasting stick or a skewer.\n\nStep 4: Once your marshmallow is toasted, you're going to place it on top of your graham cracker.\n\nStep 5: Next, you're going to place a piece of chocolate on top of the marshmallow.\n\nStep 6: Finally, you're going to place another graham"
The result form the LLM seems ok.