keywords [Please go ahead]

Release of a model trained on one_keyword_included 0.9, translation 0.1 - mae only.

Full Changelog: checkpoint-all_tasks...checkpoint-keywords

Jirka: This model is trained on MAE modality ONLY!, passing in other modalities does nothing. It was trained on mostly 1-keyword task and a little bit of translation.

Jirka: I needed to patch the sign_public_api.py file to make it run. It's attached under this release.

Installation into Demo

To install this checkpoint into a set-up demo backend, just go to the demo/backend folder and run:

rm -rf models/Sign_LLaVA
git clone git@github.com:JSALT2024/Sign_LLaVA.git models/Sign_LLaVA
(cd models/Sign_LLaVA && git reset --hard 8734172efc7a27209fff6252675cdcc35dfb8277)
.venv/bin/python3 -m pip install --no-deps --editable ./models/Sign_LLaVA
rm models/Sign_LLaVA/llava/sign_public_api.py
(cd models/Sign_LLaVA/llava && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-keywords/sign_public_api.py")
rm -rf checkpoints/Sign_LLaVA
mkdir -p checkpoints/Sign_LLaVA
(cd checkpoints/Sign_LLaVA && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-keywords/mae_keywords09_lowlr.zip")
(cd checkpoints/Sign_LLaVA && unzip mae_keywords09_lowlr.zip)
mv checkpoints/Sign_LLaVA/mae_keywords09_lowlr/* checkpoints/Sign_LLaVA/

Then run the model test:

.venv/bin/python3 -m app.debug.test_sign_llava

And the test should end with:

...
The LLM says: "Please go ahead and provide the ASL video, and I'll do my best to translate it into English."
The result form the LLM seems ok.