all_tasks [Let's get started]

Release of a model trained on all tasks - mae.

Jirka:. This model is trained on the MAE modality ONLY!, passing in other modalities does nothing. It was trained mostly on keyword detection.

Jirka: I needed to patch the sign_public_api.py file to make it run. It's attached under this release.

Installation into Demo

To install this checkpoint into a set-up demo backend, just go to the demo/backend folder and run:

rm -rf models/Sign_LLaVA
git clone git@github.com:JSALT2024/Sign_LLaVA.git models/Sign_LLaVA
(cd models/Sign_LLaVA && git reset --hard 658c608105337d9691a459b68f220bb3175d7a0b)
.venv/bin/python3 -m pip install --no-deps --editable ./models/Sign_LLaVA
rm models/Sign_LLaVA/llava/sign_public_api.py
(cd models/Sign_LLaVA/llava && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-all_tasks/sign_public_api.py")
rm -rf checkpoints/Sign_LLaVA
mkdir -p checkpoints/Sign_LLaVA
(cd checkpoints/Sign_LLaVA && wget "https://github.com/JSALT2024/Sign_LLaVA/releases/download/checkpoint-all_tasks/all_tasks-7851619.zip")
(cd checkpoints/Sign_LLaVA && unzip all_tasks-7851619.zip)
mv checkpoints/Sign_LLaVA/all_tasks-7851619/* checkpoints/Sign_LLaVA/

Then run the model test:

.venv/bin/python3 -m app.debug.test_sign_llava

And the test should end with:

...
The LLM says: "Let's get started."                                   
The result form the LLM seems ok.