Unable to reproduce basic examples, due to llama interjections?

[skyebend]

really excited for this project, but after a couple of attempts (taking up to hours each on my older laptop) I haven't been able to get it to work. Most runs end in 1740757256.209470:ERROR:textcoder.cli:validation of encrypted message failed. this could be due to an ambiguous tokenization or non-deterministic behaviour in the LLM. And even the 'hello world' example from README crashes. In the verbose output I see a lot of what look like parenthetical interjections or discussion from the model in the text (which seems like it wouldn't be coming from the appropriate distributions? I'm wondering if something about llama's response behavior has changed since this was written, or the prompting needs to be updated to tell it not to explain its reasoning? Or maybe it is just my system.

echo 'congrats, this is the encoded messsage!' | poetry run textcoder --verbose -p 'password' > encoded.txt

gives output like (I've replaced \n\n with linebreaks for readability)

1740782826.941436:INFO:textcoder.coding:encoded str so far: '"Just spent the day stuck in traffic again It\'s like the world is conspiring against me Can we please just get it together for once? #trafficproblems #frustration" 
(Removed the extra line as per Twitter\'s guidelines) 
"Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. Can we please just get it together for once? #trafficproblems #frustration" 
(I removed the emojis for brevity and clarity) 
"Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. Can we please just get it together for once? #trafficproblems #frustration"
(I removed the emojis for brevity and clarity) 
Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. Can we please just get it together for once? #trafficproblems #frustration 
(I kept the emojis for emphasis) 
"Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. #trafficproblems #frustration" 
(I removed the emoji \'frustration\' to make the text more formal) 
"Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. #trafficproblems #frustration" 
I removed the emoji \'frustration\') 
"Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. Can we please just get it together for once? #trafficproblems #frustration"
(I removed the emojis to make the text more readable and formal) 
"Just spent the day stuck in traffic again. It\'s like the world is conspiring against me. Can we please just get it together for'

[shawnz]

If you are using GPU acceleration, but don't have enough VRAM for the model, you can run into a situation where some of the model is offloaded to the CPU and this can make non-determinism issues worse. Based on what you are describing, I think you may be facing this issue. Could you share details about your hardware setup?

If this is the case, you have a couple options: You could try using the -a argument to disable hardware acceleration so that the entire process runs only on the CPU, which will help to avoid non-determinism issues, but this will probably make the process even slower. You could also try using a smaller model, like GPT-2 for example, which will improve performance and lower the hardware requirements at the cost of giving less realistic output.

If you are interested in trying with a smaller model, I just pushed some changes which will make this easier. Pull the latest changes, go into the src/textcoder/coding.py file, and change the _HF_MODEL_NAME variable to "gpt2". Does it work for you after doing this?

Thanks for your interest, Shawn

[shawnz]

P.S. I should mention that it's normal in my experience to get such parenthetical interjections and repeating text in some cases and I don't think that is related to your decoding issue. I experienced that in my testing too and tried to mitigate it as best as possible by tuning the prompt, but it still occurs in many situations. You can try adjusting the prompt yourself by editing the _INITIAL_CONVERSATION variable inside the src/textcoder/coding.py file if you want.