panic with inference_session split_at

[spion]

When presenting / feeding certain tokens I'm getting:

thread 'main' panicked at 'assertion failed: mid <= self.len()', 
  /home/spion/.cargo/git/checkouts/llm-d8a8bbe144aa0546/e52a102/crates/llm-base/src/inference_session.rs:101:51

Steps to reproduce:

Feed the following string using feed_prompt

да гледам ќе идев

to any model that has llama like tokens.

Commit: latest master (e52a102)

[LLukas22]

Are you using the Huggingface tokenizer or the tokenizer implemented directly into rustformers?

[spion]

Oops, I forgot to meniton that really important bit. I'm using the HuggingFace tokenizer.

[philpax]

Okay, took a bit of time, but I managed to find the root cause. The end of the UTF-8 decoding of the tokens with the Hugging Face tokenizer can actually change as more tokens are added (including being a shorter string!), leading to panics when what's already been decoded doesn't match what the new decoding has provided.

I've requested help here: huggingface/tokenizers#1141 (comment) - hoping that we can get to the bottom of this soon!