LLaVA predictions are always empty

[icrto]

Thanks for the awesome paper and repo!

I was trying out LLaVA and noticed that the model predictions were always empty strings, and was able to narrow it down to this line

VL-ICL/utils/model_inference.py

Line 76 in 6ad043d

predicted_answers = tokenizer.batch_decode(generated_ids[:, input_token_len:], skip_special_tokens=True)[0]

.

It seems LLaVA already outputs only the tokens it generated, and not the whole context tokens + generated tokens.

The fix is quite easy, just decode the whole generated sentence, without truncating it first with the input_token_len.

Is my thinking correct or am I missing something?

[ys-zong]

Hi, thanks for your interest. I think that's due to different versions of Llava - the older version I used when developing this repo seems to need this slicing (I'll need to double check it). But if decoding the whole generated sentence works for you, that's totally fine to just go without truncation.

[icrto]

Ah I see thanks!