Generation framework should be tokens in tokens out

[parthchadha]

Describe the bug

To avoid discrepancies of different tokenizer versions between training and generation framework, generation framework should just get the tokens in, special token_id's (ex: stop, eos token) and it should return just tokens.

Steps/Code to reproduce bug

The current vllm implementation has reference to the tokenizer.

The fix for this issue should include a test which ensures we don't add tokenizer to any generation backend.

[SahilJain314]

I believe we've more or less made peace with the amount of tokenizer involvement in vllm here. Can we close this @parthchadha ?

[parthchadha]

Yes, closing this issue.