[Feature Request] Stream tokens to the console as soon as they are generated

[swolchok]

This would significantly increase perceived performance IMO, because you can read the message as it's generated (at above human typing speed) rather than waiting until it's all done to start reading.

[mikekgfb]

This is for chat? we used to have callback() to print as we go, at least for chat?

cc: @JacobSzwejbka

[swolchok]

This is for chat

arguably we should do it for the regular generate flow if stdout is a terminal

[swolchok]

perhaps even if stdout is not a terminal, looks like it wouldn't hurt anything.

[JacobSzwejbka]

So you just want it to be immediate instead of batches of 4 @swolchok ?

[JacobSzwejbka]

Oh this is for generate. We can just use this callback here https://github.com/pytorch/torchchat/blob/main/generate.py#L654 in more cases

[metascroy]

@JacobSzwejbka are you taking a look at this?

[metascroy]

PR here: #571