Modify `generate()` method in `GPT2CausalLM` to support chatbot

[chenmoneygithub]

The gap we have is about the end_token_id. In a chatbot system like DialoGPT , the user prompt needs to be concatenated by an end_token before generation, otherwise it will just generate end_token and stop. One simple fix is to add an argument to generate() method, e.g., append_end_token=False, when which is True the prompt will be appended by an end_token before calling our sampler.

@mattdangerw This should be an easy fix, but we need to make a call on the arg name and if we want to solve it at generate() level.

[kanpuriyanawab]

Sounds interesting!

[mattdangerw]

@chenmoneygithub what does the training data look like for DialoGPT? Is it "multi segment" in that inputs should be passed in as tuples and joined with an end token?

If that's the case our preprocessing offering doesn't really support that right now for GPT.

[TheAthleticCoder]

Hey!
According to the paper, the input training data is segmented and joined by an end-of-sentence token.

I think something which can be even more helpful to understand their data flow is in their official code, this example here which I tested out here. The colab link shared shows an example conversation of three turns where after every input given and generated sequence, an <eos> token is passed(in the code 50256).

[mattdangerw]

Thinking about this more, we need to take it on more broadly. It isn't really enough to cover handling generation() well for seq2seq to seek case, we should really be covering an entire fine-tuning and inference flow.

In some ways, this would be easiest if we could do the special token packing in string space, because we know that in some cases that is what people will want to do...

imdb_ds = load_imdb()
causal_lm = keras_nlp.models.GPT2CausalLM.from_preset("gpt2_base_en")
causal_lm.fit(imdb_ds)
causal_lm.generate("That movie was")

cnn_dailymail_ds = load_cnn_dailymail()
combined_ds = cnn_dailymail_ds.map(
    lambda s: tf.strings.join(s["article"], "TL;DR", s["highlights"]),
)
causal_lm = keras_nlp.models.GPT2CausalLM.from_preset("gpt2_base_en")
causal_lm.fit(combined_ds)
causal_lm.generate("The situation in Congress [...entire paragraph...] TL;DR")

chat_response_ds = load_chat_response()
combined_ds = chat_response_ds.map(
    lambda s: tf.strings.join(s["context"], "<|endoftext|>", s["reply"]),
)
causal_lm = keras_nlp.models.GPT2CausalLM.from_preset("gpt2_base_en")
causal_lm.fit(combined_ds)
causal_lm.generate("What is the capital of Argentina? <|endoftext|>")

[chenmoneygithub]

@mattdangerw I agree, it's also possible that users want to convert dialogue training data to something like "<|endoftext|><|endoftext|><|endoftext|>", doing which on string space is much easier.

[mattdangerw]

Worth testing out, but I believe the workflows above #844 (comment) should now be working. Tentatively closing this, but we can reopen if needed.