multi-turn dialog format

[LHolten]

Section 3.1 of the paper states that dialog turns of the same session are concatenated into a long text, ended by the end-of-text token.

Does this mean that there are no special tokens in between dialog turns?

How do I separate dialog turns?

[dreasysnail]

There ARE special tokens (<|endoftext|>, id=50256) between dialogue turns in multi-turn setup. Your input format should be like this:

Turn1 <|endoftext|> Turn2 <|endoftext|> ... TurnN

Let us know if you have any further concerns.

[LHolten]

Thanks for the clarification, that makes sense.

[LHolten]

What are these token_type_ids for?

DialoGPT/prepro.py

Line 107 in 5688864

token_type_ids += [i] * (len(s) + 1)

It looks like they are different for the different turns in a session and get passed to the model while training.
This would be very weird as the GPT2 model from the transformers library converts token_type_ids as if they are regular input_ids and uses the same embedding for them.

[intersun]

Yes, they are different for different turns when we prepared the data. I believe we did some experiments about it, and end up using the default token_type_id (by setting token_type_id = None) during training.

[LHolten]

I see it now:

DialoGPT/LSP_train.py

Lines 279 to 280 in 5688864

	if args.no_token_id:
	token_ids = None

[abisee]

Turn1 <|endoftext|> Turn2 <|endoftext|> ... TurnN

@dreasysnail: should there be spaces around the <|endoftext|> or not?

[abisee]

@dreasysnail, was DialoGPT trained with any kind of padding (e.g., if the entire dialogue doesn't fill up the max length)? Or did the multi-turn dialogue always fill up the entire max length (as in GPT2 training)?