Questions regarding the code

[dimeldo]

1. What does this line check?

if sum([len(item) for item in batch[0][1]]) > 1024:

2. What is the maximum number of turns a dialogue can have? Or is it set by maximum length a dialogue can have? If so, where is it specified? I saw a number of constants number who can be contenders for that:

train_data = [data[idx] for idx in indices[100:]]
val_data = [data[idx] for idx in indices[:100]]

self.tokenizer.max_len = 1500
        # tokenizer weird behavior

[qywu]

1. Since GPT2 only supports sequences shorter 1024 tokens, this line skips dialogs longer than that. Though depending on your dataset, you can use a fix-sized window to extract dialogs.

2. Yes. It is set by the maximum length. If your dialog is longer than that, you can use a window to sample from the dialogs.

[dimeldo]

Thanks for answering.

1. Does it check if there’s a dialogue longer than 1024 in the batch, and if so, it skips the entire batch? I thought it’s this way since I saw my GPU not being utilized fully even on 128 batch size. Or is it just skipping per dialogue? Or is checking the sum of all the length of dialogues in a batch and then skipping them all if that’s higher than 1024?

2. What’s the 100 and the 1500 constant numbers represent then?

Please, excuse my lack of understanding. Thanks again.

[qywu]

The batch size is forced to be 1 now, which probably consumes all the memory of 11GB GPU. However, I have included batch support in the actual implementation. You can check out camrest and multiwoz. If there is a dialogue longer than 1024, it will skip the dialogue (batch size=1).

100 means dataset split. The first 100 dialogues are used for validation. 1500 is maximum length of your dialogue. It is only used for the tokenizer to tokenize your dialogue.

[dimeldo]

Okay, thanks a lot!