data size of enfr_wmt32k dataset

[apeterswu]

Hi,

I use t2t-data_gen to download and process the enfr_wmt32k dataset, but find the dataset is about 40M, not 36M instead. Does it mean that I still need to further preprocess like filtering data? If yes, how to do the further preprocess? Thanks.

[gsoul]

Hey, that's the excellent question. I'll be happy to listen to the answers from the experts.

On my side I can only mention that some time before I found such reply to this question in FairSeq issue queue: facebookresearch/fairseq#59

Hope this may help you.

[apeterswu]

@gsoul Thanks, I think according to fairseq prepocess, the dataset will be 34M instead of 36M either. So I think we still need to wait for experts' answers.

[lukaszkaiser]

We don't do any further pre-processing in our experiments. Note that things like "remove sentences longer than 175 words" may happen during training if you set max_len to that value. But the results, e.g. in Attention is All You Need (https://arxiv.org/abs/1706.03762) are without any other pre-processing.

[JinjieNi]

@lukaszkaiser Hey, actually you still didn't answer why in Attention is All You Need (https://arxiv.org/abs/1706.03762) the training set claims to be 36M, while the actual size (before preprocessing) is 40.8M. Would you kindly help to explain that?

[JinjieNi]

@Psycoy PS: In most papers, they would claim that the WMT2014 ENDE's training set has the size of 4.5M and WMT2014 ENFR has the size of 36M. While the actual sizes of the downloaded training sets are 4.5M (3.9M after preprocessing) for WMT2014 ENDE and 40.8M (35.8M after preprocessing) for WMT2014 ENFR respectively.

It seems that the size of WMT2014 ENDE before preprocessing and the size of WMT2014 ENFR after preprocessing are the numbers that are claimed by most of the papers, which is strange because the papers should have reported the sizes under the same conditions (both not preprocessed or both preprocessed).
Here "preprocessing" means deleting the data samples that exceed a certain length, which will make the set smaller.

Here are the corpora I used for the two training sets (WMT2014 ENDE and WMT2014 ENFR), which is also the same as WMT2014 homepage:

WMT2014 ENDE:

1. Europarl v7
2. Common Crawl corpus
3. News Commentary

WMT2014 ENFR:

1. Europarl v7
2. Common Crawl corpus
3. UN corpus
4. News Commentary
5. giga French-English corpus

Any problems?
Can anyone explain it?