Wiki Chinese dump preprocessing: #lines not matching

[LiweiPeng]

The BERT Chinese model was trained from wikipedia dump. According to #155 , BERT original model got 25M lines/sentences after pre-processing.

However, I got 12.5M lines(sentences) after pre-processing. Can someone let me know what could be wrong with my steps?

What I did:

• download https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2
• use WikiExtractor.py to extract the paragraphs. I got total 14.7M lines.
• use a script to split the paragraphs into sentences. Ignore empty lines in wiki extracts. Add new line for wiki only. I got total 12.5M lines.

[LiweiPeng]

I found the reason for my issue. I need to include both simplified and traditional Chinese versions. That'll be totally 25M.

[VictorSanh]

The BERT Chinese model was trained from wikipedia dump. According to #155 , BERT original model got 25M lines/sentences after pre-processing.

However, I got 12.5M lines(sentences) after pre-processing. Can someone let me know what could be wrong with my steps?

What I did:

• download https://dumps.wikimedia.org/zhwiki/latest/zhwiki-latest-pages-articles.xml.bz2
• use WikiExtractor.py to extract the paragraphs. I got total 14.7M lines.
• use a script to split the paragraphs into sentences. Ignore empty lines in wiki extracts. Add new line for wiki only. I got total 12.5M lines.

Hey @LiweiPeng,
I am not sure where to get the simplified Chinese version of Wikipedia.
I can't find the code (like zh) for simplified Chinese, or do I need to use an automatic converter?

Victor

[LiweiPeng]

@VictorSanh I used the opencc tool (https://github.com/BYVoid/OpenCC) to convert all wikipedia Chinese text into simplified Chinese version and traditional Chinese version. Then use BERT to train both versions together.

I did other tests where only simplified Chinese wiki are used. The results are not as good as using both simplified and traditional for XNLI dataset.

[VictorSanh]

Thank you @LiweiPeng for you answer!
Just to make sure, which config files (the json files) are you calling? t2s.json?
Did you use the Taiwan standard?

[LiweiPeng]

t2s json to convert to simplified Chinese. s2t.json to convert to traditional Chinese.

[VictorSanh]

So you are applying t2s.json to the output of wikiextractor to get the simplified Chinese version and then reapply s2t.json to the latter to get the traditional Chinese version (i.e. two steps process)?

[notabigfish]

her tests where only simplified Chinese wiki are us

Hi, this method works for me, thanks! But I have a few questions. Did you split tfrecord
into train and test?
Here is what I did after generating simplified and traditional chinese text files:

1. concate two txt files using "cat simplified.txt traditional.txt > all.txt"
2. generate tf record files using

python create_pretraining_data.py \
  --input_file=data/download/wikicorpus_zh/all.txt \
  --output_file=data/download/wikicorpus_zh/all.tfrecord \
  --vocab_file=data/download/google_pretrained_weights/chinese_L-12_H-768_A-12/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=5

In this way, I could train using all.tfrecord as training set. However, no test or eval set is generated. May I ask, how did you split zhwiki into different sets? Thank you!