Speed data reader for IMDB dataset.

[qingqing01]

Fix #7001

• Test Env:

  • model config: https://github.com/dzhwinter/benchmark/blob/master/fluid/understand_sentiment_dynamic_lstm.py
  • CPU, 2 epoc

• Total Time:

  • 109.56469s -> 51.86136s

[chengduoZH]

buffer_size becomes useless.

[dzhwinter]

buffer_size is never used. Even in the previous experiment, people only set the shuffle buffer.

[qingqing01]

Done. Remove the buffer size. And I test the time for whether to use two threads.

• Not use two threads: 16.65757s
• Use two threads: 25 - 27s. I'm not sure why this is slower, the code is as follows:

def reader_creator(pos_pattern, neg_pattern, word_idx, buffer_size):
    start_time = time.time()
    UNK = word_idx['<unk>']

    POS = []
    NEG = []

    def load(pattern, out, label):
        for doc in tokenize(pattern):
            out.append(([word_idx.get(w, UNK) for w in doc], label))

    # Creates two threads that loads positive and negative samples
    # into qs.
    t0 = threading.Thread(
        target=load, args=(
            pos_pattern,
            POS, 0, ))
    t0.daemon = True
    t0.start()

    t1 = threading.Thread(
        target=load, args=(
            neg_pattern,
            NEG, 1, ))
    t1.daemon = True
    t1.start()

    t0.join()
    t1.join()

    INS = POS + NEG
    random.shuffle(INS)
    duration = time.time() - start_time
    print('\nTotal time: %.5f ' % (duration))

    def reader():
        for doc, label in INS:
            yield doc, label

    return reader

[dzhwinter]

Great enhance

[reyoung]

Excellent jobs. Thanks