Non deterministic NLU training on GPU

[fparga]

Rasa version: 1.9.4

Python version: 3.6.9

Operating system (windows, osx, ...): linux

Issue:
NLU training on GPU is non reproducible, each training with the same pipeline and the same train set gives a model that perform differently. I get that GPU training is by nature non-deterministic (although I think some progress have been made on that front), but the variations we see here are significant and makes it really hard to compare the performance impact of pipeline configuration/parameter tuning.

We don't have this problem when training on CPU.

Note: we have 2701 intent examples (5 distinct intents)

Command or request that led to the issue:

rasa train nlu  --nlu training_data.json
rasa test nlu -m models --successes --no-plot --nlu test_data.json

Content of configuration file (config.yml):

language: fr
pipeline:
  - name: WhitespaceTokenizer
    case_sensitive: False
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
    random_seed: 4321

[sara-tagger]

Thanks for the issue, @alwx will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

[fparga]

Some more context, from our experiments, the predicted intents seems to be the same when running tests on two models trained with the same train set (and pipeline) on GPU. It's the confidence scores that varies wildly, whereas they are identical when the training is done on CPU.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please create a new issue if you need more help.