Add flag to disable training on GPU for TEDPolicy

[indam23]

What problem are you trying to solve?

Since the bump to Tensorflow 2.6, training on TED on GPU is significantly (in one case 4X) slower than before. CPU training time is not affected to the same degree.

Because it is still advantageous to use GPU for DIETClassifier, it's desirable to disable training on GPU for TED only to save time.

What's your suggested solution?

A parameter for TED like use_gpu that controls whether available GPUs are used or not.

Examples (if relevant)

No response

Is anything blocking this from being implemented? (if relevant)

No response

Definition of Done

• flag added

[WashingtonBispo]

Exalate commented:

WashingtonBispo commented:

Hello There. I have interest on work on this issue.

[indam23]

Exalate commented:

melindaloubser1 commented:

@WashingtonBispo Would you like to contribute a PR for this feature or do you mean you're interested in future work on it?

[WashingtonBispo]

Exalate commented:

WashingtonBispo commented:

I intent to contribute a PR. I will work on it this weekend.

[indam23]

Exalate commented:

melindaloubser1 commented:

Great, thanks! Please tag me in the PR and I'll make sure it gets reviewed

[WashingtonBispo]

Exalate commented:

WashingtonBispo commented:

Done. @melindaloubser1

[indam23]

Exalate commented:

melindaloubser1 commented:

For anyone who needs a workaround until the PR is merged, you can train NLU on GPU and core on CPU separately, by exporting CUDA_VISIBLE_DEVICES=-1 before training core. Then you can combine the models:

1. Default GPU NLU training
   rasa train nlu --fixed-model-name nlu

2. GPU excluded core training

3. If this is inside a docker container possible that you'll additionally need NVIDIA_VISIBLE_DEVICES=none
   export CUDA_VISIBLE_DEVICES="-1"
   rasa train core --fixed-model-name core

4. Combine NLU & Core models
   cd models
   mkdir nlu-model
   tar xvf nlu.tar.gz -C nlu-model
   mkdir core-model
   tar xvf core.tar.gz -C core-model
   mkdir combined
   mv core-model/core combined
   mv nlu-model/nlu combined
   mv core-model/fingerprint.json combined
   tar -C combined -cvzf combined.tar.gz .

[dakshvar22]

Closed by #10944