What's the proper version of tensorflow for this project?

[mindmapper15]

I saw the Dockerfile and it automatically downloads the newest version of Tensorflow.
Recently, Tensorflow revealed 2.0 version and it's quiet different from the older version.
Could you let me know the proper version of tensorflow to run this project?

[kennethnakasone]

I can confirm that tensorflow-gpu 1.13.1, 1.14, and 1.15rc2 work (at least for training Tacotron). I haven't gotten around to doing wavenet training yet. : \

[JasonWei512]

Use Tensorflow 1.10.

I trained Wavenet on Tensorflow 1.14, and I got an error when I tried to do inference from checkpoint file. It seemed that some variables were missing from checkpoint file. (#421)
I retrained WaveNet with Tensorflow 1.10 and the problem is gone.

[kennethnakasone]

I can't do Wavenet inference with tensorflow 1.14. There's a bug with checkpoint loading.
Tensorflow 1.10 works fine for me.

Good to know. Thanks.
(Does this mean I have to re-train my tacotron model that was trained on 1.13?)

[JasonWei512]

I can't do Wavenet inference with tensorflow 1.14. There's a bug with checkpoint loading.
Tensorflow 1.10 works fine for me.

Good to know. Thanks.
(Does this mean I have to re-train my tacotron model that was trained on 1.13?)

You don't have to. My tacotron model was trained with TF 1.14, and now I'm training WaveNet with it on TF 1.10. I see no problem so far.

[kennethnakasone]

You don't have to. My tacotron model was trained with TF 1.14, and now I'm training WaveNet with it on TF 1.10. I see no problem so far.

Awesome! (I was worried I was going to have to re-do a lot of it). Thanks for the info

[mindmapper15]

I can't do Wavenet inference with tensorflow 1.14. There's a bug with checkpoint loading.
Tensorflow 1.10 works fine for me.

Thank's for the info! I'll have to check the newest as possible version of Tensorflow for WaveNet...

[orascheg]

Can it be that at saving of the checkpoints at WaveNet training the eval model (5,3M) of Wavenet is used instead the synth model (3,2M)?
I am also experiencing this problem (tensorflow 1.14 GPU) and have put the following line into the function create_shadow_saver() in wavenet_vocoder/train.py
log('Shadow variables {}'.format(shadow_variables))
When comparing the output of the shadow variables, I find that they are different if I start the training and the synthesis. Is it making sense to initialize the sh_saver directly after creating the training model (model, stats = model_train_mode(args, feeder, hparams, global_step)) and before the creation of the eval model (eval_model = model_test_mode(args, feeder, hparams, global_step))? I am not fully into the wirings of tensorflow, however I could imagine that the saver uses the latest model created. Perhaps this was different in tf 1.10

[orascheg]

Actually I have tried it. In the original check out from git (12th December 2019 head master ab5cb08) the output of the shadow variables with command line "python3 train.py --model='WaveNet'" was:
'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/input_convolution/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/input_convolution/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_skip_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_skip_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_out_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_out_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_causal_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_causal_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_cin_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_cin_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_skip_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_skip_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_out_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_out_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_causal_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_causal_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_cin_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_cin_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_skip_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_skip_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_out_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_out_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_causal_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_causal_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_cin_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_cin_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_skip_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_skip_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_out_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_out_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_causal_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_causal_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_cin_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_cin_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_skip_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_skip_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_out_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_out_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_causal_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_causal_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_cin_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_cin_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_skip_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_skip_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_out_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_out_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_causal_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_causal_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_cin_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_cin_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_skip_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_skip_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_out_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_out_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_causal_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_causal_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_cin_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_cin_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_skip_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_skip_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_out_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_out_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_causal_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_causal_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_cin_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_cin_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_skip_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_skip_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_out_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_out_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_causal_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_causal_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_cin_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_cin_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_skip_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_skip_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_out_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_out_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_causal_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_causal_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_cin_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_cin_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_skip_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_skip_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_out_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_out_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_causal_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_causal_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_cin_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_cin_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_skip_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_skip_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_out_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_out_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_causal_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_causal_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_cin_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_cin_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_skip_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_skip_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_out_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_out_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_causal_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_causal_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_cin_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_cin_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_skip_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_skip_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_out_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_out_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_causal_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_causal_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_cin_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_cin_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_skip_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_skip_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_out_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_out_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_causal_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_causal_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_cin_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_cin_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_skip_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_skip_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_out_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_out_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_causal_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_causal_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_cin_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_cin_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_skip_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_skip_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_out_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_out_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_causal_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_causal_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_cin_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_cin_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_skip_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_skip_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_out_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_out_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_causal_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_causal_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_cin_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_cin_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_skip_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_skip_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_out_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_out_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_causal_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_causal_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_cin_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_cin_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_skip_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_skip_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_out_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_out_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_2/bias/ExponentialMovingAverage'

When starting synthsis with the command line "python3 synthesize.py --model='WaveNet' --wavenet_name='WaveNet'" the output was:
'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/SubPixelConvolution_layer_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/input_convolution/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/input_convolution/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_cin_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_skip_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_skip_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_out_conv_ResidualConv1DGLU_0/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_out_conv_ResidualConv1DGLU_0/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_cin_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_cin_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_skip_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_skip_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_out_conv_ResidualConv1DGLU_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_1/residual_block_out_conv_ResidualConv1DGLU_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_cin_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_cin_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_skip_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_skip_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_out_conv_ResidualConv1DGLU_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_2/residual_block_out_conv_ResidualConv1DGLU_2/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_cin_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_cin_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_skip_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_skip_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_out_conv_ResidualConv1DGLU_3/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_3/residual_block_out_conv_ResidualConv1DGLU_3/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_cin_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_cin_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_skip_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_skip_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_out_conv_ResidualConv1DGLU_4/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_4/residual_block_out_conv_ResidualConv1DGLU_4/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_cin_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_cin_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_skip_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_skip_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_out_conv_ResidualConv1DGLU_5/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_5/residual_block_out_conv_ResidualConv1DGLU_5/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_cin_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_cin_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_skip_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_skip_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_out_conv_ResidualConv1DGLU_6/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_6/residual_block_out_conv_ResidualConv1DGLU_6/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_cin_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_cin_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_skip_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_skip_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_out_conv_ResidualConv1DGLU_7/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_7/residual_block_out_conv_ResidualConv1DGLU_7/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_cin_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_cin_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_skip_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_skip_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_out_conv_ResidualConv1DGLU_8/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_8/residual_block_out_conv_ResidualConv1DGLU_8/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_cin_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_cin_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_skip_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_skip_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_out_conv_ResidualConv1DGLU_9/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_9/residual_block_out_conv_ResidualConv1DGLU_9/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_cin_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_cin_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_skip_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_skip_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_out_conv_ResidualConv1DGLU_10/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_10/residual_block_out_conv_ResidualConv1DGLU_10/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_cin_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_cin_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_skip_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_skip_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_out_conv_ResidualConv1DGLU_11/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_11/residual_block_out_conv_ResidualConv1DGLU_11/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_cin_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_cin_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_skip_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_skip_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_out_conv_ResidualConv1DGLU_12/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_12/residual_block_out_conv_ResidualConv1DGLU_12/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_cin_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_cin_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_skip_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_skip_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_out_conv_ResidualConv1DGLU_13/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_13/residual_block_out_conv_ResidualConv1DGLU_13/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_cin_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_cin_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_skip_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_skip_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_out_conv_ResidualConv1DGLU_14/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_14/residual_block_out_conv_ResidualConv1DGLU_14/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_cin_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_cin_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_skip_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_skip_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_out_conv_ResidualConv1DGLU_15/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_15/residual_block_out_conv_ResidualConv1DGLU_15/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_cin_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_cin_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_skip_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_skip_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_out_conv_ResidualConv1DGLU_16/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_16/residual_block_out_conv_ResidualConv1DGLU_16/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_cin_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_cin_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_skip_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_skip_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_out_conv_ResidualConv1DGLU_17/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_17/residual_block_out_conv_ResidualConv1DGLU_17/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_cin_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_cin_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_skip_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_skip_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_out_conv_ResidualConv1DGLU_18/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_18/residual_block_out_conv_ResidualConv1DGLU_18/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_cin_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_cin_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_skip_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_skip_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_out_conv_ResidualConv1DGLU_19/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_19/residual_block_out_conv_ResidualConv1DGLU_19/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_1/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_1/bias/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_2/kernel/ExponentialMovingAverage', 'WaveNet_model/WaveNet_model/inference/final_convolution_2/bias/ExponentialMovingAverage'

After moving the creation of the writer the output is the same as in the synthesize:

#Set up model
global_step = tf.Variable(0, name='global_step', trainable=False)
model, stats = model_train_mode(args, feeder, hparams, global_step)
#GO 020120 moved so that it is created before the eval model
sh_saver = create_shadow_saver(model, global_step)
eval_model = model_test_mode(args, feeder, hparams, global_step)

I am just doing additional training and will then try if it loads without the mentioned error. Strange though because the model is given as parameter to create the shadow saver, thus I assume it is some side effect of something in tensorflow 1.14.

[orascheg]

Further investigation shows that the difference is:
Synthesize:
WaveNet_model/WaveNet_model/inference/ResidualConv1DGLU_0/residual_block_causal_conv_ResidualConv1DGLU_0
Train:
WaveNet_model/WaveNet_model/inference/residual_block_causal_conv_ResidualConv1DGLU_0

Somehow the names differ. I am still debugging where this comes from...
Must be somewhere in models/wavenet.py constructor:

	#Residual Blocks
	self.residual_layers = []
	for layer in range(hparams.layers):
		self.residual_layers.append(ResidualConv1DGLU(
		hparams.residual_channels, hparams.gate_channels,
		kernel_size=hparams.kernel_size,
		skip_out_channels=hparams.skip_out_channels,
		use_bias=hparams.use_bias,
		dilation_rate=2**(layer % layers_per_stack),
		dropout=hparams.wavenet_dropout,
		cin_channels=hparams.cin_channels,
		gin_channels=hparams.gin_channels,
		weight_normalization=hparams.wavenet_weight_normalization, 
		init=init, 
		init_scale=hparams.wavenet_init_scale,
		residual_legacy=hparams.residual_legacy,
		name='ResidualConv1DGLU_{}'.format(layer)))

[Arafat4341]

Hello everyone!
Can someone please tell me why am I getting this IndexError?

Traceback (most recent call last):
File "train.py", line 138, in
main()
File "train.py", line 132, in main
train(args, log_dir, hparams)
File "train.py", line 52, in train
checkpoint = tacotron_train(args, log_dir, hparams)
File "/content/drive/My Drive/Tacotron-2/tacotron/train.py", line 399, in tacotron_train
return train(log_dir, args, hparams)
File "/content/drive/My Drive/Tacotron-2/tacotron/train.py", line 152, in train
feeder = Feeder(coord, input_path, hparams)
File "/content/drive/My Drive/Tacotron-2/tacotron/feeder.py", line 33, in init
hours = sum([int(x[4]) for x in self._metadata]) * frame_shift_ms / (3600)
File "/content/drive/My Drive/Tacotron-2/tacotron/feeder.py", line 33, in
hours = sum([int(x[4]) for x in self._metadata]) * frame_shift_ms / (3600)
IndexError: list index out of range

Why list index would be out of range I don't understand! I checked the size of self._metadata. I don't know what's going on! Kindly help if you find the reason!
Thanks in advance!