Improve Performance with Config

[jclouet]

Hello all, new to the ML and using coqui-tts to learn the basics.

I'm currently training my own glow-tts model using LJSpeech and I am trying to get the best performance out of my machine. This is currently running on a lab device with 8 gpus mounted and running CUDA11.

However, despite this beefy hardware my epochs still take about 5 minutes to complete each. I know the LJSpeech dataset is quite large, but this still feels slow to me. I'm worried that it may be due to my misunderstanding of a basic setting in the config that is holding be back. Below is my current config.

I am running torch=1.7.1+cu110

Appreciate the help.

"model": "glow_tts",
    "run_name": "coqui_tts",
    "run_description": "",
    "epochs": 500,
    "batch_size": 128,
    "eval_batch_size": 16,
    "mixed_precision": true,
    "scheduler_after_epoch": false,
    "run_eval": true,
    "test_delay_epochs": -1,
    "print_eval": false,
    "dashboard_logger": "tensorboard",
    "print_step": 25,
    "plot_step": 100,
    "model_param_stats": false,
    "project_name": null,
    "log_model_step": null,
    "wandb_entity": null,
    "save_step": 10000,
    "checkpoint": true,
    "keep_all_best": false,
    "keep_after": 10000,
    "num_loader_workers": 8,
    "num_eval_loader_workers": 8,
    "use_noise_augment": false,
    "output_path": "path/to/model/output/",
    "distributed_backend": "nccl",
    "distributed_url": "tcp://localhost:54321",
    "audio": {
        "fft_size": 1024,
        "win_length": 1024,
        "hop_length": 256,
        "frame_shift_ms": null,
        "frame_length_ms": null,
        "stft_pad_mode": "reflect",
        "sample_rate": 22050,
        "resample": false,
        "preemphasis": 0.0,
        "ref_level_db": 20,
        "do_sound_norm": false,
        "log_func": "np.log10",
        "do_trim_silence": true,
        "trim_db": 45,
        "power": 1.5,
        "griffin_lim_iters": 60,
        "num_mels": 80,
        "mel_fmin": 0.0,
        "mel_fmax": null,
        "spec_gain": 20,
        "do_amp_to_db_linear": true,
        "do_amp_to_db_mel": true,
        "signal_norm": true,
        "min_level_db": -100,
        "symmetric_norm": true,
        "max_norm": 4.0,
        "clip_norm": true,
        "stats_path": null
    },
    "use_phonemes": true,
    "use_espeak_phonemes": true,
    "phoneme_language": "en-us",
    "compute_input_seq_cache": false,
    "text_cleaner": "phoneme_cleaners",
    "enable_eos_bos_chars": false,
    "test_sentences_file": "",
    "phoneme_cache_path": "path/to/phoneme_cache",
    "characters": {
        "pad": "_",
        "eos": "~",
        "bos": "^",
        "characters": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!'(),-.:;? ",
        "punctuations": "!'(),-.:;? ",
        "phonemes": "iy\u0268\u0289\u026fu\u026a\u028f\u028ae\u00f8\u0258\u0259\u0275\u0264o\u025b\u0153\u025c\u025e\u028c\u0254\u00e6\u0250a\u0276\u0251\u0252\u1d7b\u0298\u0253\u01c0\u0257\u01c3\u0284\u01c2\u0260\u01c1\u029bpbtd\u0288\u0256c\u025fk\u0261q\u0262\u0294\u0274\u014b\u0272\u0273n\u0271m\u0299r\u0280\u2c71\u027e\u027d\u0278\u03b2fv\u03b8\u00f0sz\u0283\u0292\u0282\u0290\u00e7\u029dx\u0263\u03c7\u0281\u0127\u0295h\u0266\u026c\u026e\u028b\u0279\u027bj\u0270l\u026d\u028e\u029f\u02c8\u02cc\u02d0\u02d1\u028dw\u0265\u029c\u02a2\u02a1\u0255\u0291\u027a\u0267\u02b2\u025a\u02de\u026b",
        "unique": true
    },
    "batch_group_size": 0,
    "loss_masking": null,
    "sort_by_audio_len": false,
    "min_seq_len": 3,
    "max_seq_len": 500,
    "compute_f0": false,
    "compute_linear_spec": false,
    "add_blank": false,
    "datasets": [
        {
            "name": "ljspeech",
            "path": "/path/to/LJSpeech-1.1/",
            "meta_file_train": "metadata.csv",
            "ununsed_speakers": null,
            "meta_file_val": "",
            "meta_file_attn_mask": ""
        }
    ],
    "optimizer": "RAdam",
    "optimizer_params": {
        "betas": [
            0.9,
            0.998
        ],
        "weight_decay": 1e-06
    },
    "lr_scheduler": "NoamLR",
    "lr_scheduler_params": {
        "warmup_steps": 4000
    },
    "test_sentences": [
        "It took me quite a long time to develop a voice, and now that I have it I'm not going to be silent.",
        "Be a voice, not an echo.",
        "I'm sorry Dave. I'm afraid I can't do that.",
        "This cake is great. It's so delicious and moist.",
        "Prior to November 22, 1963."
    ],
    "num_chars": 130,
    "encoder_type": "rel_pos_transformer",
    "encoder_params": {
        "kernel_size": 3,
        "dropout_p": 0.1,
        "num_layers": 6,
        "num_heads": 2,
        "hidden_channels_ffn": 768,
        "input_length": null
    },
    "use_encoder_prenet": true,
    "hidden_channels_enc": 192,
    "hidden_channels_dec": 192,
    "hidden_channels_dp": 256,
    "dropout_p_dp": 0.1,
    "dropout_p_dec": 0.05,
    "mean_only": true,
    "out_channels": 80,
    "num_flow_blocks_dec": 12,
    "inference_noise_scale": 0.0,
    "kernel_size_dec": 5,
    "dilation_rate": 1,
    "num_block_layers": 4,
    "num_speakers": 0,
    "c_in_channels": 0,
    "num_splits": 4,
    "num_squeeze": 2,
    "sigmoid_scale": false,
    "d_vector_dim": 0,
    "data_dep_init_steps": 10,
    "style_wav_for_test": null,
    "length_scale": 1.0,
    "use_speaker_embedding": false,
    "use_d_vector_file": false,
    "d_vector_file": false,
    "grad_clip": 5.0,
    "lr": 0.001,
    "r": 1

[erogol]

Can you post tensorboard logs for training stats like iter time, data loading time?

It'd also help to see your terminal logs for the same values to see if the bottleneck is the data loading or the computation.

[erogol]

It also helps, in general, to add these env vars to your training command

MKL_NUM_THREADS=1 OMP_NUM_THREADS=1 ...

[jclouet]

Will try this next.