Training loss is always above 200 on dataset LibriSpeech/train-clean-100

[Honghe]

Environment:

• TensorFlow GPU 2.3
• TensorFlowASR ade7891

Training command:

PYTHONPATH=`pwd` python examples/conformer/train_subword_conformer.py --tbs 2 --ebs 2 --mxp --devices 0 --cache --subwords ./output/librispeed \
--subwords_corpus \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/train-clean-100/transcripts.tsv \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/test-clean/transcripts.tsv \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/dev-clean/transcripts.tsv \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/dev-other/transcripts.tsv 

[nglehuy]

@Honghe Are you using the same config as in the example directory (only different dataset paths)?

[nglehuy]

@Honghe The model I trained on 4 RTX 2080Ti 11G with --tbs 3 on each gpu.
Regardless the batch size, this is weird because it hadn't happened to me before, the validation loss should be under 10

[Honghe]

@usimarit config.yml, I only modify dataset paths.

speech_config:
  sample_rate: 16000
  frame_ms: 25
  stride_ms: 10
  num_feature_bins: 80
  feature_type: log_mel_spectrogram
  preemphasis: 0.97
  normalize_signal: True
  normalize_feature: True
  normalize_per_feature: False

decoder_config:
  vocabulary: null
  target_vocab_size: 1024
  max_subword_length: 4
  blank_at_zero: True
  beam_width: 5
  norm_score: True

model_config:
  name: conformer
  subsampling:
    type: conv2d
    filters: 144
    kernel_size: 3
    strides: 2
  positional_encoding: sinusoid_concat
  dmodel: 144
  num_blocks: 16
  head_size: 36
  num_heads: 4
  mha_type: relmha
  kernel_size: 32
  fc_factor: 0.5
  dropout: 0.1
  embed_dim: 320
  embed_dropout: 0.1
  num_rnns: 1
  rnn_units: 320
  rnn_type: lstm
  layer_norm: True
  joint_dim: 320

learning_config:
  augmentations:
    after:
      time_masking:
        num_masks: 10
        mask_factor: 100
        p_upperbound: 0.05
      freq_masking:
        num_masks: 1
        mask_factor: 27

  dataset_config:
    train_paths:
      - /home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/train-clean-100/transcripts.tsv
    eval_paths:
      - /home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/dev-clean/transcripts.tsv
    test_paths:
      - /home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/test-clean/transcripts.tsv
    tfrecords_dir: null

  optimizer_config:
    warmup_steps: 10000
    beta1: 0.9
    beta2: 0.98
    epsilon: 1e-9

  running_config:
    batch_size: 2
    accumulation_steps: 1
    num_epochs: 20
    outdir: ./output/conformer/
    log_interval_steps: 300
    eval_interval_steps: 500
    save_interval_steps: 1000

[nglehuy]

@Honghe another time I trained librispeech 100h, it gave the validation loss smaller than 10.
Can you try checking out the branch r0.2.6 and retraining the model? I don't have available gpus this time.

[Honghe]

@usimarit Did you use --tbs 3 --ebs 3? And after how many steps when reach loss < 10?

[nglehuy]

@Honghe Actually it's about 20, this is a very old wrong implementation of character conformer, with batch size 4
For the results in the newest example, I used --tbs 3 --ebs 4 with 4 gpus, so actually it's tbs 12 and ebs 16.

[Honghe]

@usimarit The loss still high after changed to r0.2.6.
Environment:

• TensorFlowASR r0.2.6

PYTHONPATH=`pwd` python examples/conformer/train_subword_conformer.py --tbs 3 --ebs 3 --mxp --devices 0 1 --cache --subwords ./output/librispeed \
--subwords_corpus \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/train-clean-100/transcripts.tsv \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/test-clean/transcripts.tsv \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/dev-clean/transcripts.tsv \
/home/ubuntu/Data/LibriSpeechConformer/LibriSpeech/dev-other/transcripts.tsv 

[nglehuy]

@Honghe so the newest version ans r0.2.6 is the same.
I used warmup_steps = 40000 for batch size 12.
Try with that value, may be it will overcome local minima

[Honghe]

@usimarit After changed to warmup_steps = 40000, the training got a better loss.

[nglehuy]

@Honghe Yeah, problem solved 😆