CICD NeMo

[NeMo-UX] Fix bug in import_ckpt #2548

Triggered via pull request June 19, 2024 14:24

marcromeyn

labeled #9492

nemo-ux/import_ckpt_bug

Status Cancelled

Total duration 46m 54s

Artifacts –

cicd-main.yml

on: pull_request

cicd-cluster-clean

gpu-test

cicd-test-container-setup

5m 5s

L0_Unit_Tests_CPU / main

29m 30s

L2_Community_LLM_Checkpoints_tests_Llama / main

38s

L2_Community_LLM_Checkpoints_tests_StarCoder / main

1m 37s

L2_Community_LLM_Checkpoints_tests_Falcon / main

39s

ASR_dev_run_Speech_to_Text / main

46s

ASR_dev_run_Speech_to_Text_WPE_-_CitriNet / main

47s

ASR_dev_run_Speech_Pre-training_-_CitriNet / main

49s

ASR_dev_run_Speech_To_Text_Finetuning / main

49s

ASR_dev_run_Speech_to_Text_WPE_-_Conformer / main

38s

ASR_dev_run-part_two_Speech_to_Text_WPE_-_Squeezeformer / main

37s

L2_Speech_to_Text_EMA / main

54s

L2_Speaker_dev_run_Speaker_Recognition / main

35s

L2_Speaker_dev_run_Speaker_Diarization / main

36s

L2_Speaker_dev_run_Speech_to_Label / main

36s

L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference / main

30s

L2_Speaker_dev_run_Clustering_Diarizer_Inference / main

1m 11s

L2_Speaker_dev_run_Neural_Diarizer_Inference / main

39s

L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation / main

29s

L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader / main

47s

L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader / main

36s

L2_ASR_Adapters_Linear_Adapters / main

38s

L2_ASR_Adapters_RelPos_MHA_Adapters / main

38s

L2_Speech_Transcription_Speech_to_Text_Transcribe / main

45s

L2_Transducer_alignment_Running_pytest / main

52s

L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_CitriNet_with_wav / main

3m 16s

L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Ru_QN_with_mp3 / main

1m 33s

L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference / main

1m 20s

L2_G2P_Models_HeteronymClassificationModel_training_evaluation_and_inference / main

1m 22s

L2_Duplex_Text_Normalization_with_Tarred_dataset / main

47s

L2_Intent_and_Slot_Classification_Tasks_Intent_and_Slot_Classification / main

37s

L2_Intent_and_Slot_Classification_Tasks_Multi-Label_Intent_and_Slot_Classification / main

40s

L2_Parallel_NLP_Examples2_NER_finetuning_from_pretrained_Test / main

41s

L2_Parallel_NLP_Examples2_Punctuation_and_capitalization_finetuning_from_pretrained_test / main

42s

L2_Parallel_NLP_Examples2_NER_with_TurkuNLP__bert-base-finnish-cased-v1 / main

36s

L2_Parallel_NLP_Examples2_Evaluation_script_for_Token_Classification / main

1m 16s

L2_Parallel_NLP_Examples2_Evaluation_script_for_Punctuation / main

1m 15s

L2_Pretraining_BERT_pretraining_from_Text / main

37s

L2_Pretraining_BERT_from_Preprocessed / main

47s

L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Post-LN / main

1m 6s

L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Pre-LN / main

49s

L2_NMT_Attention_is_All_You_Need_Training_NMT_Multi-Validation / main

1m 2s

L2_NMT_Attention_is_All_You_Need_Inference / main

1m 28s

L2_NMT_Attention_is_All_You_Need_Finetuning / main

1m 47s

L2_NMT_Tarred_Dataset_Creation_Auto_Tarred_Dataset_Creation / main

52s

L2_NMT_Tarred_Dataset_Creation_Script_Tarred_Dataset_Creation / main

1m 18s

L2_Megatron_NMT_Training_TP2 / main

4m 20s

L2_Megatron_BART_Perceiver_MIM_Training_TP2 / main

1m 44s

L2_Megatron_Bert_Pretraining_and_Resume_Training_with_Pipeline_Parallelism

3m 4s

L2_Megatron_Bert_Pretraining_and_Resume_Training

2m 14s

L2_Megatron_Core_Bert_Pretraining_and_Resume_Training

3m 0s

L2_Megatron_RETRO_Pretraining_and_Resume_Training / main

6m 14s

L2_Legacy_Megatron_RETRO_Pretraining_and_Resume_Training / main

1m 40s

L2_RAG_Pipeline_Indexing

49s

L2_RAG_Pipeline_Generating

1m 55s

L2_BioMegatron_Bert_NER_Task / main

1m 8s

L2_Megatron_GPT_Pretraining_and_Resume_Training_TP2

4m 59s

L2_Megatron_GPT_with_Rope_Pretraining_and_Resume_Training_TP2

1m 56s

L2_Megatron_GPT_with_ALiBi_Pretraining_and_Resume_Training_TP2

2m 37s

L2_Megatron_GPT_with_KERPLE_Pretraining_and_Resume_Training_TP2

1m 56s

L2_Megatron_GPT_Pretraining_and_Resume_Training_PP2 / main

3m 57s

L2_Megatron_GPT_Finetuning_PP2 / main

3m 17s

L2_Megatron_GPT_Finetuning_StarCoder_PP1

1m 8s

L2_Megatron_GPT_Embedding / main

1m 58s

L2_Megatron_GPT_PEFT_Lora_PP2_O2 / main

1m 50s

L2_Megatron_GPT_PEFT_Lora_TP2_O1 / main

2m 39s

L2_Megatron_GPT_Eval / main

1m 6s

L2_Megatron_GPT_Eval_PP2 / main

2m 6s

L2_Megatron_GPT_SFT_Eval_inference_seq_len_greaterThan_training_seq_len / main

1m 21s

L2_Megatron_Change_Partitions_Reduce_TP_Num_Partitions_-2_to_1-_and_PP_Num_Partitions_-1_to_2 / main

50s

L2_Megatron_Change_Partitions_Increase_TP_Num_Partitions_-2_to_4-_and_PP_Num_Partitions_-1_to_2 / main

47s

L2_Megatron_T5_Pretraining_and_Resume_Training_TP2 / main

L2_Megatron_T5_with_ALiBi_Pretraining_and_Resume_Training_TP2 / main

L2_Megatron_T5_with_KERPLE_Pretraining_and_Resume_Training_TP2 / main

L2_Megatron_T5_Pretraining_and_Resume_Training_PP2 / main

L2_Megatron_T5_w_Mixture_of_Expert_Pretraining / main

L2_Megatron_UL2_Pretraining_and_Resume_Training_TP2 / main

L2_Megatron_T5_Eval / main

L2_Megatron_BART_Pretraining_and_Resume_Training_TP2 / main

2m 35s

L2_Megatron_BART_Pretraining_and_Resume_Training_PP2 / main

2m 38s

L2_Megatron_T5_PEFT_Lora_TP2 / main

L2_Megatron_Mock_Data_Generation_MockGPTDataset / main

38s

L2_Megatron_Mock_Data_Generation_MockT5Dataset / main

25s

L2_TTS_Fast_dev_runs_1_Tacotron_2 / main

1m 5s

L2_TTS_Fast_dev_runs_1_WaveGlow / main

L2_TTS_Fast_dev_runs_1_FastPitch / main

L2_TTS_Fast_dev_runs_1_Mixer-TTS / main

L2_TTS_Fast_dev_runs_1_Hifigan / main

Speech_Checkpoints_tests / main

OPTIONAL_L0_Unit_Tests_GPU / main

30m 7s

L0_Setup_Test_Data_And_Models / main

12s

L2_Community_LLM_Checkpoints_tests_Llama3 / main

40s

L2_PTQ_Llama2_Export_Only / main

35s

L2_PTQ_Llama2_FP8 / main

1m 47s

L2_PTQ_Llama2_INT8_SQ / main

39s

OPTIONAL_ASR_dev_run_Speech_To_Text_HF_Finetuning / main

1m 28s

Nemo_CICD_Test

Annotations

21 errors

OPTIONAL_L0_Unit_Tests_GPU / main

The job running on runner azure-gpu-vm-runner5 has exceeded the maximum execution time of 30 minutes.

OPTIONAL_L0_Unit_Tests_GPU / main

The operation was canceled.

L2_Megatron_T5_Pretraining_and_Resume_Training_TP2 / main