[EMAGE]Is there bug when using Tensor/array !=[] ?

[ex3ploiter]

Hi.
I think there is a bug when in the code Tensor !=[] is used.

2024-02-19 06:31:16.340301: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-19 06:31:16.340360: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-19 06:31:16.341764: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-19 06:31:16.349059: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-02-19 06:31:17.470005: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
 02-19 06:31:20 | {'a_encoder': None,
 'a_fix_pre': False,
 'a_pre_encoder': None,
 'acc_weight': 0.0,
 'additional_data': False,
 'adv_weight': 20.0,
 'ali_weight': 0.0,
 'amsgrad': False,
 'apex': False,
 'asmr': 0.0,
 'atcont': 0.0,
 'atmr': 0.0,
 'aud_prob': 1.0,
 'audio_dims': 1,
 'audio_f': 256,
 'audio_fps': 16000,
 'audio_norm': False,
 'audio_rep': 'onset+amplitude',
 'audio_sr': 16000,
 'batch_size': 64,
 'beat_align': True,
 'benchmark': True,
 'cache_only': False,
 'cache_path': '/datasets/beat_cache/beat_smplx_en_emage/',
 'cf': 0.0,
 'ch': 1.0,
 'cl': 1.0,
 'clean_final_seconds': 0,
 'clean_first_seconds': 0,
 'config': './scripts/EMAGE_2024/configs/emage.yaml',
 'csv_name': 'a2g_0',
 'cu': 1.0,
 'cudnn_enabled': True,
 'd_lr_weight': 0.2,
 'd_name': None,
 'data_path': '/content/PantoMatrix/BEATX/beat_english_v2.0.0/',
 'data_path_1': '/content/PantoMatrix/EMAGE/',
 'dataset': 'beat_sep_lower',
 'ddp': False,
 'debug': False,
 'decay_epochs': 9999,
 'decay_rate': 0.1,
 'decode_fusion': None,
 'deterministic': True,
 'disable_filtering': True,
 'div_reg_weight': 0.0,
 'dropout_prob': 0.3,
 'e_name': 'VAESKConv',
 'e_path': 'weights/AESKConv_240_100.bin',
 'emo_rep': None,
 'emotion_dims': 8,
 'emotion_f': 0,
 'epoch_stage': 0,
 'epochs': 400,
 'eval_model': 'motion_representation',
 'f_encoder': 'null',
 'f_fix_pre': False,
 'f_pre_encoder': 'null',
 'fac_prob': 1.0,
 'facial_dims': 100,
 'facial_f': 0,
 'facial_fps': 15,
 'facial_norm': False,
 'facial_rep': 'smplxflame_30',
 'fid_weight': 0.0,
 'finger_net': 'original',
 'freeze_wordembed': False,
 'fsmr': 0.0,
 'ftmr': 0.0,
 'fusion_mode': 'sum',
 'g_name': 'MAGE_Transformer',
 'gap_weight': 0.0,
 'gpus': [0],
 'grad_norm': 0.99,
 'hidden_size': 768,
 'id_rep': 'onehot',
 'input_context': 'both',
 'is_train': True,
 'ita_weight': 0.0,
 'iwa_weight': 0.0,
 'kld_aud_weight': 0.0,
 'kld_fac_weight': 0.0,
 'kld_weight': 0.0,
 'l': 4,
 'lf': 3.0,
 'lh': 3.0,
 'll': 3.0,
 'loader_workers': 0,
 'log_period': 10,
 'loss_contrastive_neg_weight': 0.005,
 'loss_contrastive_pos_weight': 0.2,
 'loss_gan_weight': 5.0,
 'loss_kld_weight': 0.1,
 'loss_physical_weight': 0.0,
 'loss_reg_weight': 0.05,
 'loss_regression_weight': 70.0,
 'lr_base': 0.0005,
 'lr_min': 1e-07,
 'lr_policy': 'step',
 'lu': 3.0,
 'm_decoder': None,
 'm_encoder': 'null',
 'm_fix_pre': False,
 'm_pre_encoder': 'null',
 'mean_pose_path': '/datasets/trinity/train/',
 'model': 'emage',
 'momentum': 0.8,
 'motion_f': 256,
 'msmr': 0.0,
 'mtmr': 0.0,
 'multi_length_training': [1.0],
 'n_layer': 1,
 'n_poses': 34,
 'n_pre_poses': 4,
 'name': '0219_063120_emage',
 'nesterov': True,
 'new_cache': False,
 'no_adv_epoch': 999,
 'notes': '',
 'opt': 'adam',
 'opt_betas': [0.5, 0.999],
 'ori_joints': 'beat_smplx_joints',
 'out_path': '/content/PantoMatrix/outputs/audio2pose/',
 'pos_encoding_type': 'sin',
 'pos_prob': 1.0,
 'pose_dims': 330,
 'pose_fps': 30,
 'pose_length': 64,
 'pose_norm': False,
 'pose_rep': 'smplxflame_30',
 'pre_frames': 4,
 'pre_type': 'zero',
 'pretrain': False,
 'project': 's2g',
 'queue_size': 1024,
 'random_seed': 2021,
 'rec_aud_weight': 0.0,
 'rec_fac_weight': 0.0,
 'rec_pos_weight': 0.0,
 'rec_txt_weight': 0.0,
 'rec_ver_weight': 0.0,
 'rec_weight': 1.0,
 'root_path': '/content/PantoMatrix/',
 'rot6d': True,
 'sem_rep': None,
 'sparse': 1,
 'speaker_dims': 4,
 'speaker_f': 0,
 'speaker_id': 'onehot',
 'stat': 'ts',
 'std_pose_path': '/datasets/trinity/train/',
 'stride': 20,
 't_encoder': 'null',
 't_fix_pre': False,
 't_pre_encoder': 'fasttext',
 'tar_joints': 'beat_smplx_full',
 'test_ckpt': 'weights/multi.bin',
 'test_data_path': '/datasets/trinity/test/',
 'test_length': 64,
 'test_period': 20,
 'train_data_path': '/datasets/trinity/train/',
 'train_trans': True,
 'trainer': 'emage',
 'training_speakers': [2],
 'tsmr': 0.0,
 'ttmr': 0.0,
 'txt_prob': 1.0,
 'use_aug': False,
 'vae_codebook_size': 256,
 'vae_grow': [1, 1, 2, 1],
 'vae_layer': 4,
 'vae_length': 240,
 'vae_quantizer_lambda': 1.0,
 'vae_test_dim': 330,
 'vae_test_len': 32,
 'vae_test_stride': 20,
 'val_data_path': '/datasets/trinity/val/',
 'variational': False,
 'vel_weight': 0.0,
 'warmup_epochs': 0,
 'warmup_lr': 0.0005,
 'wei_weight': 0.0,
 'weight_decay': 0.0,
 'word_cache': False,
 'word_dims': 300,
 'word_f': 256,
 'word_index_num': 11195,
 'word_rep': 'textgrid',
 'z_type': 'speaker'}
 02-19 06:31:20 | # ------------ 0219_063120_emage ----------- #
 02-19 06:31:20 | PyTorch version: 2.1.0+cu121
 02-19 06:31:20 | CUDA version: 12.1
 02-19 06:31:20 | 1 GPUs
 02-19 06:31:20 | Random Seed: 2021
 02-19 06:31:21 | Audio bit rate: 16000
 02-19 06:31:21 | Reading data '/content/PantoMatrix/BEATX/beat_english_v2.0.0/'...
 02-19 06:31:21 | Creating the dataset cache...
 02-19 06:31:21 | # ---- Building cache for Pose   2_scott_0_100_100 ---- #
 02-19 06:31:22 | # ---- Building cache for Facial 2_scott_0_100_100 and Pose 2_scott_0_100_100 ---- #
 02-19 06:31:22 | # ---- Building cache for Audio  2_scott_0_100_100 and Pose 2_scott_0_100_100 ---- #
 02-19 06:31:25 | # ---- Building cache for Word   2_scott_0_100_100 and Pose 2_scott_0_100_100 ---- #
 02-19 06:31:25 | An error has been caught in function '<module>', process 'MainProcess' (16470), thread 'MainThread' (132308915815040):
Traceback (most recent call last):

> File "/content/PantoMatrix/scripts/EMAGE_2024/train.py", line 307, in <module>
    main_worker(0, 1, args)
    │                 └ Namespace(config='./scripts/EMAGE_2024/configs/emage.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='em...
    └ <function main_worker at 0x785485ede5f0>

  File "/content/PantoMatrix/scripts/EMAGE_2024/train.py", line 257, in main_worker
    trainer = __import__(f"{args.trainer}_trainer", fromlist=["something"]).CustomTrainer(args) if args.trainer != "base" else BaseTrainer(args)
                                                                                          │        │    │                      │           └ Namespace(config='./scripts/EMAGE_2024/configs/emage.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='em...
                                                                                          │        │    │                      └ <class '__main__.BaseTrainer'>
                                                                                          │        │    └ 'emage'
                                                                                          │        └ Namespace(config='./scripts/EMAGE_2024/configs/emage.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='em...
                                                                                          └ Namespace(config='./scripts/EMAGE_2024/configs/emage.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='em...

  File "/content/PantoMatrix/scripts/EMAGE_2024/emage_trainer.py", line 29, in __init__
    super().__init__(args)
                     └ Namespace(config='./scripts/EMAGE_2024/configs/emage.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='em...

  File "/content/PantoMatrix/scripts/EMAGE_2024/train.py", line 43, in __init__
    self.train_data = __import__(f"dataloaders.{args.dataset}", fromlist=["something"]).CustomDataset(args, "train")
    │                                                                                                 └ Namespace(config='./scripts/EMAGE_2024/configs/emage.yaml', project='s2g', stat='ts', csv_name='a2g_0', notes='', trainer='em...
    └ <emage_trainer.CustomTrainer object at 0x785485edb040>

  File "/content/PantoMatrix/scripts/EMAGE_2024/dataloaders/beat_sep_lower.py", line 110, in __init__
    self.build_cache(preloaded_dir)
    │    │           └ '/content/PantoMatrix//datasets/beat_cache/beat_smplx_en_emage/train/smplxflame_30_cache'
    │    └ <function CustomDataset.build_cache at 0x785472bec3a0>
    └ <dataloaders.beat_sep_lower.CustomDataset object at 0x785485f02020>

  File "/content/PantoMatrix/scripts/EMAGE_2024/dataloaders/beat_sep_lower.py", line 214, in build_cache
    self.cache_generation(
    │    └ <function CustomDataset.cache_generation at 0x785472bec550>
    └ <dataloaders.beat_sep_lower.CustomDataset object at 0x785485f02020>

  File "/content/PantoMatrix/scripts/EMAGE_2024/dataloaders/beat_sep_lower.py", line 584, in cache_generation
    filtered_result = self._sample_from_clip(
                      │    └ <function CustomDataset._sample_from_clip at 0x785472bec5e0>
                      └ <dataloaders.beat_sep_lower.CustomDataset object at 0x785485f02020>

  File "/content/PantoMatrix/scripts/EMAGE_2024/dataloaders/beat_sep_lower.py", line 623, in _sample_from_clip
    if audio_each_file != []:
       └ array([[0.00039914, 0.        ],
                [0.00039914, 0.        ],
                [0.00039914, 0.        ],
                ...,
                [0.0844...

ValueError: operands could not be broadcast together with shapes (880000,2) (0,) 

[H-Liu1997]

Hi thanks for the feedback,

I am going to debug, could you share the command you run and take a look could current EMAGE works well?

[ex3ploiter]

Thank you for your reply.
I used to the command you provided in README: python scripts/EMAGE_2024/train.py --config ./scripts/EMAGE_2024/configs/emage.yaml

I'll try it.

[fcchit]

I changed if audio_each_file != []: to if len(audio_each_file) != 0:, and it works well. :)