2022-09-24 07:46:10,134 - root - INFO - AIMET 2022-09-24 07:46:10,177 - root - INFO - xgen_scripts.py 2022-09-24 07:46:10,177 - root - INFO - Training arguments: 2022-09-24 07:46:10,177 - root - INFO - arch: c3d 2022-09-24 07:46:10,177 - root - INFO - batch_size: 32 2022-09-24 07:46:10,177 - root - INFO - dataset: ucf101 2022-09-24 07:46:10,177 - root - INFO - epochs: 35 2022-09-24 07:46:10,177 - root - INFO - gpu: 1,2,3,4,5,6,7 2022-09-24 07:46:10,177 - root - INFO - log_interval: 100 2022-09-24 07:46:10,177 - root - INFO - logdir: checkpoint/ucf101_c3d 2022-09-24 07:46:10,177 - root - INFO - lr: 0.005 2022-09-24 07:46:10,177 - root - INFO - lr_milestones: [20, 30] 2022-09-24 07:46:10,177 - root - INFO - lr_scheduler: None 2022-09-24 07:46:10,177 - root - INFO - multiplier: 8 2022-09-24 07:46:10,177 - root - INFO - optim: sgd 2022-09-24 07:46:10,177 - root - INFO - pretrained: False 2022-09-24 07:46:10,177 - root - INFO - resume: False 2022-09-24 07:46:10,177 - root - INFO - smooth_eps: 0.0 2022-09-24 07:46:10,177 - root - INFO - test: False 2022-09-24 07:46:10,178 - root - INFO - test_path: None 2022-09-24 07:46:10,178 - root - INFO - transfer: False You are choose test mode, so we will create a temporary directory and will remove it after testing /root/Projects/video-classification-s2-1d/XGen_workplace/31ce2137e23ffc3c37ba3fd5a9844f0b Your current workplace is /root/Projects/video-classification-s2-1d/XGen_workplace/31ce2137e23ffc3c37ba3fd5a9844f0b start a new xgen searching! ****************************config summary************************************************ xgen-config-path: /root/Projects/video-classification-s2-1d/s2+1d_config/xgen.xgen.json xgen-workplace: /root/Projects/video-classification-s2-1d/XGen_workplace/31ce2137e23ffc3c37ba3fd5a9844f0b xgen-resume: False xgen-mode: compatible_testing xgen-pretrained-model-path: /root/Projects/video-classification-s2-1d/pre_trained_model/r2+1d_ucf101_dense_top1-94.0_unpruned.pt Detail args: {'origin': {'common_train_epochs': 35, 'root_path': './Xgen/', 'pretrain_model_weights_path': None, 'train_data_path': '/data/video-classification-s2-1d/ucf101_frame', 'train_label_path': None, 'eval_data_path': None, 'eval_label_path': None, 'arch': 'r2+1d', 'num_classes': 101, 'dataset': 'ucf101', 'batch_size': 32, 'lr': 0.005, 'optim': 'sgd', 'lr_scheduler': None, 'lr_milestones': [20, 30], 'resume': False, 'transfer': False, 'smooth_eps': 0.0, 'pretrained': False, 'log_interval': 100, 'test': False, 'test_path': None, 'gpu': '0', 'logdir': './checkpoint/', 'multiplier': 8}, 'general': {'user_id': 'test', 'work_place': '/root/Projects/video-classification-s2-1d/XGen_workplace/31ce2137e23ffc3c37ba3fd5a9844f0b'}, 'prune': {'sp_store_weights': None, 'sp_lars': False, 'sp_lars_trust_coef': 0.001, 'sp_backbone': False, 'sp_retrain': False, 'sp_admm': False, 'sp_admm_multi': False, 'sp_retrain_multi': False, 'sp_config_file': None, 'sp_subset_progressive': False, 'sp_admm_fixed_params': False, 'sp_no_harden': False, 'nv_sparse': False, 'sp_load_prune_params': None, 'sp_store_prune_params': None, 'generate_rand_seq_gap_yaml': False, 'sp_admm_update_epoch': 5, 'sp_admm_update_batch': None, 'sp_admm_rho': 0.001, 'sparsity_type': 'block_punched', 'sp_admm_lr': 0.01, 'admm_debug': False, 'sp_global_weight_sparsity': False, 'sp_prune_threshold': -1.0, 'sp_block_irregular_sparsity': '(0,0)', 'sp_block_permute_multiplier': 2, 'sp_admm_block': '(8,4)', 'sp_admm_buckets_num': 16, 'sp_admm_elem_per_row': 1, 'sp_admm_tile': None, 'sp_admm_select_number': 4, 'sp_admm_pattern_row_sub': 1, 'sp_admm_pattern_col_sub': 4, 'sp_admm_data_format': None, 'sp_admm_do_not_permute_conv': False, 'sp_gs_output_v': None, 'sp_gs_output_ptr': None, 'sp_load_frozen_weights': None, 'retrain_mask_pattern': 'weight', 'sp_update_init_method': 'weight', 'sp_mask_update_freq': 10, 'retrain_mask_sparsity': -1.0, 'retrain_mask_seed': None, 'sp_prune_before_retrain': False, 'output_compressed_format': False, 'sp_grad_update': False, 'sp_grad_decay': 0.98, 'sp_grad_restore_threshold': -1, 'sp_global_magnitude': False, 'sp_pre_defined_mask_dir': None, 'sp_prune_ratios': 0, 'admm_block': '(8,4)', 'prune_threshold': -1.0}, 'quantization': {'qt_aimet': False, 'qat': True, 'fold_layers': True, 'cross_layer_equalization': False, 'bias_correction': True, 'rounding_mode': 'nearest', 'num_quant_samples': 1000, 'num_bias_correct_samples': 1000, 'weight_bw': 8, 'act_bw': 8, 'quant_scheme': 'tf_enhanced', 'layers_to_ignore': [], 'auto_add_bias': True, 'perform_only_empirical_bias_corr': True}, 'task': {'specific_scenarios': 'BasicTest', 'pretrained_model_path': '/root/Projects/video-classification-s2-1d/pre_trained_model/r2+1d_ucf101_dense_top1-94.0_unpruned.pt', 'state': {'stage': 0, 'cycles': 0}, 'max_searching': 10}, 'user_requirements': {'power': None, 'accuracy': 80, 'accuracy_reverse_yn': 0, 'model_size': None, 'memory_size': None, 'latency': 0, 'margin': 0.1, 'target_type': 'latency', 'searching_variable': 'multiplier', 'searching_range': [1, 20], 'searching_step_size': 1}, 'train': {'accuracy_reverse_yn': 0, 'common_save_best_yn': 1, 'trained_yn': False}, 'compiler': {'input_shape': '(1,3,16,112,112)', 'opset_version': 11, 'devices': ['RF8MA1GCBQK']}} Current search has 1 stages Stage: 1 Max Searching Cycles: 1 ****************************config summary************************************************ Current state: stage: 1/1| cycles: 1/1 Total jobs:1 processing job 1/1 Training... The number of videos is 9537 (with more than 16 frames) (original: 9537) The number of videos is 3783 (with more than 16 frames) (original: 3783) 2022-09-24 07:46:10,209 - root - INFO - Train dataset size: 9537 Val dataset size: 3783 2022-09-24 07:46:10,210 - root - INFO - 2022-09-24 07:46:10,470 - root - INFO - VideoResNetPlus( (stem): R2Plus1dStem( (0): Conv3d(3, 45, kernel_size=(1, 7, 7), stride=(1, 2, 2), padding=(0, 3, 3), bias=False) (1): BatchNorm3d(45, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(45, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) (4): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU(inplace=True) ) (layer1): Sequential( (0): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(64, 144, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(144, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(64, 144, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(144, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) ) (1): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(64, 144, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(144, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(64, 144, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(144, 64, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) ) ) (layer2): Sequential( (0): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(64, 230, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(230, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(230, 128, kernel_size=(3, 1, 1), stride=(2, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(128, 230, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(230, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(230, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) (downsample): Sequential( (0): Conv3d(64, 128, kernel_size=(1, 1, 1), stride=(2, 2, 2), bias=False) (1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(128, 288, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(288, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(128, 288, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(288, 128, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) ) ) (layer3): Sequential( (0): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(128, 460, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(460, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(460, 256, kernel_size=(3, 1, 1), stride=(2, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(256, 460, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(460, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(460, 256, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) (downsample): Sequential( (0): Conv3d(128, 256, kernel_size=(1, 1, 1), stride=(2, 2, 2), bias=False) (1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(256, 576, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(576, 256, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(256, 576, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(576, 256, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) ) ) (layer4): Sequential( (0): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(256, 921, kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(921, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(921, 512, kernel_size=(3, 1, 1), stride=(2, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(512, 921, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(921, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(921, 512, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) (downsample): Sequential( (0): Conv3d(256, 512, kernel_size=(1, 1, 1), stride=(2, 2, 2), bias=False) (1): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Sequential( (0): Conv2Plus1D( (0): Conv3d(512, 1152, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(1152, 512, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) ) (conv2): Sequential( (0): Conv2Plus1D( (0): Conv3d(512, 1152, kernel_size=(1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=False) (1): BatchNorm3d(1152, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU(inplace=True) (3): Conv3d(1152, 512, kernel_size=(3, 1, 1), stride=(1, 1, 1), padding=(1, 0, 0), bias=False) ) (1): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (relu): ReLU(inplace=True) ) ) (avgpool): AdaptiveAvgPool3d(output_size=(1, 1, 1)) (fc): Linear(in_features=512, out_features=101, bias=True) ) Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2016, in xgen_load model.load_state_dict(torch.load(path, map_location=device), ) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2030, in xgen_load weights = torch.load(path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2047, in xgen_load weights = torch.load(path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2209, in xgen_load model = load_checkpoint(path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 692, in load_checkpoint sim = pickle.load(file) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "xgen_scripts.py", line 82, in xgen(training_main, run, training_script_path=training_script_path,log_path = log_path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/xgen_main.py", line 1479, in xgen internal_data = train_module(job, training_main) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/train_module.py", line 325, in train_module args_ai = model_train_main(job, training_main) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/train_module.py", line 321, in model_train_main last_args = train_script_main(args_ai) File "/root/Projects/video-classification-s2-1d/train_script_main.py", line 234, in training_main xgen_load(model.module,args_ai) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2213, in xgen_load raise Exception("can't load pretrained weights, please double check the path or weights formate!") Exception: can't load pretrained weights, please double check the path or weights formate! load model from /root/Projects/video-classification-s2-1d/pre_trained_model/r2+1d_ucf101_dense_top1-94.0_unpruned.pt Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2016, in xgen_load model.load_state_dict(torch.load(path, map_location=device), ) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2030, in xgen_load weights = torch.load(path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2047, in xgen_load weights = torch.load(path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2209, in xgen_load model = load_checkpoint(path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 692, in load_checkpoint sim = pickle.load(file) _pickle.UnpicklingError: invalid load key, 'v'. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "xgen_scripts.py", line 82, in xgen(training_main, run, training_script_path=training_script_path,log_path = log_path) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/xgen_main.py", line 1479, in xgen internal_data = train_module(job, training_main) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/train_module.py", line 325, in train_module args_ai = model_train_main(job, training_main) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/train_module.py", line 321, in model_train_main last_args = train_script_main(args_ai) File "/root/Projects/video-classification-s2-1d/train_script_main.py", line 234, in training_main xgen_load(model.module,args_ai) File "/root/miniconda3/envs/xgen/lib/python3.7/site-packages/xgen_tools-0.1-py3.7.egg/xgen_tools/model_train_tools.py", line 2213, in xgen_load raise Exception("can't load pretrained weights, please double check the path or weights formate!") Exception: can't load pretrained weights, please double check the path or weights formate! error found, please check log file at /root/Projects/video-classification-s2-1d/2022-09-24-07-46-08-log.txt