2022-09-26 17:50:46,992 - mmflow - INFO - Multi-processing start method is `fork` 2022-09-26 17:50:46,992 - mmflow - INFO - OpenCV num_threads is `20 2022-09-26 17:50:47,051 - mmflow - INFO - Environment info: ------------------------------------------------------------ sys.platform: linux Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] CUDA available: True CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.2, V10.2.89 GPU 0: GeForce RTX 2080 Ti GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.12.1 PyTorch compiling details: PyTorch built with: - GCC 7.3 - C++ Version: 201402 - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815) - OpenMP 201511 (a.k.a. OpenMP 4.5) - LAPACK is enabled (usually provided by MKL) - NNPACK is enabled - CPU capability usage: AVX2 - CUDA Runtime 10.2 - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37 - CuDNN 7.6.5 - Magma 2.5.2 - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, TorchVision: 0.13.1 OpenCV: 4.6.0 MMCV: 1.6.1 MMFlow: 0.5.1+cc140b4 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 ------------------------------------------------------------ 2022-09-26 17:50:47,052 - mmflow - INFO - Distributed training: False 2022-09-26 17:50:47,364 - mmflow - INFO - Config: model = dict( type='RAFT', num_levels=4, radius=4, cxt_channels=128, h_channels=128, encoder=dict( type='RAFTEncoder', in_channels=3, out_channels=256, net_type='Basic', norm_cfg=dict(type='IN'), init_cfg=[ dict( type='Kaiming', layer=['Conv2d'], mode='fan_out', nonlinearity='relu'), dict(type='Constant', layer=['InstanceNorm2d'], val=1, bias=0) ]), cxt_encoder=dict( type='RAFTEncoder', in_channels=3, out_channels=256, net_type='Basic', norm_cfg=dict(type='SyncBN'), init_cfg=[ dict( type='Kaiming', layer=['Conv2d'], mode='fan_out', nonlinearity='relu'), dict(type='Constant', layer=['SyncBatchNorm2d'], val=1, bias=0) ]), decoder=dict( type='RAFTDecoder', net_type='Basic', num_levels=4, radius=4, iters=12, corr_op_cfg=dict(type='CorrLookup', align_corners=True), gru_type='SeqConv', flow_loss=dict(type='SequenceLoss', gamma=0.85), act_cfg=dict(type='ReLU')), freeze_bn=True, train_cfg=dict(), test_cfg=dict(iters=32)) img_norm_cfg = dict( mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False) crop_size = (288, 960) kitti_train_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict( type='ColorJitter', asymmetric_prob=0.0, brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1592356687898089), dict(type='Erase', prob=0.5, bounds=[50, 100], max_num=3), dict( type='SpacialTransform', spacial_prob=0.8, stretch_prob=0.8, crop_size=(288, 960), min_scale=-0.2, max_scale=0.4, max_stretch=0.2), dict(type='RandomCrop', crop_size=(288, 960)), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['imgs', 'flow_gt', 'valid'], meta_keys=[ 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'filename_flow', 'ori_filename_flow', 'ori_shape', 'img_shape', 'erase_bounds', 'erase_num', 'scale_factor' ]) ] kitti_train = dict( type='KITTI2015', data_root='data/kitti2015', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict( type='ColorJitter', asymmetric_prob=0.0, brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1592356687898089), dict(type='Erase', prob=0.5, bounds=[50, 100], max_num=3), dict( type='SpacialTransform', spacial_prob=0.8, stretch_prob=0.8, crop_size=(288, 960), min_scale=-0.2, max_scale=0.4, max_stretch=0.2), dict(type='RandomCrop', crop_size=(288, 960)), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['imgs', 'flow_gt', 'valid'], meta_keys=[ 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'filename_flow', 'ori_filename_flow', 'ori_shape', 'img_shape', 'erase_bounds', 'erase_num', 'scale_factor' ]) ], test_mode=False) kitti_test_pipeline = [ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict(type='InputPad', exponent=3), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='TestFormatBundle'), dict( type='Collect', keys=['imgs'], meta_keys=[ 'flow_gt', 'valid', 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg', 'scale_factor', 'pad_shape', 'pad' ]) ] kitti2015_val_test = dict( type='KITTI2015', data_root='data/kitti2015', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict(type='InputPad', exponent=3), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='TestFormatBundle'), dict( type='Collect', keys=['imgs'], meta_keys=[ 'flow_gt', 'valid', 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg', 'scale_factor', 'pad_shape', 'pad' ]) ], test_mode=True) data = dict( train_dataloader=dict( samples_per_gpu=2, workers_per_gpu=5, drop_last=True, shuffle=True, persistent_workers=True), val_dataloader=dict( samples_per_gpu=1, workers_per_gpu=5, shuffle=False, persistent_workers=True), test_dataloader=dict(samples_per_gpu=1, workers_per_gpu=2, shuffle=False), train=dict( type='KITTI2015', data_root='data/kitti2015', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict( type='ColorJitter', asymmetric_prob=0.0, brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1592356687898089), dict(type='Erase', prob=0.5, bounds=[50, 100], max_num=3), dict( type='SpacialTransform', spacial_prob=0.8, stretch_prob=0.8, crop_size=(288, 960), min_scale=-0.2, max_scale=0.4, max_stretch=0.2), dict(type='RandomCrop', crop_size=(288, 960)), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='DefaultFormatBundle'), dict( type='Collect', keys=['imgs', 'flow_gt', 'valid'], meta_keys=[ 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'filename_flow', 'ori_filename_flow', 'ori_shape', 'img_shape', 'erase_bounds', 'erase_num', 'scale_factor' ]) ], test_mode=False), val=dict( type='KITTI2015', data_root='data/kitti2015', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict(type='InputPad', exponent=3), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='TestFormatBundle'), dict( type='Collect', keys=['imgs'], meta_keys=[ 'flow_gt', 'valid', 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg', 'scale_factor', 'pad_shape', 'pad' ]) ], test_mode=True), test=dict( type='KITTI2015', data_root='data/kitti2015', pipeline=[ dict(type='LoadImageFromFile'), dict(type='LoadAnnotations', sparse=True), dict(type='InputPad', exponent=3), dict( type='Normalize', mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=False), dict(type='TestFormatBundle'), dict( type='Collect', keys=['imgs'], meta_keys=[ 'flow_gt', 'valid', 'filename1', 'filename2', 'ori_filename1', 'ori_filename2', 'ori_shape', 'img_shape', 'img_norm_cfg', 'scale_factor', 'pad_shape', 'pad' ]) ], test_mode=True)) log_config = dict( interval=50, hooks=[dict(type='TextLoggerHook'), dict(type='TensorboardLoggerHook')]) dist_params = dict(backend='nccl') log_level = 'INFO' load_from = 'https://download.openmmlab.com/mmflow/raft/raft_8x2_100k_mixed_368x768.pth' resume_from = None workflow = [('train', 1)] optimizer = dict( type='AdamW', lr=0.000125, betas=(0.9, 0.999), eps=1e-08, weight_decay=1e-05, amsgrad=False) optimizer_config = dict(grad_clip=dict(max_norm=1.0)) lr_config = dict( policy='OneCycle', max_lr=0.000125, total_steps=50100, pct_start=0.05, anneal_strategy='linear') runner = dict(type='IterBasedRunner', max_iters=50000) checkpoint_config = dict(by_epoch=False, interval=5000) evaluation = dict(interval=5000, metric='EPE') work_dir = 'raft_kitti2015_debug' auto_resume = False gpu_ids = [0] 2022-09-26 17:50:47,365 - mmflow - INFO - Set random seed to 894413846, deterministic: False 2022-09-26 17:50:47,425 - mmflow - INFO - initialize RAFTEncoder with init_cfg [{'type': 'Kaiming', 'layer': ['Conv2d'], 'mode': 'fan_out', 'nonlinearity': 'relu'}, {'type': 'Constant', 'layer': ['InstanceNorm2d'], 'val': 1, 'bias': 0}] 2022-09-26 17:50:47,486 - mmflow - INFO - initialize RAFTEncoder with init_cfg [{'type': 'Kaiming', 'layer': ['Conv2d'], 'mode': 'fan_out', 'nonlinearity': 'relu'}, {'type': 'Constant', 'layer': ['SyncBatchNorm2d'], 'val': 1, 'bias': 0}] 2022-09-26 17:50:47,501 - mmflow - INFO - RAFT( (encoder): RAFTEncoder( (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3)) (in1): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU(inplace=True) (res_layer1): ResLayer( (0): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in1): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in2): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU() ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in1): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in2): InstanceNorm2d(64, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU() ) ) (conv2): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1)) (res_layer2): ResLayer( (0): BasicBlock( (conv1): Conv2d(64, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (in1): InstanceNorm2d(96, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (conv2): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in2): InstanceNorm2d(96, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU() (downsample): Sequential( (0): Conv2d(64, 96, kernel_size=(1, 1), stride=(2, 2)) (1): InstanceNorm2d(96, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) ) ) (1): BasicBlock( (conv1): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in1): InstanceNorm2d(96, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (conv2): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in2): InstanceNorm2d(96, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU() ) ) (res_layer3): ResLayer( (0): BasicBlock( (conv1): Conv2d(96, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (in1): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in2): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU() (downsample): Sequential( (0): Conv2d(96, 128, kernel_size=(1, 1), stride=(2, 2)) (1): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in1): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (in2): InstanceNorm2d(128, eps=1e-05, momentum=0.1, affine=False, track_running_stats=False) (relu): ReLU() ) ) ) init_cfg=[{'type': 'Kaiming', 'layer': ['Conv2d'], 'mode': 'fan_out', 'nonlinearity': 'relu'}, {'type': 'Constant', 'layer': ['InstanceNorm2d'], 'val': 1, 'bias': 0}] (decoder): RAFTDecoder( (corr_block): CorrelationPyramid( (pool): AvgPool2d(kernel_size=2, stride=2, padding=0) ) (corr_lookup): CorrLookup() (encoder): MotionEncoder( (corr_net): Sequential( (0): ConvModule( (conv): Conv2d(324, 256, kernel_size=(1, 1), stride=(1, 1)) (activate): ReLU(inplace=True) ) (1): ConvModule( (conv): Conv2d(256, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (activate): ReLU(inplace=True) ) ) (flow_net): Sequential( (0): ConvModule( (conv): Conv2d(2, 128, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3)) (activate): ReLU(inplace=True) ) (1): ConvModule( (conv): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (activate): ReLU(inplace=True) ) ) (out_net): Sequential( (0): ConvModule( (conv): Conv2d(256, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (activate): ReLU(inplace=True) ) ) ) (gru): ConvGRU( (conv_z): ModuleList( (0): ConvModule( (conv): Conv2d(384, 128, kernel_size=(1, 5), stride=(1, 1), padding=(0, 2)) (activate): Sigmoid() ) (1): ConvModule( (conv): Conv2d(384, 128, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0)) (activate): Sigmoid() ) ) (conv_r): ModuleList( (0): ConvModule( (conv): Conv2d(384, 128, kernel_size=(1, 5), stride=(1, 1), padding=(0, 2)) (activate): Sigmoid() ) (1): ConvModule( (conv): Conv2d(384, 128, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0)) (activate): Sigmoid() ) ) (conv_q): ModuleList( (0): ConvModule( (conv): Conv2d(384, 128, kernel_size=(1, 5), stride=(1, 1), padding=(0, 2)) (activate): Tanh() ) (1): ConvModule( (conv): Conv2d(384, 128, kernel_size=(5, 1), stride=(1, 1), padding=(2, 0)) (activate): Tanh() ) ) ) (flow_pred): XHead( (layers): Sequential( (0): ConvModule( (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (activate): ReLU(inplace=True) ) ) (predict_layer): Conv2d(256, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) ) (mask_pred): XHead( (layers): Sequential( (0): ConvModule( (conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (activate): ReLU(inplace=True) ) ) (predict_layer): Conv2d(256, 576, kernel_size=(1, 1), stride=(1, 1)) ) (flow_loss): SequenceLoss() ) (context): RAFTEncoder( (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3)) (bn1): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU(inplace=True) (res_layer1): ResLayer( (0): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn1): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU() ) (1): BasicBlock( (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn1): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU() ) ) (conv2): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1)) (res_layer2): ResLayer( (0): BasicBlock( (conv1): Conv2d(64, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (bn1): SyncBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): SyncBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU() (downsample): Sequential( (0): Conv2d(64, 96, kernel_size=(1, 1), stride=(2, 2)) (1): SyncBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn1): SyncBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): SyncBatchNorm(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU() ) ) (res_layer3): ResLayer( (0): BasicBlock( (conv1): Conv2d(96, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (bn1): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU() (downsample): Sequential( (0): Conv2d(96, 128, kernel_size=(1, 1), stride=(2, 2)) (1): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) ) (1): BasicBlock( (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn1): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (bn2): SyncBatchNorm(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (relu): ReLU() ) ) ) init_cfg=[{'type': 'Kaiming', 'layer': ['Conv2d'], 'mode': 'fan_out', 'nonlinearity': 'relu'}, {'type': 'Constant', 'layer': ['SyncBatchNorm2d'], 'val': 1, 'bias': 0}] ) 2022-09-26 17:50:47,872 - mmflow - INFO - dataset size 200 /home/liyang/mmflow1/mmflow/apis/train.py:132: UserWarning: SyncBN is only supported with DDP. To be compatible with DP, we convert SyncBN to BN. Please use dist_train.sh which can avoid this error. warnings.warn( 2022-09-26 17:50:49,528 - mmflow - INFO - load checkpoint from http path: https://download.openmmlab.com/mmflow/raft/raft_8x2_100k_mixed_368x768.pth 2022-09-26 17:50:49,578 - mmflow - INFO - Start running, host: liyang@server-blue, work_dir: /home/liyang/mmflow1/raft_kitti2015_debug 2022-09-26 17:50:49,578 - mmflow - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) OneCycleLrUpdaterHook (NORMAL ) CheckpointHook (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- before_train_epoch: (VERY_HIGH ) OneCycleLrUpdaterHook (LOW ) IterTimerHook (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- before_train_iter: (VERY_HIGH ) OneCycleLrUpdaterHook (LOW ) IterTimerHook -------------------- after_train_iter: (ABOVE_NORMAL) OptimizerHook (NORMAL ) CheckpointHook (LOW ) IterTimerHook (LOW ) EvalHook (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- after_train_epoch: (NORMAL ) CheckpointHook (LOW ) EvalHook (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- before_val_epoch: (LOW ) IterTimerHook (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- before_val_iter: (LOW ) IterTimerHook -------------------- after_val_iter: (LOW ) IterTimerHook -------------------- after_val_epoch: (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- after_run: (VERY_LOW ) TextLoggerHook (VERY_LOW ) TensorboardLoggerHook -------------------- 2022-09-26 17:50:49,578 - mmflow - INFO - workflow: [('train', 1)], max: 50000 iters 2022-09-26 17:50:49,578 - mmflow - INFO - Checkpoints will be saved to /home/liyang/mmflow1/raft_kitti2015_debug by HardDiskBackend. /home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1659484657607/work/aten/src/ATen/native/TensorShape.cpp:2895.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Traceback (most recent call last): File "tools/train.py", line 208, in main() File "tools/train.py", line 197, in main train_model( File "/home/liyang/mmflow1/mmflow/apis/train.py", line 238, in train_model runner.run(data_loaders, cfg.workflow) File "/home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 144, in run iter_runner(iter_loaders[i], **kwargs) File "/home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 70, in train self.call_hook('after_train_iter') File "/home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 317, in call_hook getattr(hook, fn_name)(self) File "/home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/mmcv/runner/hooks/optimizer.py", line 61, in after_train_iter runner.outputs['loss'].backward() File "/home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/torch/_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/liyang/miniconda3/envs/mmcv1_pt121_cuda102/lib/python3.8/site-packages/torch/autograd/__init__.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2, 128, 36, 120]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).