nohup: 忽略输入 2022-07-29 01:20:01,961 maskrcnn_benchmark INFO: Using 2 GPUs 2022-07-29 01:20:01,961 maskrcnn_benchmark INFO: Namespace(config_file='configs/e2e_relation_X_101_32_8_FPN_1x.yaml', distributed=True, local_rank=0, opts=['MODEL.ROI_RELATION_HEAD.USE_GT_BOX', 'False', 'MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL', 'False', 'MODEL.ROI_RELATION_HEAD.PREDICTOR', 'IMPPredictor', 'SOLVER.IMS_PER_BATCH', '12', 'TEST.IMS_PER_BATCH', '2', 'DTYPE', 'float16', 'SOLVER.MAX_ITER', '40000', 'SOLVER.VAL_PERIOD', '2000', 'SOLVER.CHECKPOINT_PERIOD', '2000', 'GLOVE_DIR', '/home/user/glove', 'MODEL.PRETRAINED_DETECTOR_CKPT', '/home/user/checkpoints/pretrained_faster_rcnn/model_final.pth', 'OUTPUT_DIR', '/home/user/checkpoints/IMP-sgdet-exmp'], skip_test=False) 2022-07-29 01:20:01,961 maskrcnn_benchmark INFO: Collecting env info (might take some time) 2022-07-29 01:20:04,609 maskrcnn_benchmark INFO: PyTorch version: 1.4.0 Is debug build: No CUDA used to build PyTorch: 10.1 OS: Ubuntu 16.04.7 LTS GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609 CMake version: Could not collect Python version: 3.7 Is CUDA available: Yes CUDA runtime version: 10.1.243 GPU models and configuration: GPU 0: GeForce RTX 2080 Ti GPU 1: GeForce RTX 2080 Ti Nvidia driver version: 430.64 cuDNN version: Could not collect Versions of relevant libraries: [pip3] numpy==1.21.5 [pip3] torch==1.4.0 [pip3] torchtext==0.4.0 [pip3] torchvision==0.5.0 [conda] blas 1.0 mkl http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] mkl 2021.4.0 h06a4308_640 http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] mkl-service 2.4.0 py37h7f8727e_0 http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] mkl_fft 1.3.1 py37hd3c417c_0 http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] mkl_random 1.2.2 py37h51133e4_0 http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main [conda] pytorch 1.4.0 py3.7_cuda10.1.243_cudnn7.6.3_0 http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch [conda] torchtext 0.4.0 pyhb384e40_1 http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch [conda] torchvision 0.5.0 py37_cu101 http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch Pillow (9.2.0) 2022-07-29 01:20:04,609 maskrcnn_benchmark INFO: Loaded configuration file configs/e2e_relation_X_101_32_8_FPN_1x.yaml 2022-07-29 01:20:04,609 maskrcnn_benchmark INFO: INPUT: MIN_SIZE_TRAIN: (600,) MAX_SIZE_TRAIN: 1000 MIN_SIZE_TEST: 600 MAX_SIZE_TEST: 1000 MODEL: META_ARCHITECTURE: "GeneralizedRCNN" WEIGHT: "catalog://ImageNetPretrained/FAIR/20171220/X-101-32x8d" BACKBONE: CONV_BODY: "R-101-FPN" # VGG-16 RESNETS: BACKBONE_OUT_CHANNELS: 256 STRIDE_IN_1X1: False NUM_GROUPS: 32 WIDTH_PER_GROUP: 8 RELATION_ON: True ATTRIBUTE_ON: False FLIP_AUG: False # if there is any left-right relation, FLIP AUG should be false RPN: USE_FPN: True ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (4, 8, 16, 32, 64) ASPECT_RATIOS: (0.23232838, 0.63365731, 1.28478321, 3.15089189) # from neural-motifs PRE_NMS_TOP_N_TRAIN: 6000 PRE_NMS_TOP_N_TEST: 6000 POST_NMS_TOP_N_TRAIN: 1000 POST_NMS_TOP_N_TEST: 1000 FPN_POST_NMS_TOP_N_TRAIN: 1000 FPN_POST_NMS_TOP_N_TEST: 1000 FPN_POST_NMS_PER_BATCH: False RPN_MID_CHANNEL: 256 ROI_HEADS: USE_FPN: True POSITIVE_FRACTION: 0.5 BG_IOU_THRESHOLD: 0.3 BATCH_SIZE_PER_IMAGE: 256 DETECTIONS_PER_IMG: 80 NMS_FILTER_DUPLICATES: True ROI_BOX_HEAD: POOLER_RESOLUTION: 7 POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125) POOLER_SAMPLING_RATIO: 2 FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor" PREDICTOR: "FPNPredictor" NUM_CLASSES: 151 # 151 for VG, 1201 for GQA MLP_HEAD_DIM: 4096 ROI_ATTRIBUTE_HEAD: FEATURE_EXTRACTOR: "FPN2MLPFeatureExtractor" PREDICTOR: "FPNPredictor" USE_BINARY_LOSS: True # choose binary, because cross_entropy loss deteriorate the box head, even with 0.1 weight POS_WEIGHT: 50.0 ATTRIBUTE_LOSS_WEIGHT: 1.0 NUM_ATTRIBUTES: 201 # 201 for VG, 501 for GQA MAX_ATTRIBUTES: 10 ATTRIBUTE_BGFG_SAMPLE: True ATTRIBUTE_BGFG_RATIO: 3 ROI_RELATION_HEAD: USE_GT_BOX: True USE_GT_OBJECT_LABEL: True REQUIRE_BOX_OVERLAP: False # for sgdet, during training, only train pairs with overlap ADD_GTBOX_TO_PROPOSAL_IN_TRAIN: True # for sgdet only, in case some gt boxes are missing NUM_CLASSES: 51 # 51 for VG, 201 for GQA (not contain "to the left of" & "to the right of") BATCH_SIZE_PER_IMAGE: 1024 # sample as much as possible POSITIVE_FRACTION: 0.25 CONTEXT_POOLING_DIM: 4096 CONTEXT_HIDDEN_DIM: 512 #1024 for VCTree POOLING_ALL_LEVELS: True LABEL_SMOOTHING_LOSS: False FEATURE_EXTRACTOR: "RelationFeatureExtractor" #################### Select Relationship Model #################### #PREDICTOR: "MotifPredictor" #PREDICTOR: "VCTreePredictor" #PREDICTOR: "TransformerPredictor" PREDICTOR: "CausalAnalysisPredictor" ################# Parameters for Motif Predictor ################## CONTEXT_OBJ_LAYER: 1 CONTEXT_REL_LAYER: 1 ############# Parameters for Causal Unbias Predictor ############## ### Implementation for paper "Unbiased Scene Graph Generation from Biased Training" CAUSAL: EFFECT_TYPE: 'none' # candicates: 'TDE', 'NIE', 'TE', 'none' FUSION_TYPE: 'sum' # candicates: 'sum', 'gate' SEPARATE_SPATIAL: False # separate spatial in union feature CONTEXT_LAYER: "motifs" # candicates: motifs, vctree, vtranse SPATIAL_FOR_VISION: True EFFECT_ANALYSIS: True ############### Parameters for Transformer Predictor ############## TRANSFORMER: DROPOUT_RATE: 0.1 OBJ_LAYER: 4 REL_LAYER: 2 NUM_HEAD: 8 KEY_DIM: 64 VAL_DIM: 64 INNER_DIM: 2048 DATASETS: TRAIN: ("VG_stanford_filtered_with_attribute_train",) VAL: ("VG_stanford_filtered_with_attribute_val",) TEST: ("VG_stanford_filtered_with_attribute_test",) DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: BIAS_LR_FACTOR: 1 BASE_LR: 0.01 WARMUP_FACTOR: 0.1 WEIGHT_DECAY: 0.0001 MOMENTUM: 0.9 GRAD_NORM_CLIP: 5.0 STEPS: (10000, 16000) MAX_ITER: 40000 VAL_PERIOD: 2000 CHECKPOINT_PERIOD: 2000 PRINT_GRAD_FREQ: 4000 SCHEDULE: # the following paramters are only used for WarmupReduceLROnPlateau TYPE: "WarmupReduceLROnPlateau" # WarmupMultiStepLR, WarmupReduceLROnPlateau PATIENCE: 2 THRESHOLD: 0.001 COOLDOWN: 0 FACTOR: 0.1 MAX_DECAY_STEP: 3 OUTPUT_DIR: './output/relation_baseline' TEST: ALLOW_LOAD_FROM_CACHE: False RELATION: SYNC_GATHER: True # turn on will slow down the evaluation to solve the sgdet test out of memory problem REQUIRE_OVERLAP: False LATER_NMS_PREDICTION_THRES: 0.5 CUSTUM_EVAL: False # eval SGDet model on custum images, output a json CUSTUM_PATH: '.' # the folder that contains the custum images, only jpg files are allowed 2022-07-29 01:20:04,609 maskrcnn_benchmark INFO: Running with config: AMP_VERBOSE: False DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: TEST: ('VG_stanford_filtered_with_attribute_test',) TO_TEST: None TRAIN: ('VG_stanford_filtered_with_attribute_train',) VAL: ('VG_stanford_filtered_with_attribute_val',) DETECTED_SGG_DIR: . DTYPE: float16 GLOVE_DIR: /home/user/glove INPUT: BRIGHTNESS: 0.0 CONTRAST: 0.0 HUE: 0.0 MAX_SIZE_TEST: 1000 MAX_SIZE_TRAIN: 1000 MIN_SIZE_TEST: 600 MIN_SIZE_TRAIN: (600,) PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] SATURATION: 0.0 TO_BGR255: True VERTICAL_FLIP_PROB_TRAIN: 0.0 MODEL: ATTRIBUTE_ON: False BACKBONE: CONV_BODY: R-101-FPN FREEZE_CONV_BODY_AT: 2 CLS_AGNOSTIC_BBOX_REG: False DEVICE: cuda FBNET: ARCH: default ARCH_DEF: BN_TYPE: bn DET_HEAD_BLOCKS: [] DET_HEAD_LAST_SCALE: 1.0 DET_HEAD_STRIDE: 0 DW_CONV_SKIP_BN: True DW_CONV_SKIP_RELU: True KPTS_HEAD_BLOCKS: [] KPTS_HEAD_LAST_SCALE: 0.0 KPTS_HEAD_STRIDE: 0 MASK_HEAD_BLOCKS: [] MASK_HEAD_LAST_SCALE: 0.0 MASK_HEAD_STRIDE: 0 RPN_BN_TYPE: RPN_HEAD_BLOCKS: 0 SCALE_FACTOR: 1.0 WIDTH_DIVISOR: 1 FLIP_AUG: False FPN: USE_GN: False USE_RELU: False GROUP_NORM: DIM_PER_GP: -1 EPSILON: 1e-05 NUM_GROUPS: 32 KEYPOINT_ON: False MASK_ON: False META_ARCHITECTURE: GeneralizedRCNN PRETRAINED_DETECTOR_CKPT: /home/user/checkpoints/pretrained_faster_rcnn/model_final.pth RELATION_ON: True RESNETS: BACKBONE_OUT_CHANNELS: 256 DEFORMABLE_GROUPS: 1 NUM_GROUPS: 32 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STAGE_WITH_DCN: (False, False, False, False) STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: False TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 8 WITH_MODULATED_DCN: False RETINANET: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (0.5, 1.0, 2.0) BBOX_REG_BETA: 0.11 BBOX_REG_WEIGHT: 4.0 BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.4 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 SCALES_PER_OCTAVE: 3 STRADDLE_THRESH: 0 USE_C5: True RETINANET_ON: False ROI_ATTRIBUTE_HEAD: ATTRIBUTE_BGFG_RATIO: 3 ATTRIBUTE_BGFG_SAMPLE: True ATTRIBUTE_LOSS_WEIGHT: 1.0 FEATURE_EXTRACTOR: FPN2MLPFeatureExtractor MAX_ATTRIBUTES: 10 NUM_ATTRIBUTES: 201 POS_WEIGHT: 50.0 PREDICTOR: FPNPredictor SHARE_BOX_FEATURE_EXTRACTOR: True USE_BINARY_LOSS: True ROI_BOX_HEAD: CONV_HEAD_DIM: 256 DILATION: 1 FEATURE_EXTRACTOR: FPN2MLPFeatureExtractor MLP_HEAD_DIM: 4096 NUM_CLASSES: 151 NUM_STACKED_CONVS: 4 POOLER_RESOLUTION: 7 POOLER_SAMPLING_RATIO: 2 POOLER_SCALES: (0.25, 0.125, 0.0625, 0.03125) PREDICTOR: FPNPredictor USE_GN: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 256 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.3 DETECTIONS_PER_IMG: 80 FG_IOU_THRESHOLD: 0.5 NMS: 0.3 NMS_FILTER_DUPLICATES: True POSITIVE_FRACTION: 0.5 POST_NMS_PER_CLS_TOPN: 300 SCORE_THRESH: 0.01 USE_FPN: True ROI_KEYPOINT_HEAD: CONV_LAYERS: (512, 512, 512, 512, 512, 512, 512, 512) FEATURE_EXTRACTOR: KeypointRCNNFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: KeypointRCNNPredictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True USE_GN: False ROI_RELATION_HEAD: ADD_GTBOX_TO_PROPOSAL_IN_TRAIN: True BATCH_SIZE_PER_IMAGE: 1024 CAUSAL: CONTEXT_LAYER: motifs EFFECT_ANALYSIS: True EFFECT_TYPE: none FUSION_TYPE: sum SEPARATE_SPATIAL: False SPATIAL_FOR_VISION: True CONTEXT_DROPOUT_RATE: 0.2 CONTEXT_HIDDEN_DIM: 512 CONTEXT_OBJ_LAYER: 1 CONTEXT_POOLING_DIM: 4096 CONTEXT_REL_LAYER: 1 EMBED_DIM: 200 FEATURE_EXTRACTOR: RelationFeatureExtractor LABEL_SMOOTHING_LOSS: False NUM_CLASSES: 51 NUM_SAMPLE_PER_GT_REL: 4 POOLING_ALL_LEVELS: True POSITIVE_FRACTION: 0.25 PREDICTOR: IMPPredictor PREDICT_USE_BIAS: True PREDICT_USE_VISION: True REL_PROP: [0.01858, 0.00057, 0.00051, 0.00109, 0.0015, 0.00489, 0.00432, 0.02913, 0.00245, 0.00121, 0.00404, 0.0011, 0.00132, 0.00172, 5e-05, 0.00242, 0.0005, 0.00048, 0.00208, 0.15608, 0.0265, 0.06091, 0.009, 0.00183, 0.00225, 0.0009, 0.00028, 0.00077, 0.04844, 0.08645, 0.31621, 0.00088, 0.00301, 0.00042, 0.00186, 0.001, 0.00027, 0.01012, 0.0001, 0.01286, 0.00647, 0.00084, 0.01077, 0.00132, 0.00069, 0.00376, 0.00214, 0.11424, 0.01205, 0.02958] REQUIRE_BOX_OVERLAP: False TRANSFORMER: DROPOUT_RATE: 0.1 INNER_DIM: 2048 KEY_DIM: 64 NUM_HEAD: 8 OBJ_LAYER: 4 REL_LAYER: 2 VAL_DIM: 64 USE_GT_BOX: False USE_GT_OBJECT_LABEL: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (4, 8, 16, 32, 64) ASPECT_RATIOS: (0.23232838, 0.63365731, 1.28478321, 3.15089189) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_PER_BATCH: False FPN_POST_NMS_TOP_N_TEST: 1000 FPN_POST_NMS_TOP_N_TRAIN: 1000 MIN_SIZE: 0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 1000 PRE_NMS_TOP_N_TEST: 6000 PRE_NMS_TOP_N_TRAIN: 6000 RPN_HEAD: SingleConvRPNHead RPN_MID_CHANNEL: 256 STRADDLE_THRESH: 0 USE_FPN: True RPN_ONLY: False VGG: VGG16_OUT_CHANNELS: 512 WEIGHT: catalog://ImageNetPretrained/FAIR/20171220/X-101-32x8d OUTPUT_DIR: /home/user/checkpoints/IMP-sgdet-exmp PATHS_CATALOG: /home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/config/paths_catalog.py PATHS_DATA: /home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/config/../data/datasets SOLVER: BASE_LR: 0.01 BIAS_LR_FACTOR: 1 CHECKPOINT_PERIOD: 2000 CLIP_NORM: 5.0 GAMMA: 0.1 GRAD_NORM_CLIP: 5.0 IMS_PER_BATCH: 12 MAX_ITER: 40000 MOMENTUM: 0.9 PRE_VAL: True PRINT_GRAD_FREQ: 4000 SCHEDULE: COOLDOWN: 0 FACTOR: 0.1 MAX_DECAY_STEP: 3 PATIENCE: 2 THRESHOLD: 0.001 TYPE: WarmupReduceLROnPlateau STEPS: (10000, 16000) TO_VAL: True UPDATE_SCHEDULE_DURING_LOAD: False VAL_PERIOD: 2000 WARMUP_FACTOR: 0.1 WARMUP_ITERS: 500 WARMUP_METHOD: linear WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0.0 TEST: ALLOW_LOAD_FROM_CACHE: False BBOX_AUG: ENABLED: False H_FLIP: False MAX_SIZE: 4000 SCALES: () SCALE_H_FLIP: False CUSTUM_EVAL: False CUSTUM_PATH: . DETECTIONS_PER_IMG: 100 EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 2 RELATION: IOU_THRESHOLD: 0.5 LATER_NMS_PREDICTION_THRES: 0.5 MULTIPLE_PREDS: False REQUIRE_OVERLAP: False SYNC_GATHER: True SAVE_PROPOSALS: False 2022-07-29 01:20:04,609 maskrcnn_benchmark INFO: Saving config into: /home/user/checkpoints/IMP-sgdet-exmp/config.yml 2022-07-29 01:20:04,623 maskrcnn_benchmark INFO: #################### prepare training #################### INIT SAVE DIR /home/user/checkpoints/IMP-sgdet-exmp get_checkpoint_file /home/user/checkpoints/IMP-sgdet-exmp/last_checkpoint last_saved /home/user/checkpoints/IMP-sgdet-exmp/model_0004000.pth 2022-07-29 01:20:07,061 maskrcnn_benchmark INFO: #################### end model construction #################### 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.union_feature_extractor.feature_extractor.pooler.reduce_channel.0.weight is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.union_feature_extractor.feature_extractor.pooler.reduce_channel.0.bias is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.union_feature_extractor.feature_extractor.fc6.weight is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.union_feature_extractor.feature_extractor.fc6.bias is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.union_feature_extractor.feature_extractor.fc7.weight is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.union_feature_extractor.feature_extractor.fc7.bias is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.box_feature_extractor.fc6.weight is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.box_feature_extractor.fc6.bias is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.box_feature_extractor.fc7.weight is slow down by ratio of 10.0. 2022-07-29 01:20:07,211 maskrcnn_benchmark INFO: SLOW HEADS: roi_heads.relation.box_feature_extractor.fc7.bias is slow down by ratio of 10.0. 2022-07-29 01:20:07,212 maskrcnn_benchmark INFO: #################### end optimizer and shcedule #################### Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods. Defaults for this optimization level are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic Processing user overrides (additional kwargs that are not None)... After processing overrides, optimization options are: enabled : True opt_level : O1 cast_model_type : None patch_torch_functions : True keep_batchnorm_fp32 : None master_weights : None loss_scale : dynamic 2022-07-29 01:20:07,229 maskrcnn_benchmark INFO: #################### end distributed #################### INIT SAVE DIR /home/user/checkpoints/IMP-sgdet-exmp get_checkpoint_file /home/user/checkpoints/IMP-sgdet-exmp/last_checkpoint last_saved /home/user/checkpoints/IMP-sgdet-exmp/model_0004000.pth 2022-07-29 01:20:07,230 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from /home/user/checkpoints/IMP-sgdet-exmp/model_0004000.pth 2022-07-29 01:20:08,617 maskrcnn_benchmark.utils.checkpoint INFO: Loading optimizer from /home/user/checkpoints/IMP-sgdet-exmp/model_0004000.pth When using more than one image per GPU you may encounter an out-of-memory (OOM) error if your GPU does not have sufficient memory. If this happens, you can reduce SOLVER.IMS_PER_BATCH (for training) or TEST.IMS_PER_BATCH (for inference). For training, you must also adjust the learning rate and schedule length according to the linear scaling rule. See for example: https://github.com/facebookresearch/Detectron/blob/master/configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml#L14 2022-07-29 01:20:08,886 maskrcnn_benchmark INFO: #################### end load checkpointer #################### 2022-07-29 01:20:08,886 maskrcnn_benchmark.data.build WARNING: When using more than one image per GPU you may encounter an out-of-memory (OOM) error if your GPU does not have sufficient memory. If this happens, you can reduce SOLVER.IMS_PER_BATCH (for training) or TEST.IMS_PER_BATCH (for inference). For training, you must also adjust the learning rate and schedule length according to the linear scaling rule. See for example: https://github.com/facebookresearch/Detectron/blob/master/configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml#L14 2022-07-29 01:20:11,404 maskrcnn_benchmark.utils.miscellaneous INFO: Saving labels mapping into /home/user/checkpoints/IMP-sgdet-exmp/labels.json 2022-07-29 01:20:12,200 maskrcnn_benchmark INFO: #################### end dataloader #################### 2022-07-29 01:20:12,200 maskrcnn_benchmark INFO: Validate before training 2022-07-29 01:20:12,200 maskrcnn_benchmark INFO: Start evaluation on VG_stanford_filtered_with_attribute_val dataset(5000 images). 0%| | 0/2500 [00:00 main() File "tools/relation_train_net.py", line 372, in main model = train(cfg, args.local_rank, args.distributed, logger) File "tools/relation_train_net.py", line 203, in train val_result = run_val(cfg, model, val_data_loaders, distributed, logger) File "tools/relation_train_net.py", line 259, in run_val logger=logger, File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/engine/inference.py", line 110, in inference predictions = compute_on_dataset(model, data_loader, device, synchronize_gather=cfg.TEST.RELATION.SYNC_GATHER, timer=inference_timer) File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/engine/inference.py", line 34, in compute_on_dataset output = model(images.to(device), targets) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/modeling/detector/generalized_rcnn.py", line 52, in forward x, result, detector_losses = self.roi_heads(features, proposals, targets, logger) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 27, in forward x, detections, loss_box = self.box(features, proposals, targets) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/box_head.py", line 74, in forward x, result = self.post_processor((x, class_logits, box_regression), proposals, relation_mode=True) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/inference.py", line 97, in forward boxlist, orig_inds, boxes_per_cls = self.filter_results(boxlist, num_classes) File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/modeling/roi_heads/box_head/inference.py", line 180, in filter_results boxlist_for_class, self.nms, max_proposals=self.post_nms_per_cls_topn, score_field='pred_scores' File "/home/user/PycharmProjects/deeplabv3+/dataloaders/datasets/__pycache__/Scene-Graph-Benchmark/maskrcnn_benchmark/structures/boxlist_ops.py", line 28, in boxlist_nms keep = _box_nms(boxes, score, nms_thresh) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/amp.py", line 25, in wrapper return wrap_fn(orig_fn, inner_cast_fn, handle)(*args, **kwargs) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/wrap.py", line 27, in wrapper kwargs) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/utils.py", line 80, in casted_args if is_fp_tensor(x): File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/utils.py", line 21, in is_fp_tensor return compat.is_tensor_like(x) and compat.is_floating_point(x) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/apex-0.1-py3.7-linux-x86_64.egg/apex/amp/compat.py", line 26, in is_floating_point return torch.is_floating_point(x) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 48413) is killed by signal: Killed. Traceback (most recent call last): File "/home/user/anaconda3/envs/sg/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/user/anaconda3/envs/sg/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in main() File "/home/user/anaconda3/envs/sg/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main cmd=cmd) subprocess.CalledProcessError: Command '['/home/user/anaconda3/envs/sg/bin/python', '-u', 'tools/relation_train_net.py', '--local_rank=1', '--config-file', 'configs/e2e_relation_X_101_32_8_FPN_1x.yaml', 'MODEL.ROI_RELATION_HEAD.USE_GT_BOX', 'False', 'MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL', 'False', 'MODEL.ROI_RELATION_HEAD.PREDICTOR', 'IMPPredictor', 'SOLVER.IMS_PER_BATCH', '12', 'TEST.IMS_PER_BATCH', '2', 'DTYPE', 'float16', 'SOLVER.MAX_ITER', '40000', 'SOLVER.VAL_PERIOD', '2000', 'SOLVER.CHECKPOINT_PERIOD', '2000', 'GLOVE_DIR', '/home/user/glove', 'MODEL.PRETRAINED_DETECTOR_CKPT', '/home/user/checkpoints/pretrained_faster_rcnn/model_final.pth', 'OUTPUT_DIR', '/home/user/checkpoints/IMP-sgdet-exmp']' returned non-zero exit status 1. ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. *****************************************