2022-07-13 19:25:03,293 fcos_core INFO: Using 1 GPUs 2022-07-13 19:25:03,293 fcos_core INFO: Namespace(config_file='configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml', distributed=False, local_rank=0, opts=[], skip_test=False, test_only=False, use_tensorboard=True) 2022-07-13 19:25:03,294 fcos_core INFO: Collecting env info (might take some time) 2022-07-13 19:25:08,129 fcos_core INFO: PyTorch version: 1.4.0 Is debug build: No CUDA used to build PyTorch: 10.1 OS: Ubuntu 18.04.6 LTS GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 CMake version: Could not collect Python version: 3.7 Is CUDA available: Yes CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA RTX A4000 GPU 1: NVIDIA RTX A4000 GPU 2: NVIDIA RTX A4000 GPU 3: NVIDIA RTX A4000 GPU 4: NVIDIA RTX A4000 GPU 5: NVIDIA RTX A4000 GPU 6: NVIDIA RTX A4000 GPU 7: NVIDIA RTX A4000 Nvidia driver version: 470.86 cuDNN version: Could not collect Versions of relevant libraries: [pip3] numpy==1.21.6 [pip3] torch==1.4.0 [pip3] torchvision==0.2.1 [conda] torch 1.4.0 pypi_0 pypi [conda] torchvision 0.2.1 pypi_0 pypi Pillow (9.2.0) 2022-07-13 19:25:08,130 fcos_core INFO: Loaded configuration file configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml 2022-07-13 19:25:08,130 fcos_core INFO: OUTPUT_DIR: './experiments/sigma/sim10k_to_cityscapes_vgg16/' MODEL: META_ARCHITECTURE: "GeneralizedRCNN" WEIGHT: 'https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth' # Initialed by imagenet # NOTE: In our cvpr version, we mistakely set FCOS.NMS_TH as 0.3, giving a 53.7 result. After setting it correctly, sigma gives 57.1 mAP.... # WEIGHT: './well_trained_models/sim10k_to_city_vgg16_53.73_mAP.pth' # Initialed by pretrained weight RPN_ONLY: True FCOS_ON: True DA_ON: True ATSS_ON: False MIDDLE_HEAD_CFG: 'GM_HEAD' MIDDLE_HEAD: CONDGRAPH_ON: True IN_NORM: 'LN' NUM_CONVS_IN: 2 GM: # matching cfg MATCHING_LOSS_CFG: 'MSE' MATCHING_CFG: 'none' WITH_SCORE_WEIGHT: False WITH_NODE_DIS: True # node sampling NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 BG_RATIO: 2 # loss weight MATCHING_LOSS_WEIGHT: 1.0 NODE_LOSS_WEIGHT: 1.0 NODE_DIS_WEIGHT: 0.1 NODE_DIS_LAMBDA: 0.02 WITH_SEMANTIC_COMPLETION: True WITH_QUADRATIC_MATCHING: True WITH_CLUSTER_UPDATE: True WITH_CTR: False WITH_COMPLETE_GRAPH: True WITH_DOMAIN_INTERACTION: True BACKBONE: CONV_BODY: "VGG-16-FPN-RETINANET" RETINANET: USE_C5: False # FCOS uses P5 instead of C5 FCOS: NUM_CONVS_REG: 4 NUM_CONVS_CLS: 4 NUM_CLASSES: 2 INFERENCE_TH: 0.05 # pre_nms_thresh (default=0.05) PRE_NMS_TOP_N: 1000 # pre_nms_top_n (default=1000) NMS_TH: 0.6 # nms_thresh (default=0.6) REG_CTR_ON: True ADV: GA_DIS_LAMBDA: 0.1 # for dis loss CON_NUM_SHARED_CONV_P7: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P3: 4 # USE_DIS_GLOBAL: True USE_DIS_P7: True USE_DIS_P6: True USE_DIS_P5: True USE_DIS_P4: True USE_DIS_P3: True GRL_WEIGHT_P7: 0.02 # for gradient GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P3: 0.02 TEST: DETECTIONS_PER_IMG: 100 # fpn_post_nms_top_n (default=100) MODE: 'common' DATASETS: TRAIN_SOURCE: ("sim10k_trainval_caronly", ) TRAIN_TARGET: ("cityscapes_train_caronly_cocostyle", ) TEST: ("cityscapes_val_caronly_cocostyle", ) INPUT: MIN_SIZE_RANGE_TRAIN: (640, 800) MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MAX_SIZE_TEST: 1333 DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: VAL_ITER: 100 ADAPT_VAL_ON: True INITIAL_AP50: 35 WEIGHT_DECAY: 0.0001 MAX_ITER: 200000 # 4 for source and 4 for target IMS_PER_BATCH: 2 CHECKPOINT_PERIOD: 100000 # BACKBONE: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" MIDDLE_HEAD: BASE_LR: 0.005 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" PLABEL_TH: (0.5, 1.0) FCOS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" # DIS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" 2022-07-13 19:25:08,133 fcos_core INFO: Running with config: CLS_MAP_PRE: softmax DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: TEST: ('cityscapes_val_caronly_cocostyle',) TRAIN_SOURCE: ('sim10k_trainval_caronly',) TRAIN_TARGET: ('cityscapes_train_caronly_cocostyle',) INPUT: MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_RANGE_TRAIN: (640, 800) MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN: (800,) PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] TO_BGR255: True MODEL: ADV: BASE_DIS_TOWER: False CA_DIS_LAMBDA: 0.1 CA_DIS_P3_NUM_CONVS: 4 CA_DIS_P4_NUM_CONVS: 4 CA_DIS_P5_NUM_CONVS: 4 CA_DIS_P6_NUM_CONVS: 4 CA_DIS_P7_NUM_CONVS: 4 CA_GRL_WEIGHT_P3: 0.1 CA_GRL_WEIGHT_P4: 0.1 CA_GRL_WEIGHT_P5: 0.1 CA_GRL_WEIGHT_P6: 0.1 CA_GRL_WEIGHT_P7: 0.1 CENTER_AWARE_TYPE: ca_feature CENTER_AWARE_WEIGHT: 20 CON_DIS_LAMBDA: 0.1 CON_FUSUIN_CFG: concat CON_NUM_SHARED_CONV_P3: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P7: 4 CON_WITH_GA: False DIS_P3_NUM_CONVS: 4 DIS_P4_NUM_CONVS: 4 DIS_P5_NUM_CONVS: 4 DIS_P6_NUM_CONVS: 4 DIS_P7_NUM_CONVS: 4 GA_DIS_LAMBDA: 0.1 GRL_APPLIED_DOMAIN: both GRL_WEIGHT_P3: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P7: 0.02 OUTMAP_OP: sigmoid OUTPUT_CENTERNESS_DA: True OUTPUT_CLS_DA: True OUTPUT_REG_DA: True OUT_DIS_LAMBDA: 0.1 OUT_LOSS: ce OUT_WEIGHT: 0.5 PATCH_STRIDE: None USE_DIS_CENTER_AWARE: False USE_DIS_CON: False USE_DIS_GLOBAL: True USE_DIS_OUT: False USE_DIS_P3: True USE_DIS_P3_CON: False USE_DIS_P4: True USE_DIS_P4_CON: False USE_DIS_P5: True USE_DIS_P5_CON: False USE_DIS_P6: True USE_DIS_P6_CON: False USE_DIS_P7: True USE_DIS_P7_CON: False ATSS: ANCHOR_SIZES: (64, 128, 256, 512, 1024) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (1.0,) BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 5.0 NMS_TH: 0.6 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 POSITIVE_TYPE: ATSS PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REGRESSION_TYPE: BOX REG_LOSS_WEIGHT: 2.0 SCALES_PER_OCTAVE: 1 STRADDLE_THRESH: 0 TOPK: 9 USE_DCN_IN_TOWER: False ATSS_ON: False BACKBONE: CONV_BODY: VGG-16-FPN-RETINANET FREEZE_CONV_BODY_AT: 2 USE_GN: False VGG_W_BN: False CLS_AGNOSTIC_BBOX_REG: False DA_ON: True DEBUG_CFG: None DEVICE: cuda FBNET: ARCH: default ARCH_DEF: BN_TYPE: bn DET_HEAD_BLOCKS: [] DET_HEAD_LAST_SCALE: 1.0 DET_HEAD_STRIDE: 0 DW_CONV_SKIP_BN: True DW_CONV_SKIP_RELU: True KPTS_HEAD_BLOCKS: [] KPTS_HEAD_LAST_SCALE: 0.0 KPTS_HEAD_STRIDE: 0 MASK_HEAD_BLOCKS: [] MASK_HEAD_LAST_SCALE: 0.0 MASK_HEAD_STRIDE: 0 RPN_BN_TYPE: RPN_HEAD_BLOCKS: 0 SCALE_FACTOR: 1.0 WIDTH_DIVISOR: 1 FCOS: FPN_STRIDES: [8, 16, 32, 64, 128] INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.6 NUM_CLASSES: 2 NUM_CONVS: 4 NUM_CONVS_CLS: 4 NUM_CONVS_REG: 4 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REG_CTR_ON: True FCOS_ON: True FPN: USE_GN: False USE_RELU: False GROUP_NORM: DIM_PER_GP: -1 EPSILON: 1e-05 NUM_GROUPS: 32 KEYPOINT_ON: False MASK_ON: False META_ARCHITECTURE: GeneralizedRCNN MIDDLE_HEAD: ACT_LOSS: None ACT_LOSS_WEIGHT: 1.0 CAT_ACT_MAP: True CONDGRAPH_ON: True COND_WITH_BIAS: False CON_LOSS_WEIGHT: 1.0 CON_TG_CFG: KLdiv GCN1_OUT_CHANNEL: 256 GCN2_OUT_CHANNEL: 256 GCN_EDGE_NORM: softmax GCN_EDGE_PROJECT: 128 GCN_LOSS_WEIGHT: 1.0 GCN_LOSS_WEIGHT_TG: 1.0 GCN_OUT_ACTIVATION: relu GCN_SHORTCUT: False GLOBAL_GRAPH_ON: False GM: BG_RATIO: 2 MATCHING_CFG: none MATCHING_LOSS_CFG: MSE MATCHING_LOSS_WEIGHT: 1.0 MIN_AP50_SAVE: 40 NODE_DIS_LAMBDA: 0.02 NODE_DIS_PLACE: feat NODE_DIS_WEIGHT: 0.1 NODE_LOSS_WEIGHT: 1.0 NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 WITH_CLUSTER_UPDATE: True WITH_COMPLETE_GRAPH: True WITH_COND_CLS: False WITH_CTR: False WITH_DOMAIN_INTERACTION: True WITH_GLOBAL_GRAPH: False WITH_NODE_DIS: True WITH_QUADRATIC_MATCHING: True WITH_SCORE_WEIGHT: False WITH_SEMANTIC_COMPLETION: True GM_ON: False IN_NORM: LN NUM_CONVS_IN: 2 NUM_CONVS_OUT: 1 PROTO_CHANNEL: 256 PROTO_MOMENTUM: 0.95 PROTO_WITH_BG: True RETURN_ACT_LOGITS: False MIDDLE_HEAD_CFG: GM_HEAD RESNETS: BACKBONE_OUT_CHANNELS: 1024 NUM_GROUPS: 1 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 64 RETINANET: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (0.5, 1.0, 2.0) BBOX_REG_BETA: 0.11 BBOX_REG_WEIGHT: 4.0 BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.4 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 SCALES_PER_OCTAVE: 3 STRADDLE_THRESH: 0 USE_C5: False RETINANET_ON: False ROI_BOX_HEAD: CONV_HEAD_DIM: 256 DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 81 NUM_STACKED_CONVS: 4 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: FastRCNNPredictor USE_GN: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.5 DETECTIONS_PER_IMG: 100 FG_IOU_THRESHOLD: 0.5 NMS: 0.5 POSITIVE_FRACTION: 0.25 SCORE_THRESH: 0.05 USE_FPN: False ROI_KEYPOINT_HEAD: CONV_LAYERS: (512, 512, 512, 512, 512, 512, 512, 512) FEATURE_EXTRACTOR: KeypointRCNNFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: KeypointRCNNPredictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True USE_GN: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (16,) ASPECT_RATIOS: (0.5, 1.0, 2.0) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_TOP_N_TEST: 2000 FPN_POST_NMS_TOP_N_TRAIN: 2000 MIN_SIZE: 0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 6000 PRE_NMS_TOP_N_TRAIN: 12000 RPN_HEAD: SingleConvRPNHead STRADDLE_THRESH: 0 USE_FPN: False RPN_ONLY: True USE_SYNCBN: False WEIGHT: https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth OUTPUT_DIR: ./experiments/sigma/sim10k_to_cityscapes_vgg16/ PATHS_CATALOG: /home/sh/SIGMA-main/fcos_core/config/paths_catalog.py SOLVER: ADAPT_VAL_ON: True BACKBONE: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) SWA: False WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant CHECKPOINT_PERIOD: 100000 DIS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant FCOS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant IMS_PER_BATCH: 2 INITIAL_AP50: 35 MAX_ITER: 200000 MIDDLE_HEAD: BASE_LR: 0.005 BIAS_LR_FACTOR: 2 GAMMA: 0.1 PLABEL_TH: (0.5, 1.0) STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant MOMENTUM: 0.9 VAL_ITER: 100 VAL_TYPE: AP50 WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0 TENSORBOARD_EXPERIMENT: ./exps/demo/logs/ TEST: DETECTIONS_PER_IMG: 100 EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 4 MODE: common 2022-07-13 19:45:46,361 fcos_core.trainer INFO: node dis setting: feat 2022-07-13 19:45:46,384 fcos_core.trainer INFO: node_dis initialized 2022-07-13 19:45:46,385 fcos_core.trainer INFO: node_cls_middle initialized 2022-07-13 19:45:46,387 fcos_core.trainer INFO: head_in_ln initialized 2022-07-13 19:45:46,699 fcos_core.utils.checkpoint INFO: Loading checkpoint from https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth 2022-07-13 19:45:46,699 fcos_core.utils.checkpoint INFO: url https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth cached in /home/sh/.torch/models/vgg16_caffe-292e1171.pth 2022-07-13 19:45:46,752 fcos_core.utils.model_serialization INFO: body.features.0.bias loaded from features.0.bias of shape (64,) 2022-07-13 19:45:46,753 fcos_core.utils.model_serialization INFO: body.features.0.weight loaded from features.0.weight of shape (64, 3, 3, 3) 2022-07-13 19:45:46,753 fcos_core.utils.model_serialization INFO: body.features.10.bias loaded from features.10.bias of shape (256,) 2022-07-13 19:45:46,753 fcos_core.utils.model_serialization INFO: body.features.10.weight loaded from features.10.weight of shape (256, 128, 3, 3) 2022-07-13 19:45:46,753 fcos_core.utils.model_serialization INFO: body.features.12.bias loaded from features.12.bias of shape (256,) 2022-07-13 19:45:46,753 fcos_core.utils.model_serialization INFO: body.features.12.weight loaded from features.12.weight of shape (256, 256, 3, 3) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.14.bias loaded from features.14.bias of shape (256,) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.14.weight loaded from features.14.weight of shape (256, 256, 3, 3) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.17.bias loaded from features.17.bias of shape (512,) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.17.weight loaded from features.17.weight of shape (512, 256, 3, 3) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.19.bias loaded from features.19.bias of shape (512,) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.19.weight loaded from features.19.weight of shape (512, 512, 3, 3) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.2.bias loaded from features.2.bias of shape (64,) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.2.weight loaded from features.2.weight of shape (64, 64, 3, 3) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.21.bias loaded from features.21.bias of shape (512,) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.21.weight loaded from features.21.weight of shape (512, 512, 3, 3) 2022-07-13 19:45:46,754 fcos_core.utils.model_serialization INFO: body.features.24.bias loaded from features.24.bias of shape (512,) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.24.weight loaded from features.24.weight of shape (512, 512, 3, 3) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.26.bias loaded from features.26.bias of shape (512,) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.26.weight loaded from features.26.weight of shape (512, 512, 3, 3) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.28.bias loaded from features.28.bias of shape (512,) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.28.weight loaded from features.28.weight of shape (512, 512, 3, 3) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.5.bias loaded from features.5.bias of shape (128,) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.5.weight loaded from features.5.weight of shape (128, 64, 3, 3) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.7.bias loaded from features.7.bias of shape (128,) 2022-07-13 19:45:46,755 fcos_core.utils.model_serialization INFO: body.features.7.weight loaded from features.7.weight of shape (128, 128, 3, 3) 2022-07-13 19:45:48,175 fcos_core.data.build WARNING: When using more than one image per GPU you may encounter an out-of-memory (OOM) error if your GPU does not have sufficient memory. If this happens, you can reduce SOLVER.IMS_PER_BATCH (for training) or TEST.IMS_PER_BATCH (for inference). For training, you must also adjust the learning rate and schedule length according to the linear scaling rule. See for example: https://github.com/facebookresearch/Detectron/blob/master/configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml#L14 2022-07-13 19:45:52,690 fcos_core.trainer INFO: Start training 2022-07-13 20:16:51,947 fcos_core INFO: Using 1 GPUs 2022-07-13 20:16:51,947 fcos_core INFO: Namespace(config_file='configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml', distributed=False, local_rank=0, opts=[], skip_test=False, test_only=False, use_tensorboard=True) 2022-07-13 20:16:51,948 fcos_core INFO: Collecting env info (might take some time) 2022-07-13 20:16:56,834 fcos_core INFO: PyTorch version: 1.4.0 Is debug build: No CUDA used to build PyTorch: 10.1 OS: Ubuntu 18.04.6 LTS GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 CMake version: Could not collect Python version: 3.7 Is CUDA available: Yes CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA RTX A4000 GPU 1: NVIDIA RTX A4000 GPU 2: NVIDIA RTX A4000 GPU 3: NVIDIA RTX A4000 GPU 4: NVIDIA RTX A4000 GPU 5: NVIDIA RTX A4000 GPU 6: NVIDIA RTX A4000 GPU 7: NVIDIA RTX A4000 Nvidia driver version: 470.86 cuDNN version: Could not collect Versions of relevant libraries: [pip3] numpy==1.21.6 [pip3] torch==1.4.0 [pip3] torchvision==0.2.1 [conda] torch 1.4.0 pypi_0 pypi [conda] torchvision 0.2.1 pypi_0 pypi Pillow (9.2.0) 2022-07-13 20:16:56,834 fcos_core INFO: Loaded configuration file configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml 2022-07-13 20:16:56,834 fcos_core INFO: OUTPUT_DIR: './experiments/sigma/sim10k_to_cityscapes_vgg16/' MODEL: META_ARCHITECTURE: "GeneralizedRCNN" WEIGHT: 'https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth' # Initialed by imagenet # NOTE: In our cvpr version, we mistakely set FCOS.NMS_TH as 0.3, giving a 53.7 result. After setting it correctly, sigma gives 57.1 mAP.... # WEIGHT: './well_trained_models/sim10k_to_city_vgg16_53.73_mAP.pth' # Initialed by pretrained weight RPN_ONLY: True FCOS_ON: True DA_ON: True ATSS_ON: False MIDDLE_HEAD_CFG: 'GM_HEAD' MIDDLE_HEAD: CONDGRAPH_ON: True IN_NORM: 'LN' NUM_CONVS_IN: 2 GM: # matching cfg MATCHING_LOSS_CFG: 'MSE' MATCHING_CFG: 'none' WITH_SCORE_WEIGHT: False WITH_NODE_DIS: True # node sampling NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 BG_RATIO: 2 # loss weight MATCHING_LOSS_WEIGHT: 1.0 NODE_LOSS_WEIGHT: 1.0 NODE_DIS_WEIGHT: 0.1 NODE_DIS_LAMBDA: 0.02 WITH_SEMANTIC_COMPLETION: True WITH_QUADRATIC_MATCHING: True WITH_CLUSTER_UPDATE: True WITH_CTR: False WITH_COMPLETE_GRAPH: True WITH_DOMAIN_INTERACTION: True BACKBONE: CONV_BODY: "VGG-16-FPN-RETINANET" RETINANET: USE_C5: False # FCOS uses P5 instead of C5 FCOS: NUM_CONVS_REG: 4 NUM_CONVS_CLS: 4 NUM_CLASSES: 2 INFERENCE_TH: 0.05 # pre_nms_thresh (default=0.05) PRE_NMS_TOP_N: 1000 # pre_nms_top_n (default=1000) NMS_TH: 0.6 # nms_thresh (default=0.6) REG_CTR_ON: True ADV: GA_DIS_LAMBDA: 0.1 # for dis loss CON_NUM_SHARED_CONV_P7: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P3: 4 # USE_DIS_GLOBAL: True USE_DIS_P7: True USE_DIS_P6: True USE_DIS_P5: True USE_DIS_P4: True USE_DIS_P3: True GRL_WEIGHT_P7: 0.02 # for gradient GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P3: 0.02 TEST: DETECTIONS_PER_IMG: 100 # fpn_post_nms_top_n (default=100) MODE: 'common' DATASETS: TRAIN_SOURCE: ("sim10k_trainval_caronly", ) TRAIN_TARGET: ("cityscapes_train_caronly_cocostyle", ) TEST: ("cityscapes_val_caronly_cocostyle", ) INPUT: MIN_SIZE_RANGE_TRAIN: (640, 800) MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MAX_SIZE_TEST: 1333 DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: VAL_ITER: 100 ADAPT_VAL_ON: True INITIAL_AP50: 35 WEIGHT_DECAY: 0.0001 MAX_ITER: 200000 # 4 for source and 4 for target IMS_PER_BATCH: 2 CHECKPOINT_PERIOD: 100000 # BACKBONE: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" MIDDLE_HEAD: BASE_LR: 0.005 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" PLABEL_TH: (0.5, 1.0) FCOS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" # DIS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" 2022-07-13 20:16:56,840 fcos_core INFO: Running with config: CLS_MAP_PRE: softmax DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: TEST: ('cityscapes_val_caronly_cocostyle',) TRAIN_SOURCE: ('sim10k_trainval_caronly',) TRAIN_TARGET: ('cityscapes_train_caronly_cocostyle',) INPUT: MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_RANGE_TRAIN: (640, 800) MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN: (800,) PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] TO_BGR255: True MODEL: ADV: BASE_DIS_TOWER: False CA_DIS_LAMBDA: 0.1 CA_DIS_P3_NUM_CONVS: 4 CA_DIS_P4_NUM_CONVS: 4 CA_DIS_P5_NUM_CONVS: 4 CA_DIS_P6_NUM_CONVS: 4 CA_DIS_P7_NUM_CONVS: 4 CA_GRL_WEIGHT_P3: 0.1 CA_GRL_WEIGHT_P4: 0.1 CA_GRL_WEIGHT_P5: 0.1 CA_GRL_WEIGHT_P6: 0.1 CA_GRL_WEIGHT_P7: 0.1 CENTER_AWARE_TYPE: ca_feature CENTER_AWARE_WEIGHT: 20 CON_DIS_LAMBDA: 0.1 CON_FUSUIN_CFG: concat CON_NUM_SHARED_CONV_P3: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P7: 4 CON_WITH_GA: False DIS_P3_NUM_CONVS: 4 DIS_P4_NUM_CONVS: 4 DIS_P5_NUM_CONVS: 4 DIS_P6_NUM_CONVS: 4 DIS_P7_NUM_CONVS: 4 GA_DIS_LAMBDA: 0.1 GRL_APPLIED_DOMAIN: both GRL_WEIGHT_P3: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P7: 0.02 OUTMAP_OP: sigmoid OUTPUT_CENTERNESS_DA: True OUTPUT_CLS_DA: True OUTPUT_REG_DA: True OUT_DIS_LAMBDA: 0.1 OUT_LOSS: ce OUT_WEIGHT: 0.5 PATCH_STRIDE: None USE_DIS_CENTER_AWARE: False USE_DIS_CON: False USE_DIS_GLOBAL: True USE_DIS_OUT: False USE_DIS_P3: True USE_DIS_P3_CON: False USE_DIS_P4: True USE_DIS_P4_CON: False USE_DIS_P5: True USE_DIS_P5_CON: False USE_DIS_P6: True USE_DIS_P6_CON: False USE_DIS_P7: True USE_DIS_P7_CON: False ATSS: ANCHOR_SIZES: (64, 128, 256, 512, 1024) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (1.0,) BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 5.0 NMS_TH: 0.6 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 POSITIVE_TYPE: ATSS PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REGRESSION_TYPE: BOX REG_LOSS_WEIGHT: 2.0 SCALES_PER_OCTAVE: 1 STRADDLE_THRESH: 0 TOPK: 9 USE_DCN_IN_TOWER: False ATSS_ON: False BACKBONE: CONV_BODY: VGG-16-FPN-RETINANET FREEZE_CONV_BODY_AT: 2 USE_GN: False VGG_W_BN: False CLS_AGNOSTIC_BBOX_REG: False DA_ON: True DEBUG_CFG: None DEVICE: cuda FBNET: ARCH: default ARCH_DEF: BN_TYPE: bn DET_HEAD_BLOCKS: [] DET_HEAD_LAST_SCALE: 1.0 DET_HEAD_STRIDE: 0 DW_CONV_SKIP_BN: True DW_CONV_SKIP_RELU: True KPTS_HEAD_BLOCKS: [] KPTS_HEAD_LAST_SCALE: 0.0 KPTS_HEAD_STRIDE: 0 MASK_HEAD_BLOCKS: [] MASK_HEAD_LAST_SCALE: 0.0 MASK_HEAD_STRIDE: 0 RPN_BN_TYPE: RPN_HEAD_BLOCKS: 0 SCALE_FACTOR: 1.0 WIDTH_DIVISOR: 1 FCOS: FPN_STRIDES: [8, 16, 32, 64, 128] INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.6 NUM_CLASSES: 2 NUM_CONVS: 4 NUM_CONVS_CLS: 4 NUM_CONVS_REG: 4 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REG_CTR_ON: True FCOS_ON: True FPN: USE_GN: False USE_RELU: False GROUP_NORM: DIM_PER_GP: -1 EPSILON: 1e-05 NUM_GROUPS: 32 KEYPOINT_ON: False MASK_ON: False META_ARCHITECTURE: GeneralizedRCNN MIDDLE_HEAD: ACT_LOSS: None ACT_LOSS_WEIGHT: 1.0 CAT_ACT_MAP: True CONDGRAPH_ON: True COND_WITH_BIAS: False CON_LOSS_WEIGHT: 1.0 CON_TG_CFG: KLdiv GCN1_OUT_CHANNEL: 256 GCN2_OUT_CHANNEL: 256 GCN_EDGE_NORM: softmax GCN_EDGE_PROJECT: 128 GCN_LOSS_WEIGHT: 1.0 GCN_LOSS_WEIGHT_TG: 1.0 GCN_OUT_ACTIVATION: relu GCN_SHORTCUT: False GLOBAL_GRAPH_ON: False GM: BG_RATIO: 2 MATCHING_CFG: none MATCHING_LOSS_CFG: MSE MATCHING_LOSS_WEIGHT: 1.0 MIN_AP50_SAVE: 40 NODE_DIS_LAMBDA: 0.02 NODE_DIS_PLACE: feat NODE_DIS_WEIGHT: 0.1 NODE_LOSS_WEIGHT: 1.0 NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 WITH_CLUSTER_UPDATE: True WITH_COMPLETE_GRAPH: True WITH_COND_CLS: False WITH_CTR: False WITH_DOMAIN_INTERACTION: True WITH_GLOBAL_GRAPH: False WITH_NODE_DIS: True WITH_QUADRATIC_MATCHING: True WITH_SCORE_WEIGHT: False WITH_SEMANTIC_COMPLETION: True GM_ON: False IN_NORM: LN NUM_CONVS_IN: 2 NUM_CONVS_OUT: 1 PROTO_CHANNEL: 256 PROTO_MOMENTUM: 0.95 PROTO_WITH_BG: True RETURN_ACT_LOGITS: False MIDDLE_HEAD_CFG: GM_HEAD RESNETS: BACKBONE_OUT_CHANNELS: 1024 NUM_GROUPS: 1 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 64 RETINANET: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (0.5, 1.0, 2.0) BBOX_REG_BETA: 0.11 BBOX_REG_WEIGHT: 4.0 BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.4 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 SCALES_PER_OCTAVE: 3 STRADDLE_THRESH: 0 USE_C5: False RETINANET_ON: False ROI_BOX_HEAD: CONV_HEAD_DIM: 256 DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 81 NUM_STACKED_CONVS: 4 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: FastRCNNPredictor USE_GN: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.5 DETECTIONS_PER_IMG: 100 FG_IOU_THRESHOLD: 0.5 NMS: 0.5 POSITIVE_FRACTION: 0.25 SCORE_THRESH: 0.05 USE_FPN: False ROI_KEYPOINT_HEAD: CONV_LAYERS: (512, 512, 512, 512, 512, 512, 512, 512) FEATURE_EXTRACTOR: KeypointRCNNFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: KeypointRCNNPredictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True USE_GN: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (16,) ASPECT_RATIOS: (0.5, 1.0, 2.0) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_TOP_N_TEST: 2000 FPN_POST_NMS_TOP_N_TRAIN: 2000 MIN_SIZE: 0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 6000 PRE_NMS_TOP_N_TRAIN: 12000 RPN_HEAD: SingleConvRPNHead STRADDLE_THRESH: 0 USE_FPN: False RPN_ONLY: True USE_SYNCBN: False WEIGHT: https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth OUTPUT_DIR: ./experiments/sigma/sim10k_to_cityscapes_vgg16/ PATHS_CATALOG: /home/sh/SIGMA-main/fcos_core/config/paths_catalog.py SOLVER: ADAPT_VAL_ON: True BACKBONE: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) SWA: False WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant CHECKPOINT_PERIOD: 100000 DIS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant FCOS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant IMS_PER_BATCH: 2 INITIAL_AP50: 35 MAX_ITER: 200000 MIDDLE_HEAD: BASE_LR: 0.005 BIAS_LR_FACTOR: 2 GAMMA: 0.1 PLABEL_TH: (0.5, 1.0) STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant MOMENTUM: 0.9 VAL_ITER: 100 VAL_TYPE: AP50 WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0 TENSORBOARD_EXPERIMENT: ./exps/demo/logs/ TEST: DETECTIONS_PER_IMG: 100 EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 4 MODE: common 2022-07-13 20:35:09,967 fcos_core.trainer INFO: node dis setting: feat 2022-07-13 20:35:10,000 fcos_core.trainer INFO: node_dis initialized 2022-07-13 20:35:10,002 fcos_core.trainer INFO: node_cls_middle initialized 2022-07-13 20:35:10,004 fcos_core.trainer INFO: head_in_ln initialized 2022-07-13 20:35:10,365 fcos_core.utils.checkpoint INFO: Loading checkpoint from https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth 2022-07-13 20:35:10,366 fcos_core.utils.checkpoint INFO: url https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth cached in /home/sh/.torch/models/vgg16_caffe-292e1171.pth 2022-07-13 20:35:10,420 fcos_core.utils.model_serialization INFO: body.features.0.bias loaded from features.0.bias of shape (64,) 2022-07-13 20:35:10,420 fcos_core.utils.model_serialization INFO: body.features.0.weight loaded from features.0.weight of shape (64, 3, 3, 3) 2022-07-13 20:35:10,420 fcos_core.utils.model_serialization INFO: body.features.10.bias loaded from features.10.bias of shape (256,) 2022-07-13 20:35:10,421 fcos_core.utils.model_serialization INFO: body.features.10.weight loaded from features.10.weight of shape (256, 128, 3, 3) 2022-07-13 20:35:10,421 fcos_core.utils.model_serialization INFO: body.features.12.bias loaded from features.12.bias of shape (256,) 2022-07-13 20:35:10,421 fcos_core.utils.model_serialization INFO: body.features.12.weight loaded from features.12.weight of shape (256, 256, 3, 3) 2022-07-13 20:35:10,421 fcos_core.utils.model_serialization INFO: body.features.14.bias loaded from features.14.bias of shape (256,) 2022-07-13 20:35:10,421 fcos_core.utils.model_serialization INFO: body.features.14.weight loaded from features.14.weight of shape (256, 256, 3, 3) 2022-07-13 20:35:10,421 fcos_core.utils.model_serialization INFO: body.features.17.bias loaded from features.17.bias of shape (512,) 2022-07-13 20:35:10,422 fcos_core.utils.model_serialization INFO: body.features.17.weight loaded from features.17.weight of shape (512, 256, 3, 3) 2022-07-13 20:35:10,422 fcos_core.utils.model_serialization INFO: body.features.19.bias loaded from features.19.bias of shape (512,) 2022-07-13 20:35:10,422 fcos_core.utils.model_serialization INFO: body.features.19.weight loaded from features.19.weight of shape (512, 512, 3, 3) 2022-07-13 20:35:10,422 fcos_core.utils.model_serialization INFO: body.features.2.bias loaded from features.2.bias of shape (64,) 2022-07-13 20:35:10,422 fcos_core.utils.model_serialization INFO: body.features.2.weight loaded from features.2.weight of shape (64, 64, 3, 3) 2022-07-13 20:35:10,422 fcos_core.utils.model_serialization INFO: body.features.21.bias loaded from features.21.bias of shape (512,) 2022-07-13 20:35:10,423 fcos_core.utils.model_serialization INFO: body.features.21.weight loaded from features.21.weight of shape (512, 512, 3, 3) 2022-07-13 20:35:10,423 fcos_core.utils.model_serialization INFO: body.features.24.bias loaded from features.24.bias of shape (512,) 2022-07-13 20:35:10,423 fcos_core.utils.model_serialization INFO: body.features.24.weight loaded from features.24.weight of shape (512, 512, 3, 3) 2022-07-13 20:35:10,423 fcos_core.utils.model_serialization INFO: body.features.26.bias loaded from features.26.bias of shape (512,) 2022-07-13 20:35:10,423 fcos_core.utils.model_serialization INFO: body.features.26.weight loaded from features.26.weight of shape (512, 512, 3, 3) 2022-07-13 20:35:10,423 fcos_core.utils.model_serialization INFO: body.features.28.bias loaded from features.28.bias of shape (512,) 2022-07-13 20:35:10,424 fcos_core.utils.model_serialization INFO: body.features.28.weight loaded from features.28.weight of shape (512, 512, 3, 3) 2022-07-13 20:35:10,424 fcos_core.utils.model_serialization INFO: body.features.5.bias loaded from features.5.bias of shape (128,) 2022-07-13 20:35:10,424 fcos_core.utils.model_serialization INFO: body.features.5.weight loaded from features.5.weight of shape (128, 64, 3, 3) 2022-07-13 20:35:10,424 fcos_core.utils.model_serialization INFO: body.features.7.bias loaded from features.7.bias of shape (128,) 2022-07-13 20:35:10,424 fcos_core.utils.model_serialization INFO: body.features.7.weight loaded from features.7.weight of shape (128, 128, 3, 3) 2022-07-13 20:35:11,656 fcos_core.data.build WARNING: When using more than one image per GPU you may encounter an out-of-memory (OOM) error if your GPU does not have sufficient memory. If this happens, you can reduce SOLVER.IMS_PER_BATCH (for training) or TEST.IMS_PER_BATCH (for inference). For training, you must also adjust the learning rate and schedule length according to the linear scaling rule. See for example: https://github.com/facebookresearch/Detectron/blob/master/configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml#L14 2022-07-13 20:35:16,037 fcos_core.trainer INFO: Start training 2022-07-13 21:04:02,040 fcos_core INFO: Using 1 GPUs 2022-07-13 21:04:02,040 fcos_core INFO: Namespace(config_file='configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml', distributed=False, local_rank=0, opts=[], skip_test=False, test_only=False, use_tensorboard=True) 2022-07-13 21:04:02,040 fcos_core INFO: Collecting env info (might take some time) 2022-07-13 21:04:06,886 fcos_core INFO: PyTorch version: 1.4.0 Is debug build: No CUDA used to build PyTorch: 10.1 OS: Ubuntu 18.04.6 LTS GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 CMake version: Could not collect Python version: 3.7 Is CUDA available: Yes CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA RTX A4000 GPU 1: NVIDIA RTX A4000 GPU 2: NVIDIA RTX A4000 GPU 3: NVIDIA RTX A4000 GPU 4: NVIDIA RTX A4000 GPU 5: NVIDIA RTX A4000 GPU 6: NVIDIA RTX A4000 GPU 7: NVIDIA RTX A4000 Nvidia driver version: 470.86 cuDNN version: Could not collect Versions of relevant libraries: [pip3] numpy==1.21.6 [pip3] torch==1.4.0 [pip3] torchvision==0.2.1 [conda] torch 1.4.0 pypi_0 pypi [conda] torchvision 0.2.1 pypi_0 pypi Pillow (9.2.0) 2022-07-13 21:04:06,887 fcos_core INFO: Loaded configuration file configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml 2022-07-13 21:04:06,887 fcos_core INFO: OUTPUT_DIR: './experiments/sigma/sim10k_to_cityscapes_vgg16/' MODEL: META_ARCHITECTURE: "GeneralizedRCNN" WEIGHT: 'https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth' # Initialed by imagenet # NOTE: In our cvpr version, we mistakely set FCOS.NMS_TH as 0.3, giving a 53.7 result. After setting it correctly, sigma gives 57.1 mAP.... # WEIGHT: './well_trained_models/sim10k_to_city_vgg16_53.73_mAP.pth' # Initialed by pretrained weight RPN_ONLY: True FCOS_ON: True DA_ON: True ATSS_ON: False MIDDLE_HEAD_CFG: 'GM_HEAD' MIDDLE_HEAD: CONDGRAPH_ON: True IN_NORM: 'LN' NUM_CONVS_IN: 2 GM: # matching cfg MATCHING_LOSS_CFG: 'MSE' MATCHING_CFG: 'none' WITH_SCORE_WEIGHT: False WITH_NODE_DIS: True # node sampling NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 BG_RATIO: 2 # loss weight MATCHING_LOSS_WEIGHT: 1.0 NODE_LOSS_WEIGHT: 1.0 NODE_DIS_WEIGHT: 0.1 NODE_DIS_LAMBDA: 0.02 WITH_SEMANTIC_COMPLETION: True WITH_QUADRATIC_MATCHING: True WITH_CLUSTER_UPDATE: True WITH_CTR: False WITH_COMPLETE_GRAPH: True WITH_DOMAIN_INTERACTION: True BACKBONE: CONV_BODY: "VGG-16-FPN-RETINANET" RETINANET: USE_C5: False # FCOS uses P5 instead of C5 FCOS: NUM_CONVS_REG: 4 NUM_CONVS_CLS: 4 NUM_CLASSES: 2 INFERENCE_TH: 0.05 # pre_nms_thresh (default=0.05) PRE_NMS_TOP_N: 1000 # pre_nms_top_n (default=1000) NMS_TH: 0.6 # nms_thresh (default=0.6) REG_CTR_ON: True ADV: GA_DIS_LAMBDA: 0.1 # for dis loss CON_NUM_SHARED_CONV_P7: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P3: 4 # USE_DIS_GLOBAL: True USE_DIS_P7: True USE_DIS_P6: True USE_DIS_P5: True USE_DIS_P4: True USE_DIS_P3: True GRL_WEIGHT_P7: 0.02 # for gradient GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P3: 0.02 TEST: DETECTIONS_PER_IMG: 100 # fpn_post_nms_top_n (default=100) MODE: 'common' DATASETS: TRAIN_SOURCE: ("sim10k_trainval_caronly", ) TRAIN_TARGET: ("cityscapes_train_caronly_cocostyle", ) TEST: ("cityscapes_val_caronly_cocostyle", ) INPUT: MIN_SIZE_RANGE_TRAIN: (640, 800) MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MAX_SIZE_TEST: 1333 DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: VAL_ITER: 100 ADAPT_VAL_ON: True INITIAL_AP50: 35 WEIGHT_DECAY: 0.0001 MAX_ITER: 200000 # 4 for source and 4 for target IMS_PER_BATCH: 2 CHECKPOINT_PERIOD: 100000 # BACKBONE: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" MIDDLE_HEAD: BASE_LR: 0.005 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" PLABEL_TH: (0.5, 1.0) FCOS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" # DIS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" 2022-07-13 21:04:06,890 fcos_core INFO: Running with config: CLS_MAP_PRE: softmax DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: TEST: ('cityscapes_val_caronly_cocostyle',) TRAIN_SOURCE: ('sim10k_trainval_caronly',) TRAIN_TARGET: ('cityscapes_train_caronly_cocostyle',) INPUT: MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_RANGE_TRAIN: (640, 800) MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN: (800,) PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] TO_BGR255: True MODEL: ADV: BASE_DIS_TOWER: False CA_DIS_LAMBDA: 0.1 CA_DIS_P3_NUM_CONVS: 4 CA_DIS_P4_NUM_CONVS: 4 CA_DIS_P5_NUM_CONVS: 4 CA_DIS_P6_NUM_CONVS: 4 CA_DIS_P7_NUM_CONVS: 4 CA_GRL_WEIGHT_P3: 0.1 CA_GRL_WEIGHT_P4: 0.1 CA_GRL_WEIGHT_P5: 0.1 CA_GRL_WEIGHT_P6: 0.1 CA_GRL_WEIGHT_P7: 0.1 CENTER_AWARE_TYPE: ca_feature CENTER_AWARE_WEIGHT: 20 CON_DIS_LAMBDA: 0.1 CON_FUSUIN_CFG: concat CON_NUM_SHARED_CONV_P3: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P7: 4 CON_WITH_GA: False DIS_P3_NUM_CONVS: 4 DIS_P4_NUM_CONVS: 4 DIS_P5_NUM_CONVS: 4 DIS_P6_NUM_CONVS: 4 DIS_P7_NUM_CONVS: 4 GA_DIS_LAMBDA: 0.1 GRL_APPLIED_DOMAIN: both GRL_WEIGHT_P3: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P7: 0.02 OUTMAP_OP: sigmoid OUTPUT_CENTERNESS_DA: True OUTPUT_CLS_DA: True OUTPUT_REG_DA: True OUT_DIS_LAMBDA: 0.1 OUT_LOSS: ce OUT_WEIGHT: 0.5 PATCH_STRIDE: None USE_DIS_CENTER_AWARE: False USE_DIS_CON: False USE_DIS_GLOBAL: True USE_DIS_OUT: False USE_DIS_P3: True USE_DIS_P3_CON: False USE_DIS_P4: True USE_DIS_P4_CON: False USE_DIS_P5: True USE_DIS_P5_CON: False USE_DIS_P6: True USE_DIS_P6_CON: False USE_DIS_P7: True USE_DIS_P7_CON: False ATSS: ANCHOR_SIZES: (64, 128, 256, 512, 1024) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (1.0,) BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 5.0 NMS_TH: 0.6 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 POSITIVE_TYPE: ATSS PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REGRESSION_TYPE: BOX REG_LOSS_WEIGHT: 2.0 SCALES_PER_OCTAVE: 1 STRADDLE_THRESH: 0 TOPK: 9 USE_DCN_IN_TOWER: False ATSS_ON: False BACKBONE: CONV_BODY: VGG-16-FPN-RETINANET FREEZE_CONV_BODY_AT: 2 USE_GN: False VGG_W_BN: False CLS_AGNOSTIC_BBOX_REG: False DA_ON: True DEBUG_CFG: None DEVICE: cuda FBNET: ARCH: default ARCH_DEF: BN_TYPE: bn DET_HEAD_BLOCKS: [] DET_HEAD_LAST_SCALE: 1.0 DET_HEAD_STRIDE: 0 DW_CONV_SKIP_BN: True DW_CONV_SKIP_RELU: True KPTS_HEAD_BLOCKS: [] KPTS_HEAD_LAST_SCALE: 0.0 KPTS_HEAD_STRIDE: 0 MASK_HEAD_BLOCKS: [] MASK_HEAD_LAST_SCALE: 0.0 MASK_HEAD_STRIDE: 0 RPN_BN_TYPE: RPN_HEAD_BLOCKS: 0 SCALE_FACTOR: 1.0 WIDTH_DIVISOR: 1 FCOS: FPN_STRIDES: [8, 16, 32, 64, 128] INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.6 NUM_CLASSES: 2 NUM_CONVS: 4 NUM_CONVS_CLS: 4 NUM_CONVS_REG: 4 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REG_CTR_ON: True FCOS_ON: True FPN: USE_GN: False USE_RELU: False GROUP_NORM: DIM_PER_GP: -1 EPSILON: 1e-05 NUM_GROUPS: 32 KEYPOINT_ON: False MASK_ON: False META_ARCHITECTURE: GeneralizedRCNN MIDDLE_HEAD: ACT_LOSS: None ACT_LOSS_WEIGHT: 1.0 CAT_ACT_MAP: True CONDGRAPH_ON: True COND_WITH_BIAS: False CON_LOSS_WEIGHT: 1.0 CON_TG_CFG: KLdiv GCN1_OUT_CHANNEL: 256 GCN2_OUT_CHANNEL: 256 GCN_EDGE_NORM: softmax GCN_EDGE_PROJECT: 128 GCN_LOSS_WEIGHT: 1.0 GCN_LOSS_WEIGHT_TG: 1.0 GCN_OUT_ACTIVATION: relu GCN_SHORTCUT: False GLOBAL_GRAPH_ON: False GM: BG_RATIO: 2 MATCHING_CFG: none MATCHING_LOSS_CFG: MSE MATCHING_LOSS_WEIGHT: 1.0 MIN_AP50_SAVE: 40 NODE_DIS_LAMBDA: 0.02 NODE_DIS_PLACE: feat NODE_DIS_WEIGHT: 0.1 NODE_LOSS_WEIGHT: 1.0 NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 WITH_CLUSTER_UPDATE: True WITH_COMPLETE_GRAPH: True WITH_COND_CLS: False WITH_CTR: False WITH_DOMAIN_INTERACTION: True WITH_GLOBAL_GRAPH: False WITH_NODE_DIS: True WITH_QUADRATIC_MATCHING: True WITH_SCORE_WEIGHT: False WITH_SEMANTIC_COMPLETION: True GM_ON: False IN_NORM: LN NUM_CONVS_IN: 2 NUM_CONVS_OUT: 1 PROTO_CHANNEL: 256 PROTO_MOMENTUM: 0.95 PROTO_WITH_BG: True RETURN_ACT_LOGITS: False MIDDLE_HEAD_CFG: GM_HEAD RESNETS: BACKBONE_OUT_CHANNELS: 1024 NUM_GROUPS: 1 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 64 RETINANET: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (0.5, 1.0, 2.0) BBOX_REG_BETA: 0.11 BBOX_REG_WEIGHT: 4.0 BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.4 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 SCALES_PER_OCTAVE: 3 STRADDLE_THRESH: 0 USE_C5: False RETINANET_ON: False ROI_BOX_HEAD: CONV_HEAD_DIM: 256 DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 81 NUM_STACKED_CONVS: 4 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: FastRCNNPredictor USE_GN: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.5 DETECTIONS_PER_IMG: 100 FG_IOU_THRESHOLD: 0.5 NMS: 0.5 POSITIVE_FRACTION: 0.25 SCORE_THRESH: 0.05 USE_FPN: False ROI_KEYPOINT_HEAD: CONV_LAYERS: (512, 512, 512, 512, 512, 512, 512, 512) FEATURE_EXTRACTOR: KeypointRCNNFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: KeypointRCNNPredictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True USE_GN: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (16,) ASPECT_RATIOS: (0.5, 1.0, 2.0) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_TOP_N_TEST: 2000 FPN_POST_NMS_TOP_N_TRAIN: 2000 MIN_SIZE: 0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 6000 PRE_NMS_TOP_N_TRAIN: 12000 RPN_HEAD: SingleConvRPNHead STRADDLE_THRESH: 0 USE_FPN: False RPN_ONLY: True USE_SYNCBN: False WEIGHT: https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth OUTPUT_DIR: ./experiments/sigma/sim10k_to_cityscapes_vgg16/ PATHS_CATALOG: /home/sh/SIGMA-main/fcos_core/config/paths_catalog.py SOLVER: ADAPT_VAL_ON: True BACKBONE: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) SWA: False WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant CHECKPOINT_PERIOD: 100000 DIS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant FCOS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant IMS_PER_BATCH: 2 INITIAL_AP50: 35 MAX_ITER: 200000 MIDDLE_HEAD: BASE_LR: 0.005 BIAS_LR_FACTOR: 2 GAMMA: 0.1 PLABEL_TH: (0.5, 1.0) STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant MOMENTUM: 0.9 VAL_ITER: 100 VAL_TYPE: AP50 WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0 TENSORBOARD_EXPERIMENT: ./exps/demo/logs/ TEST: DETECTIONS_PER_IMG: 100 EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 4 MODE: common 2022-07-13 21:23:00,588 fcos_core.trainer INFO: node dis setting: feat 2022-07-13 21:23:00,618 fcos_core.trainer INFO: node_dis initialized 2022-07-13 21:23:00,620 fcos_core.trainer INFO: node_cls_middle initialized 2022-07-13 21:23:00,622 fcos_core.trainer INFO: head_in_ln initialized 2022-07-13 21:23:01,023 fcos_core.utils.checkpoint INFO: Loading checkpoint from https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth 2022-07-13 21:23:01,024 fcos_core.utils.checkpoint INFO: url https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth cached in /home/sh/.torch/models/vgg16_caffe-292e1171.pth 2022-07-13 21:23:01,073 fcos_core.utils.model_serialization INFO: body.features.0.bias loaded from features.0.bias of shape (64,) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.0.weight loaded from features.0.weight of shape (64, 3, 3, 3) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.10.bias loaded from features.10.bias of shape (256,) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.10.weight loaded from features.10.weight of shape (256, 128, 3, 3) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.12.bias loaded from features.12.bias of shape (256,) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.12.weight loaded from features.12.weight of shape (256, 256, 3, 3) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.14.bias loaded from features.14.bias of shape (256,) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.14.weight loaded from features.14.weight of shape (256, 256, 3, 3) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.17.bias loaded from features.17.bias of shape (512,) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.17.weight loaded from features.17.weight of shape (512, 256, 3, 3) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.19.bias loaded from features.19.bias of shape (512,) 2022-07-13 21:23:01,074 fcos_core.utils.model_serialization INFO: body.features.19.weight loaded from features.19.weight of shape (512, 512, 3, 3) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.2.bias loaded from features.2.bias of shape (64,) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.2.weight loaded from features.2.weight of shape (64, 64, 3, 3) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.21.bias loaded from features.21.bias of shape (512,) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.21.weight loaded from features.21.weight of shape (512, 512, 3, 3) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.24.bias loaded from features.24.bias of shape (512,) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.24.weight loaded from features.24.weight of shape (512, 512, 3, 3) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.26.bias loaded from features.26.bias of shape (512,) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.26.weight loaded from features.26.weight of shape (512, 512, 3, 3) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.28.bias loaded from features.28.bias of shape (512,) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.28.weight loaded from features.28.weight of shape (512, 512, 3, 3) 2022-07-13 21:23:01,075 fcos_core.utils.model_serialization INFO: body.features.5.bias loaded from features.5.bias of shape (128,) 2022-07-13 21:23:01,076 fcos_core.utils.model_serialization INFO: body.features.5.weight loaded from features.5.weight of shape (128, 64, 3, 3) 2022-07-13 21:23:01,076 fcos_core.utils.model_serialization INFO: body.features.7.bias loaded from features.7.bias of shape (128,) 2022-07-13 21:23:01,076 fcos_core.utils.model_serialization INFO: body.features.7.weight loaded from features.7.weight of shape (128, 128, 3, 3) 2022-07-13 21:23:02,394 fcos_core.data.build WARNING: When using more than one image per GPU you may encounter an out-of-memory (OOM) error if your GPU does not have sufficient memory. If this happens, you can reduce SOLVER.IMS_PER_BATCH (for training) or TEST.IMS_PER_BATCH (for inference). For training, you must also adjust the learning rate and schedule length according to the linear scaling rule. See for example: https://github.com/facebookresearch/Detectron/blob/master/configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml#L14 2022-07-13 21:23:06,501 fcos_core.trainer INFO: Start training 2022-07-13 21:26:34,789 fcos_core INFO: Using 1 GPUs 2022-07-13 21:26:34,789 fcos_core INFO: Namespace(config_file='configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml', distributed=False, local_rank=0, opts=[], skip_test=False, test_only=False, use_tensorboard=True) 2022-07-13 21:26:34,789 fcos_core INFO: Collecting env info (might take some time) 2022-07-13 21:26:39,754 fcos_core INFO: PyTorch version: 1.4.0 Is debug build: No CUDA used to build PyTorch: 10.1 OS: Ubuntu 18.04.6 LTS GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 CMake version: Could not collect Python version: 3.7 Is CUDA available: Yes CUDA runtime version: Could not collect GPU models and configuration: GPU 0: NVIDIA RTX A4000 GPU 1: NVIDIA RTX A4000 GPU 2: NVIDIA RTX A4000 GPU 3: NVIDIA RTX A4000 GPU 4: NVIDIA RTX A4000 GPU 5: NVIDIA RTX A4000 GPU 6: NVIDIA RTX A4000 GPU 7: NVIDIA RTX A4000 Nvidia driver version: 470.86 cuDNN version: Could not collect Versions of relevant libraries: [pip3] numpy==1.21.6 [pip3] torch==1.4.0 [pip3] torchvision==0.2.1 [conda] torch 1.4.0 pypi_0 pypi [conda] torchvision 0.2.1 pypi_0 pypi Pillow (9.2.0) 2022-07-13 21:26:39,755 fcos_core INFO: Loaded configuration file configs/SIGMA/sigma_vgg16_sim10k_to_cityscapes.yaml 2022-07-13 21:26:39,755 fcos_core INFO: OUTPUT_DIR: './experiments/sigma/sim10k_to_cityscapes_vgg16/' MODEL: META_ARCHITECTURE: "GeneralizedRCNN" WEIGHT: 'https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth' # Initialed by imagenet # NOTE: In our cvpr version, we mistakely set FCOS.NMS_TH as 0.3, giving a 53.7 result. After setting it correctly, sigma gives 57.1 mAP.... # WEIGHT: './well_trained_models/sim10k_to_city_vgg16_53.73_mAP.pth' # Initialed by pretrained weight RPN_ONLY: True FCOS_ON: True DA_ON: True ATSS_ON: False MIDDLE_HEAD_CFG: 'GM_HEAD' MIDDLE_HEAD: CONDGRAPH_ON: True IN_NORM: 'LN' NUM_CONVS_IN: 2 GM: # matching cfg MATCHING_LOSS_CFG: 'MSE' MATCHING_CFG: 'none' WITH_SCORE_WEIGHT: False WITH_NODE_DIS: True # node sampling NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 BG_RATIO: 2 # loss weight MATCHING_LOSS_WEIGHT: 1.0 NODE_LOSS_WEIGHT: 1.0 NODE_DIS_WEIGHT: 0.1 NODE_DIS_LAMBDA: 0.02 WITH_SEMANTIC_COMPLETION: True WITH_QUADRATIC_MATCHING: True WITH_CLUSTER_UPDATE: True WITH_CTR: False WITH_COMPLETE_GRAPH: True WITH_DOMAIN_INTERACTION: True BACKBONE: CONV_BODY: "VGG-16-FPN-RETINANET" RETINANET: USE_C5: False # FCOS uses P5 instead of C5 FCOS: NUM_CONVS_REG: 4 NUM_CONVS_CLS: 4 NUM_CLASSES: 2 INFERENCE_TH: 0.05 # pre_nms_thresh (default=0.05) PRE_NMS_TOP_N: 1000 # pre_nms_top_n (default=1000) NMS_TH: 0.6 # nms_thresh (default=0.6) REG_CTR_ON: True ADV: GA_DIS_LAMBDA: 0.1 # for dis loss CON_NUM_SHARED_CONV_P7: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P3: 4 # USE_DIS_GLOBAL: True USE_DIS_P7: True USE_DIS_P6: True USE_DIS_P5: True USE_DIS_P4: True USE_DIS_P3: True GRL_WEIGHT_P7: 0.02 # for gradient GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P3: 0.02 TEST: DETECTIONS_PER_IMG: 100 # fpn_post_nms_top_n (default=100) MODE: 'common' DATASETS: TRAIN_SOURCE: ("sim10k_trainval_caronly", ) TRAIN_TARGET: ("cityscapes_train_caronly_cocostyle", ) TEST: ("cityscapes_val_caronly_cocostyle", ) INPUT: MIN_SIZE_RANGE_TRAIN: (640, 800) MAX_SIZE_TRAIN: 1333 MIN_SIZE_TEST: 800 MAX_SIZE_TEST: 1333 DATALOADER: SIZE_DIVISIBILITY: 32 SOLVER: VAL_ITER: 100 ADAPT_VAL_ON: True INITIAL_AP50: 35 WEIGHT_DECAY: 0.0001 MAX_ITER: 200000 # 4 for source and 4 for target IMS_PER_BATCH: 2 CHECKPOINT_PERIOD: 100000 # BACKBONE: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" MIDDLE_HEAD: BASE_LR: 0.005 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" PLABEL_TH: (0.5, 1.0) FCOS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" # DIS: BASE_LR: 0.0025 STEPS: (90000, ) WARMUP_ITERS: 1000 WARMUP_METHOD: "constant" 2022-07-13 21:26:39,758 fcos_core INFO: Running with config: CLS_MAP_PRE: softmax DATALOADER: ASPECT_RATIO_GROUPING: True NUM_WORKERS: 4 SIZE_DIVISIBILITY: 32 DATASETS: TEST: ('cityscapes_val_caronly_cocostyle',) TRAIN_SOURCE: ('sim10k_trainval_caronly',) TRAIN_TARGET: ('cityscapes_train_caronly_cocostyle',) INPUT: MAX_SIZE_TEST: 1333 MAX_SIZE_TRAIN: 1333 MIN_SIZE_RANGE_TRAIN: (640, 800) MIN_SIZE_TEST: 800 MIN_SIZE_TRAIN: (800,) PIXEL_MEAN: [102.9801, 115.9465, 122.7717] PIXEL_STD: [1.0, 1.0, 1.0] TO_BGR255: True MODEL: ADV: BASE_DIS_TOWER: False CA_DIS_LAMBDA: 0.1 CA_DIS_P3_NUM_CONVS: 4 CA_DIS_P4_NUM_CONVS: 4 CA_DIS_P5_NUM_CONVS: 4 CA_DIS_P6_NUM_CONVS: 4 CA_DIS_P7_NUM_CONVS: 4 CA_GRL_WEIGHT_P3: 0.1 CA_GRL_WEIGHT_P4: 0.1 CA_GRL_WEIGHT_P5: 0.1 CA_GRL_WEIGHT_P6: 0.1 CA_GRL_WEIGHT_P7: 0.1 CENTER_AWARE_TYPE: ca_feature CENTER_AWARE_WEIGHT: 20 CON_DIS_LAMBDA: 0.1 CON_FUSUIN_CFG: concat CON_NUM_SHARED_CONV_P3: 4 CON_NUM_SHARED_CONV_P4: 4 CON_NUM_SHARED_CONV_P5: 4 CON_NUM_SHARED_CONV_P6: 4 CON_NUM_SHARED_CONV_P7: 4 CON_WITH_GA: False DIS_P3_NUM_CONVS: 4 DIS_P4_NUM_CONVS: 4 DIS_P5_NUM_CONVS: 4 DIS_P6_NUM_CONVS: 4 DIS_P7_NUM_CONVS: 4 GA_DIS_LAMBDA: 0.1 GRL_APPLIED_DOMAIN: both GRL_WEIGHT_P3: 0.02 GRL_WEIGHT_P4: 0.02 GRL_WEIGHT_P5: 0.02 GRL_WEIGHT_P6: 0.02 GRL_WEIGHT_P7: 0.02 OUTMAP_OP: sigmoid OUTPUT_CENTERNESS_DA: True OUTPUT_CLS_DA: True OUTPUT_REG_DA: True OUT_DIS_LAMBDA: 0.1 OUT_LOSS: ce OUT_WEIGHT: 0.5 PATCH_STRIDE: None USE_DIS_CENTER_AWARE: False USE_DIS_CON: False USE_DIS_GLOBAL: True USE_DIS_OUT: False USE_DIS_P3: True USE_DIS_P3_CON: False USE_DIS_P4: True USE_DIS_P4_CON: False USE_DIS_P5: True USE_DIS_P5_CON: False USE_DIS_P6: True USE_DIS_P6_CON: False USE_DIS_P7: True USE_DIS_P7_CON: False ATSS: ANCHOR_SIZES: (64, 128, 256, 512, 1024) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (1.0,) BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 5.0 NMS_TH: 0.6 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 POSITIVE_TYPE: ATSS PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REGRESSION_TYPE: BOX REG_LOSS_WEIGHT: 2.0 SCALES_PER_OCTAVE: 1 STRADDLE_THRESH: 0 TOPK: 9 USE_DCN_IN_TOWER: False ATSS_ON: False BACKBONE: CONV_BODY: VGG-16-FPN-RETINANET FREEZE_CONV_BODY_AT: 2 USE_GN: False VGG_W_BN: False CLS_AGNOSTIC_BBOX_REG: False DA_ON: True DEBUG_CFG: None DEVICE: cuda FBNET: ARCH: default ARCH_DEF: BN_TYPE: bn DET_HEAD_BLOCKS: [] DET_HEAD_LAST_SCALE: 1.0 DET_HEAD_STRIDE: 0 DW_CONV_SKIP_BN: True DW_CONV_SKIP_RELU: True KPTS_HEAD_BLOCKS: [] KPTS_HEAD_LAST_SCALE: 0.0 KPTS_HEAD_STRIDE: 0 MASK_HEAD_BLOCKS: [] MASK_HEAD_LAST_SCALE: 0.0 MASK_HEAD_STRIDE: 0 RPN_BN_TYPE: RPN_HEAD_BLOCKS: 0 SCALE_FACTOR: 1.0 WIDTH_DIVISOR: 1 FCOS: FPN_STRIDES: [8, 16, 32, 64, 128] INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.6 NUM_CLASSES: 2 NUM_CONVS: 4 NUM_CONVS_CLS: 4 NUM_CONVS_REG: 4 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 REG_CTR_ON: True FCOS_ON: True FPN: USE_GN: False USE_RELU: False GROUP_NORM: DIM_PER_GP: -1 EPSILON: 1e-05 NUM_GROUPS: 32 KEYPOINT_ON: False MASK_ON: False META_ARCHITECTURE: GeneralizedRCNN MIDDLE_HEAD: ACT_LOSS: None ACT_LOSS_WEIGHT: 1.0 CAT_ACT_MAP: True CONDGRAPH_ON: True COND_WITH_BIAS: False CON_LOSS_WEIGHT: 1.0 CON_TG_CFG: KLdiv GCN1_OUT_CHANNEL: 256 GCN2_OUT_CHANNEL: 256 GCN_EDGE_NORM: softmax GCN_EDGE_PROJECT: 128 GCN_LOSS_WEIGHT: 1.0 GCN_LOSS_WEIGHT_TG: 1.0 GCN_OUT_ACTIVATION: relu GCN_SHORTCUT: False GLOBAL_GRAPH_ON: False GM: BG_RATIO: 2 MATCHING_CFG: none MATCHING_LOSS_CFG: MSE MATCHING_LOSS_WEIGHT: 1.0 MIN_AP50_SAVE: 40 NODE_DIS_LAMBDA: 0.02 NODE_DIS_PLACE: feat NODE_DIS_WEIGHT: 0.1 NODE_LOSS_WEIGHT: 1.0 NUM_NODES_PER_LVL_SR: 50 NUM_NODES_PER_LVL_TG: 50 WITH_CLUSTER_UPDATE: True WITH_COMPLETE_GRAPH: True WITH_COND_CLS: False WITH_CTR: False WITH_DOMAIN_INTERACTION: True WITH_GLOBAL_GRAPH: False WITH_NODE_DIS: True WITH_QUADRATIC_MATCHING: True WITH_SCORE_WEIGHT: False WITH_SEMANTIC_COMPLETION: True GM_ON: False IN_NORM: LN NUM_CONVS_IN: 2 NUM_CONVS_OUT: 1 PROTO_CHANNEL: 256 PROTO_MOMENTUM: 0.95 PROTO_WITH_BG: True RETURN_ACT_LOGITS: False MIDDLE_HEAD_CFG: GM_HEAD RESNETS: BACKBONE_OUT_CHANNELS: 1024 NUM_GROUPS: 1 RES2_OUT_CHANNELS: 256 RES5_DILATION: 1 STEM_FUNC: StemWithFixedBatchNorm STEM_OUT_CHANNELS: 64 STRIDE_IN_1X1: True TRANS_FUNC: BottleneckWithFixedBatchNorm WIDTH_PER_GROUP: 64 RETINANET: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDES: (8, 16, 32, 64, 128) ASPECT_RATIOS: (0.5, 1.0, 2.0) BBOX_REG_BETA: 0.11 BBOX_REG_WEIGHT: 4.0 BG_IOU_THRESHOLD: 0.4 FG_IOU_THRESHOLD: 0.5 INFERENCE_TH: 0.05 LOSS_ALPHA: 0.25 LOSS_GAMMA: 2.0 NMS_TH: 0.4 NUM_CLASSES: 81 NUM_CONVS: 4 OCTAVE: 2.0 PRE_NMS_TOP_N: 1000 PRIOR_PROB: 0.01 SCALES_PER_OCTAVE: 3 STRADDLE_THRESH: 0 USE_C5: False RETINANET_ON: False ROI_BOX_HEAD: CONV_HEAD_DIM: 256 DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 81 NUM_STACKED_CONVS: 4 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: FastRCNNPredictor USE_GN: False ROI_HEADS: BATCH_SIZE_PER_IMAGE: 512 BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0) BG_IOU_THRESHOLD: 0.5 DETECTIONS_PER_IMG: 100 FG_IOU_THRESHOLD: 0.5 NMS: 0.5 POSITIVE_FRACTION: 0.25 SCORE_THRESH: 0.05 USE_FPN: False ROI_KEYPOINT_HEAD: CONV_LAYERS: (512, 512, 512, 512, 512, 512, 512, 512) FEATURE_EXTRACTOR: KeypointRCNNFeatureExtractor MLP_HEAD_DIM: 1024 NUM_CLASSES: 17 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) PREDICTOR: KeypointRCNNPredictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True ROI_MASK_HEAD: CONV_LAYERS: (256, 256, 256, 256) DILATION: 1 FEATURE_EXTRACTOR: ResNet50Conv5ROIFeatureExtractor MLP_HEAD_DIM: 1024 POOLER_RESOLUTION: 14 POOLER_SAMPLING_RATIO: 0 POOLER_SCALES: (0.0625,) POSTPROCESS_MASKS: False POSTPROCESS_MASKS_THRESHOLD: 0.5 PREDICTOR: MaskRCNNC4Predictor RESOLUTION: 14 SHARE_BOX_FEATURE_EXTRACTOR: True USE_GN: False RPN: ANCHOR_SIZES: (32, 64, 128, 256, 512) ANCHOR_STRIDE: (16,) ASPECT_RATIOS: (0.5, 1.0, 2.0) BATCH_SIZE_PER_IMAGE: 256 BG_IOU_THRESHOLD: 0.3 FG_IOU_THRESHOLD: 0.7 FPN_POST_NMS_TOP_N_TEST: 2000 FPN_POST_NMS_TOP_N_TRAIN: 2000 MIN_SIZE: 0 NMS_THRESH: 0.7 POSITIVE_FRACTION: 0.5 POST_NMS_TOP_N_TEST: 1000 POST_NMS_TOP_N_TRAIN: 2000 PRE_NMS_TOP_N_TEST: 6000 PRE_NMS_TOP_N_TRAIN: 12000 RPN_HEAD: SingleConvRPNHead STRADDLE_THRESH: 0 USE_FPN: False RPN_ONLY: True USE_SYNCBN: False WEIGHT: https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth OUTPUT_DIR: ./experiments/sigma/sim10k_to_cityscapes_vgg16/ PATHS_CATALOG: /home/sh/SIGMA-main/fcos_core/config/paths_catalog.py SOLVER: ADAPT_VAL_ON: True BACKBONE: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) SWA: False WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant CHECKPOINT_PERIOD: 100000 DIS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant FCOS: BASE_LR: 0.0025 BIAS_LR_FACTOR: 2 GAMMA: 0.1 STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant IMS_PER_BATCH: 2 INITIAL_AP50: 35 MAX_ITER: 200000 MIDDLE_HEAD: BASE_LR: 0.005 BIAS_LR_FACTOR: 2 GAMMA: 0.1 PLABEL_TH: (0.5, 1.0) STEPS: (90000,) WARMUP_FACTOR: 0.3333333333333333 WARMUP_ITERS: 1000 WARMUP_METHOD: constant MOMENTUM: 0.9 VAL_ITER: 100 VAL_TYPE: AP50 WEIGHT_DECAY: 0.0001 WEIGHT_DECAY_BIAS: 0 TENSORBOARD_EXPERIMENT: ./exps/demo/logs/ TEST: DETECTIONS_PER_IMG: 100 EXPECTED_RESULTS: [] EXPECTED_RESULTS_SIGMA_TOL: 4 IMS_PER_BATCH: 4 MODE: common 2022-07-13 21:47:18,831 fcos_core.trainer INFO: node dis setting: feat 2022-07-13 21:47:18,854 fcos_core.trainer INFO: node_dis initialized 2022-07-13 21:47:18,855 fcos_core.trainer INFO: node_cls_middle initialized 2022-07-13 21:47:18,857 fcos_core.trainer INFO: head_in_ln initialized 2022-07-13 21:47:19,170 fcos_core.utils.checkpoint INFO: Loading checkpoint from https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth 2022-07-13 21:47:19,171 fcos_core.utils.checkpoint INFO: url https://s3.ap-northeast-2.amazonaws.com/open-mmlab/pretrain/third_party/vgg16_caffe-292e1171.pth cached in /home/sh/.torch/models/vgg16_caffe-292e1171.pth 2022-07-13 21:47:19,229 fcos_core.utils.model_serialization INFO: body.features.0.bias loaded from features.0.bias of shape (64,) 2022-07-13 21:47:19,229 fcos_core.utils.model_serialization INFO: body.features.0.weight loaded from features.0.weight of shape (64, 3, 3, 3) 2022-07-13 21:47:19,229 fcos_core.utils.model_serialization INFO: body.features.10.bias loaded from features.10.bias of shape (256,) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.10.weight loaded from features.10.weight of shape (256, 128, 3, 3) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.12.bias loaded from features.12.bias of shape (256,) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.12.weight loaded from features.12.weight of shape (256, 256, 3, 3) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.14.bias loaded from features.14.bias of shape (256,) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.14.weight loaded from features.14.weight of shape (256, 256, 3, 3) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.17.bias loaded from features.17.bias of shape (512,) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.17.weight loaded from features.17.weight of shape (512, 256, 3, 3) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.19.bias loaded from features.19.bias of shape (512,) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.19.weight loaded from features.19.weight of shape (512, 512, 3, 3) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.2.bias loaded from features.2.bias of shape (64,) 2022-07-13 21:47:19,230 fcos_core.utils.model_serialization INFO: body.features.2.weight loaded from features.2.weight of shape (64, 64, 3, 3) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.21.bias loaded from features.21.bias of shape (512,) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.21.weight loaded from features.21.weight of shape (512, 512, 3, 3) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.24.bias loaded from features.24.bias of shape (512,) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.24.weight loaded from features.24.weight of shape (512, 512, 3, 3) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.26.bias loaded from features.26.bias of shape (512,) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.26.weight loaded from features.26.weight of shape (512, 512, 3, 3) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.28.bias loaded from features.28.bias of shape (512,) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.28.weight loaded from features.28.weight of shape (512, 512, 3, 3) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.5.bias loaded from features.5.bias of shape (128,) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.5.weight loaded from features.5.weight of shape (128, 64, 3, 3) 2022-07-13 21:47:19,231 fcos_core.utils.model_serialization INFO: body.features.7.bias loaded from features.7.bias of shape (128,) 2022-07-13 21:47:19,232 fcos_core.utils.model_serialization INFO: body.features.7.weight loaded from features.7.weight of shape (128, 128, 3, 3) 2022-07-13 21:47:20,527 fcos_core.data.build WARNING: When using more than one image per GPU you may encounter an out-of-memory (OOM) error if your GPU does not have sufficient memory. If this happens, you can reduce SOLVER.IMS_PER_BATCH (for training) or TEST.IMS_PER_BATCH (for inference). For training, you must also adjust the learning rate and schedule length according to the linear scaling rule. See for example: https://github.com/facebookresearch/Detectron/blob/master/configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml#L14 2022-07-13 21:47:24,913 fcos_core.trainer INFO: Start training