Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

训练出错,使用configs/dcn/cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms配置文件训练跑不起来 #74

Closed
dragon515 opened this issue Dec 6, 2019 · 7 comments

Comments

@dragon515
Copy link

具体的复现信息如下:
/home/user/anaconda3/envs/mmlab/bin/python /home/user/PaddleDetection-release-0.1/tools/train.py -c ../configs/dcn/cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms.yml --use_tb=True --tb_log_dir=../tb_1206_1452_cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms/scalar --eval
CascadeBBoxAssigner:
[32mclass_aware[0m: true
batch_size_per_im: 512
bbox_reg_weights:

  • 10
  • 20
  • 30
    bg_thresh_hi:
  • 0.5
  • 0.6
  • 0.7
    bg_thresh_lo:
  • 0.0
  • 0.0
  • 0.0
    fg_fraction: 0.25
    fg_thresh:
  • 0.5
  • 0.6
  • 0.7
    num_classes: 81
    shuffle_before_sample: true
    CascadeBBoxHead:
    [32mhead[0m: CascadeTwoFCHead
    [32mnms[0m: MultiClassSoftNMS
    num_classes: 81
    CascadeRCNNClsAware:
    [32mbackbone[0m: ResNet
    [32mrpn_head[0m: FPNRPNHead
    bbox_assigner: CascadeBBoxAssigner
    bbox_head: CascadeBBoxHead
    fpn: FPN
    roi_extractor: FPNRoIAlign
    CascadeTwoFCHead:
    [32mmlp_dim[0m: 1024
    FPN:
    freeze_norm: false
    has_extra_convs: false
    max_level: 6
    min_level: 2
    norm_type: null
    num_chan: 256
    spatial_scale:
  • 0.03125
  • 0.0625
  • 0.125
  • 0.25
    FPNRPNHead:
    [32mrpn_target_assign[0m:
    rpn_batch_size_per_im: 256
    rpn_fg_fraction: 0.5
    rpn_negative_overlap: 0.3
    rpn_positive_overlap: 0.7
    rpn_straddle_thresh: 0.0
    [32mtest_proposal[0m:
    min_size: 0.0
    nms_thresh: 0.7
    post_nms_top_n: 1000
    pre_nms_top_n: 1000
    [32mtrain_proposal[0m:
    min_size: 0.0
    nms_thresh: 0.7
    post_nms_top_n: 2000
    pre_nms_top_n: 2000
    anchor_generator:
    anchor_sizes:
    • 32
    • 64
    • 128
    • 256
    • 512
      aspect_ratios:
    • 0.5
    • 1.0
    • 2.0
      stride:
    • 16.0
    • 16.0
      variance:
    • 1.0
    • 1.0
    • 1.0
    • 1.0
      anchor_start_size: 32
      max_level: 6
      min_level: 2
      num_chan: 256
      num_classes: 1
      FPNRoIAlign:
      [32mbox_resolution[0m: 14
      [32msampling_ratio[0m: 2
      canconical_level: 4
      canonical_size: 224
      mask_resolution: 14
      max_level: 5
      min_level: 2
      FasterRCNNEvalFeed:
      [32mbatch_transforms[0m:
  • !PadBatch
    pad_to_stride: 32
    [32mdataset[0m:
    annotation: annotations/instances_val2017.json
    dataset_dir: dataset/coco
    image_dir: val2017
    [32msample_transforms[0m:
  • !DecodeImage
    to_rgb: true
    with_mixup: false
  • !NormalizeImage
    is_channel_first: false
    is_scale: true
    mean:
    • 0.485
    • 0.456
    • 0.406
      std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage
    interp: 1
    max_size: 2000
    target_size:
    • 1200
      use_cv2: true
  • !Permute
    channel_first: true
    to_bgr: false
    batch_size: 1
    drop_last: false
    enable_aug_flip: false
    enable_multiscale: false
    fields:
  • image
  • im_info
  • im_id
  • im_shape
  • gt_box
  • gt_label
  • is_difficult
    image_shape:
  • null
  • 3
  • null
  • null
    num_scale: 1
    num_workers: 2
    samples: -1
    shuffle: false
    use_padded_im_info: true
    FasterRCNNTestFeed:
    [32mbatch_transforms[0m:
  • !PadBatch
    pad_to_stride: 32
    [32mdataset[0m:
    annotation: dataset/coco/annotations/instances_val2017.json
    batch_size: 1
    drop_last: false
    fields:
  • image
  • im_info
  • im_id
  • im_shape
    image_shape:
  • null
  • 3
  • null
  • null
    num_workers: 2
    sample_transforms:
  • !DecodeImage
    to_rgb: true
    with_mixup: false
  • !NormalizeImage
    is_channel_first: false
    is_scale: true
    mean:
    • 0.485
    • 0.456
    • 0.406
      std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage
    interp: 1
    max_size: 1333
    target_size: 800
    use_cv2: true
  • !Permute
    channel_first: true
    to_bgr: false
    samples: -1
    shuffle: false
    use_padded_im_info: true
    FasterRCNNTrainFeed:
    [32mbatch_transforms[0m:
  • !PadBatch
    pad_to_stride: 32
    [32mdataset[0m:
    annotation: annotations/instances_train2017.json
    dataset_dir: dataset/coco
    image_dir: train2017
    [32msample_transforms[0m:
  • !DecodeImage
    to_rgb: true
    with_mixup: false
  • !RandomFlipImage
    is_mask_flip: false
    is_normalized: false
    prob: 0.5
  • !NormalizeImage
    is_channel_first: false
    is_scale: true
    mean:
    • 0.485
    • 0.456
    • 0.406
      std:
    • 0.229
    • 0.224
    • 0.225
  • !ResizeImage
    interp: 1
    max_size: 1800
    target_size:
    • 416
    • 448
    • 480
    • 512
    • 544
    • 576
    • 608
    • 640
    • 672
    • 704
    • 736
    • 768
    • 800
    • 832
    • 864
    • 896
    • 928
    • 960
    • 992
    • 1024
    • 1056
    • 1088
    • 1120
    • 1152
    • 1184
    • 1216
    • 1248
    • 1280
    • 1312
    • 1344
    • 1376
    • 1408
      use_cv2: true
  • !Permute
    channel_first: true
    to_bgr: false
    batch_size: 1
    bufsize: 10
    class_aware_sampling: false
    drop_last: false
    fields:
  • image
  • im_info
  • im_id
  • gt_box
  • gt_label
  • is_crowd
    image_shape:
  • null
  • 3
  • null
  • null
    memsize: null
    num_workers: 2
    samples: -1
    shuffle: true
    use_process: false
    LearningRate:
    [32mschedulers[0m:
  • !PiecewiseDecay
    gamma: 0.1
    milestones:
    • 340000
    • 440000
      values: null
  • !LinearWarmup
    start_factor: 0.1
    steps: 1000
    base_lr: 0.01
    MultiClassSoftNMS:
    background_label: 0
    keep_top_k: 300
    normalized: false
    score_threshold: 0.01
    softnms_sigma: 0.5
    OptimizerBuilder:
    optimizer:
    momentum: 0.9
    type: Momentum
    regularizer:
    factor: 0.0001
    type: L2
    ResNet:
    [32mdcn_v2_stages[0m:
  • 3
  • 4
  • 5
    [32mdepth[0m: 200
    [32mnonlocal_stages[0m:
  • 4
    [32mnorm_type[0m: bn
    [32mvariant[0m: d
    feature_maps:
  • 2
  • 3
  • 4
  • 5
    freeze_at: 2
    freeze_norm: true
    norm_decay: 0.0
    weight_prefix_name: ''
    architecture: CascadeRCNNClsAware
    eval_feed: FasterRCNNEvalFeed
    log_smooth_window: 20
    max_iters: 460000
    metric: COCO
    num_classes: 81
    pretrain_weights: https://paddle-imagenet-models-name.bj.bcebos.com/ResNet200_vd_pretrained.tar
    save_dir: output
    snapshot_iter: 10000
    test_feed: FasterRCNNTestFeed
    train_feed: FasterRCNNTrainFeed
    use_gpu: true
    weights: output/cascade_rcnn_cls_aware_r200_vd_fpn_dcnv2_nonlocal_softnms/model_final

Traceback (most recent call last):
File "/home/user/PaddleDetection-release-0.1/tools/train.py", line 341, in
main()
File "/home/user/PaddleDetection-release-0.1/tools/train.py", line 129, in main
train_fetches = model.train(feed_vars)
File "/home/user/PaddleDetection-release-0.1/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 178, in train
return self.build(feed_vars, 'train')
File "/home/user/PaddleDetection-release-0.1/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 88, in build
body_feats = self.backbone(im)
File "/home/user/PaddleDetection-release-0.1/ppdet/modeling/backbones/resnet.py", line 432, in call
res = self.layer_warp(res, i)
File "/home/user/PaddleDetection-release-0.1/ppdet/modeling/backbones/resnet.py", line 383, in layer_warp
nonlocal_name + '_{}'.format(i), int(dim_in / 2) )
File "/home/user/PaddleDetection-release-0.1/ppdet/modeling/backbones/nonlocal_helper.py", line 152, in add_space_nonlocal
conv = space_nonlocal(input, dim_in, dim_out, prefix, dim_inner)
File "/home/user/PaddleDetection-release-0.1/ppdet/modeling/backbones/nonlocal_helper.py", line 101, in space_nonlocal
t_re = fluid.layers.reshape(t, shape=list(theta_shape), actual_shape=theta_shape_op )
File "/home/user/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 8976, in reshape
attrs["shape"] = get_attr_shape(shape)
File "/home/user/anaconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layers/nn.py", line 8949, in get_attr_shape
"be -1. But received shape[%d] is also -1." % dim_idx)
AssertionError: Only one dimension value of 'shape' in reshape can be -1. But received shape[2] is also -1.

Process finished with exit code 1

@heavengate
Copy link
Collaborator

看着是reshape的时候指定多个维度是-1没法推断具体shape,是否修改了这块相关代码?

@dragon515
Copy link
Author

配置文件和训练代码都没有修改,数据集采用coco

@heavengate
Copy link
Collaborator

你使用的paddle版本是多少呢,我本地在1.6.1上跑是能跑通的

@dragon515
Copy link
Author

您好,我的用的paddlepaddle-gpu 1.6.1.post107

@heavengate
Copy link
Collaborator

@hoodpan
Copy link

hoodpan commented Dec 15, 2019

我也是提示reshape这个问题。。。用的是paddlepaddle-gpu 1.6.2.post97

@littletomatodonkey
Copy link
Collaborator

这是因为新的reader与之前reshape op的写法不兼容造成的(涉及到编译时shape),master分支已经修复了这个问题,现在使用master分支代码或者release0.1分支代码应该都是可以正常运行的~ @hoodpan @dragon515

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants