trainning AP is always 0.000 #337

chaerlo · 2021-08-03T11:17:56Z

I have pulled the latest code.

training command is: 'python3 tools/train.py -f exps/default/yolox_l.py -d 1 -b 8 --fp16 -o -c yolox/weights/yolox_l.pth.tar'

Joker316701882 · 2021-08-03T13:00:37Z

@XieLeo11 Could you show us your details about exps/default/yolox_l.py?

chaerlo · 2021-08-04T01:06:56Z

@XieLeo11 Could you show us your details about exps/default/yolox_l.py?

GOATmessi7 · 2021-08-04T06:22:36Z

Plz check your dataset and show us your training log

chaerlo · 2021-08-04T06:47:57Z

Plz check your dataset and show us your training log
`2021-08-03 18:30:35.514 | INFO | yolox.core.trainer:before_train:125 - args: Namespace(batch_size=8, ckpt='yolox/weights/yolox_l.pth.tar', devices=1, dist_backend='nccl', dist_url=None, exp_file='exps/default/yolox_l.py', experiment_name='yolox_l', fp16=True, local_rank=0, machine_rank=0, name=None, num_machines=1, occupy=True, opts=[], resume=False, start_epoch=None)
2021-08-03 18:30:35.516 | INFO | yolox.core.trainer:before_train:126 - exp value:
╒══════════════════╤════════════════════════════╕
│ keys │ values │
╞══════════════════╪════════════════════════════╡
│ seed │ None │
├──────────────────┼────────────────────────────┤
│ output_dir │ './YOLOX_outputs' │
├──────────────────┼────────────────────────────┤
│ print_interval │ 100 │
├──────────────────┼────────────────────────────┤
│ eval_interval │ 2 │
├──────────────────┼────────────────────────────┤
│ num_classes │ 4 │
├──────────────────┼────────────────────────────┤
│ depth │ 1.0 │
├──────────────────┼────────────────────────────┤
│ width │ 1.0 │
├──────────────────┼────────────────────────────┤
│ data_num_workers │ 4 │
├──────────────────┼────────────────────────────┤
│ input_size │ (640, 640) │
├──────────────────┼────────────────────────────┤
│ random_size │ (14, 26) │
├──────────────────┼────────────────────────────┤
│ data_dir │ None │
├──────────────────┼────────────────────────────┤
│ train_ann │ 'instances_train2017.json' │
├──────────────────┼────────────────────────────┤
│ val_ann │ 'instances_val2017.json' │
├──────────────────┼────────────────────────────┤
│ degrees │ 10.0 │
├──────────────────┼────────────────────────────┤
│ translate │ 0.1 │
├──────────────────┼────────────────────────────┤
│ scale │ (0.1, 2) │
├──────────────────┼────────────────────────────┤
│ mscale │ (0.8, 1.6) │
├──────────────────┼────────────────────────────┤
│ shear │ 2.0 │
├──────────────────┼────────────────────────────┤
│ perspective │ 0.0 │
├──────────────────┼────────────────────────────┤
│ enable_mixup │ True │
├──────────────────┼────────────────────────────┤
│ warmup_epochs │ 5 │
├──────────────────┼────────────────────────────┤
│ max_epoch │ 300 │
├──────────────────┼────────────────────────────┤
│ warmup_lr │ 0 │
├──────────────────┼────────────────────────────┤
│ basic_lr_per_img │ 0.00015625 │
├──────────────────┼────────────────────────────┤
│ scheduler │ 'yoloxwarmcos' │
├──────────────────┼────────────────────────────┤
│ no_aug_epochs │ 15 │
├──────────────────┼────────────────────────────┤
│ min_lr_ratio │ 0.05 │
├──────────────────┼────────────────────────────┤
│ ema │ True │
├──────────────────┼────────────────────────────┤
│ weight_decay │ 0.0005 │
├──────────────────┼────────────────────────────┤
│ momentum │ 0.9 │
├──────────────────┼────────────────────────────┤
│ exp_name │ 'yolox_l' │
├──────────────────┼────────────────────────────┤
│ test_size │ (640, 640) │
├──────────────────┼────────────────────────────┤
│ test_conf │ 0.01 │
├──────────────────┼────────────────────────────┤
│ nmsthre │ 0.65 │
╘══════════════════╧════════════════════════════╛
2021-08-03 18:30:36.115 | INFO | yolox.core.trainer:before_train:132 - Model Summary: Params: 54.15M, Gflops: 155.32
2021-08-03 18:30:38.821 | INFO | apex.amp.frontend:initialize:328 - Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.
2021-08-03 18:30:38.822 | INFO | apex.amp.frontend:initialize:329 - Defaults for this optimization level are:
2021-08-03 18:30:38.823 | INFO | apex.amp.frontend:initialize:331 - enabled : True
2021-08-03 18:30:38.824 | INFO | apex.amp.frontend:initialize:331 - opt_level : O1
2021-08-03 18:30:38.824 | INFO | apex.amp.frontend:initialize:331 - cast_model_type : None
2021-08-03 18:30:38.824 | INFO | apex.amp.frontend:initialize:331 - patch_torch_functions : True
2021-08-03 18:30:38.824 | INFO | apex.amp.frontend:initialize:331 - keep_batchnorm_fp32 : None
2021-08-03 18:30:38.824 | INFO | apex.amp.frontend:initialize:331 - master_weights : None
2021-08-03 18:30:38.825 | INFO | apex.amp.frontend:initialize:331 - loss_scale : dynamic
2021-08-03 18:30:38.825 | INFO | apex.amp.frontend:initialize:336 - Processing user overrides (additional kwargs that are not None)...
2021-08-03 18:30:38.825 | INFO | apex.amp.frontend:initialize:354 - After processing overrides, optimization options are:
2021-08-03 18:30:38.825 | INFO | apex.amp.frontend:initialize:356 - enabled : True
2021-08-03 18:30:38.825 | INFO | apex.amp.frontend:initialize:356 - opt_level : O1
2021-08-03 18:30:38.825 | INFO | apex.amp.frontend:initialize:356 - cast_model_type : None
2021-08-03 18:30:38.826 | INFO | apex.amp.frontend:initialize:356 - patch_torch_functions : True
2021-08-03 18:30:38.826 | INFO | apex.amp.frontend:initialize:356 - keep_batchnorm_fp32 : None
2021-08-03 18:30:38.826 | INFO | apex.amp.frontend:initialize:356 - master_weights : None
2021-08-03 18:30:38.826 | INFO | apex.amp.frontend:initialize:356 - loss_scale : dynamic
2021-08-03 18:30:38.837 | INFO | yolox.core.trainer:resume_train:296 - loading checkpoint for fine tuning
2021-08-03 18:30:39.174 | WARNING | yolox.utils.checkpoint:load_ckpt:27 - Shape of head.cls_preds.0.weight in checkpoint is torch.Size([80, 256, 1, 1]), while shape of head.cls_preds.0.weight in model is torch.Size([4, 256, 1, 1]).
2021-08-03 18:30:39.175 | WARNING | yolox.utils.checkpoint:load_ckpt:27 - Shape of head.cls_preds.0.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.0.bias in model is torch.Size([4]).
2021-08-03 18:30:39.175 | WARNING | yolox.utils.checkpoint:load_ckpt:27 - Shape of head.cls_preds.1.weight in checkpoint is torch.Size([80, 256, 1, 1]), while shape of head.cls_preds.1.weight in model is torch.Size([4, 256, 1, 1]).
2021-08-03 18:30:39.175 | WARNING | yolox.utils.checkpoint:load_ckpt:27 - Shape of head.cls_preds.1.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.1.bias in model is torch.Size([4]).
2021-08-03 18:30:39.175 | WARNING | yolox.utils.checkpoint:load_ckpt:27 - Shape of head.cls_preds.2.weight in checkpoint is torch.Size([80, 256, 1, 1]), while shape of head.cls_preds.2.weight in model is torch.Size([4, 256, 1, 1]).
2021-08-03 18:30:39.176 | WARNING | yolox.utils.checkpoint:load_ckpt:27 - Shape of head.cls_preds.2.bias in checkpoint is torch.Size([80]), while shape of head.cls_preds.2.bias in model is torch.Size([4]).
2021-08-03 18:30:39.236 | INFO | yolox.data.datasets.coco:init:44 - loading annotations into memory...
2021-08-03 18:30:39.533 | INFO | yolox.data.datasets.coco:init:44 - Done (t=0.30s)
2021-08-03 18:30:39.534 | INFO | pycocotools.coco:init:88 - creating index...
2021-08-03 18:30:39.557 | INFO | pycocotools.coco:init:88 - index created!
2021-08-03 18:30:40.932 | INFO | yolox.core.trainer:before_train:152 - init prefetcher, this might take one minute or less...
2021-08-03 18:30:48.218 | INFO | yolox.data.datasets.coco:init:44 - loading annotations into memory...
2021-08-03 18:30:48.244 | INFO | yolox.data.datasets.coco:init:44 - Done (t=0.03s)
2021-08-03 18:30:48.245 | INFO | pycocotools.coco:init:88 - creating index...
2021-08-03 18:30:48.247 | INFO | pycocotools.coco:init:88 - index created!
2021-08-03 18:30:48.405 | INFO | yolox.core.trainer:before_train:182 - Training start...
2021-08-03 18:30:48.408 | INFO | yolox.core.trainer:before_train:183 -
YOLOX(
(backbone): YOLOPAFPN(
(backbone): CSPDarknet(
(stem): Focus(
(conv): BaseConv(
(conv): Conv2d(12, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(dark2): Sequential(
(0): BaseConv(
(conv): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(64, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark3): Sequential(
(0): BaseConv(
(conv): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(3): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(4): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(5): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(6): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(7): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(8): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark4): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(3): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(4): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(5): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(6): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(7): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(8): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(dark5): Sequential(
(0): BaseConv(
(conv): Conv2d(512, 1024, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): SPPBottleneck(
(conv1): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): ModuleList(
(0): MaxPool2d(kernel_size=5, stride=1, padding=2, dilation=1, ceil_mode=False)
(1): MaxPool2d(kernel_size=9, stride=1, padding=4, dilation=1, ceil_mode=False)
(2): MaxPool2d(kernel_size=13, stride=1, padding=6, dilation=1, ceil_mode=False)
)
(conv2): BaseConv(
(conv): Conv2d(2048, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
)
(upsample): Upsample(scale_factor=2.0, mode=nearest)
(lateral_conv0): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_p4): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(reduce_conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_p3): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(128, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(bu_conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_n3): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
(bu_conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(C3_n4): CSPLayer(
(conv1): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv3): BaseConv(
(conv): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(1024, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(m): Sequential(
(0): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(conv2): BaseConv(
(conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(512, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
)
)
(head): YOLOXHead(
(cls_convs): ModuleList(
(0): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
(reg_convs): ModuleList(
(0): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(1): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(2): Sequential(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
)
(cls_preds): ModuleList(
(0): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1))
)
(reg_preds): ModuleList(
(0): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(256, 4, kernel_size=(1, 1), stride=(1, 1))
)
(obj_preds): ModuleList(
(0): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1))
(1): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1))
(2): Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1))
)
(stems): ModuleList(
(0): BaseConv(
(conv): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(1): BaseConv(
(conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
(2): BaseConv(
(conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): BatchNorm2d(256, eps=0.001, momentum=0.03, affine=True, track_running_stats=True)
(act): SiLU(inplace=True)
)
)
(l1_loss): L1Loss()
(bcewithlog_loss): BCEWithLogitsLoss()
(iou_loss): IOUloss()
)
)
2021-08-03 18:30:48.409 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch1
2021-08-03 18:30:52.759 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 32768.0
2021-08-03 18:30:55.351 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 18:31:17.313 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 8192.0
2021-08-03 18:32:11.637 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 100/700, mem: 11631Mb, iter_time: 0.831s, data_time: 0.001s, total_loss: 13.8, iou_loss: 3.7, l1_loss: 0.0, conf_loss: 8.0, cls_loss: 2.1, lr: 1.020e-06, size: 704, ETA: 2 days, 0:28:45
2021-08-03 18:33:18.327 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 200/700, mem: 11631Mb, iter_time: 0.666s, data_time: 0.001s, total_loss: 11.5, iou_loss: 4.1, l1_loss: 0.0, conf_loss: 6.0, cls_loss: 1.4, lr: 4.082e-06, size: 672, ETA: 1 day, 19:38:09
2021-08-03 18:34:22.784 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 300/700, mem: 11631Mb, iter_time: 0.644s, data_time: 0.001s, total_loss: 9.4, iou_loss: 3.7, l1_loss: 0.0, conf_loss: 4.6, cls_loss: 1.1, lr: 9.184e-06, size: 640, ETA: 1 day, 17:34:27
2021-08-03 18:35:16.393 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 400/700, mem: 11631Mb, iter_time: 0.535s, data_time: 0.001s, total_loss: 10.6, iou_loss: 3.6, l1_loss: 0.0, conf_loss: 5.8, cls_loss: 1.2, lr: 1.633e-05, size: 768, ETA: 1 day, 14:57:18
2021-08-03 18:36:06.604 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 500/700, mem: 11631Mb, iter_time: 0.501s, data_time: 0.002s, total_loss: 9.3, iou_loss: 3.5, l1_loss: 0.0, conf_loss: 4.8, cls_loss: 1.0, lr: 2.551e-05, size: 768, ETA: 1 day, 12:58:56
2021-08-03 18:37:01.581 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 600/700, mem: 11631Mb, iter_time: 0.549s, data_time: 0.001s, total_loss: 9.8, iou_loss: 3.5, l1_loss: 0.0, conf_loss: 5.2, cls_loss: 1.0, lr: 3.673e-05, size: 800, ETA: 1 day, 12:07:29
2021-08-03 18:37:55.036 | INFO | yolox.core.trainer:after_iter:254 - epoch: 1/300, iter: 700/700, mem: 11631Mb, iter_time: 0.534s, data_time: 0.001s, total_loss: 8.0, iou_loss: 3.5, l1_loss: 0.0, conf_loss: 3.5, cls_loss: 1.0, lr: 5.000e-05, size: 576, ETA: 1 day, 11:22:53
2021-08-03 18:37:55.037 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 18:37:55.885 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch2
2021-08-03 18:38:56.248 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 100/700, mem: 11631Mb, iter_time: 0.603s, data_time: 0.001s, total_loss: 7.8, iou_loss: 3.0, l1_loss: 0.0, conf_loss: 3.9, cls_loss: 0.9, lr: 6.531e-05, size: 800, ETA: 1 day, 11:19:20
2021-08-03 18:39:53.239 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 200/700, mem: 11631Mb, iter_time: 0.569s, data_time: 0.001s, total_loss: 7.3, iou_loss: 3.2, l1_loss: 0.0, conf_loss: 3.3, cls_loss: 0.8, lr: 8.265e-05, size: 640, ETA: 1 day, 11:03:17
2021-08-03 18:40:50.273 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 300/700, mem: 11631Mb, iter_time: 0.569s, data_time: 0.001s, total_loss: 6.6, iou_loss: 2.9, l1_loss: 0.0, conf_loss: 2.9, cls_loss: 0.8, lr: 1.020e-04, size: 768, ETA: 1 day, 10:50:25
2021-08-03 18:41:45.868 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 400/700, mem: 11631Mb, iter_time: 0.555s, data_time: 0.001s, total_loss: 6.5, iou_loss: 2.9, l1_loss: 0.0, conf_loss: 2.8, cls_loss: 0.8, lr: 1.235e-04, size: 608, ETA: 1 day, 10:35:08
2021-08-03 18:42:52.910 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 500/700, mem: 11631Mb, iter_time: 0.670s, data_time: 0.001s, total_loss: 6.7, iou_loss: 2.9, l1_loss: 0.0, conf_loss: 3.0, cls_loss: 0.8, lr: 1.469e-04, size: 512, ETA: 1 day, 10:55:28
2021-08-03 18:43:44.674 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 600/700, mem: 11631Mb, iter_time: 0.517s, data_time: 0.001s, total_loss: 5.7, iou_loss: 2.7, l1_loss: 0.0, conf_loss: 2.3, cls_loss: 0.7, lr: 1.724e-04, size: 512, ETA: 1 day, 10:31:35
2021-08-03 18:44:38.603 | INFO | yolox.core.trainer:after_iter:254 - epoch: 2/300, iter: 700/700, mem: 11631Mb, iter_time: 0.538s, data_time: 0.002s, total_loss: 6.0, iou_loss: 2.8, l1_loss: 0.0, conf_loss: 2.5, cls_loss: 0.8, lr: 2.000e-04, size: 576, ETA: 1 day, 10:16:23
2021-08-03 18:44:38.605 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 18:44:54.125 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 18:44:55.227 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 18:44:55.729 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.50s)
2021-08-03 18:44:55.730 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 18:44:55.762 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 18:44:56.326 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 13.84 ms, Average NMS time: 1.14 ms, Average inference time: 14.98 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

2021-08-03 18:44:56.327 | INFO 2021-08-03 18:44:57.388 | INFO 2021-08-03 18:46:00.209 | INFO 2021-08-03 18:46:58.807 | INFO 2021-08-03 18:47:54.314 | INFO 2021-08-03 18:48:55.416 | INFO 2021-08-03 18:49:48.368 | INFO 2021-08-03 18:50:46.390 | INFO 2021-08-03 18:51:40.492 | INFO 2021-08-03 18:51:40.494 | INFO 2021-08-03 18:51:42.488 | INFO 2021-08-03 18:52:40.516 | INFO 2021-08-03 18:53:36.909 | INFO 2021-08-03 18:54:37.040 | INFO 2021-08-03 18:55:30.734 | INFO 2021-08-03 18:56:25.397 | INFO 2021-08-03 18:57:16.552 | INFO 2021-08-03 18:58:13.293 | INFO 2021-08-03 18:58:13.295 | INFO 2021-08-03 18:58:27.422 | INFO 2021-08-03 18:58:27.860 | INFO 2021-08-03 18:58:28.057 | INFO 2021-08-03 18:58:28.057 | INFO 2021-08-03 18:58:28.070 | INFO 2021-08-03 18:58:28.365 | INFO Average forward time: Average Precision (AP) Average Precision (AP) @[ IoU=0.50 Average Precision (AP) @[ IoU=0.75 Average Precision (AP) Average Precision (AP) Average Precision (AP) Average Recall (AR) Average Recall (AR) Average Recall (AR) Average Recall (AR) Average Recall (AR) Average Recall (AR) | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
| yolox.core.trainer:before_epoch:193 - ---> start train epoch3
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 100/700, mem: 11631Mb, iter_time: 0.627s, data_time: 0.001s, total_loss: 4.8, iou_loss: 2.4, l1_loss: 0.0, conf_loss: 1.8, cls_loss: 0.7, lr: 2.296e-04, size: 672, ETA: 1 day, 10:23:42
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 200/700, mem: 11631Mb, iter_time: 0.585s, data_time: 0.001s, total_loss: 5.1, iou_loss: 2.3, l1_loss: 0.0, conf_loss: 2.2, cls_loss: 0.6, lr: 2.612e-04, size: 672, ETA: 1 day, 10:20:48
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 300/700, mem: 11631Mb, iter_time: 0.554s, data_time: 0.001s, total_loss: 5.2, iou_loss: 2.4, l1_loss: 0.0, conf_loss: 2.1, cls_loss: 0.7, lr: 2.949e-04, size: 576, ETA: 1 day, 10:11:48
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 400/700, mem: 11631Mb, iter_time: 0.610s, data_time: 0.001s, total_loss: 5.7, iou_loss: 2.5, l1_loss: 0.0, conf_loss: 2.5, cls_loss: 0.7, lr: 3.306e-04, size: 736, ETA: 1 day, 10:14:29
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 500/700, mem: 11631Mb, iter_time: 0.529s, data_time: 0.002s, total_loss: 4.4, iou_loss: 2.2, l1_loss: 0.0, conf_loss: 1.6, cls_loss: 0.6, lr: 3.684e-04, size: 640, ETA: 1 day, 10:01:55
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 600/700, mem: 11631Mb, iter_time: 0.579s, data_time: 0.001s, total_loss: 5.2, iou_loss: 2.4, l1_loss: 0.0, conf_loss: 2.1, cls_loss: 0.6, lr: 4.082e-04, size: 832, ETA: 1 day, 9:59:18
| yolox.core.trainer:after_iter:254 - epoch: 3/300, iter: 700/700, mem: 11631Mb, iter_time: 0.540s, data_time: 0.001s, total_loss: 4.3, iou_loss: 2.1, l1_loss: 0.0, conf_loss: 1.6, cls_loss: 0.6, lr: 4.500e-04, size: 544, ETA: 1 day, 9:50:22
| yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
| yolox.core.trainer:before_epoch:193 - ---> start train epoch4
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 100/700, mem: 11631Mb, iter_time: 0.579s, data_time: 0.001s, total_loss: 4.4, iou_loss: 2.1, l1_loss: 0.0, conf_loss: 1.6, cls_loss: 0.6, lr: 4.939e-04, size: 672, ETA: 1 day, 9:48:22
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 200/700, mem: 11631Mb, iter_time: 0.563s, data_time: 0.001s, total_loss: 4.6, iou_loss: 2.5, l1_loss: 0.0, conf_loss: 1.5, cls_loss: 0.6, lr: 5.398e-04, size: 512, ETA: 1 day, 9:43:58
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 300/700, mem: 11631Mb, iter_time: 0.600s, data_time: 0.001s, total_loss: 4.4, iou_loss: 2.2, l1_loss: 0.0, conf_loss: 1.6, cls_loss: 0.6, lr: 5.878e-04, size: 576, ETA: 1 day, 9:45:16
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 400/700, mem: 11631Mb, iter_time: 0.536s, data_time: 0.001s, total_loss: 4.0, iou_loss: 2.2, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.6, lr: 6.378e-04, size: 512, ETA: 1 day, 9:37:28
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 500/700, mem: 11631Mb, iter_time: 0.546s, data_time: 0.001s, total_loss: 4.0, iou_loss: 2.1, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.6, lr: 6.898e-04, size: 480, ETA: 1 day, 9:31:29
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 600/700, mem: 11631Mb, iter_time: 0.511s, data_time: 0.001s, total_loss: 4.0, iou_loss: 2.1, l1_loss: 0.0, conf_loss: 1.4, cls_loss: 0.6, lr: 7.439e-04, size: 832, ETA: 1 day, 9:21:23
| yolox.core.trainer:after_iter:254 - epoch: 4/300, iter: 700/700, mem: 11631Mb, iter_time: 0.566s, data_time: 0.001s, total_loss: 4.5, iou_loss: 2.2, l1_loss: 0.0, conf_loss: 1.7, cls_loss: 0.6, lr: 8.000e-04, size: 640, ETA: 1 day, 9:18:50
| yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
| yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
| yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
| yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.20s)
| pycocotools.coco:loadRes:363 - creating index...
| pycocotools.coco:loadRes:363 - index created!
| yolox.core.trainer:evaluate_and_save_model:313 -
13.99 ms, Average NMS time: 1.42 ms, Average inference time: 15.41 ms
@[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
| area= all | maxDets=100 ] = 0.000
| area= all | maxDets=100 ] = 0.000
@[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
@[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
@[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
@[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
@[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
@[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
@[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
@[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
@[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 18:58:28.366 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 18:58:33.508 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch5
2021-08-03 18:59:34.365 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 100/700, mem: 11631Mb, iter_time: 0.608s, data_time: 0.001s, total_loss: 4.1, iou_loss: 2.2, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.6, lr: 8.582e-04, size: 448, ETA: 1 day, 9:21:18
2021-08-03 19:00:32.544 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 200/700, mem: 11631Mb, iter_time: 0.581s, data_time: 0.001s, total_loss: 4.5, iou_loss: 2.4, l1_loss: 0.0, conf_loss: 1.5, cls_loss: 0.6, lr: 9.184e-04, size: 448, ETA: 1 day, 9:20:27
2021-08-03 19:01:30.324 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 300/700, mem: 11631Mb, iter_time: 0.577s, data_time: 0.001s, total_loss: 4.1, iou_loss: 2.0, l1_loss: 0.0, conf_loss: 1.5, cls_loss: 0.5, lr: 9.806e-04, size: 512, ETA: 1 day, 9:19:10
2021-08-03 19:02:21.770 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 400/700, mem: 11631Mb, iter_time: 0.513s, data_time: 0.001s, total_loss: 3.4, iou_loss: 1.9, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.5, lr: 1.045e-03, size: 672, ETA: 1 day, 9:11:03
2021-08-03 19:03:15.921 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 500/700, mem: 11631Mb, iter_time: 0.541s, data_time: 0.001s, total_loss: 4.1, iou_loss: 2.1, l1_loss: 0.0, conf_loss: 1.4, cls_loss: 0.6, lr: 1.111e-03, size: 832, ETA: 1 day, 9:06:13
2021-08-03 19:04:13.863 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 600/700, mem: 11631Mb, iter_time: 0.579s, data_time: 0.001s, total_loss: 4.1, iou_loss: 2.3, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.6, lr: 1.180e-03, size: 480, ETA: 1 day, 9:05:27
2021-08-03 19:05:03.825 | INFO | yolox.core.trainer:after_iter:254 - epoch: 5/300, iter: 700/700, mem: 11631Mb, iter_time: 0.499s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 576, ETA: 1 day, 8:56:50
2021-08-03 19:05:03.827 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:05:06.506 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch6
2021-08-03 19:05:58.201 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 100/700, mem: 11631Mb, iter_time: 0.516s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.9, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 448, ETA: 1 day, 8:50:17
2021-08-03 19:06:55.884 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 200/700, mem: 11631Mb, iter_time: 0.576s, data_time: 0.001s, total_loss: 4.0, iou_loss: 2.0, l1_loss: 0.0, conf_loss: 1.5, cls_loss: 0.5, lr: 1.250e-03, size: 800, ETA: 1 day, 8:49:37
2021-08-03 19:07:55.008 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 300/700, mem: 11631Mb, iter_time: 0.590s, data_time: 0.001s, total_loss: 4.7, iou_loss: 2.4, l1_loss: 0.0, conf_loss: 1.6, cls_loss: 0.6, lr: 1.250e-03, size: 640, ETA: 1 day, 8:50:15
2021-08-03 19:08:55.817 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 400/700, mem: 11631Mb, iter_time: 0.607s, data_time: 0.001s, total_loss: 3.7, iou_loss: 1.9, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.5, lr: 1.250e-03, size: 608, ETA: 1 day, 8:52:17
2021-08-03 19:09:53.447 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 500/700, mem: 11631Mb, iter_time: 0.575s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 800, ETA: 1 day, 8:51:25
2021-08-03 19:10:34.144 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 19:10:53.561 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 600/700, mem: 11631Mb, iter_time: 0.600s, data_time: 0.001s, total_loss: 3.4, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.250e-03, size: 576, ETA: 1 day, 8:52:38
2021-08-03 19:11:59.282 | INFO | yolox.core.trainer:after_iter:254 - epoch: 6/300, iter: 700/700, mem: 11631Mb, iter_time: 0.656s, data_time: 0.001s, total_loss: 4.1, iou_loss: 2.1, l1_loss: 0.0, conf_loss: 1.5, cls_loss: 0.5, lr: 1.250e-03, size: 448, ETA: 1 day, 8:58:20
2021-08-03 19:11:59.283 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:12:13.623 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 19:12:13.800 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 19:12:13.836 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.04s)
2021-08-03 19:12:13.836 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 19:12:13.841 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 19:12:14.009 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.10 ms, Average NMS time: 1.09 ms, Average inference time: 15.18 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 19:12:14.010 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:12:18.546 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch7
2021-08-03 19:13:16.108 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 100/700, mem: 11631Mb, iter_time: 0.575s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 576, ETA: 1 day, 8:57:13
2021-08-03 19:14:17.708 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 200/700, mem: 11631Mb, iter_time: 0.615s, data_time: 0.001s, total_loss: 3.3, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.5, lr: 1.250e-03, size: 736, ETA: 1 day, 8:59:14
2021-08-03 19:15:11.908 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 300/700, mem: 11631Mb, iter_time: 0.541s, data_time: 0.001s, total_loss: 3.6, iou_loss: 1.9, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.6, lr: 1.250e-03, size: 448, ETA: 1 day, 8:55:29
2021-08-03 19:16:10.501 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 400/700, mem: 11631Mb, iter_time: 0.585s, data_time: 0.001s, total_loss: 3.4, iou_loss: 1.7, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 512, ETA: 1 day, 8:55:08
2021-08-03 19:17:06.833 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 500/700, mem: 11631Mb, iter_time: 0.562s, data_time: 0.001s, total_loss: 3.0, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.250e-03, size: 576, ETA: 1 day, 8:53:07
2021-08-03 19:18:06.971 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 600/700, mem: 11631Mb, iter_time: 0.600s, data_time: 0.001s, total_loss: 3.3, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 832, ETA: 1 day, 8:53:51
2021-08-03 19:19:06.691 | INFO | yolox.core.trainer:after_iter:254 - epoch: 7/300, iter: 700/700, mem: 11631Mb, iter_time: 0.596s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 736, ETA: 1 day, 8:54:13
2021-08-03 19:19:06.692 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:19:09.507 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch8
2021-08-03 19:20:06.377 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 100/700, mem: 11631Mb, iter_time: 0.568s, data_time: 0.001s, total_loss: 3.8, iou_loss: 1.9, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.5, lr: 1.250e-03, size: 736, ETA: 1 day, 8:52:35
2021-08-03 19:21:04.008 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 200/700, mem: 11631Mb, iter_time: 0.575s, data_time: 0.001s, total_loss: 3.2, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.250e-03, size: 640, ETA: 1 day, 8:51:30
2021-08-03 19:21:56.904 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 300/700, mem: 11631Mb, iter_time: 0.528s, data_time: 0.001s, total_loss: 3.2, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.250e-03, size: 448, ETA: 1 day, 8:47:18
2021-08-03 19:22:55.071 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 400/700, mem: 11631Mb, iter_time: 0.581s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.5, lr: 1.250e-03, size: 736, ETA: 1 day, 8:46:37
2021-08-03 19:23:56.654 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 500/700, mem: 11631Mb, iter_time: 0.615s, data_time: 0.001s, total_loss: 3.0, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.250e-03, size: 800, ETA: 1 day, 8:48:05
2021-08-03 19:24:50.165 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 600/700, mem: 11631Mb, iter_time: 0.534s, data_time: 0.001s, total_loss: 3.4, iou_loss: 1.7, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.5, lr: 1.250e-03, size: 800, ETA: 1 day, 8:44:28
2021-08-03 19:25:54.492 | INFO | yolox.core.trainer:after_iter:254 - epoch: 8/300, iter: 700/700, mem: 11631Mb, iter_time: 0.642s, data_time: 0.001s, total_loss: 3.9, iou_loss: 2.0, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.5, lr: 1.250e-03, size: 640, ETA: 1 day, 8:47:31
2021-08-03 19:25:54.493 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:26:08.330 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 19:26:08.502 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 19:26:08.537 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.03s)
2021-08-03 19:26:08.538 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 19:26:08.543 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 19:26:08.707 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.31 ms, Average NMS time: 1.17 ms, Average inference time: 15.48 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

2021-08-03 19:26:08.707 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:26:11.510 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch9
2021-08-03 19:27:06.751 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 100/700, mem: 11631Mb, iter_time: 0.551s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.9, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.250e-03, size: 448, ETA: 1 day, 8:45:00
2021-08-03 19:28:06.396 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 200/700, mem: 11631Mb, iter_time: 0.596s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.5, lr: 1.250e-03, size: 800, ETA: 1 day, 8:45:07
2021-08-03 19:29:10.839 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 300/700, mem: 11631Mb, iter_time: 0.643s, data_time: 0.001s, total_loss: 3.2, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.250e-03, size: 640, ETA: 1 day, 8:47:58
2021-08-03 19:30:11.873 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 400/700, mem: 11631Mb, iter_time: 0.609s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.5, lr: 1.250e-03, size: 480, ETA: 1 day, 8:48:45
2021-08-03 19:30:59.898 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 19:31:11.473 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 500/700, mem: 11631Mb, iter_time: 0.595s, data_time: 0.001s, total_loss: 3.5, iou_loss: 1.7, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.249e-03, size: 832, ETA: 1 day, 8:48:41
2021-08-03 19:32:04.465 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 600/700, mem: 11631Mb, iter_time: 0.529s, data_time: 0.001s, total_loss: 3.6, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.3, cls_loss: 0.5, lr: 1.249e-03, size: 736, ETA: 1 day, 8:44:57
2021-08-03 19:32:55.247 | INFO | yolox.core.trainer:after_iter:254 - epoch: 9/300, iter: 700/700, mem: 11631Mb, iter_time: 0.507s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.249e-03, size: 448, ETA: 1 day, 8:40:07
2021-08-03 19:32:55.249 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:32:57.513 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch10
2021-08-03 19:33:55.712 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 100/700, mem: 11631Mb, iter_time: 0.581s, data_time: 0.004s, total_loss: 3.2, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.5, lr: 1.249e-03, size: 832, ETA: 1 day, 8:39:21
2021-08-03 19:34:55.694 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 200/700, mem: 11631Mb, iter_time: 0.599s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.249e-03, size: 512, ETA: 1 day, 8:39:31
2021-08-03 19:35:48.604 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 300/700, mem: 11631Mb, iter_time: 0.528s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.5, lr: 1.249e-03, size: 736, ETA: 1 day, 8:36:00
2021-08-03 19:36:48.675 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 400/700, mem: 11631Mb, iter_time: 0.600s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.249e-03, size: 544, ETA: 1 day, 8:36:12
2021-08-03 19:37:45.844 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 500/700, mem: 11631Mb, iter_time: 0.571s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.249e-03, size: 544, ETA: 1 day, 8:34:54
2021-08-03 19:38:40.661 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 600/700, mem: 11631Mb, iter_time: 0.547s, data_time: 0.001s, total_loss: 3.3, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.5, lr: 1.249e-03, size: 512, ETA: 1 day, 8:32:28
2021-08-03 19:39:36.425 | INFO | yolox.core.trainer:after_iter:254 - epoch: 10/300, iter: 700/700, mem: 11631Mb, iter_time: 0.557s, data_time: 0.001s, total_loss: 3.0, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.5, lr: 1.249e-03, size: 448, ETA: 1 day, 8:30:32
2021-08-03 19:39:36.427 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:39:50.632 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 19:39:50.777 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 19:39:50.806 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.03s)
2021-08-03 19:39:50.807 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 19:39:50.811 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 19:39:50.964 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.10 ms, Average NMS time: 1.25 ms, Average inference time: 15.36 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 19:39:50.964 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:39:53.505 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch11
2021-08-03 19:40:49.319 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 100/700, mem: 11631Mb, iter_time: 0.557s, data_time: 0.001s, total_loss: 3.4, iou_loss: 1.8, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.249e-03, size: 576, ETA: 1 day, 8:28:39
2021-08-03 19:41:47.194 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 200/700, mem: 11631Mb, iter_time: 0.578s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.249e-03, size: 544, ETA: 1 day, 8:27:46
2021-08-03 19:42:39.392 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 300/700, mem: 11631Mb, iter_time: 0.521s, data_time: 0.001s, total_loss: 3.0, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.5, lr: 1.249e-03, size: 832, ETA: 1 day, 8:24:15
2021-08-03 19:43:37.723 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 400/700, mem: 11631Mb, iter_time: 0.582s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.5, lr: 1.249e-03, size: 512, ETA: 1 day, 8:23:36
2021-08-03 19:44:32.898 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 500/700, mem: 11631Mb, iter_time: 0.551s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.249e-03, size: 736, ETA: 1 day, 8:21:32
2021-08-03 19:45:28.179 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 600/700, mem: 11631Mb, iter_time: 0.552s, data_time: 0.001s, total_loss: 4.7, iou_loss: 2.2, l1_loss: 0.0, conf_loss: 1.9, cls_loss: 0.6, lr: 1.249e-03, size: 768, ETA: 1 day, 8:19:32
2021-08-03 19:46:29.704 | INFO | yolox.core.trainer:after_iter:254 - epoch: 11/300, iter: 700/700, mem: 11631Mb, iter_time: 0.614s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.5, lr: 1.249e-03, size: 480, ETA: 1 day, 8:20:18
2021-08-03 19:46:29.705 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:46:32.503 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch12
2021-08-03 19:47:25.946 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 100/700, mem: 11631Mb, iter_time: 0.533s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.5, lr: 1.249e-03, size: 544, ETA: 1 day, 8:17:32
2021-08-03 19:48:25.425 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 200/700, mem: 11631Mb, iter_time: 0.594s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.249e-03, size: 576, ETA: 1 day, 8:17:23
2021-08-03 19:49:19.265 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 300/700, mem: 11631Mb, iter_time: 0.537s, data_time: 0.001s, total_loss: 3.0, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.4, lr: 1.248e-03, size: 800, ETA: 1 day, 8:14:50
2021-08-03 19:50:16.233 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 400/700, mem: 11631Mb, iter_time: 0.569s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 704, ETA: 1 day, 8:13:37
2021-08-03 19:50:27.054 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 19:51:08.613 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 500/700, mem: 11631Mb, iter_time: 0.523s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 640, ETA: 1 day, 8:10:33
2021-08-03 19:52:04.639 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 600/700, mem: 11631Mb, iter_time: 0.559s, data_time: 0.004s, total_loss: 2.6, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 800, ETA: 1 day, 8:09:00
2021-08-03 19:52:56.722 | INFO | yolox.core.trainer:after_iter:254 - epoch: 12/300, iter: 700/700, mem: 11631Mb, iter_time: 0.520s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 672, ETA: 1 day, 8:05:53
2021-08-03 19:52:56.723 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:53:11.173 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 19:53:11.316 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 19:53:11.445 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.13s)
2021-08-03 19:53:11.446 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 19:53:11.450 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 19:53:11.603 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.11 ms, Average NMS time: 1.19 ms, Average inference time: 15.29 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 19:53:11.603 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:53:15.543 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch13
2021-08-03 19:54:07.515 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 100/700, mem: 11631Mb, iter_time: 0.519s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 768, ETA: 1 day, 8:02:46
2021-08-03 19:55:06.049 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 200/700, mem: 11631Mb, iter_time: 0.584s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 768, ETA: 1 day, 8:02:17
2021-08-03 19:56:05.636 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 300/700, mem: 11631Mb, iter_time: 0.595s, data_time: 0.001s, total_loss: 3.1, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.2, cls_loss: 0.4, lr: 1.248e-03, size: 768, ETA: 1 day, 8:02:11
2021-08-03 19:57:03.298 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 400/700, mem: 11631Mb, iter_time: 0.576s, data_time: 0.001s, total_loss: 3.3, iou_loss: 1.7, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.248e-03, size: 832, ETA: 1 day, 8:01:20
2021-08-03 19:57:58.711 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 500/700, mem: 11631Mb, iter_time: 0.553s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.248e-03, size: 512, ETA: 1 day, 7:59:38
2021-08-03 19:58:49.530 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 600/700, mem: 11631Mb, iter_time: 0.507s, data_time: 0.002s, total_loss: 2.5, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.248e-03, size: 512, ETA: 1 day, 7:56:15
2021-08-03 19:59:45.449 | INFO | yolox.core.trainer:after_iter:254 - epoch: 13/300, iter: 700/700, mem: 11631Mb, iter_time: 0.558s, data_time: 0.003s, total_loss: 2.7, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.248e-03, size: 576, ETA: 1 day, 7:54:47
2021-08-03 19:59:45.450 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 19:59:47.512 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch14
2021-08-03 20:00:42.173 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 100/700, mem: 11631Mb, iter_time: 0.546s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.248e-03, size: 480, ETA: 1 day, 7:52:52
2021-08-03 20:01:34.572 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 200/700, mem: 11631Mb, iter_time: 0.523s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.247e-03, size: 512, ETA: 1 day, 7:50:10
2021-08-03 20:02:33.547 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 300/700, mem: 11631Mb, iter_time: 0.589s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.247e-03, size: 512, ETA: 1 day, 7:49:51
2021-08-03 20:03:27.640 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 400/700, mem: 11631Mb, iter_time: 0.540s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.247e-03, size: 608, ETA: 1 day, 7:47:48
2021-08-03 20:04:21.218 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 500/700, mem: 11631Mb, iter_time: 0.535s, data_time: 0.003s, total_loss: 2.6, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.247e-03, size: 544, ETA: 1 day, 7:45:36
2021-08-03 20:05:21.258 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 600/700, mem: 11631Mb, iter_time: 0.599s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.247e-03, size: 640, ETA: 1 day, 7:45:38
2021-08-03 20:06:22.363 | INFO | yolox.core.trainer:after_iter:254 - epoch: 14/300, iter: 700/700, mem: 11631Mb, iter_time: 0.610s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.247e-03, size: 544, ETA: 1 day, 7:46:02
2021-08-03 20:06:22.364 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:06:36.501 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 20:06:36.632 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 20:06:36.658 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.03s)
2021-08-03 20:06:36.659 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 20:06:36.662 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 20:06:36.810 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.12 ms, Average NMS time: 1.16 ms, Average inference time: 15.28 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 20:06:36.810 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:06:39.510 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch15
2021-08-03 20:07:37.169 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 100/700, mem: 11631Mb, iter_time: 0.576s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.247e-03, size: 832, ETA: 1 day, 7:45:13
2021-08-03 20:08:34.392 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 200/700, mem: 11631Mb, iter_time: 0.571s, data_time: 0.001s, total_loss: 3.2, iou_loss: 1.7, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.5, lr: 1.247e-03, size: 576, ETA: 1 day, 7:44:16
2021-08-03 20:09:29.601 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 300/700, mem: 11631Mb, iter_time: 0.551s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.247e-03, size: 768, ETA: 1 day, 7:42:39
2021-08-03 20:10:01.593 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 20:10:29.414 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 400/700, mem: 11631Mb, iter_time: 0.597s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.4, lr: 1.247e-03, size: 704, ETA: 1 day, 7:42:33
2021-08-03 20:11:26.526 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 500/700, mem: 11631Mb, iter_time: 0.570s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.246e-03, size: 704, ETA: 1 day, 7:41:34
2021-08-03 20:12:25.829 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 600/700, mem: 11631Mb, iter_time: 0.592s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.246e-03, size: 608, ETA: 1 day, 7:41:17
2021-08-03 20:13:25.414 | INFO | yolox.core.trainer:after_iter:254 - epoch: 15/300, iter: 700/700, mem: 11631Mb, iter_time: 0.595s, data_time: 0.001s, total_loss: 3.6, iou_loss: 1.7, l1_loss: 0.0, conf_loss: 1.4, cls_loss: 0.5, lr: 1.246e-03, size: 832, ETA: 1 day, 7:41:04
2021-08-03 20:13:25.415 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:13:27.510 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch16
2021-08-03 20:14:27.048 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 100/700, mem: 11631Mb, iter_time: 0.594s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.246e-03, size: 640, ETA: 1 day, 7:40:50
2021-08-03 20:15:19.174 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 200/700, mem: 11631Mb, iter_time: 0.520s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.246e-03, size: 576, ETA: 1 day, 7:38:16
2021-08-03 20:16:14.581 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 300/700, mem: 11631Mb, iter_time: 0.553s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.246e-03, size: 768, ETA: 1 day, 7:36:45
2021-08-03 20:17:15.773 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 400/700, mem: 11631Mb, iter_time: 0.611s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.246e-03, size: 768, ETA: 1 day, 7:37:01
2021-08-03 20:18:08.747 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 500/700, mem: 11631Mb, iter_time: 0.529s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.246e-03, size: 640, ETA: 1 day, 7:34:46
2021-08-03 20:19:03.386 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 600/700, mem: 11631Mb, iter_time: 0.545s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.5, lr: 1.246e-03, size: 480, ETA: 1 day, 7:33:02
2021-08-03 20:19:57.641 | INFO | yolox.core.trainer:after_iter:254 - epoch: 16/300, iter: 700/700, mem: 11631Mb, iter_time: 0.542s, data_time: 0.001s, total_loss: 1.9, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.3, lr: 1.245e-03, size: 576, ETA: 1 day, 7:31:13
2021-08-03 20:19:57.644 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:20:12.018 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 20:20:12.137 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 20:20:12.162 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.02s)
2021-08-03 20:20:12.162 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 20:20:12.165 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 20:20:12.309 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.08 ms, Average NMS time: 1.20 ms, Average inference time: 15.28 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 20:20:12.310 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:20:14.572 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch17
2021-08-03 20:21:15.146 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 100/700, mem: 11631Mb, iter_time: 0.605s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.245e-03, size: 800, ETA: 1 day, 7:31:16
2021-08-03 20:22:08.782 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 200/700, mem: 11631Mb, iter_time: 0.535s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.245e-03, size: 640, ETA: 1 day, 7:29:16
2021-08-03 20:23:06.772 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 300/700, mem: 11631Mb, iter_time: 0.579s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.5, lr: 1.245e-03, size: 448, ETA: 1 day, 7:28:34
2021-08-03 20:24:02.756 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 400/700, mem: 11631Mb, iter_time: 0.559s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.245e-03, size: 544, ETA: 1 day, 7:27:16
2021-08-03 20:24:57.898 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 500/700, mem: 11631Mb, iter_time: 0.551s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.245e-03, size: 544, ETA: 1 day, 7:25:45
2021-08-03 20:25:55.679 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 600/700, mem: 11631Mb, iter_time: 0.577s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.245e-03, size: 512, ETA: 1 day, 7:24:58
2021-08-03 20:26:52.829 | INFO | yolox.core.trainer:after_iter:254 - epoch: 17/300, iter: 700/700, mem: 11631Mb, iter_time: 0.571s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.245e-03, size: 480, ETA: 1 day, 7:24:01
2021-08-03 20:26:52.830 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:26:55.506 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch18
2021-08-03 20:27:51.983 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 100/700, mem: 11631Mb, iter_time: 0.564s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.244e-03, size: 736, ETA: 1 day, 7:22:53
2021-08-03 20:28:45.599 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 200/700, mem: 11631Mb, iter_time: 0.535s, data_time: 0.001s, total_loss: 2.2, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.244e-03, size: 480, ETA: 1 day, 7:20:58
2021-08-03 20:29:29.354 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 20:29:33.015 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 300/700, mem: 11631Mb, iter_time: 0.473s, data_time: 0.002s, total_loss: 3.2, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.244e-03, size: 800, ETA: 1 day, 7:17:23
2021-08-03 20:30:33.723 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 400/700, mem: 11631Mb, iter_time: 0.606s, data_time: 0.001s, total_loss: 2.2, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.244e-03, size: 800, ETA: 1 day, 7:17:25
2021-08-03 20:31:31.331 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 500/700, mem: 11631Mb, iter_time: 0.575s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.244e-03, size: 448, ETA: 1 day, 7:16:37
2021-08-03 20:32:28.510 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 600/700, mem: 11631Mb, iter_time: 0.571s, data_time: 0.001s, total_loss: 2.1, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.244e-03, size: 800, ETA: 1 day, 7:15:42
2021-08-03 20:33:26.298 | INFO | yolox.core.trainer:after_iter:254 - epoch: 18/300, iter: 700/700, mem: 11631Mb, iter_time: 0.577s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.244e-03, size: 608, ETA: 1 day, 7:14:56
2021-08-03 20:33:26.301 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:33:40.242 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 20:33:40.359 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 20:33:40.383 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.02s)
2021-08-03 20:33:40.383 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 20:33:40.386 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 20:33:40.527 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.09 ms, Average NMS time: 1.12 ms, Average inference time: 15.21 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 20:33:40.528 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:33:42.517 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch19
2021-08-03 20:34:40.901 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 100/700, mem: 11631Mb, iter_time: 0.583s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.244e-03, size: 768, ETA: 1 day, 7:14:19
2021-08-03 20:35:39.874 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 200/700, mem: 11631Mb, iter_time: 0.589s, data_time: 0.002s, total_loss: 2.5, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.243e-03, size: 448, ETA: 1 day, 7:13:51
2021-08-03 20:36:37.352 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 300/700, mem: 11631Mb, iter_time: 0.574s, data_time: 0.002s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.243e-03, size: 640, ETA: 1 day, 7:13:00
2021-08-03 20:37:32.729 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 400/700, mem: 11631Mb, iter_time: 0.553s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.243e-03, size: 640, ETA: 1 day, 7:11:36
2021-08-03 20:38:30.080 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 500/700, mem: 11631Mb, iter_time: 0.573s, data_time: 0.002s, total_loss: 1.9, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.3, lr: 1.243e-03, size: 608, ETA: 1 day, 7:10:43
2021-08-03 20:39:28.336 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 600/700, mem: 11631Mb, iter_time: 0.582s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.243e-03, size: 448, ETA: 1 day, 7:10:04
2021-08-03 20:40:21.313 | INFO | yolox.core.trainer:after_iter:254 - epoch: 19/300, iter: 700/700, mem: 11631Mb, iter_time: 0.529s, data_time: 0.001s, total_loss: 2.9, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 1.0, cls_loss: 0.4, lr: 1.243e-03, size: 768, ETA: 1 day, 7:08:05
2021-08-03 20:40:21.315 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:40:23.514 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch20
2021-08-03 20:41:15.350 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 100/700, mem: 11631Mb, iter_time: 0.517s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.243e-03, size: 800, ETA: 1 day, 7:05:52
2021-08-03 20:42:16.343 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 200/700, mem: 11631Mb, iter_time: 0.609s, data_time: 0.001s, total_loss: 3.2, iou_loss: 1.6, l1_loss: 0.0, conf_loss: 1.1, cls_loss: 0.5, lr: 1.242e-03, size: 512, ETA: 1 day, 7:05:52
2021-08-03 20:43:07.856 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 300/700, mem: 11631Mb, iter_time: 0.514s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.242e-03, size: 832, ETA: 1 day, 7:03:35
2021-08-03 20:44:12.749 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 400/700, mem: 11631Mb, iter_time: 0.648s, data_time: 0.001s, total_loss: 2.0, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.3, lr: 1.242e-03, size: 704, ETA: 1 day, 7:04:31
2021-08-03 20:45:18.216 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 500/700, mem: 11631Mb, iter_time: 0.654s, data_time: 0.001s, total_loss: 1.9, iou_loss: 1.0, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.3, lr: 1.242e-03, size: 736, ETA: 1 day, 7:05:33
2021-08-03 20:46:24.121 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 600/700, mem: 11631Mb, iter_time: 0.658s, data_time: 0.001s, total_loss: 2.1, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.4, lr: 1.242e-03, size: 448, ETA: 1 day, 7:06:40
2021-08-03 20:47:18.977 | INFO | yolox.core.trainer:after_iter:254 - epoch: 20/300, iter: 700/700, mem: 11631Mb, iter_time: 0.548s, data_time: 0.001s, total_loss: 2.0, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.5, cls_loss: 0.4, lr: 1.242e-03, size: 640, ETA: 1 day, 7:05:10
2021-08-03 20:47:18.978 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:47:33.655 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 20:47:33.769 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 20:47:33.891 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.12s)
2021-08-03 20:47:33.891 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 20:47:33.894 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 20:47:34.032 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.05 ms, Average NMS time: 1.11 ms, Average inference time: 15.16 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 20:47:34.033 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:47:39.511 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch21
2021-08-03 20:48:36.684 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 100/700, mem: 11631Mb, iter_time: 0.571s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.241e-03, size: 448, ETA: 1 day, 7:04:12
2021-08-03 20:49:30.205 | INFO | apex.amp.handle:skip_step:140 - Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 16384.0
2021-08-03 20:49:31.990 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 200/700, mem: 11631Mb, iter_time: 0.552s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.241e-03, size: 512, ETA: 1 day, 7:02:49
2021-08-03 20:50:28.176 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 300/700, mem: 11631Mb, iter_time: 0.561s, data_time: 0.001s, total_loss: 2.1, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.241e-03, size: 544, ETA: 1 day, 7:01:38
2021-08-03 20:51:25.076 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 400/700, mem: 11631Mb, iter_time: 0.568s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.241e-03, size: 768, ETA: 1 day, 7:00:38
2021-08-03 20:52:25.803 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 500/700, mem: 11631Mb, iter_time: 0.606s, data_time: 0.001s, total_loss: 2.1, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.241e-03, size: 800, ETA: 1 day, 7:00:29
2021-08-03 20:53:21.542 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 600/700, mem: 11631Mb, iter_time: 0.556s, data_time: 0.012s, total_loss: 2.6, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.241e-03, size: 480, ETA: 1 day, 6:59:12
2021-08-03 20:54:20.394 | INFO | yolox.core.trainer:after_iter:254 - epoch: 21/300, iter: 700/700, mem: 11631Mb, iter_time: 0.588s, data_time: 0.001s, total_loss: 2.2, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.240e-03, size: 768, ETA: 1 day, 6:58:37
2021-08-03 20:54:20.396 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 20:54:22.507 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch22
2021-08-03 20:55:20.701 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 100/700, mem: 11631Mb, iter_time: 0.581s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.5, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.240e-03, size: 480, ETA: 1 day, 6:57:53
2021-08-03 20:56:17.168 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 200/700, mem: 11631Mb, iter_time: 0.564s, data_time: 0.001s, total_loss: 2.4, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.240e-03, size: 448, ETA: 1 day, 6:56:46
2021-08-03 20:57:19.873 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 300/700, mem: 11631Mb, iter_time: 0.626s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.240e-03, size: 608, ETA: 1 day, 6:57:01
2021-08-03 20:58:18.637 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 400/700, mem: 11631Mb, iter_time: 0.587s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.240e-03, size: 832, ETA: 1 day, 6:56:24
2021-08-03 20:59:19.180 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 500/700, mem: 11631Mb, iter_time: 0.605s, data_time: 0.001s, total_loss: 2.2, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.240e-03, size: 608, ETA: 1 day, 6:56:09
2021-08-03 21:00:14.573 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 600/700, mem: 11631Mb, iter_time: 0.553s, data_time: 0.001s, total_loss: 2.1, iou_loss: 1.1, l1_loss: 0.0, conf_loss: 0.6, cls_loss: 0.4, lr: 1.239e-03, size: 704, ETA: 1 day, 6:54:48
2021-08-03 21:01:15.645 | INFO | yolox.core.trainer:after_iter:254 - epoch: 22/300, iter: 700/700, mem: 11631Mb, iter_time: 0.610s, data_time: 0.001s, total_loss: 2.6, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.8, cls_loss: 0.4, lr: 1.239e-03, size: 736, ETA: 1 day, 6:54:39
2021-08-03 21:01:15.646 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 21:01:29.791 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:172 - Evaluate in main process...
2021-08-03 21:01:29.903 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - Loading and preparing results...
2021-08-03 21:01:29.926 | INFO | yolox.evaluators.coco_evaluator:evaluate_prediction:205 - DONE (t=0.02s)
2021-08-03 21:01:29.926 | INFO | pycocotools.coco:loadRes:363 - creating index...
2021-08-03 21:01:29.929 | INFO | pycocotools.coco:loadRes:363 - index created!
2021-08-03 21:01:30.172 | INFO | yolox.core.trainer:evaluate_and_save_model:313 -
Average forward time: 14.08 ms, Average NMS time: 1.34 ms, Average inference time: 15.42 ms
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001

2021-08-03 21:01:30.173 | INFO | yolox.core.trainer:save_ckpt:322 - Save weights to ./YOLOX_outputs/yolox_l
2021-08-03 21:01:35.504 | INFO | yolox.core.trainer:before_epoch:193 - ---> start train epoch23
2021-08-03 21:02:44.451 | INFO | yolox.core.trainer:after_iter:254 - epoch: 23/300, iter: 100/700, mem: 11631Mb, iter_time: 0.689s, data_time: 0.001s, total_loss: 2.7, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.239e-03, size: 736, ETA: 1 day, 6:56:08
2021-08-03 21:03:43.635 | INFO | yolox.core.trainer:after_iter:254 - epoch: 23/300, iter: 200/700, mem: 11631Mb, iter_time: 0.591s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.2, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.239e-03, size: 512, ETA: 1 day, 6:55:34
2021-08-03 21:04:37.597 | INFO | yolox.core.trainer:after_iter:254 - epoch: 23/300, iter: 300/700, mem: 11631Mb, iter_time: 0.539s, data_time: 0.001s, total_loss: 2.3, iou_loss: 1.3, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.239e-03, size: 544, ETA: 1 day, 6:53:54
2021-08-03 21:05:39.145 | INFO | yolox.core.trainer:after_iter:254 - epoch: 23/300, iter: 400/700, mem: 11631Mb, iter_time: 0.615s, data_time: 0.001s, total_loss: 2.5, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.7, cls_loss: 0.4, lr: 1.238e-03, size: 512, ETA: 1 day, 6:53:49
2021-08-03 21:06:32.963 | INFO | yolox.core.trainer:after_iter:254 - epoch: 23/300, iter: 500/700, mem: 11631Mb, iter_time: 0.537s, data_time: 0.001s, total_loss: 2.8, iou_loss: 1.4, l1_loss: 0.0, conf_loss: 0.9, cls_loss: 0.4, lr: 1.238e-03, size: 448, ETA: 1 day, 6:52:08
`

GOATmessi7 · 2021-08-04T15:13:23Z

I think there is something wrong with your labels or evaluations. Plz check your dataset and visualize some labels first. Make sure the annos follow the coco format of (x0, y0, w, h).

shineway14 · 2021-08-06T06:51:53Z

找到问题了吗，我这个loss都正常减少，就是ap始终为0，用训练模型推理也是啥框都没，太奇怪了

TUDelftHao · 2021-08-06T08:02:57Z

找到问题了吗，我这个loss都正常减少，就是ap始终为0，用训练模型推理也是啥框都没，太奇怪了

同，我的loss甚至一直在10左右徘徊，刚开始mAP还是有40多，第三轮之后就一直是0了

GOATmessi7 · 2021-08-06T08:18:13Z

如果是训练自己的数据，建议按以下几步检查一下问题：

pull最新的代码，我们已经在coco数据集上完整验证过，所以还有问题的话可以排除训练代码的问题；
检测你的数据集和标注，可视化gt、对其coco或者voc的格式等等；
一定记得加载预训练好的COCO权重；
观察training loss，同时修改eval_interval=1，每个epoch都评测一次。如果training loss还是在10以上徘徊，建议返回第2步再看看（COCO数据集上前3个epoch之后total loss能降到7~8左右，如果你的数据集类别低于80类，loss只会更低）
如果AP还有明显的先升后降这种形式，可以考虑调小lr，调小max_epoch等等（特别是如果你的数据集只有几百张图，那还是别训300 epoch了吧）

shineway14 · 2021-08-06T09:05:57Z

@ruinmessi ，那我这有点奇怪了，loss前2个epoch都从30多降到6了，预训练模型也加了，ap还是0，是不是要训练到100多个epoch才有结果

GOATmessi7 · 2021-08-06T14:34:25Z

@ruinmessi ，那我这有点奇怪了，loss前2个epoch都从30多降到6了，预训练模型也加了，ap还是0，是不是要训练到100多个epoch才有结果

初始的loss也就20左右，如果加载了预训练只会更低

zhangming8 · 2021-08-06T16:00:07Z

I think you should check the train/val annotation, then evaluate on train dataset, and confirm whether the AP is still 0 on train dataset

shineway14 · 2021-08-07T02:07:10Z

@ruinmessi ，那我这有点奇怪了，loss前2个epoch都从30多降到6了，预训练模型也加了，ap还是0，是不是要训练到100多个epoch才有结果

初始的loss也就20左右，如果加载了预训练只会更低

可能是目标太小了吧，训练coco数据集是没问题的

shineway14 · 2021-08-07T02:19:22Z

数据集是用的链接的yolo2coo 里面的转的，yolov5训练时没问题的，看了vision画的框也是对对的

hhngdcz · 2021-08-07T14:30:52Z

如果是voc数据集的话，把voc.py中54行左右的name = obj.find("name").text.lower().strip()改成name = obj.find("name").text.strip()试试

TUDelftHao · 2021-08-07T15:40:24Z

找到问题了吗，我这个loss都正常减少，就是ap始终为0，用训练模型推理也是啥框都没，太奇怪了

同，我的loss甚至一直在10左右徘徊，刚开始mAP还是有40多，第三轮之后就一直是0了

重拉一下最新代码吧，我在coco和自己的数据集上面测了，mAP都正常了，加载预训练模型会收敛更快

MangoloD · 2021-08-12T05:03:13Z

找到问题了吗，我这个loss都正常减少，就是ap始终为0，用训练模型推理也是啥框都没，太奇怪了

问题已解决

Hezhexi2002 · 2021-08-15T03:06:46Z

找到问题了吗，我这个loss都正常减少，就是ap始终为0，用训练模型推理也是啥框都没，太奇怪了

问题已解决

请问你是怎样解决的，我现在也遇到了相同问题

MangoloD · 2021-08-16T07:22:28Z

找到问题了吗，我这个loss都正常减少，就是ap始终为0，用训练模型推理也是啥框都没，太奇怪了

问题已解决

请问你是怎样解决的，我现在也遇到了相同问题

如果你是voc数据集的话，应该就是验证集的路径写错了

Joker316701882 assigned GOATmessi7 Aug 4, 2021

Joker316701882 mentioned this issue Aug 8, 2021

修改voc.py去掉标签转小写 #413

Merged

FateScript closed this as completed Oct 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

trainning AP is always 0.000 #337

trainning AP is always 0.000 #337

chaerlo commented Aug 3, 2021

Joker316701882 commented Aug 3, 2021

chaerlo commented Aug 4, 2021

GOATmessi7 commented Aug 4, 2021

chaerlo commented Aug 4, 2021

GOATmessi7 commented Aug 4, 2021

shineway14 commented Aug 6, 2021

TUDelftHao commented Aug 6, 2021

GOATmessi7 commented Aug 6, 2021

shineway14 commented Aug 6, 2021

GOATmessi7 commented Aug 6, 2021

zhangming8 commented Aug 6, 2021

shineway14 commented Aug 7, 2021

shineway14 commented Aug 7, 2021

hhngdcz commented Aug 7, 2021

TUDelftHao commented Aug 7, 2021

MangoloD commented Aug 12, 2021

Hezhexi2002 commented Aug 15, 2021

MangoloD commented Aug 16, 2021

trainning AP is always 0.000 #337

trainning AP is always 0.000 #337

Comments

chaerlo commented Aug 3, 2021

Joker316701882 commented Aug 3, 2021

chaerlo commented Aug 4, 2021

GOATmessi7 commented Aug 4, 2021

chaerlo commented Aug 4, 2021

GOATmessi7 commented Aug 4, 2021

shineway14 commented Aug 6, 2021

TUDelftHao commented Aug 6, 2021

GOATmessi7 commented Aug 6, 2021

shineway14 commented Aug 6, 2021

GOATmessi7 commented Aug 6, 2021

zhangming8 commented Aug 6, 2021

shineway14 commented Aug 7, 2021

shineway14 commented Aug 7, 2021

hhngdcz commented Aug 7, 2021

TUDelftHao commented Aug 7, 2021

MangoloD commented Aug 12, 2021

Hezhexi2002 commented Aug 15, 2021

MangoloD commented Aug 16, 2021