MultiDatasetEvaluator [Bug]

[Kyriio]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmpose).

Environment

OrderedDict([('sys.platform', 'win32'), ('Python', '3.8.19 (default, Mar 20 2024, 19:55:45) [MSC v.1916 64 bit (AMD64)]'), ('CUDA available', True), ('MUSA available', False), ('numpy_random_seed', 2147483648), ('GPU 0', 'NVIDIA RTX 3500 Ada Generation Laptop GPU'), ('CUDA_HOME', 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4'), ('NVCC', 'Cuda compilation tools, release 12.4, V12.4.99'), ('GCC', 'n/a'), ('PyTorch', '2.2.2'), ('PyTorch compiling details', 'PyTorch built with:\n - C++ Version: 201703\n - MSVC 192930151\n - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications\n - Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)\n - OpenMP 2019\n - LAPACK is enabled (usually provided by MKL)\n - CPU capability usage: AVX2\n - CUDA Runtime 12.1\n - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n - CuDNN 8.8.1 (built against CUDA 12.0)\n - Magma 2.5.4\n - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.8.1, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /Zc:__cplusplus /bigobj /FS /utf-8 -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n'), ('TorchVision', '0.17.2'), ('OpenCV', '4.9.0'), ('MMEngine', '0.10.3'), ('MMPose', '1.3.1+5a3be94')])

mmcv 2.1.0
mmdet 3.3.0
mmengine 0.10.3
mmpose 1.3.1

Reproduces the problem - code sample

base = ['../../configs/base/default_runtime.py']

runtime

train_cfg = dict(max_epochs=210, val_interval=1)

optimizer

optim_wrapper = dict(optimizer=dict(
type='Adam',
lr=5e-4,
))

learning policy

param_scheduler = [
dict(
type='LinearLR', begin=0, end=500, start_factor=0.001,
by_epoch=False), # warm-up
dict(
type='MultiStepLR',
begin=0,
end=210,
milestones=[100, 150],
gamma=0.1,
by_epoch=True)
]

automatically scaling LR based on the actual training batch size

auto_scale_lr = dict(base_batch_size=512)

hooks

default_hooks = dict(checkpoint=dict(save_best='coco/AP', rule='greater'))

codec settings

codec = dict(
type='MSRAHeatmap', input_size=(384, 384), heatmap_size=(96, 96), sigma=3)

model settings

model = dict(
type='TopdownPoseEstimator',
data_preprocessor=dict(
type='PoseDataPreprocessor',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
bgr_to_rgb=True),
backbone=dict(
type='HourglassNet',
num_stacks=1,
init_cfg=dict(
type='Pretrained',
prefix="backbone.",
checkpoint='models/hg/model1.pth'
),
),
head=dict(
type='CPMHead',
in_channels=256,
out_channels=17,
num_stages=1,
deconv_out_channels=None,
loss=dict(type='KeypointMSELoss', use_target_weight=True),
decoder=codec),
test_cfg=dict(
flip_test=True,
flip_mode='heatmap',
shift_heatmap=True,
))

base dataset settings

dataset_type = 'HumanArtDataset'
data_mode = 'topdown'
data_root = 'data/humanart/'

visualizer = dict(vis_backends=[
dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend'),
])

pipelines

train_pipeline = [
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(type='RandomHalfBody'),
dict(type='RandomBBoxTransform'),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='GenerateTarget', encoder=codec),
dict(type='PackPoseInputs')
]
val_pipeline = [
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(type='TopdownAffine', input_size=codec['input_size']),
dict(type='PackPoseInputs')
]

####### TRAIN PART###########

train_water = dict(
type=dataset_type,
data_root=data_root,
ann_file='HumanArt/annotations/training_humanart_watercolor.json',
pipeline=[],
)
train_oil = dict(
type=dataset_type,
data_root=data_root,
ann_file='HumanArt/annotations/training_humanart_oil_painting.json',
pipeline=[],
)
train_ink = dict(
type=dataset_type,
data_root=data_root,
ann_file='HumanArt/annotations/training_humanart_ink_painting.json',
pipeline=[],
)

Data loaders

train_dataloader = dict(
batch_size=32,
num_workers=2,
persistent_workers=True,sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
dataset=dict(
type='CombinedDataset',
metainfo=dict(from_file='configs/base/datasets/coco.py'),
datasets=[train_water, train_oil, train_ink],
pipeline=train_pipeline,
test_mode=False,
),
)

######## VAL PART ###########
water_colo_val = dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file='data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
test_mode=True,
pipeline=[]
)

oil_val = dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file='data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
test_mode=True,
pipeline=[]
)

ink_val = dict(
type=dataset_type,
data_root=data_root,
data_mode=data_mode,
ann_file='HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file='data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
test_mode=True,
pipeline=[]
)

val_dataset = dict(
type='CombinedDataset',
metainfo=dict(from_file='configs/base/datasets/coco.py'),
datasets=[water_colo_val, oil_val, ink_val],
pipeline=val_pipeline,
test_mode=True,
)

val_dataloader = dict(
batch_size=32,
num_workers=2,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False, round_up=False),
dataset=val_dataset,
)

test_dataloader = val_dataloader

evaluators

val_evaluator = dict(
type='MultiDatasetEvaluator',
metrics=[
dict(
type='CocoMetric',
ann_file=data_root + 'HumanArt/annotations/validation_humanart_watercolor.json',
),
dict(
type='CocoMetric',
ann_file=data_root + 'HumanArt/annotations/validation_humanart_oil_painting.json',

    ),
    dict(
        type='CocoMetric',
        ann_file=data_root + 'HumanArt/annotations/validation_humanart_ink_painting.json',
    ),
],
datasets=val_dataset['datasets'],

)

test_evaluator = val_evaluator

Reproduces the problem - command or script

python tools/train.py models/new_/model1.py --work-dir models/new_/train/ --resume

Reproduces the problem - error message

05/28 00:55:25 - mmengine - INFO -

System environment:
sys.platform: win32
Python: 3.8.19 (default, Mar 20 2024, 19:55:45) [MSC v.1916 64 bit (AMD64)]
CUDA available: True
MUSA available: False
numpy_random_seed: 1986547594
GPU 0: NVIDIA RTX 3500 Ada Generation Laptop GPU
CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4
NVCC: Cuda compilation tools, release 12.4, V12.4.99
GCC: n/a
PyTorch: 2.2.2
PyTorch compiling details: PyTorch built with:

• C++ Version: 201703

• MSVC 192930151

• Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications

• Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)

• OpenMP 2019

• LAPACK is enabled (usually provided by MKL)

• CPU capability usage: AVX2

• CUDA Runtime 12.1

• NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90

• CuDNN 8.8.1 (built against CUDA 12.0)

• Magma 2.5.4

• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.8.1, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /Zc:__cplusplus /bigobj /FS /utf-8 -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 /wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 /wd4804 /wd4273, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

  TorchVision: 0.17.2
  OpenCV: 4.9.0
  MMEngine: 0.10.3

Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: 1986547594
Distributed launcher: none
Distributed training: False
GPU number: 1

05/28 00:55:29 - mmengine - INFO - Config:
auto_scale_lr = dict(base_batch_size=512)
backend_args = dict(backend='local')
codec = dict(
heatmap_size=(
96,
96,
),
input_size=(
384,
384,
),
sigma=3,
type='MSRAHeatmap')
custom_hooks = [
dict(type='SyncBuffersHook'),
]
data_mode = 'topdown'
data_root = 'data/humanart/'
dataset_type = 'HumanArtDataset'
default_hooks = dict(
badcase=dict(
badcase_thr=5,
enable=False,
metric_type='loss',
out_dir='badcase',
type='BadCaseAnalysisHook'),
checkpoint=dict(
interval=10,
rule='greater',
save_best='coco/AP',
type='CheckpointHook'),
logger=dict(interval=50, type='LoggerHook'),
param_scheduler=dict(type='ParamSchedulerHook'),
sampler_seed=dict(type='DistSamplerSeedHook'),
timer=dict(type='IterTimerHook'),
visualization=dict(enable=False, type='PoseVisualizationHook'))
default_scope = 'mmpose'
env_cfg = dict(
cudnn_benchmark=False,
dist_cfg=dict(backend='nccl'),
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
ink_val = dict(
ann_file='HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset')
launcher = 'none'
load_from = None
log_level = 'INFO'
log_processor = dict(
by_epoch=True, num_digits=6, type='LogProcessor', window_size=50)
model = dict(
backbone=dict(
init_cfg=dict(
checkpoint='models/hg/model1.pth',
prefix='backbone.',
type='Pretrained'),
num_stacks=1,
type='HourglassNet'),
data_preprocessor=dict(
bgr_to_rgb=True,
mean=[
123.675,
116.28,
103.53,
],
std=[
58.395,
57.12,
57.375,
],
type='PoseDataPreprocessor'),
head=dict(
decoder=dict(
heatmap_size=(
96,
96,
),
input_size=(
384,
384,
),
sigma=3,
type='MSRAHeatmap'),
deconv_out_channels=None,
in_channels=256,
loss=dict(type='KeypointMSELoss', use_target_weight=True),
num_stages=1,
out_channels=17,
type='CPMHead'),
test_cfg=dict(flip_mode='heatmap', flip_test=True, shift_heatmap=True),
type='TopdownPoseEstimator')
oil_val = dict(
ann_file='HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset')
optim_wrapper = dict(optimizer=dict(lr=0.0005, type='Adam'))
param_scheduler = [
dict(
begin=0, by_epoch=False, end=500, start_factor=0.001, type='LinearLR'),
dict(
begin=0,
by_epoch=True,
end=210,
gamma=0.1,
milestones=[
100,
150,
],
type='MultiStepLR'),
]
resume = True
test_cfg = dict()
test_dataloader = dict(
batch_size=32,
dataset=dict(
datasets=[
dict(
ann_file=
'HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
],
metainfo=dict(from_file='configs/base/datasets/coco.py'),
pipeline=[
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(input_size=(
384,
384,
), type='TopdownAffine'),
dict(type='PackPoseInputs'),
],
test_mode=True,
type='CombinedDataset'),
drop_last=False,
num_workers=2,
persistent_workers=True,
sampler=dict(round_up=False, shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
datasets=[
dict(
ann_file='HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
],
metrics=[
dict(
ann_file=
'data/humanart/HumanArt/annotations/validation_humanart_watercolor.json',
prefix='water',
type='CocoMetric'),
dict(
ann_file=
'data/humanart/HumanArt/annotations/validation_humanart_oil_painting.json',
prefix='oil',
type='CocoMetric'),
dict(
ann_file=
'data/humanart/HumanArt/annotations/validation_humanart_ink_painting.json',
prefix='ink',
type='CocoMetric'),
],
type='MultiDatasetEvaluator')
train_cfg = dict(by_epoch=True, max_epochs=210, val_interval=1)
train_dataloader = dict(
batch_size=32,
dataset=dict(
datasets=[
dict(
ann_file=
'HumanArt/annotations/training_humanart_watercolor.json',
data_root='data/humanart/',
pipeline=[],
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/training_humanart_oil_painting.json',
data_root='data/humanart/',
pipeline=[],
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/training_humanart_ink_painting.json',
data_root='data/humanart/',
pipeline=[],
type='HumanArtDataset'),
],
metainfo=dict(from_file='configs/base/datasets/coco.py'),
pipeline=[
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(type='RandomHalfBody'),
dict(type='RandomBBoxTransform'),
dict(input_size=(
384,
384,
), type='TopdownAffine'),
dict(
encoder=dict(
heatmap_size=(
96,
96,
),
input_size=(
384,
384,
),
sigma=3,
type='MSRAHeatmap'),
type='GenerateTarget'),
dict(type='PackPoseInputs'),
],
test_mode=False,
type='CombinedDataset'),
num_workers=2,
persistent_workers=True,
sampler=dict(round_up=False, shuffle=False, type='DefaultSampler'))
train_ink = dict(
ann_file='HumanArt/annotations/training_humanart_ink_painting.json',
data_root='data/humanart/',
pipeline=[],
type='HumanArtDataset')
train_oil = dict(
ann_file='HumanArt/annotations/training_humanart_oil_painting.json',
data_root='data/humanart/',
pipeline=[],
type='HumanArtDataset')
train_pipeline = [
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(type='RandomHalfBody'),
dict(type='RandomBBoxTransform'),
dict(input_size=(
384,
384,
), type='TopdownAffine'),
dict(
encoder=dict(
heatmap_size=(
96,
96,
),
input_size=(
384,
384,
),
sigma=3,
type='MSRAHeatmap'),
type='GenerateTarget'),
dict(type='PackPoseInputs'),
]
train_water = dict(
ann_file='HumanArt/annotations/training_humanart_watercolor.json',
data_root='data/humanart/',
pipeline=[],
type='HumanArtDataset')
val_cfg = dict()
val_dataloader = dict(
batch_size=32,
dataset=dict(
datasets=[
dict(
ann_file=
'HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
],
metainfo=dict(from_file='configs/base/datasets/coco.py'),
pipeline=[
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(input_size=(
384,
384,
), type='TopdownAffine'),
dict(type='PackPoseInputs'),
],
test_mode=True,
type='CombinedDataset'),
drop_last=False,
num_workers=2,
persistent_workers=True,
sampler=dict(round_up=False, shuffle=False, type='DefaultSampler'))
val_dataset = dict(
datasets=[
dict(
ann_file='HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
],
metainfo=dict(from_file='configs/base/datasets/coco.py'),
pipeline=[
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(input_size=(
384,
384,
), type='TopdownAffine'),
dict(type='PackPoseInputs'),
],
test_mode=True,
type='CombinedDataset')
val_evaluator = dict(
datasets=[
dict(
ann_file='HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_oil_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_oil_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
dict(
ann_file=
'HumanArt/annotations/validation_humanart_ink_painting.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_ink_painting_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset'),
],
metrics=[
dict(
ann_file=
'data/humanart/HumanArt/annotations/validation_humanart_watercolor.json',
prefix='water',
type='CocoMetric'),
dict(
ann_file=
'data/humanart/HumanArt/annotations/validation_humanart_oil_painting.json',
prefix='oil',
type='CocoMetric'),
dict(
ann_file=
'data/humanart/HumanArt/annotations/validation_humanart_ink_painting.json',
prefix='ink',
type='CocoMetric'),
],
type='MultiDatasetEvaluator')
val_pipeline = [
dict(type='LoadImage'),
dict(type='GetBBoxCenterScale'),
dict(input_size=(
384,
384,
), type='TopdownAffine'),
dict(type='PackPoseInputs'),
]
vis_backends = [
dict(type='LocalVisBackend'),
]
visualizer = dict(
name='visualizer',
type='PoseLocalVisualizer',
vis_backends=[
dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend'),
])
water_colo_val = dict(
ann_file='HumanArt/annotations/validation_humanart_watercolor.json',
bbox_file=
'data/humanart/HumanArt/person_detection_results/HumanArt_watercolor_validation_detections_AP_H_56_person.json',
data_mode='topdown',
data_root='data/humanart/',
pipeline=[],
test_mode=True,
type='HumanArtDataset')
work_dir = 'models/new_/train/'

05/28 00:55:40 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
05/28 00:55:40 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) RuntimeInfoHook
(BELOW_NORMAL) LoggerHook

before_train:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook

before_train_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook

before_train_iter:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook

after_train_iter:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_train_epoch:
(NORMAL ) IterTimerHook
(NORMAL ) SyncBuffersHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

before_val:
(VERY_HIGH ) RuntimeInfoHook

before_val_epoch:
(NORMAL ) IterTimerHook
(NORMAL ) SyncBuffersHook

before_val_iter:
(NORMAL ) IterTimerHook

after_val_iter:
(NORMAL ) IterTimerHook
(NORMAL ) PoseVisualizationHook
(BELOW_NORMAL) LoggerHook

after_val_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_val:
(VERY_HIGH ) RuntimeInfoHook

after_train:
(VERY_HIGH ) RuntimeInfoHook
(VERY_LOW ) CheckpointHook

before_test:
(VERY_HIGH ) RuntimeInfoHook

before_test_epoch:
(NORMAL ) IterTimerHook

before_test_iter:
(NORMAL ) IterTimerHook

after_test_iter:
(NORMAL ) IterTimerHook
(NORMAL ) PoseVisualizationHook
(NORMAL ) BadCaseAnalysisHook
(BELOW_NORMAL) LoggerHook

after_test_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) BadCaseAnalysisHook
(BELOW_NORMAL) LoggerHook

after_test:
(VERY_HIGH ) RuntimeInfoHook

after_run:
(BELOW_NORMAL) LoggerHook

loading annotations into memory...
Done (t=0.07s)
creating index...
index created!
loading annotations into memory...
Done (t=0.11s)
creating index...
index created!
loading annotations into memory...
Done (t=0.09s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
05/28 00:55:51 - mmengine - INFO - load backbone. in model from: models/hg/model1.pth
Loads checkpoint by local backend from path: models/hg/model1.pth
Did not find last_checkpoint to be resumed.
05/28 00:55:52 - mmengine - INFO - Auto resumed from the latest checkpoint None.
05/28 00:55:52 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
05/28 00:55:52 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
05/28 00:55:52 - mmengine - INFO - Checkpoints will be saved to C:\Users\Me\mmpose\models\new_\train.
05/28 00:57:54 - mmengine - INFO - Epoch(train) [1][ 50/487] lr: 4.954910e-05 eta: 2 days, 21:23:05 time: 2.443607 data_time: 0.182674 memory: 10300 loss: 0.002342 loss_kpt: 0.002342 acc_pose: 0.280381
05/28 00:59:58 - mmengine - INFO - Epoch(train) [1][100/487] lr: 9.959920e-05 eta: 2 days, 22:03:22 time: 2.493321 data_time: 0.006862 memory: 10300 loss: 0.001887 loss_kpt: 0.001887 acc_pose: 0.734947
05/28 01:02:09 - mmengine - INFO - Epoch(train) [1][150/487] lr: 1.496493e-04 eta: 2 days, 23:23:07 time: 2.612638 data_time: 0.018283 memory: 10300 loss: 0.001341 loss_kpt: 0.001341 acc_pose: 0.665533
05/28 01:04:30 - mmengine - INFO - Epoch(train) [1][200/487] lr: 1.996994e-04 eta: 3 days, 1:25:52 time: 2.810086 data_time: 0.195886 memory: 10300 loss: 0.001245 loss_kpt: 0.001245 acc_pose: 0.679139
05/28 01:06:52 - mmengine - INFO - Epoch(train) [1][250/487] lr: 2.497495e-04 eta: 3 days, 2:50:52 time: 2.846212 data_time: 0.231822 memory: 10300 loss: 0.001181 loss_kpt: 0.001181 acc_pose: 0.809415
05/28 01:09:19 - mmengine - INFO - Epoch(train) [1][300/487] lr: 2.997996e-04 eta: 3 days, 4:12:30 time: 2.937147 data_time: 0.152995 memory: 10300 loss: 0.001093 loss_kpt: 0.001093 acc_pose: 0.761654
05/28 01:11:45 - mmengine - INFO - Epoch(train) [1][350/487] lr: 3.498497e-04 eta: 3 days, 5:06:45 time: 2.923319 data_time: 0.122444 memory: 10300 loss: 0.001260 loss_kpt: 0.001260 acc_pose: 0.441407
05/28 01:14:07 - mmengine - INFO - Epoch(train) [1][400/487] lr: 3.998998e-04 eta: 3 days, 5:29:32 time: 2.841782 data_time: 0.038016 memory: 10300 loss: 0.001600 loss_kpt: 0.001600 acc_pose: 0.578416
05/28 01:16:30 - mmengine - INFO - Epoch(train) [1][450/487] lr: 4.499499e-04 eta: 3 days, 5:50:05 time: 2.859576 data_time: 0.034563 memory: 10300 loss: 0.001354 loss_kpt: 0.001354 acc_pose: 0.672502
05/28 01:18:06 - mmengine - INFO - Exp name: model1_20240528_005523
05/28 01:19:46 - mmengine - INFO - Epoch(val) [1][ 50/260] eta: 0:06:58 time: 1.993454 data_time: 1.335836 memory: 10300
05/28 01:21:43 - mmengine - INFO - Epoch(val) [1][100/260] eta: 0:05:47 time: 2.351338 data_time: 1.687622 memory: 2921
05/28 01:22:41 - mmengine - INFO - Epoch(val) [1][150/260] eta: 0:03:21 time: 1.156793 data_time: 0.563125 memory: 2921
05/28 01:24:48 - mmengine - INFO - Epoch(val) [1][200/260] eta: 0:02:00 time: 2.538494 data_time: 1.845335 memory: 2921
05/28 01:25:51 - mmengine - INFO - Epoch(val) [1][250/260] eta: 0:00:18 time: 1.263815 data_time: 0.646265 memory: 2921
05/28 01:25:57 - mmengine - WARNING - CocoMetric got empty self.results. Please ensure that the processed results are properly added into self.results in process method.

Traceback (most recent call last):
File "tools/train.py", line 162, in
main()
File "tools/train.py", line 158, in main
runner.train()
File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\runner\runner.py", line 1777, in train
model = self.train_loop.run() # type: ignore
File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\runner\loops.py", line 102, in run
self.runner.val_loop.run()
File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\runner\loops.py", line 374, in run
metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\evaluator\evaluator.py", line 79, in evaluate
_results = metric.evaluate(size)
File "C:\Users\Me\miniconda3\envs\openmmlab\lib\site-packages\mmengine\evaluator\metric.py", line 133, in evaluate
_metrics = self.compute_metrics(results) # type: ignore
File "c:\users\Me\mmpose\mmpose\evaluation\metrics\coco_metric.py", line 381, in compute_metrics
preds, gts = zip(*results)
ValueError: not enough values to unpack (expected 2, got 0)

Additional information

Hi,

First of all, thanks for everything you're doing ! It's amazing !

I'm currently using the Hourglass model, I didn't had problem until I try to evaluate across multiples datasets ( In fact, this is not multiple datasets but just some part of HumanArt Dataset) this is made like Coco (points and so on)

I followed the documentation : https://mmpose.readthedocs.io/en/latest/user_guides/train_and_test.html?highlight=Evaluator#evaluating-across-multiple-datasets

And I saw that someone noticed it before me here : #2674

And on the forum too : #3024

I would like to know if you could help me to know what's wrong ? It seems like he doesn't have any value or maybe I'm doing something wrong ?

Thanks a lot !