Error building DATASET and print heatmaps annotations in PoseC3D

[christina05011]

Branch

main branch (1.x version, such as v1.0.0, or dev-1.x branch)

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the documentation but cannot get the expected help.
• The bug has not been fixed in the latest version.

Environment

I have this versions:

• mmengine 0.7.4
• mmcv 2.0.0
• mmdet 3.0.0
• mmpose 1.0.0
• mmaction2 1.0.0

Describe the bug

Hi, I have a problem when I am building datasets in PoseC3D.

First, I did the steps according to the tutorial , but I want to print only heatmaps annotations.

Reproduces the problem - code sample

I run:

dataloader_cfg = copy.deepcopy(cfg.train_dataloader)
dataset_cfg = dataloader_cfg.pop('dataset')
dataset = DATASETS.build(dataset_cfg)
print(next(iter(dataset)))

Reproduces the problem - command or script

No response

Reproduces the problem - error message

When I run those commands with config tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py, I have results resized and not empty.

But when I run with config slowonly_r50_8xb16-u48-240e_ntu60-xsub-keypoint.py, I only have zeros. Also, when I train, I only have zeros too.

Additional information

• How can I print only heatmaps annotations? Did I do it correctly?
• Why does it only print zeros (none data)?

Thank you for the answers.

[rodp63]

Hello @cir7, same problem here. Do you have any update on this? Thank you!

[christina05011]

Hi @cir7 @hukkai . Do you have any update on this? Thanks!

[cir7]

Could you share the config file you are using for training? And are you using a custom dataset?

[christina05011]

Hi @cir7, thank you for the answer.

I use this config file, but I changed the numbers of workers to 2 and the batch size to 1 in dataloaders. I made this configurations for one GPU in Colab like the tutorial.

And, I use the same .pkl file ntu60_2d.pkl.

[cir7]

Ok, I found that the heatmap you got is reasonable. PoseC3D uses a special heatmap annotation, which maps the keypoints to an all-zero image according to the spatial coordinates, so most values of the input are zeros, except for the keypoints, this visualization shows the sparsity. You could refer to the paper for mode details. While the general 3D models take rgb images as input, which has a mean value of 128.

[christina05011]

Thanks for the reply @cir7.

I check the output and I have the keypoints (non-zero data). But, if I only want the heatmap annotations, is it correct to print with DATASETS.build() (like I run) or with Runner.build_dataloader(dataloader=train_dataloader_cfg)? What is the difference?

Another question, when I print the shape of the dataset with next(iter(dataset))['inputs'].shape returns torch.Size([1, 17, 48, 56, 56]), but in the paper says that the input size is 17 x 32 x 56 x 56 (C x T x H x W). However in the experiments it says that shape 48 × 56 × 56 was used as inputs. So, what is the correct shape please?

[cir7]

1. dataloader stack batch of samples, which could add a new dim, while dataset returns a single sample
2. You could easily ignore the first dim of shape [1, 17, 48, 56, 56], different T dim (32 vs 48) means different sample strategies, the paper takes both for experiments.

[christina05011]

Thanks @cir7 for the answers. You solved my doubts!