Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor loader_cfg. #214

Merged
merged 7 commits into from
Mar 1, 2021
Merged

Refactor loader_cfg. #214

merged 7 commits into from
Mar 1, 2021

Conversation

Yshuo-Li
Copy link
Collaborator

Refactor loader_cfg of main() in tools/test.py.

There is a bug in the master branch:
TypeError: type object got multiple values for keyword argument 'samples_per_gpu'.

That is because loader_cfg contains two 'samples_per_gpu':

  1. samples_per_gpu=1,
  2. **cfg.data.get('test_dataloader', {}),, while cfg.data['test_dataloader']=dict(samples_per_gpu=1),

The second one has been replaced by loader_cfg.update(cfg.data.get('test_dataloader', {})).

@codecov
Copy link

codecov bot commented Feb 27, 2021

Codecov Report

Merging #214 (857a3b2) into master (4f90c0c) will not change coverage.
The diff coverage is 0.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #214   +/-   ##
=======================================
  Coverage   82.27%   82.27%           
=======================================
  Files         145      145           
  Lines        6732     6732           
  Branches     1004     1004           
=======================================
  Hits         5539     5539           
  Misses       1086     1086           
  Partials      107      107           
Flag Coverage Δ
unittests 82.27% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmedit/apis/train.py 20.98% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f90c0c...0011f39. Read the comment docs.

@innerlee
Copy link
Contributor

Thanks! I will check shortly

@innerlee
Copy link
Contributor

Yeah thanks for the findings. It indeed causes error. I got another fancy way of merging dicts, do you want to check it out?

loader_cfg = {
    **dict((k, cfg.data[k]) for k in ['workers_per_gpu'] if k in cfg.data),
    **dict(samples_per_gpu=1,
           drop_last=False,
           shuffle=False,
           dist=distributed),
    **cfg.data.get('train_dataloader', {})}

@innerlee
Copy link
Contributor

Thanks, there are other places need to be modified: train.py L104, L170 and L278.

@innerlee
Copy link
Contributor

pls install pre-commit and let it pass the lint :)

drop_last=False,
dist=True,
),
**dict({} if torch.__version__ != 'parrots' else dict(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No there is no need to add this dict

@innerlee
Copy link
Contributor

Tests are failing

shuffle=False,
drop_last=False,
val_loader_cfg = {
**dict(loader_cfg, shuffle=False, drop_last=False),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

loader_cfg is a dict. To better demostrate the priority, put it outside

    **loader_cfg,
    **dict(shuffle=False, drop_last=False),

@innerlee
Copy link
Contributor

innerlee commented Mar 1, 2021

Thanks a lot!

@innerlee innerlee merged commit 41c1a49 into open-mmlab:master Mar 1, 2021
@Yshuo-Li Yshuo-Li deleted the ysli branch April 8, 2021 08:18
Yshuo-Li added a commit to Yshuo-Li/mmediting that referenced this pull request Jul 15, 2022
* Refactor loader_cfg.

* Refactor loader_cfg in main()

* Refactor 'loader_cfg', 'val_loader_cfg', and 'loader_cfg'.

* Refactor 'loader_cfg's and 'val_loader_cfg'.

* Refactor loader_cfg

* Refactor loader_cfg and format code.

* Refactor val_loader_cfg.

Co-authored-by: root <root@cn0014004493l.domain.sensetime.com>
Co-authored-by: 李尹硕 <SENSETIME\liyinshuo@cn0014004493l.domain.sensetime.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants