Enable custom default config for dataset tests

[pmeier]

This came up in #3423 (comment).

TL;DR: With the current architecture, it is not possible to test a dataset that has a positional parameter, which should change depending on the config.

To overcome this, this PR makes the DEFAULT_CONFIG a first class member. It still defaults to the keyword arguments extracted from the dataset constructor, but can now be safely overwritten if needed.

[pmeier]

I've included some other documentation upgrades. If you want these in separate PRs, let me know.

[codecov]

Codecov Report

Merging #3578 (c00c4d3) into master (bb2805a) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head c00c4d3 differs from pull request most recent head 085745e. Consider uploading reports for the commit 085745e to get more accurate results

@@            Coverage Diff             @@
##           master    #3578      +/-   ##
==========================================
- Coverage   79.07%   79.07%   -0.01%     
==========================================
  Files         105      105              
  Lines        9787     9786       -1     
  Branches     1572     1572              
==========================================
- Hits         7739     7738       -1     
  Misses       1567     1567              
  Partials      481      481              

Impacted Files	Coverage Δ
torchvision/datasets/voc.py	94.44% <100.00%> (-0.07%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb2805a...085745e. Read the comment docs.

[NicolasHug]

Thanks @pmeier , I made a few comments. I find the logic a bit tricky to follow overall, as the OO implementation is very stateful. It might not be worth considering a refactoring, but I think at least DEFAULT_CONFIG could be renamed in something more descriptive.

[pmeier]

There seems to be some confusion about how the configs are handled. I hope I can make it more clear with my comments.

[NicolasHug]

Thanks for these precisions @pmeier , I made a second pass

[pmeier]

@NicolasHug

Following our internal discussion, I refactored a little more:

• We now can have a DEFAULT_CONFIG as well as ADDITIONAL_CONFIGS. Neither of them need to be complete or will be overwritten.
• test_all_configs will now also test the DEFAULT_CONFIG. By default it also tries to remove duplicate configs. If this fails, for example if a value is not hashable, it falls back to testing all configs.

Let me know what you think.

[NicolasHug]

Thanks @pmeier , some last comments but LGTM!

[pmeier]

@NicolasHug I've addressed all your comments. Thanks a lot for the input! IMO it made the PR a lot better 🙂 Merge if you and the CI are happy.

[NicolasHug]

Thanks @pmeier !
I think the CI failures are unrelated