Identifying tests in need of a refactor to unify test landscape

[githubnemo]

We recently discussed the importance of a unified test landscape so that contributors can easily orient themselves on the existing tests when writing new ones and thus reducing the review burden on the maintainer side. For this we think it is vital to firstly have a list of tests that need to be refactored. Indicators for this could be the use of unittest instead of pytest, lots of private model class definitions instead of global ones that make sense for most of the tests, missing use of hub_online_once and complicated, test-local skip rules that can be refactored into global checks.

Since this also involves human judgement we thought that possibly @DaoyuanLi2816 wants to help in creating said list and possibly, at a later point, the refactoring itself once the list is vetted and ranked. WDYT? :)