prototyping test_this function. Not yet for merge, faciliate design d…

[Benudek]

@stas00, submitting this code snippet to register Tests per Api ### as base for design discussion not meant to be merged already.

One design decision to make is how often to write the test-per-API entries to the json file.

1. load & save new information to file with every call of test_this. This would seem not very elegant and on some platforms, this would be an anti-pattern. But since this is ‘just’ a one-time all test run, provided performance does not suffer so much, this might be the simplest option wrt code: we just call this_tests and inside this_tests we open, write and close the json file
2. load & save information once per test file. This is the pattern I implemented testwise for this PR. It does require 2 pseudo test method to load and save at the start and end of the file, so it's not super elegant and compared to (1) is more IO efficient but clutters the code.
3. save all generated Tests per API entries once at the end of all tests: In the Makefile after the tests run (make test), we could call a save()method once (would e.g. allow calling save via a command line comment). From the remarks in the wiki, this is probably what you had in mind and it would be the most efficient I/O method and leave the test scripts simple. The issue I see is that I would not be sure, how to cache the information collected. As far as I can see from the docs a class variable as I used it just gets dropped after the test ran one test class, so I need to save then (see option 2).

Maybe there is some 'cool' cache-across-test-file-runs or file-write-design pattern to use here best?

Other points worthwhile checking:

• format dictionary: took json, key is the API and there could be several tests per API
• Name json file: need to agree on name (now: TestAPIRegister )
• Path: need to find a $PATH var to store under fastai folder to avoid hardcoding the path
• Open To do: extension of MANIFEST file
• new python file ‘register’ agree on file name,
• new python file ‘register’ sits under path tests/utils. Saw your comment to place this under gen_docs. To me it seemed placing under test_utils could also be good (and I did have some issues getting to the package gen_docs / register with imports from tests, which should be solvable though)

Thx for your help!