Run standalone tests in batches

[carmocca]

What does this PR do?

Launch the standalone tests in batches.

Standalone tests are launched in separate processes because they have interactions with others. pytest runs all tests together in one process by default so we need to launch them 1-by-1.

But we can still batch the tests ourselves but launch them in separate processes. This is what this PR does.

batch size	Standalone runtime
master (1)	22m 30s
4	8m 15s
6	7m 2s
8	failed allclose on a test

We might be able to push the batch size higher. But there seem to be interactions between the deepspeed tests. As a drawback, this opens up a potential source of flakiness depending on how the tests get batched.

Discussed with @tchaton

There's a failing test in standalone tests:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/run.py", line 728, in <module>
    main()
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/run.py", line 724, in main
    run(args)
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/run.py", line 715, in run
    elastic_launch(
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.9/dist-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

But it's happening in master too.

Does your PR introduce any breaking changes? If yes, please list them.

None

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• [n/a] Did you make sure to update the documentation with your changes? (if necessary)
• [n/a] Did you write any new necessary tests? (not for typos and docs)
• [n/a] Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

cc @carmocca @akihironitta @Borda

[Borda]

just curious, if we set it all in one batch, then what is the difference to run standards tests?

[carmocca]

See the issue description

[tchaton]

Love it !

[Borda]

We might be able to push the batch size higher. But there seem to be interactions between tests. As a drawback, this opens up a potential source of flakiness depending on how the tests get batched.

yes, I see, so shall we bit more experiments what is the right size for min interactions?

[carmocca]

shall we bit more experiments what is the right size for min interactions?

I've done it already. See table at the to. "6" is the highest we can run at the moment without interactions. I would like to experiment further in a follow-up PR.

We can always tune it higher or lower in the future. And setting it to "1" should be equal to reverting this PR

[awaelchli]

If each test is still standalone, launches its own processes and lets them terminate, then there should be absolutely no interactions between tests (assuming only writing to tmpdir). Is this the case? This property has to remain, otherwise we will have problems.