New xformers test runner

[DN6]

What does this PR do?

xformers breaks cuDNN drivers with torch 2.1. This PR

1. Creates a separate runner for xformer related tests with a pinned torch version
2. Updates a few tests that had entangled xformers tests

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[patrickvonplaten]

Suggested change

	python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile -s -v -k "xformers" --make-reports=tests_torch_xformers_cuda tests/
	python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile -s -v -k "xformers" --make-reports=tests_torch_xformers_cuda tests/

ok for now, but I'm not a huge fan of using the -k "xformers" option in our runners as this is quite brittle. I'd prefer to start marking the tests like we do in Transformers: https://github.com/huggingface/transformers/blob/1e3c9ddacc7fc4142253bc9ddcba85c4d5b977e7/tests/test_modeling_common.py#L2746 so that we can then run:

pytest -m xformers tests/....

[patrickvonplaten]

Actually why do we have pytorch-lightning in here? We never use this library IMO. Can we remove it from all places in the dockers?

[DN6]

Sounds good. I'll make the change.