Add gpu memory wait before test_async_tp

[angelayi]

Attempting to fix test failure

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[ProExpertProg]

Seems like this works, can you add it to E2E tests as well? Or maybe better add a test util file that just calls this function, so that we can insert it into the test pipeline and not pollute the tests?

[ProExpertProg]

I like the envvar approach! I think you should take into account the CUDA_VISIBLE_DEVICES variable, as I think that these tests should only be running on 2xH200 at a time, and the CI failure implies that the fixture is waiting on all 8 GPUs on the node.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @angelayi.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ProExpertProg]

Okay this looks good now! We finally fixed the failure with e2e model tests. There's now a DBO failure which I assume snuck in from Lucas's refactor PR because this test failed before the DBO tests could run