[ROCm] Add diskspace check for rocm CI nodes #93032

amdfaa · 2023-01-26T00:49:59Z

Fixes #92822

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport

pytorch-bot · 2023-01-26T00:50:02Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/93032

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7288d74:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

linux-foundation-easycla · 2023-01-26T00:50:04Z

The committers listed above are authorized under a signed CLA.

✅ login: amdfaa (4307c3c)

huydhn · 2023-01-26T18:00:26Z

/easycla

huydhn

Overall LGTM! But as this step is generic and can be used by others as well. It makes sense to move it to an action, i.e. https://github.com/pytorch/pytorch/blob/master/.github/actions/get-workflow-job-id/action.yml, then invoke the action wherever it is needed. The action could accept a diskspace cutoff param defaults to 70%

Also there is a minor linter check.

huydhn · 2023-01-26T18:17:46Z

Also please sign the CLA first

amdfaa · 2023-01-26T18:32:48Z

I'm signing the easycla but it's not updating on the ticket. I submitted a help request for the issue.

huydhn · 2023-01-26T18:38:51Z

I'm signing the easycla but it's not updating on the ticket. I submitted a help request for the issue.

Oh once you have signed it, I would just need to invoke the check again to make it pass

amdfaa · 2023-01-26T18:40:39Z

Sounds good! Please check if it works.

huydhn · 2023-01-26T18:59:37Z

/easycla

huydhn · 2023-01-26T19:06:49Z

Weird, invoking /easycla doesn't seem to do anything. You're right to submit a help request. May be also confirm that: your email is the correct one used in your GitHub account.

.github/actions/teardown-rocm/action.yml

malfet

Looks good to me, but please make sure to sign CLA (PR authors email must be covered by CLA)

amdfaa · 2023-01-26T19:58:32Z

Weird, invoking /easycla doesn't seem to do anything. You're right to submit a help request. May be also confirm that: your email is the correct one used in your GitHub account.

Sounds good!

.github/actions/setup-rocm/action.yml

malfet

Hmm, actually, any tips how one can test it?
Because, afaik $diskspace -ge diskspace_cutoff would always evaluate to true, aren't it?

$ diskspace=1 ;if [[ $diskspace -ge diskspace_cutoff ]]; then echo "Ha ha ha $diskspace"; fi
Ha ha ha 1

malfet · 2023-01-26T20:02:52Z

@ZainRizvi , Hmm, don't we have lintrunner to detect such mistakes?

ZainRizvi · 2023-01-26T20:17:38Z

The committers listed above are authorized under a signed CLA.

✅ login: amdfaa (4307c3c, 77a11ef)

ZainRizvi · 2023-01-26T20:19:32Z

@ZainRizvi , Hmm, don't we have lintrunner to detect such mistakes?

@kit1980, do we have a shell lintrunner to catch issues like the one mentioned here?

amdfaa · 2023-01-26T23:21:08Z

The support team stated the following: To resolve this [issue], the community usually make sure the GitHub Global Configs are properly configured, and run a git rebase to update the committer/author information on that specific commit.

huydhn · 2023-01-27T01:32:29Z

@pytorchbot rebase

pytorchmergebot · 2023-01-27T01:34:36Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-01-27T01:34:41Z

Successfully rebased rocm-diskspace-check onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm-diskspace-check && git pull --rebase)

huydhn · 2023-01-27T01:47:28Z

/easycla

.github/actions/setup-rocm/action.yml

jithunnair-amd · 2023-02-01T23:19:19Z

@pytorchbot rebase

pytorchmergebot · 2023-02-01T23:21:51Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-02-01T23:21:57Z

Successfully rebased rocm-diskspace-check onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout rocm-diskspace-check && git pull --rebase)

huydhn

LGTM! Thank you for fixing this issue!

jithunnair-amd · 2023-02-02T04:15:00Z

@amdfaa Added ciflow/trunk label to trigger ROCm test jobs on trunk workflow. Please check them when they finish to see that the diskspace check is being executed correctly.

amdfaa · 2023-02-03T22:36:19Z

Hmm, actually, any tips how one can test it?

I have locally tested this pr on jwr-20 with a lower cutoff and the script executed successfully.

jithunnair-amd · 2023-02-03T22:37:09Z

@pytorchbot merge

pytorchmergebot · 2023-02-03T22:38:53Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

amdfaa requested a review from a team as a code owner January 26, 2023 00:50

pytorch-bot bot added module: rocm AMD GPU support for Pytorch topic: not user facing topic category labels Jan 26, 2023

pytorchbot added the open source label Jan 26, 2023

huydhn added the test-config/default label Jan 26, 2023

huydhn reviewed Jan 26, 2023

View reviewed changes

jithunnair-amd reviewed Jan 26, 2023

View reviewed changes

.github/actions/teardown-rocm/action.yml Show resolved Hide resolved

malfet approved these changes Jan 26, 2023

View reviewed changes

malfet reviewed Jan 26, 2023

View reviewed changes

.github/actions/setup-rocm/action.yml Outdated Show resolved Hide resolved

malfet reviewed Jan 26, 2023

View reviewed changes

.github/actions/setup-rocm/action.yml Outdated Show resolved Hide resolved

malfet requested changes Jan 26, 2023

View reviewed changes

pytorchmergebot force-pushed the rocm-diskspace-check branch from 6ae95c3 to c42e549 Compare January 27, 2023 01:34

huydhn reviewed Feb 1, 2023

View reviewed changes

.github/actions/setup-rocm/action.yml Outdated Show resolved Hide resolved

amdfaa and others added 8 commits February 1, 2023 23:21

Add diskspace check for rocm CI nodes

4aa3707

Creates an action for diskspace cleanup

c2f8b36

Fixes variables calls in if statements

2853730

lint

a153d9f

Adding default cutoff of 70

a148296

Use default disk cutoff for ROCm

00b5b2b

Use fully qualified name for action

793a5ec

Use fully qualified name for action and use default disk cutoff for ROCm

667adac

pytorchmergebot force-pushed the rocm-diskspace-check branch from 634a6a2 to 667adac Compare February 1, 2023 23:22

huydhn approved these changes Feb 1, 2023

View reviewed changes

malfet approved these changes Feb 1, 2023

View reviewed changes

jithunnair-amd added the ciflow/trunk Trigger trunk jobs on your pull request label Feb 2, 2023

amdfaa added 7 commits February 2, 2023 09:24

fixing merge conflict

8860307

fixes merge issues 2

3bff23d

Fixes 'uses' statement file path in actions

79c6927

Fixes 'uses' statement file path in actions 2

e31e2ee

Fixes 'uses' statement file path in actions 3

764c412

Fixes 'uses' statement file path in actions 4

2c62424

Fixes 'uses' statement file path in actions 5

7288d74

pytorchmergebot added the Merged label Feb 3, 2023

pytorchmergebot closed this in 6d597c5 Feb 3, 2023

[ROCm] Add diskspace check for rocm CI nodes #93032

[ROCm] Add diskspace check for rocm CI nodes #93032

Uh oh!

Conversation

amdfaa commented Jan 26, 2023 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jan 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/93032

✅ No Failures

Uh oh!

linux-foundation-easycla bot commented Jan 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huydhn commented Jan 26, 2023

Uh oh!

huydhn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huydhn commented Jan 26, 2023

Uh oh!

amdfaa commented Jan 26, 2023

Uh oh!

huydhn commented Jan 26, 2023

Uh oh!

amdfaa commented Jan 26, 2023

Uh oh!

huydhn commented Jan 26, 2023

Uh oh!

huydhn commented Jan 26, 2023

Uh oh!

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

amdfaa commented Jan 26, 2023

Uh oh!

Uh oh!

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

malfet commented Jan 26, 2023

Uh oh!

ZainRizvi commented Jan 26, 2023 • edited by linux-foundation-easycla bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZainRizvi commented Jan 26, 2023

Uh oh!

amdfaa commented Jan 26, 2023

Uh oh!

huydhn commented Jan 27, 2023

Uh oh!

pytorchmergebot commented Jan 27, 2023

Uh oh!

pytorchmergebot commented Jan 27, 2023

Uh oh!

huydhn commented Jan 27, 2023

Uh oh!

Uh oh!

jithunnair-amd commented Feb 1, 2023

Uh oh!

pytorchmergebot commented Feb 1, 2023

Uh oh!

pytorchmergebot commented Feb 1, 2023

Uh oh!

huydhn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd commented Feb 2, 2023

Uh oh!

amdfaa commented Feb 3, 2023

Uh oh!

jithunnair-amd commented Feb 3, 2023

Uh oh!

pytorchmergebot commented Feb 3, 2023

amdfaa commented Jan 26, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jan 26, 2023 •

edited

Loading

linux-foundation-easycla bot commented Jan 26, 2023 •

edited

Loading

huydhn left a comment •

edited

Loading

ZainRizvi commented Jan 26, 2023 •

edited by linux-foundation-easycla bot

Loading

huydhn left a comment •

edited

Loading