Fix test_ind_worker_queue by setting max_num_worker based on system resource #63779

ejguan · 2021-08-23T16:57:53Z

Stack from ghstack:

Fix test_ind_worker_queue by setting max_num_worker based on system resource #63779 Fix test_ind_worker_queue by setting max_num_worker based on system resource

Prevent freezing test for CI.

Confirm this test is flacky due to the limitation of CPU per machine. Did reproduce the hanging workers with ASAN build of PyTorch periodically.

After this PR, the number of worker will be limited based on OS.

Differential Revision: D30494185

cc @ssnl @VitalyFedyunin @ejguan

…esource [ghstack-poisoned]

facebook-github-bot · 2021-08-23T16:57:59Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/63779

💊 CI failures summary and remediations

As of commit 12a23e2 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

…on system resource" [ghstack-poisoned]

…on system resource" Fixes #63657 [ghstack-poisoned]

…esource ghstack-source-id: 4d6ab53 Pull Request resolved: #63779

ejguan · 2021-08-23T19:42:31Z

@ejguan has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…on system resource" Fixes #63657 Differential Revision: [D30494185](https://our.internmc.facebook.com/intern/diff/D30494185) [ghstack-poisoned]

…esource ghstack-source-id: c0d3b70 Pull Request resolved: #63779

ejguan · 2021-08-24T14:58:12Z

@ejguan has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…on system resource" Fixes #63657 Differential Revision: [D30494185](https://our.internmc.facebook.com/intern/diff/D30494185) [ghstack-poisoned]

…esource ghstack-source-id: ea61e8d Pull Request resolved: #63779

ejguan · 2021-08-25T15:34:39Z

@ejguan has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…on system resource" Fixes #63657 Prevent freezing test for CI. Confirm this test is flacky due to the limitation of CPU per machine. Did reproduce the hanging workers with ASAN build of PyTorch periodically. After this PR, the number of worker will be limited based on OS. Differential Revision: [D30494185](https://our.internmc.facebook.com/intern/diff/D30494185) [ghstack-poisoned]

…esource ghstack-source-id: 0af894d Pull Request resolved: #63779

ejguan · 2021-08-26T16:17:19Z

@ejguan has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

VitalyFedyunin · 2021-09-01T20:02:43Z

test/test_dataloader.py

        for batch_size in (8, 16, 32, 64):
-            for num_workers in range(1, 6):


nit: min(6, max_num_workers)

…on system resource" Fixes #63657 Prevent freezing test for CI. Confirm this test is flacky due to the limitation of CPU per machine. Did reproduce the hanging workers with ASAN build of PyTorch periodically. After this PR, the number of worker will be limited based on OS. Differential Revision: [D30494185](https://our.internmc.facebook.com/intern/diff/D30494185) [ghstack-poisoned]

…esource ghstack-source-id: 96818b7 Pull Request resolved: #63779

ejguan · 2021-09-02T16:17:48Z

@ejguan has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-09-02T19:37:14Z

@ejguan merged this pull request in 3cd0a4a.

Fix test_ind_worker_queue by setting max_num_worker based on system r…

3a459fb

…esource [ghstack-poisoned]

facebook-github-bot added the cla signed label Aug 23, 2021

Update on "Fix test_ind_worker_queue by setting max_num_worker based …

beaba01

…on system resource" [ghstack-poisoned]

ejguan linked an issue Aug 23, 2021 that may be closed by this pull request

DISABLED test_ind_worker_queue (__main__.TestIndividualWorkerQueue) #63657

Closed

ejguan added ci/master module: dataloader Related to torch.utils.data.DataLoader and Sampler and removed ci/master labels Aug 23, 2021

Update on "Fix test_ind_worker_queue by setting max_num_worker based …

e29ff76

…on system resource" Fixes #63657 [ghstack-poisoned]

ejguan added a commit that referenced this pull request Aug 23, 2021

Fix test_ind_worker_queue by setting max_num_worker based on system r…

752c12c

…esource ghstack-source-id: 4d6ab53 Pull Request resolved: #63779

Update on "Fix test_ind_worker_queue by setting max_num_worker based …

82fa1ac

…on system resource" Fixes #63657 Differential Revision: [D30494185](https://our.internmc.facebook.com/intern/diff/D30494185) [ghstack-poisoned]

ejguan added a commit that referenced this pull request Aug 24, 2021

Fix test_ind_worker_queue by setting max_num_worker based on system r…

31d6694

…esource ghstack-source-id: c0d3b70 Pull Request resolved: #63779

Update on "Fix test_ind_worker_queue by setting max_num_worker based …

556dc9c

…on system resource" Fixes #63657 Differential Revision: [D30494185](https://our.internmc.facebook.com/intern/diff/D30494185) [ghstack-poisoned]

ejguan added a commit that referenced this pull request Aug 25, 2021

Fix test_ind_worker_queue by setting max_num_worker based on system r…

7ad6087

…esource ghstack-source-id: ea61e8d Pull Request resolved: #63779

ejguan requested review from VitalyFedyunin and NivekT August 25, 2021 15:33

ejguan added a commit that referenced this pull request Aug 26, 2021

Fix test_ind_worker_queue by setting max_num_worker based on system r…

49c09f5

…esource ghstack-source-id: 0af894d Pull Request resolved: #63779

VitalyFedyunin reviewed Sep 1, 2021

View reviewed changes

VitalyFedyunin approved these changes Sep 1, 2021

View reviewed changes

ejguan added a commit that referenced this pull request Sep 2, 2021

Fix test_ind_worker_queue by setting max_num_worker based on system r…

d2cbb87

…esource ghstack-source-id: 96818b7 Pull Request resolved: #63779

facebook-github-bot closed this in 3cd0a4a Sep 2, 2021

facebook-github-bot added the Merged label Sep 2, 2021

facebook-github-bot deleted the gh/ejguan/82/head branch September 6, 2021 14:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix test_ind_worker_queue by setting max_num_worker based on system resource #63779

Fix test_ind_worker_queue by setting max_num_worker based on system resource #63779

Uh oh!

ejguan commented Aug 23, 2021 •

edited by pytorch-probot bot

Loading

Uh oh!

facebook-github-bot commented Aug 23, 2021 •

edited

Loading

Uh oh!

ejguan commented Aug 23, 2021

Uh oh!

ejguan commented Aug 24, 2021

Uh oh!

ejguan commented Aug 25, 2021

Uh oh!

ejguan commented Aug 26, 2021

Uh oh!

VitalyFedyunin Sep 1, 2021

Uh oh!

ejguan commented Sep 2, 2021

Uh oh!

facebook-github-bot commented Sep 2, 2021

Uh oh!

Uh oh!

		for batch_size in (8, 16, 32, 64):
		for num_workers in range(1, 6):

Fix test_ind_worker_queue by setting max_num_worker based on system resource #63779

Fix test_ind_worker_queue by setting max_num_worker based on system resource #63779

Uh oh!

Conversation

ejguan commented Aug 23, 2021 • edited by pytorch-probot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Aug 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

Uh oh!

ejguan commented Aug 23, 2021

Uh oh!

ejguan commented Aug 24, 2021

Uh oh!

ejguan commented Aug 25, 2021

Uh oh!

ejguan commented Aug 26, 2021

Uh oh!

VitalyFedyunin Sep 1, 2021

Choose a reason for hiding this comment

Uh oh!

ejguan commented Sep 2, 2021

Uh oh!

facebook-github-bot commented Sep 2, 2021

Uh oh!

Uh oh!

ejguan commented Aug 23, 2021 •

edited by pytorch-probot bot

Loading

facebook-github-bot commented Aug 23, 2021 •

edited

Loading