Shard python_functions.cpp #62186

peterbell10 · 2021-07-26T14:29:57Z

Stack from ghstack:

This file takes 6 minutes on its own to compile and is the limiting factor for
building libtorch_python on a 32-core threadripper. This splits the file into
5 shards which take around 50 seconds each to compile.

Differential Revision: D29962046

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

facebook-github-bot · 2021-07-26T14:30:04Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/62186
📄 Preview docs built from this PR

💊 CI failures summary and remediations

As of commit 5c142d0 (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

win-vs2019-cpu-py3 / test (default, 1, 2, windows.4xlarge) (1/1)

Step: "Run test scripts" (full log | diagnosis details | 🔁 rerun)

2021-08-05T21:48:06.9508176Z RuntimeError: test_jit failed!

2021-08-05T21:48:05.9136404Z   test_pattern_based_rewrite (__main__.TestJit) ... ok (0.006s)
2021-08-05T21:48:05.9952443Z   test_pattern_based_rewrite_with_source_range_preserved (__main__.TestJit) ... ok (0.075s)
2021-08-05T21:48:06.0069770Z   test_peephole_optimize_shape_ops (__main__.TestJit) ... skip (0.016s)
2021-08-05T21:48:06.1360339Z   test_pretty_printer (__main__.TestJit) ... ok (0.131s)
2021-08-05T21:48:06.1381737Z   test_print_op_module (__main__.TestJit) ... ok (0.000s)
2021-08-05T21:48:06.7595966Z   test_profiler (__main__.TestJit) ... Traceback (most recent call last):
2021-08-05T21:48:06.7597591Z   File "run_test.py", line 1092, in <module>
2021-08-05T21:48:06.7597950Z     main()
2021-08-05T21:48:06.7598409Z   File "run_test.py", line 1071, in main
2021-08-05T21:48:06.7598840Z     raise RuntimeError(err_message)
2021-08-05T21:48:06.9508176Z RuntimeError: test_jit failed!
2021-08-05T21:48:06.9508499Z 
2021-08-05T21:48:06.9509085Z (base) C:\actions-runner\_work\pytorch\pytorch\pytorch-1102785398\test>if ERRORLEVEL 1 exit /b 1 
2021-08-05T21:48:06.9531490Z + cleanup
2021-08-05T21:48:06.9531807Z + retcode=1
2021-08-05T21:48:06.9532076Z + set +x
2021-08-05T21:48:06.9562781Z ##[error]Process completed with exit code 1.
2021-08-05T21:48:06.9720849Z ##[group]Run # -ir => recursive include all files in pattern
2021-08-05T21:48:06.9721492Z �[36;1m# -ir => recursive include all files in pattern�[0m
2021-08-05T21:48:06.9722059Z �[36;1m7z a "test-reports-$Env:COMMIT_SHA1-$Env:WORKFLOW_ID.zip" -ir'!test\*.xml'�[0m
2021-08-05T21:48:06.9739740Z shell: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.EXE -command ". '{0}'"

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

albanD

LGTM
Thanks for working on this!!

tools/autograd/templates/python_functions.h

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. ghstack-source-id: cd820f8f90e8bbfede1ac512549625641b462393 Pull Request resolved: pytorch#62186

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

albanD

LGTM

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

albanD · 2021-07-28T14:44:12Z

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. Differential Revision: [D29962046](https://our.internmc.facebook.com/intern/diff/D29962046) [ghstack-poisoned]

albanD · 2021-08-05T16:58:08Z

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. Differential Revision: [D29962046](https://our.internmc.facebook.com/intern/diff/D29962046) [ghstack-poisoned]

albanD · 2021-08-06T17:03:42Z

@albanD has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-08-11T16:23:15Z

@albanD merged this pull request in 5c00091.

Summary: Pull Request resolved: #62186 This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. Test Plan: Imported from OSS Reviewed By: bdhirsh Differential Revision: D29962046 Pulled By: albanD fbshipit-source-id: df13cfaebd54296f10609f67ae74a850c329bd37

Shard python_functions.cpp

161ceb1

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

peterbell10 requested review from albanD and soulitzer as code owners July 26, 2021 14:29

facebook-github-bot added the cla signed label Jul 26, 2021

This was referenced Jul 26, 2021

Refactor codegen file sharding #62184

Closed

Shard Operators.cpp #62185

Closed

Shard python_torch_functions.cpp #62187

Closed

Refactor structured set_output in Register{DispatchKey}.cpp #62188

Closed

pytorchbot added the open source label Jul 26, 2021

Update on "Shard python_functions.cpp"

6f112bc

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

albanD reviewed Jul 26, 2021

View reviewed changes

tools/autograd/templates/python_functions.h Outdated Show resolved Hide resolved

peterbell10 added 2 commits July 26, 2021 16:35

Update on "Shard python_functions.cpp"

42008c7

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

Update on "Shard python_functions.cpp"

97739ea

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

Update on "Shard python_functions.cpp"

7b58d36

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

Update on "Shard python_functions.cpp"

3c1ee9b

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

Update on "Shard python_functions.cpp"

9b02c77

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

albanD approved these changes Jul 26, 2021

View reviewed changes

peterbell10 added 2 commits July 27, 2021 17:51

Update on "Shard python_functions.cpp"

025c452

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

Update on "Shard python_functions.cpp"

d45d680

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

peterbell10 mentioned this pull request Jul 27, 2021

Codegen: Fix operator::name on windows #62278

Closed

Update on "Shard python_functions.cpp"

5681d8f

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

peterbell10 added the module: codegen Issues related to the codegen for Aten and Autograd label Jul 28, 2021

Update on "Shard python_functions.cpp"

124c8d2

This file takes 6 minutes on its own to compile and is the limiting factor for building `libtorch_python` on a 32-core threadripper. This splits the file into 5 shards which take around 50 seconds each to compile. [ghstack-poisoned]

peterbell10 added 2 commits August 5, 2021 16:01

facebook-github-bot closed this in 5c00091 Aug 11, 2021

facebook-github-bot added the Merged label Aug 11, 2021

facebook-github-bot deleted the gh/peterbell10/103/head branch August 15, 2021 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shard python_functions.cpp #62186

Shard python_functions.cpp #62186

peterbell10 commented Jul 26, 2021 •

edited by albanD

Loading

facebook-github-bot commented Jul 26, 2021 •

edited

Loading

albanD left a comment

albanD left a comment

albanD commented Jul 28, 2021

albanD commented Aug 5, 2021

albanD commented Aug 6, 2021

facebook-github-bot commented Aug 11, 2021

Shard python_functions.cpp #62186

Shard python_functions.cpp #62186

Conversation

peterbell10 commented Jul 26, 2021 • edited by albanD Loading

facebook-github-bot commented Jul 26, 2021 • edited Loading

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

win-vs2019-cpu-py3 / test (default, 1, 2, windows.4xlarge) (1/1)

albanD left a comment

Choose a reason for hiding this comment

albanD left a comment

Choose a reason for hiding this comment

albanD commented Jul 28, 2021

albanD commented Aug 5, 2021

albanD commented Aug 6, 2021

facebook-github-bot commented Aug 11, 2021

peterbell10 commented Jul 26, 2021 •

edited by albanD

Loading

facebook-github-bot commented Jul 26, 2021 •

edited

Loading