Skip to content

Conversation

dhruvbird
Copy link
Contributor

@dhruvbird dhruvbird commented Apr 14, 2021

Stack from ghstack:

We want to be able to drop the dependence of full-jit deps in the auto-generated unit tests for 2 reasons:

  1. Running bloaty on the auto-generated unit tests should be somewhat representative of the actual size.
  2. The runtime environment of the auto-generated unit tests should be as close to the production environment as possible to ensure that we are running the tests in a production-like runtime.

Due to the dependece on full-jit, we aren't there yet. For the auto-generated tests, we probably don't need to depend on _export_operator_list() evetually, but for now we do since it is used to decide whether the model being run is a Metal GPU model or a CPU model, and gates whether the test runs that model or not.

Eventually, we can stop doing this in the test and do it in the codegen from PTM-CLI instead (by fetching the operators from that tool, and writing out to the BUCK file which backend(s) this model is targeting). However, that will take some time to land, so in the spirit of expediency, this change is being proposed.

Discussed this offline with @myuan

Differential Revision: D27694781

NOTE FOR REVIEWERS: This PR has internal Facebook specific changes or comments, please review them on Phabricator!

…serialization/export_module.cpp to mobile/import.cpp

We want to be able to drop the dependence of full-jit deps in the auto-generated unit tests for 2 reasons:

1. Running bloaty on the auto-generated unit tests should be somewhat representative of the actual size.
2. The runtime environment of the auto-generated unit tests should be as close to the production environment as possible to ensure that we are running the tests in a production-like runtime.

Due to the dependece on full-jit, we aren't there yet. For the auto-generated tests, we probably don't need to depend on `_export_operator_list()` evetually, but for now we do since it is used to decide whether the model being run is a Metal GPU model or a CPU model, and gates whether the test runs that model or not.

Eventually, we can stop doing this in the test and do it in the codegen from PTM-CLI instead (by fetching the operators from that tool, and writing out to the BUCK file which backend(s) this model is targeting). However, that will take some time to land, so in the spirit of expediency, this change is being proposed.

Discussed this offline with @myuan

Differential Revision: [D27694781](https://our.internmc.facebook.com/intern/diff/D27694781/)

**NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27694781/)!

[ghstack-poisoned]
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Apr 14, 2021

💊 CI failures summary and remediations

As of commit 42d1bf6 (more details on the Dr. CI page):


  • 2/2 failures introduced in this PR

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_doc_test (1/2)

Step: "Doc test" (full log | diagnosis details | 🔁 rerun)

Apr 15 22:14:31 sccache: error: couldn't connect to server
Apr 15 22:14:31 ++++ eval 'extract_trap_cmd '
Apr 15 22:14:31 +++++ extract_trap_cmd
Apr 15 22:14:31 +++++ printf '%s\n' ''
Apr 15 22:14:31 ++++ printf '%s\n' cleanup
Apr 15 22:14:31 +++ trap -- '
Apr 15 22:14:31 cleanup' EXIT
Apr 15 22:14:31 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build != *pytorch-win-* ]]
Apr 15 22:14:31 +++ which sccache
Apr 15 22:14:31 +++ sccache --stop-server
Apr 15 22:14:31 Stopping sccache server...
Apr 15 22:14:31 sccache: error: couldn't connect to server
Apr 15 22:14:31 sccache: caused by: Connection refused (os error 111)
Apr 15 22:14:31 +++ true
Apr 15 22:14:31 +++ rm /var/lib/jenkins/sccache_error.log
Apr 15 22:14:31 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build == *rocm* ]]
Apr 15 22:14:31 +++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
Apr 15 22:14:31 +++ SCCACHE_IDLE_TIMEOUT=1200
Apr 15 22:14:31 +++ RUST_LOG=sccache::server=error
Apr 15 22:14:31 +++ sccache --start-server
Apr 15 22:14:31 sccache: Starting the server...
Apr 15 22:14:31 +++ sccache --zero-stats

See CircleCI build pytorch_python_doc_build (2/2)

Step: "Doc Build and Push" (full log | diagnosis details | 🔁 rerun)

Apr 15 22:23:39 Makefile:38: recipe for target 'html' failed
Apr 15 22:23:37 
Apr 15 22:23:37 copying static files... ... done
Apr 15 22:23:37 copying extra files... done
Apr 15 22:23:38 dumping search index in English (code: en)... done
Apr 15 22:23:38 dumping object inventory... done
Apr 15 22:23:38 build finished with problems, 1 warning.
Apr 15 22:23:38 /var/lib/jenkins/workspace/docs/src/pytorch-sphinx-theme/pytorch_sphinx_theme/search.html:21: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
Apr 15 22:23:38   <p class="last">
Apr 15 22:23:38 /var/lib/jenkins/workspace/docs/src/pytorch-sphinx-theme/pytorch_sphinx_theme/search.html:24: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
Apr 15 22:23:38   </p>
Apr 15 22:23:39 Makefile:38: recipe for target 'html' failed
Apr 15 22:23:39 make: *** [html] Error 1
Apr 15 22:23:39 =========================
Apr 15 22:23:39 ++ code=2
Apr 15 22:23:39 ++ '[' 2 -ne 0 ']'
Apr 15 22:23:39 ++ set +x
Apr 15 22:23:39 ++ return 2
Apr 15 22:23:39 WARNING: autodoc: failed to import module 'launch' from module 'torch.distributed'; the following exception was raised:
Apr 15 22:23:39 =========================
Apr 15 22:23:39 Docs build failed. If the failure is not clear, scan back in the log
Apr 15 22:23:39 for any WARNINGS or for the line build finished with problems

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

@facebook-github-bot facebook-github-bot added oncall: jit Add this issue/PR to JIT oncall triage queue cla signed labels Apr 14, 2021
dhruvbird added a commit that referenced this pull request Apr 14, 2021
…serialization/export_module.cpp to mobile/import.cpp

We want to be able to drop the dependence of full-jit deps in the auto-generated unit tests for 2 reasons:

1. Running bloaty on the auto-generated unit tests should be somewhat representative of the actual size.
2. The runtime environment of the auto-generated unit tests should be as close to the production environment as possible to ensure that we are running the tests in a production-like runtime.

Due to the dependece on full-jit, we aren't there yet. For the auto-generated tests, we probably don't need to depend on `_export_operator_list()` evetually, but for now we do since it is used to decide whether the model being run is a Metal GPU model or a CPU model, and gates whether the test runs that model or not.

Eventually, we can stop doing this in the test and do it in the codegen from PTM-CLI instead (by fetching the operators from that tool, and writing out to the BUCK file which backend(s) this model is targeting). However, that will take some time to land, so in the spirit of expediency, this change is being proposed.

Discussed this offline with @myuan

Differential Revision: [D27694781](https://our.internmc.facebook.com/intern/diff/D27694781/)

**NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27694781/)!

ghstack-source-id: 126529184
Pull Request resolved: #56044
…ist() from serialization/export_module.cpp to mobile/import.cpp"

We want to be able to drop the dependence of full-jit deps in the auto-generated unit tests for 2 reasons:

1. Running bloaty on the auto-generated unit tests should be somewhat representative of the actual size.
2. The runtime environment of the auto-generated unit tests should be as close to the production environment as possible to ensure that we are running the tests in a production-like runtime.

Due to the dependece on full-jit, we aren't there yet. For the auto-generated tests, we probably don't need to depend on `_export_operator_list()` evetually, but for now we do since it is used to decide whether the model being run is a Metal GPU model or a CPU model, and gates whether the test runs that model or not.

Eventually, we can stop doing this in the test and do it in the codegen from PTM-CLI instead (by fetching the operators from that tool, and writing out to the BUCK file which backend(s) this model is targeting). However, that will take some time to land, so in the spirit of expediency, this change is being proposed.

Discussed this offline with @myuan

Differential Revision: [D27694781](https://our.internmc.facebook.com/intern/diff/D27694781/)

**NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27694781/)!

[ghstack-poisoned]
dhruvbird added a commit that referenced this pull request Apr 15, 2021
…serialization/export_module.cpp to mobile/import.cpp

Pull Request resolved: #56044

We want to be able to drop the dependence of full-jit deps in the auto-generated unit tests for 2 reasons:

1. Running bloaty on the auto-generated unit tests should be somewhat representative of the actual size.
2. The runtime environment of the auto-generated unit tests should be as close to the production environment as possible to ensure that we are running the tests in a production-like runtime.

Due to the dependece on full-jit, we aren't there yet. For the auto-generated tests, we probably don't need to depend on `_export_operator_list()` evetually, but for now we do since it is used to decide whether the model being run is a Metal GPU model or a CPU model, and gates whether the test runs that model or not.

Eventually, we can stop doing this in the test and do it in the codegen from PTM-CLI instead (by fetching the operators from that tool, and writing out to the BUCK file which backend(s) this model is targeting). However, that will take some time to land, so in the spirit of expediency, this change is being proposed.

Discussed this offline with @myuan
ghstack-source-id: 126656877

Differential Revision: [D27694781](https://our.internmc.facebook.com/intern/diff/D27694781/)

**NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D27694781/)!
@facebook-github-bot
Copy link
Contributor

This pull request has been merged in bd3c63a.

@facebook-github-bot facebook-github-bot deleted the gh/dhruvbird/46/head branch April 19, 2021 14:15
krshrimali pushed a commit to krshrimali/pytorch that referenced this pull request May 19, 2021
…serialization/export_module.cpp to mobile/import.cpp (pytorch#56044)

Summary:
Pull Request resolved: pytorch#56044

We want to be able to drop the dependence of full-jit deps in the auto-generated unit tests for 2 reasons:

1. Running bloaty on the auto-generated unit tests should be somewhat representative of the actual size.
2. The runtime environment of the auto-generated unit tests should be as close to the production environment as possible to ensure that we are running the tests in a production-like runtime.

Due to the dependece on full-jit, we aren't there yet. For the auto-generated tests, we probably don't need to depend on `_export_operator_list()` evetually, but for now we do since it is used to decide whether the model being run is a Metal GPU model or a CPU model, and gates whether the test runs that model or not.

Eventually, we can stop doing this in the test and do it in the codegen from PTM-CLI instead (by fetching the operators from that tool, and writing out to the BUCK file which backend(s) this model is targeting). However, that will take some time to land, so in the spirit of expediency, this change is being proposed.

Discussed this offline with iseeyuan
ghstack-source-id: 126656877

Test Plan: Build + BSB.

Reviewed By: iseeyuan

Differential Revision: D27694781

fbshipit-source-id: f31a2dfd40803c02f4fd19c45a3cc6fb9bdf9697
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed Merged oncall: jit Add this issue/PR to JIT oncall triage queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants