[AOTInductor] Include constants in AOTInductor .so file. #108473

muchulee8 · 2023-09-02T04:44:12Z

Summary:
Include constants in AOTInductor .so file.
Added some difference:

serialize with ctypes instead of the native of torch.storage
Use the underlying for_blob instead of from_blob to construct Tensor.

Test Plan:
Unit tests:

test/inductor/test_aot_inductor.py

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @ngimel @yf225 @chenyang78 @kadeng @aakhundov @anijain2305

pytorch-bot · 2023-09-02T04:44:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/108473

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 1868687 with merge base c458fa0 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_smoketest_perf, 1, 1, linux.gcp.a100) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2023-09-02T04:44:23Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

facebook-github-bot · 2023-09-04T18:33:43Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

facebook-github-bot · 2023-09-04T18:34:51Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

facebook-github-bot · 2023-09-05T17:53:15Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

facebook-github-bot · 2023-09-05T17:53:21Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

facebook-github-bot · 2023-09-05T21:50:22Z

This pull request was exported from Phabricator. Differential Revision: D48927532

facebook-github-bot · 2023-09-05T21:50:23Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

facebook-github-bot · 2023-09-05T23:18:09Z

This pull request was exported from Phabricator. Differential Revision: D48927532

) Summary: Include constants in AOTInductor .so file. Added some difference: 1) serialize with ctypes instead of the native of torch.storage 2) Use the underlying for_blob instead of from_blob to construct Tensor. Test Plan: Unit tests: ``` test/inductor/test_aot_inductor.py ``` fb: MRS tests (https://fburl.com/gdoc/ffllzw72): ``` LOGLEVEL=DEBUG TORCHINDUCTOR_MAX_AUTOTUNE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 ../buck-out/v2/gen/fbcode/3408cf5f8424049a/hpc/new/models/feed/benchmark/__feed_lower_benchmark__/feed_lower_benchmark.par --load=manifold://ig_inference_model/tree/user/facebook/fblearner/predictor/966480198/289/gpu_lowering/input.predictor --skip-trt --sync-mode=0 --enable-aot-inductor ``` Previous failed buck tests: ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark -- --exact 'caffe2/torch/fb/model_transform/experimental/benchmark/test:test_aot_inductor_benchmark - test_aot_inductor_benchmark_oemae (caffe2.torch.fb.model_transform.experimental.benchmark.test.test_aot_inductor_benchmark.AOTInductorBenchmark)' ``` Differential Revision: D48927532

pytorchmergebot · 2023-09-07T22:15:01Z

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

muchulee8 · 2023-09-07T22:36:51Z

@pytorchbot label "release notes: export"

muchulee8 · 2023-09-07T22:37:52Z

@pytorchbot merge

pytorchmergebot · 2023-09-07T22:41:08Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

facebook-github-bot · 2023-09-08T00:00:52Z

@muchulee8 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

pytorchmergebot · 2023-09-08T00:02:22Z

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team

Raised by workflow job

muchulee8 · 2023-09-08T00:10:02Z

@pytorchbot merge

pytorchmergebot · 2023-09-08T00:12:26Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…108473)" Original commit changeset: 9494d031e3ac Original Phabricator Diff: D49075977 Differential Revision: [D49243049](https://our.internmc.facebook.com/intern/diff/D49243049/) [ghstack-poisoned]

…108473)" Original commit changeset: 9494d031e3ac Original Phabricator Diff: D49075977 Differential Revision: [D49243049](https://our.internmc.facebook.com/intern/diff/D49243049/) ghstack-source-id: 200617295 Pull Request resolved: #109243

…108473)" Original commit changeset: 9494d031e3ac Original Phabricator Diff: D49075977 Differential Revision: [D49243049](https://our.internmc.facebook.com/intern/diff/D49243049/) ghstack-source-id: 6f57b61f2d11162ba2dfa8dcebecc3d4cb74f2aa Pull Request resolved: #109243

…in AOTInductor .so file. (#108473)"" Original commit changeset: 9494d031e3ac Original Phabricator Diff: D49075977 Differential Revision: [D49243049](https://our.internmc.facebook.com/intern/diff/D49243049/) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ipiszy ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]

…so file. (#108473)"" Original commit changeset: 9494d031e3ac Original Phabricator Diff: D49075977 Differential Revision: [D49243049](https://our.internmc.facebook.com/intern/diff/D49243049/) cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ipiszy ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]

Summary: Same as #109560, made a new PR because we need to land from internal Previously during performance benchmark testing, we would create an AOTInductorModelContainerHandle every time the compiled function is run with new inputs. However after #108473 we now load the constants needed in the runtime when initializing the AOTInductorModelContainerHandle. This resulted in our benchmarks displaying a ~0.4x speedup. This diff moves the initialization of AOTInductorModelContainerHandle outside of the code where we run the compiled function with different inputs. For example, ``` python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 0 --only AlbertForMaskedLM ``` results in `1.359x` speedup. Specifically, this adds a `create_container_handle` and `delete_container_handle` function which need to called before `run`. We call `create_container_handle` to initialize the AOTInductorModelContainerHandle, call `run` to run the compiled .so with different inputs, and then `delete_container_handle` to delete it. [Updated dashboard results](https://hud.pytorch.org/benchmark/compilers?startTime=Wed%2C%2013%20Sep%202023%2021%3A03%3A55%20GMT&stopTime=Wed%2C%2020%20Sep%202023%2021%3A03%3A55%20GMT&granularity=hour&suite=torchbench&mode=inference&dtype=bfloat16&lBranch=angelayi/aot_inductor_benchmark&lCommit=f9aa49c4c9a1a140b6f0c4520d1d6d99b57e12fa&rBranch=main&rCommit=015be4cedba357eb931e24bf188479235db7c5c8) Test Plan: CI Differential Revision: D49513934 Pull Request resolved: #109820 Approved by: https://github.com/desertfire

facebook-github-bot added the fb-exported label Sep 2, 2023

github-actions bot added module: inductor module: dynamo ciflow/inductor module: export labels Sep 2, 2023

muchulee8 force-pushed the export-D48927532 branch from afaad01 to 814a456 Compare September 4, 2023 18:33

muchulee8 force-pushed the export-D48927532 branch from 814a456 to df24558 Compare September 4, 2023 18:34

muchulee8 force-pushed the export-D48927532 branch from df24558 to 2293345 Compare September 5, 2023 17:53

muchulee8 force-pushed the export-D48927532 branch from 2293345 to 30bb690 Compare September 5, 2023 17:53

angelayi approved these changes Sep 5, 2023

View reviewed changes

muchulee8 force-pushed the export-D48927532 branch from 30bb690 to 51e49d1 Compare September 5, 2023 21:50

muchulee8 force-pushed the export-D48927532 branch from 51e49d1 to 365e48c Compare September 5, 2023 21:50

muchulee8 force-pushed the export-D48927532 branch from 365e48c to b419fc9 Compare September 5, 2023 23:18

pytorchmergebot added the merging label Sep 7, 2023

pytorchmergebot removed the merging label Sep 7, 2023

pytorch deleted a comment from facebook-github-bot Sep 7, 2023

pytorch-bot bot added the release notes: export label Sep 7, 2023

pytorchmergebot added the merging label Sep 7, 2023

Merge branch 'main' into export-D48927532

1868687

pytorchmergebot removed the merging label Sep 8, 2023

pytorchmergebot added the merging label Sep 8, 2023

pytorchmergebot added Merged and removed merging labels Sep 8, 2023

pytorchmergebot closed this in 30a33b7 Sep 8, 2023

huydhn mentioned this pull request Sep 8, 2023

[inductor] Switch to use the runtime interface for AOTInductor testing #108663

Closed

This was referenced Sep 18, 2023

[AOTInductor] Update performance benchmark code #109560

Closed

[aotinductor] Update performance benchmark code (109560) #109820

Closed

ezyang mentioned this pull request Feb 12, 2024

Properly setup tracing context for aot_compilation #119712

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AOTInductor] Include constants in AOTInductor .so file. #108473

[AOTInductor] Include constants in AOTInductor .so file. #108473

muchulee8 commented Sep 2, 2023 •

edited

pytorch-bot bot commented Sep 2, 2023 •

edited

facebook-github-bot commented Sep 2, 2023

facebook-github-bot commented Sep 4, 2023

facebook-github-bot commented Sep 4, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

pytorchmergebot commented Sep 7, 2023

muchulee8 commented Sep 7, 2023

muchulee8 commented Sep 7, 2023

pytorchmergebot commented Sep 7, 2023

facebook-github-bot commented Sep 8, 2023

pytorchmergebot commented Sep 8, 2023

muchulee8 commented Sep 8, 2023

pytorchmergebot commented Sep 8, 2023

[AOTInductor] Include constants in AOTInductor .so file. #108473

[AOTInductor] Include constants in AOTInductor .so file. #108473

Conversation

muchulee8 commented Sep 2, 2023 • edited

pytorch-bot bot commented Sep 2, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/108473

✅ You can merge normally! (1 Unrelated Failure)

facebook-github-bot commented Sep 2, 2023

facebook-github-bot commented Sep 4, 2023

facebook-github-bot commented Sep 4, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

facebook-github-bot commented Sep 5, 2023

pytorchmergebot commented Sep 7, 2023

Merge failed

muchulee8 commented Sep 7, 2023

muchulee8 commented Sep 7, 2023

pytorchmergebot commented Sep 7, 2023

Merge started

facebook-github-bot commented Sep 8, 2023

pytorchmergebot commented Sep 8, 2023

Merge failed

muchulee8 commented Sep 8, 2023

pytorchmergebot commented Sep 8, 2023

Merge started

muchulee8 commented Sep 2, 2023 •

edited

pytorch-bot bot commented Sep 2, 2023 •

edited