move XNNPACK buck build to shared build file #77941

linbinyu · 2022-05-20T04:30:36Z

Summary:
This diff moved the XNNPACK buck build to a shared build file in xplat/caffe2/third_party, so it can be reused by OSS buck CI in the future. There's no functionality change.

Background: as we are moving to github-first, we want community to receive more signals from our internal build. XNNPACK is part of pytorch mobile build so we want to add it to OSS BUCK CI.

How it works: all XNNPACK targets are defined in xplat/caffe2/third_party/xnnpack_defs.bzl. When we build it internally, the XNNPACK source is still at xplat/third-party/XNNPACK and we will load that bzl file in xplat/third-party/XNNPACK/BUCK. Everything should work as before. In OSS build, XNNPACK is a submodule in xplat/caffe2/third_party and we will load the same bzl file in pytorch/third_party/BUILD.buck.

Wrapper Generation: the wrapper generation script is moved to xplat/caffe2/third_party/generate-xnnpack-wrappers.py. It will take an optional argument for the path of XNNPACK (they are different in internal build and OSS build). The wrapper files will always be generated at the parent folder of XNNPACK source. But the src_defs.bzl and wrapper_defs.bzl will always be in xplat/caffe2/third_party/ (they are now called xnnpack_src_defs.bzl and xnnpack_wrapper_defs.bzl). For OSS build this script will only be used in CI, and the generated files will not be committed.

Next Steps: Once landed, I will try to build XNNPACK in OSS BUCK using xnnpack_defs.bzl. Meta-specific symbols need to be resolved, so there will be some refactors to the build file.

Test Plan: buck build xplat/third-party/XNNPACK:XNNPACK

Differential Revision: D36529332

facebook-github-bot · 2022-05-20T04:30:42Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/77941
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours
↩️ [fb-only] Re-run with SSH instructions

❌ 1 New Failures

As of commit 31aac6f (more details on the Dr. CI page):

Expand to see more

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

pull / linux-xenial-cuda11.3-py3.7-gcc7 / test (default, 2, 4, linux.4xlarge.nvidia.gpu) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-05-28T00:40:24.3304981Z FAIL [3.007s]: test_mem_leak (__main__.TestProfilerCUDA)

2022-05-28T00:40:24.2454094Z STAGE:2022-05-28 00:40:24 5176:5176 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-05-28T00:40:24.2629656Z ok (0.069s)
2022-05-28T00:40:24.2657814Z   test_datapipe_with_record_function_fork (__main__.TestRecordFunction) ... STAGE:2022-05-28 00:40:24 5176:5176 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-05-28T00:40:24.3086616Z STAGE:2022-05-28 00:40:24 5176:5176 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-05-28T00:40:24.3242722Z ok (0.061s)
2022-05-28T00:40:24.3274350Z   test_record_function (__main__.TestRecordFunction) ... STAGE:2022-05-28 00:40:24 5176:5176 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-05-28T00:40:24.3290426Z STAGE:2022-05-28 00:40:24 5176:5176 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-05-28T00:40:24.3303194Z ok (0.006s)
2022-05-28T00:40:24.3303967Z 
2022-05-28T00:40:24.3304358Z ======================================================================
2022-05-28T00:40:24.3304981Z FAIL [3.007s]: test_mem_leak (__main__.TestProfilerCUDA)
2022-05-28T00:40:24.3305772Z Checks that there's no memory leak when using profiler with CUDA
2022-05-28T00:40:24.3306507Z ----------------------------------------------------------------------
2022-05-28T00:40:24.3307319Z Traceback (most recent call last):
2022-05-28T00:40:24.3308316Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 1287, in wrap_fn
2022-05-28T00:40:24.3308734Z     return fn(self, *args, **kwargs)
2022-05-28T00:40:24.3309084Z   File "test_profiler.py", line 69, in test_mem_leak
2022-05-28T00:40:24.3309943Z     msg='memory usage is increasing, {}'.format(str(last_rss)))
2022-05-28T00:40:24.3310898Z AssertionError: False is not true : memory usage is increasing, deque([2066194432, 2066272256, 2066276352, 2066280448, 2067730432], maxlen=5)
2022-05-28T00:40:24.3311481Z 
2022-05-28T00:40:24.3311827Z ----------------------------------------------------------------------

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

facebook-github-bot · 2022-05-20T04:30:58Z