Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable use of Pytorch codegen from external libraries #73212

Closed
zou3519 opened this issue Feb 22, 2022 · 4 comments
Closed

Enable use of Pytorch codegen from external libraries #73212

zou3519 opened this issue Feb 22, 2022 · 4 comments
Labels
feature A request for a proper, new feature. module: codegen Issues related to the codegen for Aten and Autograd triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@zou3519
Copy link
Contributor

zou3519 commented Feb 22, 2022 •
edited

馃殌 The feature, motivation and pitch

External libraries that have C++ extensions that provide new per-operator functionality will find it helpful to use PyTorch's codegen model that parses native_functions.yaml and has common routines for things like determining if an operation is in-place or if something is being aliased. In particular, in functorch, we really wish we had access to PyTorch's codegen model.

We should offer a way for external libraries to call PyTorch's codegen API to make their lives easier.

Alternatives

Brainstorming with @bdhirsh offered the following ideas:

  • functorch (or an external library) can clone PyTorch as a part of its build workflow. This is not ideal due to needing the additional step
  • Copy-paste the codegen library into functorch. Also not ideal because the codegen can go out-of-sync.
  • Provide a pytorch-codegen package
  • Provide a torch.codegen module

Additional context

No response

cc @ezyang @bhosmer @bdhirsh
@zou3519 zou3519 added the module: codegen Issues related to the codegen for Aten and Autograd label Feb 22, 2022
@bdhirsh
Copy link
Contributor

bdhirsh commented Feb 22, 2022

One thing @zou3519 and I agreed is worth thinking about is how to deal with (or at least properly advertise) lack of BC. If an out-of-tree codebase like functorch wants to use codegen utilities, we probably can't make any promises that the utilities will be stable API's.

@ezyang
Copy link
Contributor

ezyang commented Feb 22, 2022

No BC guarantees, but I feel comfortable making it available. Will be helpful to have unified CI so when we refactor I can fix functorch's code.

BTW, functorch may also consider putting their codegen in pytorch/pytorch, if it is relatively stable; that's what we did for XLA and it works ok.

@VitalyFedyunin VitalyFedyunin added feature A request for a proper, new feature. triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Feb 26, 2022
zou3519 added a commit to pytorch/functorch that referenced this issue Feb 28, 2022
@zou3519
vmap codegen, for review
0cc1df5 
Here's how the output looks:
- https://github.com/pytorch/functorch/blob/main/functorch/csrc/VmapGeneratedPlumbing.h
- (yes, we check in the codegen as a short-term solution because we
didn't have time to add it as a step to setup.py)

short-term / long-term plans
- we've copy-pasted tools/codegen from pytorch/pytorch into functorch
- In the long term, pytorch/pytorch will package its codegen somehow
(pytorch/pytorch#73212)

Questions for reviewer:
- Are we using PyTorch's codegen abstractions the right way?
@zou3519 zou3519 mentioned this issue Feb 28, 2022
Merged
@zou3519
Copy link
Contributor Author

zou3519 commented Mar 7, 2022

Our options right now so far seem to be:

  • kick tools/codegen (including native_functions.yaml!) into a torch.utils._codegen package
  • figure out how to package tools/codegen into its own pip installable
  • move functorch codegen into pytorch/pytorch

I'm not comfortable with moving functorch codegen into pytorch/pytorch because I want to be able to refactor it with less than 1 day (i.e. waiting for the pytorch nightly to update) turnaround.

Setting up packaging also doesn't seem like fun, so, the easiest way to do this is probably to kick tools/codegen (including native_functions.yaml!) into a torch.utils._codegen folder. This might be a little painful in terms of needing to update PyTorch's build system though

zou3519 added a commit to pytorch/functorch that referenced this issue Mar 7, 2022
@zou3519
vmap codegen, for review
544a963 
Here's how the output looks:
- https://github.com/pytorch/functorch/blob/main/functorch/csrc/VmapGeneratedPlumbing.h
- (yes, we check in the codegen as a short-term solution because we
didn't have time to add it as a step to setup.py)

short-term / long-term plans
- we've copy-pasted tools/codegen from pytorch/pytorch into functorch
- In the long term, pytorch/pytorch will package its codegen somehow
(pytorch/pytorch#73212)

Questions for reviewer:
- Are we using PyTorch's codegen abstractions the right way?
zou3519 added a commit to pytorch/functorch that referenced this issue Mar 15, 2022
@zou3519
vmap codegen, for review
549c3ec 
Here's how the output looks:
- https://github.com/pytorch/functorch/blob/main/functorch/csrc/VmapGeneratedPlumbing.h
- (yes, we check in the codegen as a short-term solution because we
didn't have time to add it as a step to setup.py)

short-term / long-term plans
- we've copy-pasted tools/codegen from pytorch/pytorch into functorch
- In the long term, pytorch/pytorch will package its codegen somehow
(pytorch/pytorch#73212)

Questions for reviewer:
- Are we using PyTorch's codegen abstractions the right way?
zou3519 added a commit to pytorch/functorch that referenced this issue Mar 16, 2022
@zou3519
vmap codegen, for review (#546)
376ca1a 
Here's how the output looks:
- https://github.com/pytorch/functorch/blob/main/functorch/csrc/VmapGeneratedPlumbing.h
- (yes, we check in the codegen as a short-term solution because we
didn't have time to add it as a step to setup.py)

short-term / long-term plans
- we've copy-pasted tools/codegen from pytorch/pytorch into functorch
- In the long term, pytorch/pytorch will package its codegen somehow
(pytorch/pytorch#73212)

Questions for reviewer:
- Are we using PyTorch's codegen abstractions the right way?
@zou3519
Copy link
Contributor Author

zou3519 commented Apr 18, 2022

Offline discussion with @ezyang and co said:

  • We're calling the codegen module torchgen (this is still bikeshed-able)
  • This will live in a top-level torchgen/ folder in the PyTorch repo.
  • We will distribute torchgen/ with the pytorch binaries (so out-of-tree package maintainers don't need to figure out which version of torchgen to pair with which version of PyTorch).

@ezyang ezyang mentioned this issue Apr 18, 2022
Closed
ezyang added a commit that referenced this issue Apr 18, 2022
@ezyang
Rename tools/codegen to torchgen
0088d90 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edit to tools/test/test_gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 18, 2022
@ezyang
Update on "Rename tools/codegen to torchgen"
37ed9be 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 18, 2022
@ezyang
Update base for Update on "Rename tools/codegen to torchgen"
d6f38fe 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 18, 2022
@ezyang
Update on "Rename tools/codegen to torchgen"
f3138ff 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 18, 2022
@ezyang
Rename tools/codegen to torchgen
ac85634 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edit to tools/test/test_gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

ghstack-source-id: e7eed085aac1bc25d25799fdb89c3f65eca45fbc
Pull Request resolved: #75972
ezyang added a commit that referenced this issue Apr 19, 2022
@ezyang
Update base for Update on "Rename tools/codegen to torchgen"
e8f458e 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 19, 2022
@ezyang
Update on "Rename tools/codegen to torchgen"
32f3220 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 19, 2022
@ezyang
Rename tools/codegen to torchgen
a43dc52 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edit to tools/test/test_gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

ghstack-source-id: 4cb1819d37ca17c3fb855751ae418f4a0c312ac3
Pull Request resolved: #75972
ezyang added a commit that referenced this issue Apr 20, 2022
@ezyang
Update base for Update on "Rename tools/codegen to torchgen"
cfedb6a 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
--- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401
 
 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')
 
 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
--- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):
 
     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
--- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:
 
     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")
 
     def make_file_manager(install_dir: str) -> FileManager:
```

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 20, 2022
@ezyang
Update on "Rename tools/codegen to torchgen"
c4e127f 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
--- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401
 
 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')
 
 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
--- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):
 
     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
--- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:
 
     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")
 
     def make_file_manager(install_dir: str) -> FileManager:
```

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 20, 2022
@ezyang
Update base for Update on "Rename tools/codegen to torchgen"
417a772 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
--- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401
 
 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')
 
 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
--- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):
 
     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
--- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:
 
     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")
 
     def make_file_manager(install_dir: str) -> FileManager:
```

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 20, 2022
@ezyang
Update on "Rename tools/codegen to torchgen"
3a5e159 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
--- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401
 
 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')
 
 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
--- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):
 
     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
--- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:
 
     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")
 
     def make_file_manager(install_dir: str) -> FileManager:
```

Signed-off-by: Edward Z. Yang <ezyangfb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 20, 2022
@ezyang
Rename tools/codegen to torchgen
2f370f8 
In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edit to tools/test/test_gen_backend_stubs.py

Signed-off-by: Edward Z. Yang <ezyangfb.com>

ghstack-source-id: 48d6fa0475fb2fbd3159ebe56159ab4b0337f912
Pull Request resolved: #75972
@ezyang ezyang mentioned this issue Apr 24, 2022
Closed
ezyang added a commit to ezyang/pytorch that referenced this issue Apr 24, 2022
@ezyang @facebook-github-bot
Rename tools/codegen to torchgen (pytorch#76275)
3c33ed4 
Summary:
Pull Request resolved: pytorch#76275

In preparation for addressing
pytorch#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
 diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
 --- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401

 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')

 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
 diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
 --- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):

     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
 diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
 --- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:

     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")

     def make_file_manager(install_dir: str) -> FileManager:
```

run_all_fbandroid_tests

Test Plan: sandcastle

Reviewed By: albanD, ngimel

Differential Revision: D35770317

fbshipit-source-id: b77d0c2af80765289a6294f8aa4481951a1e6fb0
facebook-github-bot pushed a commit that referenced this issue Apr 25, 2022
@ezyang @facebook-github-bot
Rename tools/codegen to torchgen (#76275)
c6d485d 
Summary:
Pull Request resolved: #76275

In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
 diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
 --- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401

 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')

 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
 diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
 --- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):

     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
 diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
 --- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:

     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")

     def make_file_manager(install_dir: str) -> FileManager:
```

run_all_fbandroid_tests

Test Plan: sandcastle

Reviewed By: albanD, ngimel

Differential Revision: D35770317

fbshipit-source-id: 153ac4a7fef15b1e750812a90bfafdbc8f1ebcdf
pytorchmergebot pushed a commit that referenced this issue Apr 25, 2022
@ezyang @pytorchmergebot
Rename tools/codegen to torchgen (#76275)
36420b5 
Summary:
Pull Request resolved: #76275

In preparation for addressing
#73212

Diff was generated with:

```
git mv tools/codegen torchgen
git grep -l 'tools.codegen' | xargs sed -i 's/tools.codegen/torchgen/g'
sed -i "s/\${TOOLS_PATH}\/codegen/\${TORCH_ROOT}\/torchgen/g" caffe2/CMakeLists.txt
```

and a manual edits to:

* tools/test/test_gen_backend_stubs.py
* torchgen/build.bzl
* torchgen/gen_backend_stubs.py

aka this diff:

```
 diff --git a/tools/test/test_gen_backend_stubs.py b/tools/test/test_gen_backend_stubs.py
index 3dc26c6d2d..104054575e 100644
 --- a/tools/test/test_gen_backend_stubs.py
+++ b/tools/test/test_gen_backend_stubs.py
@@ -9,7 +9,7 @@ from torchgen.gen_backend_stubs import run
 from torchgen.gen import _GLOBAL_PARSE_NATIVE_YAML_CACHE  # noqa: F401

 path = os.path.dirname(os.path.realpath(__file__))
-gen_backend_stubs_path = os.path.join(path, '../torchgen/gen_backend_stubs.py')
+gen_backend_stubs_path = os.path.join(path, '../../torchgen/gen_backend_stubs.py')

 # gen_backend_stubs.py is an integration point that is called directly by external backends.
 # The tests here are to confirm that badly formed inputs result in reasonable error messages.
 diff --git a/torchgen/build.bzl b/torchgen/build.bzl
index ed04e35a43..d00078a3cf 100644
 --- a/torchgen/build.bzl
+++ b/torchgen/build.bzl
@@ -1,6 +1,6 @@
 def define_targets(rules):
     rules.py_library(
-        name = "codegen",
+        name = "torchgen",
         srcs = rules.glob(["**/*.py"]),
         deps = [
             rules.requirement("PyYAML"),
@@ -11,6 +11,6 @@ def define_targets(rules):

     rules.py_binary(
         name = "gen",
-        srcs = [":codegen"],
+        srcs = [":torchgen"],
         visibility = ["//visibility:public"],
     )
 diff --git a/torchgen/gen_backend_stubs.py b/torchgen/gen_backend_stubs.py
index c1a672a655..beee7a15e0 100644
 --- a/torchgen/gen_backend_stubs.py
+++ b/torchgen/gen_backend_stubs.py
@@ -474,7 +474,7 @@ def run(
 ) -> None:

     # Assumes that this file lives at PYTORCH_ROOT/torchgen/gen_backend_stubs.py
-    pytorch_root = pathlib.Path(__file__).parent.parent.parent.absolute()
+    pytorch_root = pathlib.Path(__file__).parent.parent.absolute()
     template_dir = os.path.join(pytorch_root, "aten/src/ATen/templates")

     def make_file_manager(install_dir: str) -> FileManager:
```

run_all_fbandroid_tests

Test Plan: sandcastle

Reviewed By: albanD, ngimel

Differential Revision: D35770317

fbshipit-source-id: 153ac4a7fef15b1e750812a90bfafdbc8f1ebcdf
(cherry picked from commit c6d485d)
@ezyang ezyang mentioned this issue Apr 25, 2022
Closed
ezyang added a commit that referenced this issue Apr 25, 2022
@ezyang
Distribute torchgen as part of PyTorch package
ae972e2 
Fixes #73212

Signed-off-by: Edward Z. Yang <ezyang@fb.com>

[ghstack-poisoned]
ezyang added a commit that referenced this issue Apr 25, 2022
@ezyang
Distribute torchgen as part of PyTorch package
2df59dd 
Fixes #73212

Signed-off-by: Edward Z. Yang <ezyangfb.com>

ghstack-source-id: 4ff0c4062861fbaf0354094e3bd2f00e6be24b08
Pull Request resolved: #76306
facebook-github-bot pushed a commit that referenced this issue Apr 26, 2022
@ezyang @facebook-github-bot
Distribute torchgen as part of PyTorch package (#76306)
e5ab7b6 
Summary:
Fixes #73212

Signed-off-by: Edward Z. Yang <ezyangfb.com>

Pull Request resolved: #76306

Approved by: https://github.com/zou3519

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/5109d81fc542dad888e3a2ad637d7d976c306ee9

Reviewed By: osalpekar

Differential Revision: D35938191

Pulled By: ezyang

fbshipit-source-id: 4ee04007dfab8b5c168cb453f9c81f11dd5bd0b8
zou3519 added a commit to zou3519/pytorch that referenced this issue Jul 20, 2022
@zou3519
[functorch] vmap codegen, for review (pytorch/functorch#546)
fe64615 
Here's how the output looks:
- https://github.com/pytorch/functorch/blob/main/functorch/csrc/VmapGeneratedPlumbing.h
- (yes, we check in the codegen as a short-term solution because we
didn't have time to add it as a step to setup.py)

short-term / long-term plans
- we've copy-pasted tools/codegen from pytorch/pytorch into functorch
- In the long term, pytorch/pytorch will package its codegen somehow
(pytorch#73212)

Questions for reviewer:
- Are we using PyTorch's codegen abstractions the right way?
bigfootjon pushed a commit that referenced this issue Jul 21, 2022
@zou3519 @bigfootjon
[functorch] vmap codegen, for review (pytorch/functorch#546)
f23ff68 
Here's how the output looks:
- https://github.com/pytorch/functorch/blob/main/functorch/csrc/VmapGeneratedPlumbing.h
- (yes, we check in the codegen as a short-term solution because we
didn't have time to add it as a step to setup.py)

short-term / long-term plans
- we've copy-pasted tools/codegen from pytorch/pytorch into functorch
- In the long term, pytorch/pytorch will package its codegen somehow
(#73212)

Questions for reviewer:
- Are we using PyTorch's codegen abstractions the right way?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature A request for a proper, new feature. module: codegen Issues related to the codegen for Aten and Autograd triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ezyang @VitalyFedyunin @zou3519 @bdhirsh