generate C++ API for meta functions using at::meta:: #58570

bdhirsh · 2021-05-19T17:27:13Z

What the PR does
Generate a fast-path at::meta::{op} API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore).

Details
In order to avoid naming collisions I had to make two small changes:

rename MetaFunctions.h template -> NativeMetaFunctions.h (this is the file that declares the impl() function for every structured operator).
rename the meta class: at::meta::{op}::meta() -> at::meta::structured_{op}::meta()

I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h.

Why I made the change
This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see comment). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack.

Stack from ghstack:

move all external kernels into a class for better compiler error messages #59839 move all external kernels into a class for better compiler error messages
add a boxed CPU fallback kernel #58065 add a boxed CPU fallback kernel
beef up at::_ops API #59115 beef up at::_ops API
generate C++ API for meta functions using at::meta:: #58570 generate C++ API for meta functions using at::meta::
bugfix: ensure that at::{dispatch_key}:: API gets external linkage #58569 bugfix: ensure that at::{dispatch_key}:: API gets external linkage

Differential Revision: D28711299

[ghstack-poisoned]

facebook-github-bot · 2021-05-19T17:27:22Z

💊 CI failures summary and remediations

As of commit 6aee40f (more details on the Dr. CI page):

2/2 failures possibly* introduced in this PR
- 1/2 non-scanned failure(s)

1 failure not recognized by patterns:

Job	Step	Action
^{pytorch_cpp_doc_build}	^{Doc Build and Push}	🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

ghstack-source-id: 8cff248 Pull Request resolved: pytorch#58570

[ghstack-poisoned]

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change doesn't feel ambiguous so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. [ghstack-poisoned]

bdhirsh · 2021-05-19T21:57:36Z

Looks like this made the bazel build unhappy

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. [ghstack-poisoned]

ezyang

One q I have is whether or not we can easily make at::meta operators overloaded, in the sense that you can directly pass them CPU/CUDA tensors and it will still give you a meta tensor in the end. Food for thought...

bdhirsh · 2021-05-20T17:56:36Z

One q I have is whether or not we can easily make at::meta operators overloaded, in the sense that you can directly pass them CPU/CUDA tensors and it will still give you a meta tensor in the end. Food for thought...

Is the overloading that you're talking about with mixed device types? E.g. at::meta::add(cuda_tensor, cuda_tensor) should just work, but at::meta::add(cpu_tensor, cuda_tensor) will fail for the same reason that if fails normally: most ops don't support mixed device inputs (except for cpu scalars, which we special case). I dunno if that's something we want to change though, since it seems useful to have the meta function perform the same set of metadata-related error checks that the operator normally runs (including device checks).

ezyang · 2021-05-21T04:36:44Z

Looks like functional is safe because we just unconditionally ignore the device specified in options of the set_output call

            if self.backend_index.dispatch_key == DispatchKey.Meta:
                # TODO: dedupe this with below
                return """
if (strides.empty()) {
    outputs_[output_idx] = at::empty(sizes, options.device(at::kMeta));
} else {
    outputs_[output_idx] = at::empty_strided(sizes, strides, options.device(at::kMeta));
}
"""

the out variant may need some rejiggering, it seems to always test the provided tensor is consistent

TORCH_CHECK(options.device() == out.device(),
    "Expected out tensor to have device ", options.device(), ", but got ", out.device(), " instead");

(this is just a bug, easy enough to fix)

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. [ghstack-poisoned]

bdhirsh · 2021-05-26T15:24:13Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-02T13:11:41Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-02T14:05:42Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-10T22:54:21Z

the out variant may need some rejiggering, it seems to always test the provided tensor is consistent
TORCH_CHECK(options.device() == out.device(),
    "Expected out tensor to have device ", options.device(), ", but got ", out.device(), " instead");
(this is just a bug, easy enough to fix)

I played around with this locally. For TI ops, I don't think that fixing up that torch check for at::meta:: helps because TI has its own checks (here) for a common device, and it looks like the check includes the out tensor if you provide one.

>>> a = torch.ones(2, dtype=torch.float)
>>> b = torch.ones(2, dtype=torch.float)
>>> out = torch.tensor(0, device='meta', dtype=torch.float)
>>> torch.add(a, b, out=out)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, meta and cpu!

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-10T23:34:23Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-11T00:51:09Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-11T15:14:35Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-11T21:43:32Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-14T14:08:13Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

bdhirsh · 2021-06-14T19:37:35Z

@bdhirsh has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

**What the PR does** Generate a fast-path `at::meta::{op}` API for calling meta functions without having to go through the dispatcher. This will be important for perf for external backends that want to use meta functions for shape checking (which seems likely to be what we end up doing for LazyTensorCore). **Details** In order to avoid naming collisions I had to make two small changes: - rename `MetaFunctions.h` template -> `NativeMetaFunctions.h` (this is the file that declares the impl() function for every structured operator). - rename the meta class: `at::meta::{op}::meta()` -> `at::meta::structured_{op}::meta()` I also deleted a few unnecessary includes, since any file that includes NativeFunctions.h will automatically include NativeMetaFunctions.h. **Why I made the change** This change isn't actually immediately used anywhere; I already started writing it because I thought it would be useful for structured composite ops, but that isn't actually true (see [comment](#58266 (comment))). The change feels useful and unambiguous though so I think it's safe to add. I added explicit tests for C++ meta function calls just to ensure that I wrote it correctly - which is actually how I hit the internal linkage issue in the PR below this in the stack. Differential Revision: [D28711299](https://our.internmc.facebook.com/intern/diff/D28711299) [ghstack-poisoned]

facebook-github-bot · 2021-06-15T23:55:11Z

@bdhirsh merged this pull request in 27a3204.

generate C++ API for meta functions using at::meta::

228d403

[ghstack-poisoned]

facebook-github-bot added the cla signed label May 19, 2021

dgl-intel pushed a commit to dgl-intel/pytorch that referenced this pull request May 19, 2021

generate C++ API for meta functions using at::meta::

eba4199

ghstack-source-id: 8cff248 Pull Request resolved: pytorch#58570

Update on "generate C++ API for meta functions using at::meta::"

cd57843

[ghstack-poisoned]

bdhirsh requested review from ezyang and bhosmer May 19, 2021 19:09

bdhirsh added 2 commits May 19, 2021 15:12

ezyang approved these changes May 20, 2021

View reviewed changes

bdhirsh mentioned this pull request May 27, 2021

beef up at::_ops API #59115

Closed

bdhirsh added 4 commits June 2, 2021 07:13

bdhirsh mentioned this pull request Jun 9, 2021

Add TorchScript lazy tensor backend #59582

Merged

bdhirsh mentioned this pull request Jun 10, 2021

move all external kernels into a class for better compiler error messages #59839

Closed

bdhirsh added 2 commits June 10, 2021 17:52

bdhirsh added 2 commits June 15, 2021 10:00

facebook-github-bot closed this in 27a3204 Jun 15, 2021

facebook-github-bot added the Merged label Jun 15, 2021

facebook-github-bot deleted the gh/bdhirsh/121/head branch June 19, 2021 14:17

generate C++ API for meta functions using at::meta:: #58570

generate C++ API for meta functions using at::meta:: #58570

Uh oh!

Conversation

bdhirsh commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

1 failure not recognized by patterns:

Uh oh!

bdhirsh commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

bdhirsh commented May 20, 2021

Uh oh!

ezyang commented May 21, 2021

Uh oh!

bdhirsh commented May 26, 2021

Uh oh!

bdhirsh commented Jun 2, 2021

Uh oh!

bdhirsh commented Jun 2, 2021

Uh oh!

bdhirsh commented Jun 10, 2021

Uh oh!

bdhirsh commented Jun 10, 2021

Uh oh!

bdhirsh commented Jun 11, 2021

Uh oh!

bdhirsh commented Jun 11, 2021

Uh oh!

bdhirsh commented Jun 11, 2021

Uh oh!

bdhirsh commented Jun 14, 2021

Uh oh!

bdhirsh commented Jun 14, 2021

Uh oh!

facebook-github-bot commented Jun 15, 2021

Uh oh!

Uh oh!

bdhirsh commented May 19, 2021 •

edited

Loading

facebook-github-bot commented May 19, 2021 •

edited

Loading

bdhirsh commented May 19, 2021 •

edited

Loading