Clean up tensor_util and torch_util #68427

desertfire · 2021-11-16T14:59:47Z

Merge torch_util into tensor_util to reduce the number of util files
Remove unused functions in tensor_util.cpp
Move TensorHash into hash.h
Move TensorCompare into lazy_graph_executor.cpp

- Merge torch_util into tensor_util to reduce the number of util files - Remove unused functions in tensor_util.cpp - Move TensorHash into hash.h - Move TensorCompare into lazy_graph_executor.cpp

pytorch-probot · 2021-11-16T14:59:50Z

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/99b6d487eb720ea406dd09a628931b376656e185/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/noarch`, `ciflow/xla`	✅ triggered
linux-vulkan-bionic-py3.6-clang9	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/vulkan`	✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3-clang5-mobile-build	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	`ciflow/all`, `ciflow/default`, `ciflow/linux`, `ciflow/mobile`	✅ triggered
linux-xenial-py3.6-clang7-asan	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/sanitizers`	✅ triggered
linux-xenial-py3.6-clang7-onnx	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`, `ciflow/onnx`	✅ triggered
linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc7	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
linux-xenial-py3.6-gcc7-bazel-test	`ciflow/all`, `ciflow/bazel`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
pytorch-linux-xenial-py3-clang5-android-ndk-r19c-gradle-custom-build-single-full-jit	`ciflow/all`, `ciflow/android`, `ciflow/cpu`, `ciflow/default`, `ciflow/linux`	✅ triggered
win-vs2019-cpu-py3	`ciflow/all`, `ciflow/cpu`, `ciflow/default`, `ciflow/win`	✅ triggered
win-vs2019-cuda11.3-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/default`, `ciflow/win`	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`	🚫 skipped
docker-builds	`ciflow/all`	🚫 skipped
ios-12-5-1-arm64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-custom-ops	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-arm64-metal	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-x86-64	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-x86-64-coreml	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
ios-12-5-1-x86-64-full-jit	`ciflow/all`, `ciflow/ios`, `ciflow/macos`	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/slow`	🚫 skipped
macos-10-15-py3-arm64	`ciflow/all`, `ciflow/macos`	🚫 skipped
macos-10-15-py3-lite-interpreter-x86-64	`ciflow/all`, `ciflow/macos`	🚫 skipped
macos-10-15-py3-x86-64	`ciflow/all`, `ciflow/macos`	🚫 skipped
parallelnative-linux-xenial-py3.6-gcc5.4	`ciflow/all`, `ciflow/cpu`, `ciflow/linux`	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/libtorch`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`, `ciflow/slow`, `ciflow/slow-gradcheck`	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7	`ciflow/all`, `ciflow/cuda`, `ciflow/linux`, `ciflow/scheduled`	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	`ciflow/all`, `ciflow/cuda`, `ciflow/scheduled`, `ciflow/win`	🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands:

# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

facebook-github-bot · 2021-11-16T14:59:54Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/68427
📄 Preview docs built from this PR
📄 Preview C++ docs built from this PR
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 99b6d48 (more details on the Dr. CI page):

4/4 failures possibly* introduced in this PR
- 1/4 non-scanned failure(s)

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

linux-xenial-py3.6-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2021-11-16T15:15:50.8926698Z The PR is introduc...m to confirm whether this change is wanted or not.

2021-11-16T15:15:50.8913131Z processing existing schema:  alltoall_base(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor _1, Tensor _2, int[] _3, int[] _4) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-16T15:15:50.8914536Z processing existing schema:  alltoall(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, Tensor[] _2) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-16T15:15:50.8915875Z processing existing schema:  send(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-16T15:15:50.8917200Z processing existing schema:  recv(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-16T15:15:50.8918563Z processing existing schema:  recv_anysource(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-16T15:15:50.8919858Z processing existing schema:  barrier(__torch__.torch.classes.dist_c10d.ProcessGroup _0) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-16T15:15:50.8920966Z processing existing schema:  __init__(__torch__.torch.classes.dist_c10d.frontend _0) -> (NoneType _0)
2021-11-16T15:15:50.8922427Z processing existing schema:  new_process_group_helper(__torch__.torch.classes.dist_c10d.frontend _0, int _1, int _2, int[] _3, str _4, __torch__.torch.classes.dist_c10d.Store _5, str? _6, int _7) -> (__torch__.torch.classes.dist_c10d.ProcessGroup _0)
2021-11-16T15:15:50.8924021Z processing existing schema:  get_process_group_by_name(__torch__.torch.classes.dist_c10d.frontend _0, str _1) -> (__torch__.torch.classes.dist_c10d.ProcessGroup _0)
2021-11-16T15:15:50.8925488Z processing existing schema:  get_name_of_process_group(__torch__.torch.classes.dist_c10d.frontend _0, __torch__.torch.classes.dist_c10d.ProcessGroup _1) -> (str _0)
2021-11-16T15:15:50.8926698Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2021-11-16T15:15:50.8927316Z 
2021-11-16T15:15:50.8927569Z Broken ops: [
2021-11-16T15:15:50.8928525Z 	aten::_upsample_nearest_exact2d_backward(Tensor grad_output, int[2] output_size, int[4] input_size, float? scales_h=None, float? scales_w=None) -> (Tensor)
2021-11-16T15:15:50.8929881Z 	aten::_upsample_nearest_exact2d_backward.grad_input(Tensor grad_output, int[2] output_size, int[4] input_size, float? scales_h=None, float? scales_w=None, *, Tensor(a!) grad_input) -> (Tensor(a!))
2021-11-16T15:15:50.8931198Z 	aten::_upsample_nearest_exact2d_backward.vec(Tensor grad_output, int[]? output_size, int[] input_size, float[]? scale_factors) -> (Tensor)
2021-11-16T15:15:50.8932409Z 	aten::_upsample_nearest_exact1d_backward(Tensor grad_output, int[1] output_size, int[3] input_size, float? scales=None) -> (Tensor)
2021-11-16T15:15:50.8933652Z 	aten::_upsample_nearest_exact1d_backward.grad_input(Tensor grad_output, int[1] output_size, int[3] input_size, float? scales=None, *, Tensor(a!) grad_input) -> (Tensor(a!))
2021-11-16T15:15:50.8934905Z 	aten::_upsample_nearest_exact1d_backward.vec(Tensor grad_output, int[]? output_size, int[] input_size, float[]? scale_factors) -> (Tensor)
2021-11-16T15:15:50.8936082Z 	aten::_upsample_nearest_exact3d_backward.vec(Tensor grad_output, int[]? output_size, int[] input_size, float[]? scale_factors) -> (Tensor)
2021-11-16T15:15:50.8937366Z 	aten::_upsample_nearest_exact3d_backward(Tensor grad_output, int[3] output_size, int[5] input_size, float? scales_d=None, float? scales_h=None, float? scales_w=None) -> (Tensor)

2 failures not recognized by patterns:

Job	Step	Action
^{Lint / mypy}	^{Run mypy}	🔁 rerun
^{Lint / flake8-py3}	^{Fail if there were any warnings}	🔁 rerun

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm4.3.1-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

wconstab · 2021-11-16T19:14:23Z

torch/csrc/lazy/core/hash.h

+  int64_t size = ctensor.numel() * ctensor.element_size();
+  switch (ctensor.scalar_type()) {
+    case at::ScalarType::Bool:
+      return DataHash(ctensor.data_ptr<bool>(), size);


this seems needlessly complicated doesn't it? wonder why we can't just

assert scalar_type() is supported (maybe not necessary, or maybe use c10 util for it)

treat data_ptr as uint8_t* and call DataHAsh

not sure why it was necessary to handle all the cases separately.. but maybe it's better not to change it now...

Summary: Some changes to torch/csrc/lazy/core were done on the lazy_tensor_staging branch (#68427). Merge those back into the trunck. [ghstack-poisoned]

…/lazy/core" Summary: Some changes to torch/csrc/lazy/core were done on the lazy_tensor_staging branch (#68427). Merge those back into the trunck. [ghstack-poisoned]

Summary: Some changes to torch/csrc/lazy/core were done on the lazy_tensor_staging branch (#68427). Merge those back into the trunck. [ghstack-poisoned]

Summary: Some changes to torch/csrc/lazy/core were done on the lazy_tensor_staging branch (#68427). Merge those back into the trunck. ghstack-source-id: 06d1ee7 Pull Request resolved: #69012

Summary: Pull Request resolved: #69012 Some changes to torch/csrc/lazy/core were done on the lazy_tensor_staging branch (#68427). Merge those back into the trunk. Test Plan: Imported from OSS Reviewed By: wconstab Differential Revision: D32708696 Pulled By: desertfire fbshipit-source-id: e54b978f2bdb9c7db27880f60246fdf1e8b41019

Clean up tensor_util and torch_util

99b6d48

- Merge torch_util into tensor_util to reduce the number of util files - Remove unused functions in tensor_util.cpp - Move TensorHash into hash.h - Move TensorCompare into lazy_graph_executor.cpp

pytorch-probot bot added the ciflow/default label Nov 16, 2021

facebook-github-bot added the cla signed label Nov 16, 2021

desertfire requested review from alanwaketan and wconstab November 16, 2021 18:33

wconstab reviewed Nov 16, 2021

View reviewed changes

wconstab approved these changes Nov 16, 2021

View reviewed changes

desertfire merged commit cd63069 into lazy_tensor_staging Nov 17, 2021

desertfire mentioned this pull request Nov 29, 2021

[LT] Sync LTC branch changes on torch/csrc/lazy/core #69012

Closed

desertfire added a commit that referenced this pull request Nov 29, 2021

[LT] Sync LTC branch changes on torch/csrc/lazy/core

6462c76

Summary: Some changes to torch/csrc/lazy/core were done on the lazy_tensor_staging branch (#68427). Merge those back into the trunck. [ghstack-poisoned]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Clean up tensor_util and torch_util #68427

Clean up tensor_util and torch_util #68427

Uh oh!

desertfire commented Nov 16, 2021

Uh oh!

pytorch-probot bot commented Nov 16, 2021

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Nov 16, 2021 •

edited

Loading

Uh oh!

wconstab Nov 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Clean up tensor_util and torch_util #68427

Clean up tensor_util and torch_util #68427

Uh oh!

Conversation

desertfire commented Nov 16, 2021

Uh oh!

pytorch-probot bot commented Nov 16, 2021

⚛️ CI Flow

Uh oh!

facebook-github-bot commented Nov 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

linux-xenial-py3.6-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (1/1)

2 failures not recognized by patterns:

ci.pytorch.org: 1 failed

Uh oh!

wconstab Nov 16, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

facebook-github-bot commented Nov 16, 2021 •

edited

Loading