Making ops c10-full: ConstQuantizerPtr

[smessmer]

Stack from ghstack:

• Remove use_c10_dispatcher: full lines #49259 Remove use_c10_dispatcher: full lines
• Remove generated_unboxing_wrappers and setManuallyBoxedKernel #49251 Remove generated_unboxing_wrappers and setManuallyBoxedKernel
• Remove .impl_UNBOXED() and functionalities associated with it #49220 Remove .impl_UNBOXED() and functionalities associated with it
• Remove codegen logic to support non-c10-full ops #49164 Remove codegen logic to support non-c10-full ops
• Making ops c10-full: list of optional tensors #49138 [wip] Making ops c10-full: list of optional tensors
• Making ops c10-full: ConstQuantizerPtr #49150 Making ops c10-full: ConstQuantizerPtr
• Making ops c10-full: Storage arguments #49146 Making ops c10-full: Storage arguments
• Making ops c10-full: optional lists #49088 Making ops c10-full: optional lists
• Making ops c10 full: optional out arguments #49083 Making ops c10 full: optional out arguments

Add support for ConstQuantizerPtr to the JIT typing system and make ops that were blocked on that c10-full.

Differential Revision: D25457294

[dr-ci]

💊 CI failures summary and remediations

As of commit 9cb9a2c (more details on the Dr. CI page):

---

• 10/10 failures possibly* introduced in this PR
  • 2/10 non-CircleCI failure(s)

---

🕵️ 8 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_backward_compatibility_check_test (1/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Dec 15 23:41:27 The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not.

Dec 15 23:41:27 processing existing schema:  gather(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, Tensor _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  scatter(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor _1, Tensor[] _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  reduce_scatter(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor _1, Tensor[] _2) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  alltoall_base(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor _1, Tensor _2, int[] _3, int[] _4) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  alltoall(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, Tensor[] _2) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  send(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  recv(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  recv_anysource(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  barrier(__torch__.torch.classes.dist_c10d.ProcessGroup _0) -> (__torch__.torch.classes.dist_c10d.Work _0) 
Dec 15 23:41:27 processing existing schema:  __init__(__torch__.torch.classes.dist_rpc.WorkerInfo _0, str _1, int _2) -> (None _0) 
Dec 15 23:41:27 The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not.  
Dec 15 23:41:27  
Dec 15 23:41:27 Broken ops: [ 
Dec 15 23:41:27 	aten::set_quantizer_(Tensor(a!) self, int quantizer) -> (Tensor(a!)) 
Dec 15 23:41:27 	aten::set_.source_Storage(Tensor(a!) self, int source) -> (Tensor(a!)) 
Dec 15 23:41:27 	aten::set_.source_Storage_storage_offset(Tensor(a!) self, int source, int storage_offset, int[] size, int[] stride=[]) -> (Tensor(a!)) 
Dec 15 23:41:27 ] 
Dec 15 23:41:27 =================== sccache compilation log =================== 
Dec 15 23:41:27 + cleanup 
Dec 15 23:41:27 + retcode=1 
Dec 15 23:41:27 + set +x 

pytorch_linux_bionic_py3_6_clang9_test (2/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Dec 16 01:51:50 [E request_callback_no_python.cpp:636] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future

Dec 16 01:51:50 At: 
Dec 16 01:51:50   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(120): serialize 
Dec 16 01:51:50   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(172): serialize 
Dec 16 01:51:50  
Dec 16 01:51:50 [E request_callback_no_python.cpp:636] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future 
Dec 16 01:51:50  
Dec 16 01:51:50 At: 
Dec 16 01:51:50   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(120): serialize 
Dec 16 01:51:50   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(172): serialize 
Dec 16 01:51:50  
Dec 16 01:51:50 [E request_callback_no_python.cpp:636] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future 
Dec 16 01:51:50  
Dec 16 01:51:50 At: 
Dec 16 01:51:50   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(120): serialize 
Dec 16 01:51:50   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(172): serialize 
Dec 16 01:51:50  
Dec 16 01:51:50 ok (1.637s) 
Dec 16 01:51:51   test_return_future_remote (__main__.ProcessGroupRpcTestWithSpawn) ... RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Dec 16 01:51:51 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Dec 16 01:51:51 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Dec 16 01:51:51 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 

pytorch_linux_bionic_py3_8_gcc9_coverage_test2 (3/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Dec 16 00:22:06 unknown type specifier:

Dec 16 00:22:06   test_backward_compatible_structure (__main__.TestFunctionSchema) ... ok (0.004s) 
Dec 16 00:22:06   test_serialize_and_deserialize (__main__.TestFunctionSchema) ... ERROR (0.039s) 
Dec 16 00:22:06  
Dec 16 00:22:06 ====================================================================== 
Dec 16 00:22:06 ERROR [0.039s]: test_serialize_and_deserialize (__main__.TestFunctionSchema) 
Dec 16 00:22:06 ---------------------------------------------------------------------- 
Dec 16 00:22:06 Traceback (most recent call last): 
Dec 16 00:22:06   File "test_function_schema.py", line 13, in test_serialize_and_deserialize 
Dec 16 00:22:06     parsed_schema = parse_schema(str(schema)) 
Dec 16 00:22:06 RuntimeError:  
Dec 16 00:22:06 unknown type specifier: 
Dec 16 00:22:06 aten::set_quantizer_(Tensor(a!) self, QuantizerPtr quantizer) -> (Tensor(a!)) 
Dec 16 00:22:06                                       ~~~~~~~~~~~~ <--- HERE 
Dec 16 00:22:06  
Dec 16 00:22:06  
Dec 16 00:22:06 ---------------------------------------------------------------------- 
Dec 16 00:22:06 Ran 4 tests in 0.152s 
Dec 16 00:22:06  
Dec 16 00:22:06 FAILED (errors=1) 
Dec 16 00:22:06  
Dec 16 00:22:06 Generating XML reports... 

pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test2 (4/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Dec 16 02:27:55 unknown type specifier:

Dec 16 02:27:55   test_backward_compatible_structure (__main__.TestFunctionSchema) ... ok (0.001s) 
Dec 16 02:27:55   test_serialize_and_deserialize (__main__.TestFunctionSchema) ... ERROR (0.022s) 
Dec 16 02:27:55  
Dec 16 02:27:55 ====================================================================== 
Dec 16 02:27:55 ERROR [0.022s]: test_serialize_and_deserialize (__main__.TestFunctionSchema) 
Dec 16 02:27:55 ---------------------------------------------------------------------- 
Dec 16 02:27:55 Traceback (most recent call last): 
Dec 16 02:27:55   File "test_function_schema.py", line 13, in test_serialize_and_deserialize 
Dec 16 02:27:55     parsed_schema = parse_schema(str(schema)) 
Dec 16 02:27:55 RuntimeError:  
Dec 16 02:27:55 unknown type specifier: 
Dec 16 02:27:55 aten::set_quantizer_(Tensor(a!) self, QuantizerPtr quantizer) -> (Tensor(a!)) 
Dec 16 02:27:55                                       ~~~~~~~~~~~~ <--- HERE 
Dec 16 02:27:55  
Dec 16 02:27:55  
Dec 16 02:27:55 ---------------------------------------------------------------------- 
Dec 16 02:27:55 Ran 4 tests in 0.075s 
Dec 16 02:27:55  
Dec 16 02:27:55 FAILED (errors=1) 
Dec 16 02:27:55  
Dec 16 02:27:55 Generating XML reports... 

pytorch_linux_xenial_py3_6_gcc5_4_test (5/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Dec 16 00:37:45 [E request_callback_no_python.cpp:636] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future

Dec 16 00:37:45 At: 
Dec 16 00:37:45   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(120): serialize 
Dec 16 00:37:45   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(172): serialize 
Dec 16 00:37:45  
Dec 16 00:37:45 [E request_callback_no_python.cpp:636] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future 
Dec 16 00:37:45  
Dec 16 00:37:45 At: 
Dec 16 00:37:45   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(120): serialize 
Dec 16 00:37:45   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(172): serialize 
Dec 16 00:37:45  
Dec 16 00:37:45 [E request_callback_no_python.cpp:636] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future 
Dec 16 00:37:45  
Dec 16 00:37:45 At: 
Dec 16 00:37:45   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(120): serialize 
Dec 16 00:37:45   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(172): serialize 
Dec 16 00:37:45  
Dec 16 00:37:45 ok (1.640s) 
Dec 16 00:37:47   test_return_future_remote (__main__.ProcessGroupRpcTestWithSpawn) ... RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Dec 16 00:37:47 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Dec 16 00:37:47 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Dec 16 00:37:47 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 

pytorch_macos_10_13_py3_test (6/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Dec 16 01:41:11 RuntimeError: test_function_schema failed!

Dec 16 01:41:11  
Dec 16 01:41:11 FAILED (errors=1) 
Dec 16 01:41:11  
Dec 16 01:41:11 Generating XML reports... 
Dec 16 01:41:11 Generated XML report: test-reports/dist-gloo/TEST-TestFunctionSchema-20201216014111.xml 
Dec 16 01:41:11 Traceback (most recent call last): 
Dec 16 01:41:11   File "test/run_test.py", line 897, in <module> 
Dec 16 01:41:11     main() 
Dec 16 01:41:11   File "test/run_test.py", line 880, in main 
Dec 16 01:41:11     raise RuntimeError(err_message) 
Dec 16 01:41:11 RuntimeError: test_function_schema failed! 
Dec 16 01:41:11 + cleanup 
Dec 16 01:41:11 + retcode=1 
Dec 16 01:41:11 + set +x 

pytorch_windows_vs2019_py36_cuda10.1_test2 (7/8)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_function_schema failed!

 
FAILED (errors=1) 
 
Generating XML reports... 
Generated XML report: test-reports\dist-gloo\TEST-TestFunctionSchema-20201216021954.xml 
Traceback (most recent call last): 
  File "run_test.py", line 897, in <module> 
    main() 
  File "run_test.py", line 880, in main 
    raise RuntimeError(err_message) 
RuntimeError: test_function_schema failed! 
 
(base) circleci@PACKER-5FAA87D3 C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

pytorch_linux_xenial_py3_clang5_asan_test2 (8/8)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Dec 16 01:24:52 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:15:3 in

Dec 16 01:24:52     #7 0x55d9f92ce70b in PyEval_EvalCode /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:731 
Dec 16 01:24:52     #8 0x55d9f934e573 in run_mod /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:1025 
Dec 16 01:24:52     #9 0x55d9f934e60c in PyRun_StringFlags /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:949 
Dec 16 01:24:52     #10 0x55d9f934e66e in PyRun_SimpleStringFlags /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:445 
Dec 16 01:24:52     #11 0x55d9f9352472 in run_command /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:301 
Dec 16 01:24:52     #12 0x55d9f9352472 in Py_Main /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:749 
Dec 16 01:24:52     #13 0x55d9f921c43d in main /tmp/build/80754af9/python_1599604603603/work/Programs/python.c:69 
Dec 16 01:24:52     #14 0x7fe7c0c6483f in __libc_start_main /build/glibc-e6zv40/glibc-2.23/csu/../csu/libc-start.c:291 
Dec 16 01:24:52     #15 0x55d9f92fbd0a in _start /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103 
Dec 16 01:24:52  
Dec 16 01:24:52 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/Utils.cpp:15:3 in  
Dec 16 01:24:52 + retcode=1 
Dec 16 01:24:52 + set -e 
Dec 16 01:24:52 + return 1 
Dec 16 01:24:52 + [[ pytorch-linux-xenial-py3-clang5-asan-test2 == *-NO_AVX-* ]] 
Dec 16 01:24:52 + [[ pytorch-linux-xenial-py3-clang5-asan-test2 == *-NO_AVX2-* ]] 
Dec 16 01:24:52 + '[' -n https://github.com/pytorch/pytorch/pull/49150 ']' 
Dec 16 01:24:52 + [[ pytorch-linux-xenial-py3-clang5-asan-test2 != *coverage* ]] 
Dec 16 01:24:52 ++ mktemp 
Dec 16 01:24:52 + DETERMINE_FROM=/tmp/tmp.lNzXPaAMEd 
Dec 16 01:24:52 + file_diff_from_base /tmp/tmp.lNzXPaAMEd 

---

Extra GitHub checks: 1 failed

• Failed: GitHub Actions - clang-format

---

ci.pytorch.org: 1 failed

• Failed: pr/pytorch-linux-bionic-rocm3.9-py3.6

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 126 times.

[ezyang]

QuantizerPtr or Quantizer? We don't say TensorPtr.

[smessmer]

replaced by #49463