Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow converting CharTensor to np arrays #14710

Closed
wants to merge 1 commit into from
Closed

Allow converting CharTensor to np arrays #14710

wants to merge 1 commit into from

Conversation

ssnl
Copy link
Collaborator

@ssnl ssnl commented Dec 3, 2018

The other direction of #14700

cc @soumith

@ssnl ssnl changed the title Allow converting ShorTensor and CharTensor to np arrays Allow converting CharTensor to np arrays Dec 3, 2018
facebook-github-bot
facebook-github-bot reviewed Dec 3, 2018
View reviewed changes
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soumith is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ezyang
Copy link
Contributor

ezyang commented Dec 10, 2018

Unfortunately this diff causes UBSAN failures:

Dec 03 19:14:22 test_toNumpy (__main__.TestTorch) ... /var/lib/jenkins/workspace/aten/src/ATen/native/Copy.cpp:16:64: runtime error: 487.474 is outside the range of representable values of type 'signed char'
Dec 03 19:14:22     #0 0x7f42e7356f45 in void (anonymous namespace)::_copy__cpu<signed char>(at::Tensor&, at::Tensor const&)::{lambda()#1}::operator()() const (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x672df45)
Dec 03 19:14:22     #1 0x7f42e73359ad in at::native::_s_copy__cpu(at::Tensor&, at::Tensor const&, bool) (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x670c9ad)
Dec 03 19:14:22     #2 0x7f42e751de2a in at::CPUCharType::s_copy_(at::Tensor&, at::Tensor const&, bool) const (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x68f4e2a)
Dec 03 19:14:22     #3 0x7f42de59a04b in torch::autograd::VariableType::s_copy_(at::Tensor&, at::Tensor const&, bool) const (/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1+0x1c8704b)
Dec 03 19:14:22     #4 0x7f42e784e70c in at::TypeDefault::copy_(at::Tensor&, at::Tensor const&, bool) const (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x6c2570c)
Dec 03 19:14:22     #5 0x7f42e6f61bdb in at::Tensor::copy_(at::Tensor const&, bool) (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x6338bdb)
Dec 03 19:14:22     #6 0x7f42e784ef18 in at::TypeDefault::copy(at::Tensor const&, bool, c10::optional<c10::Device>) const (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x6c25f18)
Dec 03 19:14:22     #7 0x7f42e6f65f42 in at::native::_cast_Char(at::Tensor const&, bool) (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x633cf42)
Dec 03 19:14:22     #8 0x7f42e7900f26 in at::TypeDefault::_cast_Char(at::Tensor const&, bool) const (/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so+0x6cd7f26)
Dec 03 19:14:22     #9 0x7f42de3fdb4e in torch::autograd::VariableType::_cast_Char(at::Tensor const&, bool) const (/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so.1+0x1aeab4e)
Dec 03 19:14:22     #10 0x7f42f5b90548 in torch::utils::dispatch_type_conversion(at::Tensor const&, at::Type const&, c10::optional<c10::Device>, bool) (/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so+0x1591548)
Dec 03 19:14:22     #11 0x7f42f54b3c69 in torch::autograd::THPVariable_type(_object*, _object*, _object*) (/opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so+0xeb4c69)
Dec 03 19:14:22     #12 0x55ccc5ced3d3 in _PyCFunction_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/methodobject.c:231
Dec 03 19:14:22     #13 0x55ccc5d76c1b in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4837
Dec 03 19:14:22     #14 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #15 0x55ccc5d7042d in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #16 0x55ccc5d70fc0 in fast_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4978
Dec 03 19:14:22     #17 0x55ccc5d76cf4 in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4858
Dec 03 19:14:22     #18 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #19 0x55ccc5d70123 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #20 0x55ccc5d716b6 in _PyFunction_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5070
Dec 03 19:14:22     #21 0x55ccc5ced79e in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2310
Dec 03 19:14:22     #22 0x55ccc5cf2302 in _PyObject_Call_Prepend /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2373
Dec 03 19:14:22     #23 0x55ccc5ced1dd in PyObject_Call /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2261
Dec 03 19:14:22     #24 0x55ccc5d9ad99 in do_call_core /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5106
Dec 03 19:14:22     #25 0x55ccc5d9ad99 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3404
Dec 03 19:14:22     #26 0x55ccc5d70123 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #27 0x55ccc5d7149b in _PyFunction_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5070
Dec 03 19:14:22     #28 0x55ccc5ced79e in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2310
Dec 03 19:14:22     #29 0x55ccc5cf2302 in _PyObject_Call_Prepend /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2373
Dec 03 19:14:22     #30 0x55ccc5ced1dd in PyObject_Call /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2261
Dec 03 19:14:22     #31 0x55ccc5d4b126 in slot_tp_call /tmp/build/80754af9/python_1540319457073/work/Objects/typeobject.c:6207
Dec 03 19:14:22     #32 0x55ccc5ced5ba in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2331
Dec 03 19:14:22     #33 0x55ccc5d76d6d in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4861
Dec 03 19:14:22     #34 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #35 0x55ccc5d70123 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #36 0x55ccc5d716b6 in _PyFunction_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5070
Dec 03 19:14:22     #37 0x55ccc5ced79e in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2310
Dec 03 19:14:22     #38 0x55ccc5cf2302 in _PyObject_Call_Prepend /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2373
Dec 03 19:14:22     #39 0x55ccc5ced1dd in PyObject_Call /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2261
Dec 03 19:14:22     #40 0x55ccc5d9ad99 in do_call_core /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5106
Dec 03 19:14:22     #41 0x55ccc5d9ad99 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3404
Dec 03 19:14:22     #42 0x55ccc5d70123 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #43 0x55ccc5d7149b in _PyFunction_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5070
Dec 03 19:14:22     #44 0x55ccc5ced79e in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2310
Dec 03 19:14:22     #45 0x55ccc5cf2302 in _PyObject_Call_Prepend /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2373
Dec 03 19:14:22     #46 0x55ccc5ced1dd in PyObject_Call /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2261
Dec 03 19:14:22     #47 0x55ccc5d4b126 in slot_tp_call /tmp/build/80754af9/python_1540319457073/work/Objects/typeobject.c:6207
Dec 03 19:14:22     #48 0x55ccc5ced5ba in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2331
Dec 03 19:14:22     #49 0x55ccc5d76d6d in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4861
Dec 03 19:14:22     #50 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #51 0x55ccc5d70d8a in _PyFunction_FastCall /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4919
Dec 03 19:14:22     #52 0x55ccc5d70d8a in fast_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4954
Dec 03 19:14:22     #53 0x55ccc5d76cf4 in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4858
Dec 03 19:14:22     #54 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #55 0x55ccc5d70d8a in _PyFunction_FastCall /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4919
Dec 03 19:14:22     #56 0x55ccc5d70d8a in fast_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4954
Dec 03 19:14:22     #57 0x55ccc5d76cf4 in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4858
Dec 03 19:14:22     #58 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #59 0x55ccc5d70620 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #60 0x55ccc5d716b6 in _PyFunction_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:5070
Dec 03 19:14:22     #61 0x55ccc5ced79e in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2310
Dec 03 19:14:22     #62 0x55ccc5cf2302 in _PyObject_Call_Prepend /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2373
Dec 03 19:14:22     #63 0x55ccc5ced1dd in PyObject_Call /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2261
Dec 03 19:14:22     #64 0x55ccc5d4a78a in slot_tp_init /tmp/build/80754af9/python_1540319457073/work/Objects/typeobject.c:6420
Dec 03 19:14:22     #65 0x55ccc5d76f56 in type_call /tmp/build/80754af9/python_1540319457073/work/Objects/typeobject.c:915
Dec 03 19:14:22     #66 0x55ccc5ced5ba in _PyObject_FastCallDict /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2331
Dec 03 19:14:22     #67 0x55ccc5d712a9 in _PyObject_FastCallKeywords /tmp/build/80754af9/python_1540319457073/work/Objects/abstract.c:2496
Dec 03 19:14:22     #68 0x55ccc5d76d6d in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4861
Dec 03 19:14:22     #69 0x55ccc5d9a4d7 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3351
Dec 03 19:14:22     #70 0x55ccc5d1c2ea in _PyFunction_FastCall /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4919
Dec 03 19:14:22     #71 0x55ccc5d76cf4 in call_function /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4858
Dec 03 19:14:22     #72 0x55ccc5d99719 in _PyEval_EvalFrameDefault /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:3335
Dec 03 19:14:22     #73 0x55ccc5d71ad8 in _PyEval_EvalCodeWithName /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4166
Dec 03 19:14:22     #74 0x55ccc5d71ad8 in PyEval_EvalCodeEx /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:4187
Dec 03 19:14:22     #75 0x55ccc5d7287b in PyEval_EvalCode /tmp/build/80754af9/python_1540319457073/work/Python/ceval.c:731
Dec 03 19:14:22     #76 0x55ccc5df3073 in run_mod /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:1025
Dec 03 19:14:22     #77 0x55ccc5df3470 in PyRun_FileExFlags /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:978
Dec 03 19:14:22     #78 0x55ccc5df3672 in PyRun_SimpleFileExFlags /tmp/build/80754af9/python_1540319457073/work/Python/pythonrun.c:419
Dec 03 19:14:22     #79 0x55ccc5df71af in run_file /tmp/build/80754af9/python_1540319457073/work/Modules/main.c:340
Dec 03 19:14:22     #80 0x55ccc5df71af in Py_Main /tmp/build/80754af9/python_1540319457073/work/Modules/main.c:811
Dec 03 19:14:22     #81 0x55ccc5cbeb4d in main /tmp/build/80754af9/python_1540319457073/work/Programs/python.c:69
Dec 03 19:14:22     #82 0x7f43081e782f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
Dec 03 19:14:22     #83 0x55ccc5da01a7 in _start /home/rdonnelly/mc/conda-bld/compilers_linux-64_1534865402226/work/.build/src/glibc-2.12.2/csu/../sysdeps/x86_64/elf/start.S:103
Dec 03 19:14:22 
Dec 03 19:14:22 SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /var/lib/jenkins/workspace/aten/src/ATen/native/Copy.cpp:16:64 in

@ezyang
Copy link
Contributor

ezyang commented Dec 10, 2018

Unlanding

@ssnl
Copy link
Collaborator Author

ssnl commented Dec 11, 2018

@ezyang Sorry for breaking UBSAN. So it seems that we don't check for UB when converting tensors to a different type (tensor(487.474).type(torch.CharTensor) caused the failure). Should we do the check? cc @gchanan

@ssnl ssnl mentioned this pull request Dec 11, 2018
Closed
facebook-github-bot pushed a commit that referenced this pull request Dec 24, 2018
@ssnl @facebook-github-bot
Allow converting char tensor to numpy; add [fi]info.min (#15046)
521894c 
Summary:
#14710 with test fixed.

Also added `finfo.min` and `iinfo.min` to get castable tensors.

cc soumith
Pull Request resolved: #15046

Reviewed By: soumith

Differential Revision: D13429388

Pulled By: SsnL

fbshipit-source-id: 9a08004419c83bc5ef51d03b6df3961a9f5dbf47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

@soumith soumith soumith approved these changes

Assignees
No one assigned
Labels
open source
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@ssnl @ezyang @soumith @facebook-github-bot