Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make NumPy dependency dynamic #52794

Closed

Conversation

malfet
Copy link
Contributor

@malfet malfet commented Feb 24, 2021

Move NumPy initialization from initModule() to singleton inside
torch::utils::is_numpy_available() function.
This singleton will print a warning, that NumPy integration is not
available, rather than fails to import torch altogether.
The warning be printed only once, and will look something like the
following:

UserWarning: Failed to initialize NumPy: No module named 'numpy.core' (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:66.)

This is helpful if PyTorch was compiled with wrong NumPy version, of
NumPy is not commonly available on the platform (which is often the case
on AARCH64 or Apple M1)

Test that PyTorch is usable after numpy is uninstalled at the end of
_test1 CI config.

Move NumPy initialization from `initModule()` to singleton inside
`torch::utils::is_numpy_available()` function.
This singleton will print a warning, that NumPy integration is not
available, rather than fails to import torch altogether.
The warning be printed only once, and will look something like the
following:
```
UserWarning: Failed to initialize NumPy: No module named 'numpy.core' (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:66.)
```

This is helpful if PyTorch was compiled with wrong NumPy version, of
NumPy is not commonly available on the platform (which is often the case
on AARCH64 or Apple M1)

Test that PyTorch is usable after numpy is uninstalled at the end of
`_test1` CI config.
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Feb 24, 2021

💊 CI failures summary and remediations

As of commit 35dfec9 (more details on the Dr. CI page):


  • 2/2 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See GitHub Actions build clang-tidy (1/1)

Step: "Add annotations" (full log | diagnosis details | 🔁 rerun)

2021-02-25T23:51:09.6843704Z update-alternatives: error: no alternatives for nsight-sys
2021-02-25T23:51:09.6187288Z Setting up x11proto-damage-dev (1:2018.4-4) ...
2021-02-25T23:51:09.6230237Z Setting up cuda-nvjpeg-10-2 (10.2.89-1) ...
2021-02-25T23:51:09.6321623Z Setting up libxcb-dri3-dev:amd64 (1.13-2~ubuntu18.04) ...
2021-02-25T23:51:09.6364117Z Setting up libcublas10 (10.2.3.254-1) ...
2021-02-25T23:51:09.6401111Z Setting up libcublas-dev (10.2.3.254-1) ...
2021-02-25T23:51:09.6443805Z Setting up cuda-cufft-10-2 (10.2.89-1) ...
2021-02-25T23:51:09.6543125Z Setting up cuda-nsight-compute-10-2 (10.2.89-1) ...
2021-02-25T23:51:09.6593219Z Setting up nsight-systems-2020.4.3 (2020.4.3.7-10543b6) ...
2021-02-25T23:51:09.6740015Z update-alternatives: using /opt/nvidia/nsight-systems/2020.4.3/target-linux-x64/nsys to provide /usr/local/bin/nsys (nsys) in auto mode
2021-02-25T23:51:09.6833431Z update-alternatives: error: alternative path /opt/nvidia/nsight-systems/2020.4.3/host-linux-x64/nsight-sys doesn't exist
2021-02-25T23:51:09.6843704Z update-alternatives: error: no alternatives for nsight-sys
2021-02-25T23:51:09.6901663Z update-alternatives: using /opt/nvidia/nsight-systems/2020.4.3/host-linux-x64/nsys-ui to provide /usr/local/bin/nsys-ui (nsys-ui) in auto mode
2021-02-25T23:51:09.6962714Z Setting up libxcb-shape0-dev:amd64 (1.13-2~ubuntu18.04) ...
2021-02-25T23:51:09.7019557Z Setting up cuda-cusparse-10-2 (10.2.89-1) ...
2021-02-25T23:51:09.7124162Z Setting up cuda-cuobjdump-10-2 (10.2.89-1) ...
2021-02-25T23:51:09.7180299Z Setting up libxcb-glx0-dev:amd64 (1.13-2~ubuntu18.04) ...
2021-02-25T23:51:09.7234041Z Setting up libgles2:amd64 (1.0.0-2ubuntu2.3) ...
2021-02-25T23:51:09.7273055Z Setting up libglu1-mesa:amd64 (9.0.0-2.1build1) ...
2021-02-25T23:51:09.7322475Z Setting up libegl-mesa0:amd64 (20.0.8-0ubuntu1~18.04.1) ...
2021-02-25T23:51:09.7425625Z Setting up cuda-sanitizer-api-10-2 (10.2.89-1) ...
2021-02-25T23:51:09.7524542Z Setting up cuda-nvjpeg-dev-10-2 (10.2.89-1) ...

1 failure not recognized by patterns:

Job Step Action
CircleCI pytorch_linux_bionic_py3_6_clang9_test Run tests 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malfet has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

.jenkins/pytorch/test.sh Outdated Show resolved Hide resolved
Copy link
Collaborator

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. Note that there are also a bunch of unprotected numpy imports in Python code, e.g. in tensorboard, datapipes and ddp_comm_hooks. There's only one import which is protected with a try-except.

.jenkins/pytorch/test.sh Outdated Show resolved Hide resolved
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@malfet has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@malfet merged this pull request in a0a1bb0.

@malfet malfet deleted the malfet/make-numpy-dependency-dynamic branch February 26, 2021 20:23
aocsa pushed a commit to Quansight/pytorch that referenced this pull request Mar 15, 2021
Summary:
Move NumPy initialization from `initModule()` to singleton inside
`torch::utils::is_numpy_available()` function.
This singleton will print a warning, that NumPy integration is not
available, rather than fails to import torch altogether.
The warning be printed only once, and will look something like the
following:
```
UserWarning: Failed to initialize NumPy: No module named 'numpy.core' (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:66.)
```

This is helpful if PyTorch was compiled with wrong NumPy version, of
NumPy is not commonly available on the platform (which is often the case
on AARCH64 or Apple M1)

Test that PyTorch is usable after numpy is uninstalled at the end of
`_test1` CI config.

Pull Request resolved: pytorch#52794

Reviewed By: seemethere

Differential Revision: D26650509

Pulled By: malfet

fbshipit-source-id: a2d98769ef873862c3704be4afda075d76d3ad06
xsacha pushed a commit to xsacha/pytorch that referenced this pull request Mar 31, 2021
Summary:
Move NumPy initialization from `initModule()` to singleton inside
`torch::utils::is_numpy_available()` function.
This singleton will print a warning, that NumPy integration is not
available, rather than fails to import torch altogether.
The warning be printed only once, and will look something like the
following:
```
UserWarning: Failed to initialize NumPy: No module named 'numpy.core' (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:66.)
```

This is helpful if PyTorch was compiled with wrong NumPy version, of
NumPy is not commonly available on the platform (which is often the case
on AARCH64 or Apple M1)

Test that PyTorch is usable after numpy is uninstalled at the end of
`_test1` CI config.

Pull Request resolved: pytorch#52794

Reviewed By: seemethere

Differential Revision: D26650509

Pulled By: malfet

fbshipit-source-id: a2d98769ef873862c3704be4afda075d76d3ad06
dstein64 added a commit to dstein64/neuralart that referenced this pull request Jun 15, 2021
As of pytorch/pytorch#52794, numpy is a dynamic
dependency and is not installed automatically when installing torch.
dstein64 added a commit to dstein64/pastiche that referenced this pull request Jun 15, 2021
As of pytorch/pytorch#52794, numpy is a dynamic
dependency and is not installed automatically when installing torch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants