Fix failing test due to a bug in NumPy when using OpenBLAS

[lezcano]

Stack from ghstack:

• -> Fix failing test due to a bug in NumPy when using OpenBLAS #67679

implementations

Fixes #67675

cc @mruberry

Differential Revision: D32368698

[pytorch-probot]

CI Flow Status

⚛️ CI Flow

Ruleset - Version: v1
Ruleset - File: https://github.com/pytorch/pytorch/blob/a63ee85d5406c6a39436f28f4ab3a056d00b5237/.github/generated-ciflow-ruleset.json
PR ciflow labels: ciflow/default

Workflows	Labels (bold enabled)	Status
Triggered Workflows
linux-bionic-py3.6-clang9	ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/noarch, ciflow/xla	✅ triggered
linux-vulkan-bionic-py3.6-clang9	ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/vulkan	✅ triggered
linux-xenial-cuda11.3-py3.6-gcc7	ciflow/all, ciflow/cuda, ciflow/default, ciflow/linux	✅ triggered
linux-xenial-py3-clang5-mobile-build	ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-dynamic	ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile	✅ triggered
linux-xenial-py3-clang5-mobile-custom-build-static	ciflow/all, ciflow/default, ciflow/linux, ciflow/mobile	✅ triggered
linux-xenial-py3.6-clang7-asan	ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/sanitizers	✅ triggered
linux-xenial-py3.6-clang7-onnx	ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux, ciflow/onnx	✅ triggered
linux-xenial-py3.6-gcc5.4	ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux	✅ triggered
linux-xenial-py3.6-gcc7	ciflow/all, ciflow/cpu, ciflow/default, ciflow/linux	✅ triggered
linux-xenial-py3.6-gcc7-bazel-test	ciflow/all, ciflow/bazel, ciflow/cpu, ciflow/default, ciflow/linux	✅ triggered
win-vs2019-cpu-py3	ciflow/all, ciflow/cpu, ciflow/default, ciflow/win	✅ triggered
win-vs2019-cuda11.3-py3	ciflow/all, ciflow/cuda, ciflow/default, ciflow/win	✅ triggered
Skipped Workflows
caffe2-linux-xenial-py3.6-gcc5.4	ciflow/all, ciflow/cpu, ciflow/linux	🚫 skipped
docker-builds	ciflow/all	🚫 skipped
libtorch-linux-xenial-cuda10.2-py3.6-gcc7	ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux	🚫 skipped
libtorch-linux-xenial-cuda11.3-py3.6-gcc7	ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux	🚫 skipped
linux-bionic-cuda10.2-py3.9-gcc7	ciflow/all, ciflow/cuda, ciflow/linux, ciflow/slow	🚫 skipped
linux-xenial-py3-clang5-mobile-code-analysis	ciflow/all, ciflow/linux, ciflow/mobile	🚫 skipped
parallelnative-linux-xenial-py3.6-gcc5.4	ciflow/all, ciflow/cpu, ciflow/linux	🚫 skipped
periodic-libtorch-linux-xenial-cuda11.1-py3.6-gcc7	ciflow/all, ciflow/cuda, ciflow/libtorch, ciflow/linux, ciflow/scheduled	🚫 skipped
periodic-linux-xenial-cuda10.2-py3-gcc7-slow-gradcheck	ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled, ciflow/slow, ciflow/slow-gradcheck	🚫 skipped
periodic-linux-xenial-cuda11.1-py3.6-gcc7	ciflow/all, ciflow/cuda, ciflow/linux, ciflow/scheduled	🚫 skipped
periodic-win-vs2019-cuda11.1-py3	ciflow/all, ciflow/cuda, ciflow/scheduled, ciflow/win	🚫 skipped

You can add a comment to the PR and tag @pytorchbot with the following commands:

# ciflow rerun, "ciflow/default" will always be added automatically
@pytorchbot ciflow rerun

# ciflow rerun with additional labels "-l <ciflow/label_name>", which is equivalent to adding these labels manually and trigger the rerun
@pytorchbot ciflow rerun -l ciflow/scheduled -l ciflow/slow

For more information, please take a look at the CI Flow Wiki.

[facebook-github-bot]

🔗 Helpful links

• 🧪  See artifacts and rendered test results at hud.pytorch.org/pr/67679
• 📄  Preview docs built from this PR
• 📄  Preview C++ docs built from this PR
• 🔧  Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit a63ee85 (more details on the Dr. CI page):

---

• 3/3 failures introduced in this PR

---

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_py3_6_gcc5_4_build (1/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/scripts/cpp_doc_push_script.sh
Auto-merging .circleci/scripts/cpp_doc_push_script.sh
CONFLICT (add/add): Merge conflict in .circleci/generate_config_yml.py
Auto-merging .circleci/generate_config_yml.py
CONFLICT (add/add): Merge conflict in .circleci/docker/common/install_rocm.sh
Auto-merging .circleci/docker/common/install_rocm.sh
CONFLICT (add/add): Merge conflict in .circleci/config.yml
Auto-merging .circleci/config.yml
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/android_definitions.py
Auto-merging .circleci/cimodel/data/simple/android_definitions.py
Automatic merge failed; fix conflicts and then commit the result.


Exited with code exit status 1

pytorch_xla_linux_bionic_py3_6_clang9_build (2/3)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.

CONFLICT (add/add): Merge conflict in .circleci/scripts/cpp_doc_push_script.sh
Auto-merging .circleci/scripts/cpp_doc_push_script.sh
CONFLICT (add/add): Merge conflict in .circleci/generate_config_yml.py
Auto-merging .circleci/generate_config_yml.py
CONFLICT (add/add): Merge conflict in .circleci/docker/common/install_rocm.sh
Auto-merging .circleci/docker/common/install_rocm.sh
CONFLICT (add/add): Merge conflict in .circleci/config.yml
Auto-merging .circleci/config.yml
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/android_definitions.py
Auto-merging .circleci/cimodel/data/simple/android_definitions.py
Automatic merge failed; fix conflicts and then commit the result.


Exited with code exit status 1

linux-xenial-py3.6-gcc5.4 / test (backwards_compat, 1, 1, linux.2xlarge) (3/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2021-11-11T08:37:18.6524704Z The PR is introduc...m to confirm whether this change is wanted or not.

2021-11-11T08:37:18.6510235Z processing existing schema:  alltoall_base(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor _1, Tensor _2, int[] _3, int[] _4) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-11T08:37:18.6511717Z processing existing schema:  alltoall(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, Tensor[] _2) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-11T08:37:18.6513521Z processing existing schema:  send(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-11T08:37:18.6515480Z processing existing schema:  recv(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2, int _3) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-11T08:37:18.6517181Z processing existing schema:  recv_anysource(__torch__.torch.classes.dist_c10d.ProcessGroup _0, Tensor[] _1, int _2) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-11T08:37:18.6518416Z processing existing schema:  barrier(__torch__.torch.classes.dist_c10d.ProcessGroup _0) -> (__torch__.torch.classes.dist_c10d.Work _0)
2021-11-11T08:37:18.6519450Z processing existing schema:  __init__(__torch__.torch.classes.dist_c10d.frontend _0) -> (NoneType _0)
2021-11-11T08:37:18.6520800Z processing existing schema:  new_process_group_helper(__torch__.torch.classes.dist_c10d.frontend _0, int _1, int _2, int[] _3, str _4, __torch__.torch.classes.dist_c10d.Store _5, str? _6, int _7) -> (__torch__.torch.classes.dist_c10d.ProcessGroup _0)
2021-11-11T08:37:18.6522283Z processing existing schema:  get_process_group_by_name(__torch__.torch.classes.dist_c10d.frontend _0, str _1) -> (__torch__.torch.classes.dist_c10d.ProcessGroup _0)
2021-11-11T08:37:18.6523601Z processing existing schema:  get_name_of_process_group(__torch__.torch.classes.dist_c10d.frontend _0, __torch__.torch.classes.dist_c10d.ProcessGroup _1) -> (str _0)
2021-11-11T08:37:18.6524704Z The PR is introducing backward incompatible changes to the operator library. Please contact PyTorch team to confirm whether this change is wanted or not. 
2021-11-11T08:37:18.6525258Z 
2021-11-11T08:37:18.6525519Z Broken ops: [
2021-11-11T08:37:18.6526451Z 	aten::searchsorted.Tensor(Tensor sorted_sequence, Tensor self, *, bool out_int32=False, bool right=False, str? side=None, Tensor? sorter=None) -> (Tensor)
2021-11-11T08:37:18.6527714Z 	aten::searchsorted.Scalar(Tensor sorted_sequence, Scalar self, *, bool out_int32=False, bool right=False, str? side=None, Tensor? sorter=None) -> (Tensor)
2021-11-11T08:37:18.6529144Z 	aten::searchsorted.Tensor_out(Tensor sorted_sequence, Tensor self, *, bool out_int32=False, bool right=False, str? side=None, Tensor? sorter=None, Tensor(a!) out) -> (Tensor(a!))
2021-11-11T08:37:18.6530203Z 	aten::_foreach_norm.Scalar(Tensor[] tensors, Scalar ord=2) -> (Tensor[])
2021-11-11T08:37:18.6530964Z 	aten::split_with_sizes(Tensor(a -> *) self, int[] split_sizes, int dim=0) -> (Tensor[])
2021-11-11T08:37:18.6531732Z 	aten::split.Tensor(Tensor(a -> *) self, int split_size, int dim=0) -> (Tensor[])
2021-11-11T08:37:18.6532445Z 	aten::unbind.int(Tensor(a -> *) self, int dim=0) -> (Tensor[])
2021-11-11T08:37:18.6533115Z 	aten::unbind.Dimname(Tensor(a -> *) self, str dim) -> (Tensor[])

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[IvanYashchuk]

It's possible to get the information about which LAPACK implementation is used in NumPy with numpy.show_config(). Maybe the test should pass only when OpenBLAS is used.

[lezcano]

I've looked into numpy.show_config() but it looks like parsing it would make this into a very brittle test. I vote for leave the fix as it is, as I don't think that we'll encounter many more NumPy bugs in that particular piece of code.

[lezcano]

@mruberry this should be ready.

[mruberry]

Hey @lezcano this looks great but looks like it picked up some kineto updates, too.

Would you remove those and just reping me to merge this?

[lezcano]

Submodules are my worst enemy, or at least, certainly in the top 5.
Fixed @mruberry

[mruberry]

Cool! Thanks @lezcano

[mruberry]

@mruberry has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[facebook-github-bot]

@mruberry merged this pull request in 43874d7.