New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing psi4 api breaks numpy linear algebra #748

Closed
rmcgibbo opened this Issue Jun 27, 2017 · 38 comments

Comments

Projects
None yet
4 participants
@rmcgibbo
Contributor

rmcgibbo commented Jun 27, 2017

This is probably the strangest issue I've seen yet.

When numpy is imported after psi4 in a Python script, the numpy SVD functions breaks (and becomes nondeterministic). This can be salvaged, strangely, by importing numpy before importing psi4.

I see this issue on an OS X 10.12.3 laptop, with Python 3.5 (installed through conda), with psi4 installed via conda install psi4 psi4-rt -c psi4/label/dev -c psi4. Numpy was also installed through conda (default channel). It's at version 1.11.3. I do not see this issue on the other platform I've tested so far (Python 2.7, CentOS 7, psi4 installed from source).

The specific version of psi4, according to conda list, is

psi4                      1.2a1.dev249+623ad64          py35_0    psi4/label/dev

Here's the script to reproduce:

import psi4  # flipping the order of these two imports "fixes" the problem
import numpy as np

def main():
	random = np.random.RandomState(0)
	N = 50
	A = random.randn(N, N)

	U, s, VT = np.linalg.svd(A, full_matrices=True)
	A_reconstructed = U.dot(np.diag(s)).dot(VT)
	if not np.allclose(A, A_reconstructed):
		raise ValueError("SVD reconstruction failed. difference: %.3f" % np.linalg.norm(A - A_reconstructed))

if __name__ == '__main__':
	main()

Example output:

$ python simple-test.py
/Users/mcgibbon/miniconda/lib/python3.5/site-packages/v2rdm_casscf/v2rdm_casscf.so loaded.
Traceback (most recent call last):
  File "simple-test.py", line 16, in <module>
    main()
  File "simple-test.py", line 13, in main
    raise ValueError("SVD reconstruction failed. difference: %.3f" % np.linalg.norm(A - A_reconstructed))
ValueError: SVD reconstruction failed. difference: 51031.324

$ python simple-test.py
/Users/mcgibbon/miniconda/lib/python3.5/site-packages/v2rdm_casscf/v2rdm_casscf.so loaded.
Traceback (most recent call last):
  File "simple-test.py", line 16, in <module>
    main()
  File "simple-test.py", line 13, in main
    raise ValueError("SVD reconstruction failed. difference: %.3f" % np.linalg.norm(A - A_reconstructed))
ValueError: SVD reconstruction failed. difference: 266529.466
@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 27, 2017

Contributor

Update: I have also tried

  • downgrading to the latest stable psi4 release, following the instructions here (just conda update psi4 -c psi4), which pulled 1.1+add49b9-py35_0 psi4, and the issue persists.
  • switching to conda Python 2.7 (mkl: 2017.0.1-0, numpy: 1.13.0-py27_0, psi4: 1.1+add49b9-py27_0 psi4). Same issue.
Contributor

rmcgibbo commented Jun 27, 2017

Update: I have also tried

  • downgrading to the latest stable psi4 release, following the instructions here (just conda update psi4 -c psi4), which pulled 1.1+add49b9-py35_0 psi4, and the issue persists.
  • switching to conda Python 2.7 (mkl: 2017.0.1-0, numpy: 1.13.0-py27_0, psi4: 1.1+add49b9-py27_0 psi4). Same issue.
@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 27, 2017

Contributor

Update: Installing a version of numpy that doesn't link against MKL "fixes" the problem in the short term, but there's still something deeply fishy going on. (1.13.0 py27_nomkl_0 [nomkl])

Contributor

rmcgibbo commented Jun 27, 2017

Update: Installing a version of numpy that doesn't link against MKL "fixes" the problem in the short term, but there's still something deeply fishy going on. (1.13.0 py27_nomkl_0 [nomkl])

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 27, 2017

Member

Intriguing. I had the reverse problem last week where a MKL-linked numpy was inoperable in psi4 when an openblas was installed in same conda env. I'll see about recreating your env locally.

Member

loriab commented Jun 27, 2017

Intriguing. I had the reverse problem last week where a MKL-linked numpy was inoperable in psi4 when an openblas was installed in same conda env. I'll see about recreating your env locally.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

One fishy thing is that core.so from psi4 1.2a1.dev263+ae466b2 is linked to /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib and /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (from otool -L), and not linked to MKL.

Contributor

rmcgibbo commented Jun 28, 2017

One fishy thing is that core.so from psi4 1.2a1.dev263+ae466b2 is linked to /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib and /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (from otool -L), and not linked to MKL.

@andysim

This comment has been minimized.

Show comment
Hide comment
@andysim

andysim Jun 28, 2017

Member

Oh, I bet that's it. Because the BLAS/LAPACK API is standard, loading psi4 probably clobbers the namespace occupied by MKL's dependencies. If true, numpy is really calling the system BLAS instead of MKL, and perhaps the instability resides in there? Is there a way we can link psi4 to MKL in the conda build, to test this?

Member

andysim commented Jun 28, 2017

Oh, I bet that's it. Because the BLAS/LAPACK API is standard, loading psi4 probably clobbers the namespace occupied by MKL's dependencies. If true, numpy is really calling the system BLAS instead of MKL, and perhaps the instability resides in there? Is there a way we can link psi4 to MKL in the conda build, to test this?

@dgasmith

This comment has been minimized.

Show comment
Hide comment
@dgasmith

dgasmith Jun 28, 2017

Member

A little dangerous, but you could try changing the link library with the install_name_tool -change command.

Member

dgasmith commented Jun 28, 2017

A little dangerous, but you could try changing the link library with the install_name_tool -change command.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

Oh, I bet that's it. Because the BLAS/LAPACK API is standard, loading psi4 probably clobbers the namespace occupied by MKL's dependencies.

The way the dynamic linker's lookups work on linux, this shouldn't happen on Linux unless the extensions were loaded with RTLD_GLOBAL. But mac could be different. On Linux, two python extensions can definitely have symbols with the same name and things work fine (more detail than one could ever want is in https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf).

Contributor

rmcgibbo commented Jun 28, 2017

Oh, I bet that's it. Because the BLAS/LAPACK API is standard, loading psi4 probably clobbers the namespace occupied by MKL's dependencies.

The way the dynamic linker's lookups work on linux, this shouldn't happen on Linux unless the extensions were loaded with RTLD_GLOBAL. But mac could be different. On Linux, two python extensions can definitely have symbols with the same name and things work fine (more detail than one could ever want is in https://software.intel.com/sites/default/files/m/a/1/e/dsohowto.pdf).

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

Also, I can use LLDB to break on dgesdd_ (the lapack svd function), and confirm that numpy is still using the one in MKL.

Contributor

rmcgibbo commented Jun 28, 2017

Also, I can use LLDB to break on dgesdd_ (the lapack svd function), and confirm that numpy is still using the one in MKL.

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 28, 2017

Member

Right, the psi4 conda package is built to link to Mac native Accelerate blas/lapack, not MKL. The MKL is present in the conda env for numpy's benefit. The reason for this is that until very recently all the supporting language libraries and headers for MKL weren't available on conda – they seem to be now through packages on the Intel conda channel, but I haven't switched over.

I was thinking that when psi4 is imported before numpy that the latter may be trying to match symbols in the already loaded Accelerate when it should instead be importing mkl.

Member

loriab commented Jun 28, 2017

Right, the psi4 conda package is built to link to Mac native Accelerate blas/lapack, not MKL. The MKL is present in the conda env for numpy's benefit. The reason for this is that until very recently all the supporting language libraries and headers for MKL weren't available on conda – they seem to be now through packages on the Intel conda channel, but I haven't switched over.

I was thinking that when psi4 is imported before numpy that the latter may be trying to match symbols in the already loaded Accelerate when it should instead be importing mkl.

@andysim

This comment has been minimized.

Show comment
Hide comment
@andysim

andysim Jun 28, 2017

Member

@rmcgibbo thanks for the info - I wasn't aware of those fine details of dynamic loading. So it seems the input to the SVD call is subtly different then, if both paths end up in MKL routines. This is quite a mystery.

Member

andysim commented Jun 28, 2017

@rmcgibbo thanks for the info - I wasn't aware of those fine details of dynamic loading. So it seems the input to the SVD call is subtly different then, if both paths end up in MKL routines. This is quite a mystery.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

Has anyone else tried to repro and either succeeded or failed?

Contributor

rmcgibbo commented Jun 28, 2017

Has anyone else tried to repro and either succeeded or failed?

@andysim

This comment has been minimized.

Show comment
Hide comment
@andysim

andysim Jun 28, 2017

Member

I only have a version compiled from source right now (macOS) and it works correctly. I'll try and get a conda test on my linux box later this afternoon.

Member

andysim commented Jun 28, 2017

I only have a version compiled from source right now (macOS) and it works correctly. I'll try and get a conda test on my linux box later this afternoon.

@dgasmith

This comment has been minimized.

Show comment
Hide comment
@dgasmith

dgasmith Jun 28, 2017

Member

I can reproduce this using a conda Psi4. Interestingly this does not happen with my compiled version. My links look identical to @rmcgibbo's only difference I see is:

Compiled version:

/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)

Conda Version:

/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)

The imported NumPy is still linked to MKL:

daniel:~/Desktop/trials$ otool -L /Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-darwin.so
/Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-darwin.so:
	@rpath/libmkl_intel_lp64.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_intel_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_core.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
daniel:~/Desktop/trials$ otool -L /Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-darwin.so
/Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-darwin.so:
	@rpath/libmkl_intel_lp64.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_intel_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_core.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

I tried both NumPy 1.12 and 1.13 on the conda and compiled version to no effect.

Member

dgasmith commented Jun 28, 2017

I can reproduce this using a conda Psi4. Interestingly this does not happen with my compiled version. My links look identical to @rmcgibbo's only difference I see is:

Compiled version:

/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)

Conda Version:

/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)

The imported NumPy is still linked to MKL:

daniel:~/Desktop/trials$ otool -L /Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-darwin.so
/Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/_umath_linalg.cpython-36m-darwin.so:
	@rpath/libmkl_intel_lp64.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_intel_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_core.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
daniel:~/Desktop/trials$ otool -L /Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-darwin.so
/Users/daniel/anaconda3/lib/python3.6/site-packages/numpy/linalg/lapack_lite.cpython-36m-darwin.so:
	@rpath/libmkl_intel_lp64.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_intel_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_core.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libiomp5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

I tried both NumPy 1.12 and 1.13 on the conda and compiled version to no effect.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

@andysim: I think the issue might be OSX only.

Contributor

rmcgibbo commented Jun 28, 2017

@andysim: I think the issue might be OSX only.

@andysim

This comment has been minimized.

Show comment
Hide comment
@andysim

andysim Jun 28, 2017

Member

Sorry, I overlooked that statement in your original message. Note to self: learn to read bug reports more carefully in future

Member

andysim commented Jun 28, 2017

Sorry, I overlooked that statement in your original message. Note to self: learn to read bug reports more carefully in future

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

A few small data points:

  • I can reproduce the issue by replacing import psi4 with import core, and copying psi4's core.so into my current directory. So the issue isn't in anything the psi python code is doing.
  • The initial otool -L for core.so is:
$ otool -L core.so
core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpcm.1.dylib (compatibility version 1.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	@rpath/libgdma.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libdkh.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/liberd.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libsimint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libefp.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libchemps2.2.dylib (compatibility version 2.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
  • I can play a few tricks with install_name_tool to basically get rid of the dependency on libLAPACK.dylib and libBLAS.dylib. I did that by changing core.so to reference libz in their place:
$ install_name_tool -change /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib libz.dylib core.so
$ install_name_tool -change /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib libz.dylib core.so
$ otool -L core.so
core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpcm.1.dylib (compatibility version 1.0.0, current version 0.0.0)
	@rpath/libxc.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	@rpath/libgdma.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libdkh.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/liberd.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libsimint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libefp.dylib (compatibility version 0.0.0, current version 0.0.0)
	libz.dylib (compatibility version 1.0.0, current version 1.0.0)
	libz.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libchemps2.2.dylib (compatibility version 2.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

Now, this will lead to a ton of undefined symbols. But I don't really care, since I don't want to actually call any functions from core.so anyways. So I change the sys.setdlopenflags to RTLD_LAZY. i.e.

import sys
flags = sys.getdlopenflags()
sys.setdlopenflags(0x1)  # RTLD_LAZY
import core
sys.setdlopenflags(flags)

import numpy as np

random = np.random.RandomState(0)
N = 46
A = random.randn(N, N)

U, s, VT = np.linalg.svd(A, full_matrices=True)
A_reconstructed = U.dot(np.diag(s)).dot(VT)
print(np.linalg.norm(A-A_reconstructed))

And I still get the error.

Contributor

rmcgibbo commented Jun 28, 2017

A few small data points:

  • I can reproduce the issue by replacing import psi4 with import core, and copying psi4's core.so into my current directory. So the issue isn't in anything the psi python code is doing.
  • The initial otool -L for core.so is:
$ otool -L core.so
core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpcm.1.dylib (compatibility version 1.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	@rpath/libgdma.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libdkh.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/liberd.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libsimint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libefp.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib (compatibility version 1.0.0, current version 1.0.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libchemps2.2.dylib (compatibility version 2.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
  • I can play a few tricks with install_name_tool to basically get rid of the dependency on libLAPACK.dylib and libBLAS.dylib. I did that by changing core.so to reference libz in their place:
$ install_name_tool -change /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib libz.dylib core.so
$ install_name_tool -change /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib libz.dylib core.so
$ otool -L core.so
core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libpcm.1.dylib (compatibility version 1.0.0, current version 0.0.0)
	@rpath/libxc.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	@rpath/libgdma.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libdkh.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/liberd.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libsimint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libefp.dylib (compatibility version 0.0.0, current version 0.0.0)
	libz.dylib (compatibility version 1.0.0, current version 1.0.0)
	libz.dylib (compatibility version 1.0.0, current version 1.0.0)
	@rpath/libchemps2.2.dylib (compatibility version 2.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

Now, this will lead to a ton of undefined symbols. But I don't really care, since I don't want to actually call any functions from core.so anyways. So I change the sys.setdlopenflags to RTLD_LAZY. i.e.

import sys
flags = sys.getdlopenflags()
sys.setdlopenflags(0x1)  # RTLD_LAZY
import core
sys.setdlopenflags(flags)

import numpy as np

random = np.random.RandomState(0)
N = 46
A = random.randn(N, N)

U, s, VT = np.linalg.svd(A, full_matrices=True)
A_reconstructed = U.dot(np.diag(s)).dot(VT)
print(np.linalg.norm(A-A_reconstructed))

And I still get the error.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

So now loading core.so never loads a potentially-conflicting BLAS. But it still causes numpy to return garbage.

Contributor

rmcgibbo commented Jun 28, 2017

So now loading core.so never loads a potentially-conflicting BLAS. But it still causes numpy to return garbage.

@dgasmith

This comment has been minimized.

Show comment
Hide comment
@dgasmith

dgasmith Jun 28, 2017

Member

os.environ["MKL_NUM_THREADS"] = "1" makes everything magically work.

Member

dgasmith commented Jun 28, 2017

os.environ["MKL_NUM_THREADS"] = "1" makes everything magically work.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

@dgasmith: Yes! I can confirm. Still have no idea what's going on, but this is a great lead.

Contributor

rmcgibbo commented Jun 28, 2017

@dgasmith: Yes! I can confirm. Still have no idea what's going on, but this is a great lead.

@dgasmith

This comment has been minimized.

Show comment
Hide comment
@dgasmith

dgasmith Jun 28, 2017

Member

Yea, scratching my head over that one too. I was knee deep in lsof loads and thought we should check the simple things. The other weird thing is you can export either MKL or OMP threads and it fixes the error. I wonder what the precedence formkl/omp_set_num_threads/environ is. It could help track down who is setting what.

Member

dgasmith commented Jun 28, 2017

Yea, scratching my head over that one too. I was knee deep in lsof loads and thought we should check the simple things. The other weird thing is you can export either MKL or OMP threads and it fixes the error. I wonder what the precedence formkl/omp_set_num_threads/environ is. It could help track down who is setting what.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

@dgasmith: when you say that this problem doesn't happen with your self-compiled version on OS X, and only happens using the conda Psi4, were you compiling locally with icc or clang?

Contributor

rmcgibbo commented Jun 28, 2017

@dgasmith: when you say that this problem doesn't happen with your self-compiled version on OS X, and only happens using the conda Psi4, were you compiling locally with icc or clang?

@dgasmith

This comment has been minimized.

Show comment
Hide comment
@dgasmith

dgasmith Jun 28, 2017

Member

Locally was compiled with apple-clang. @loriab I think the Mac conda is compiled with clang as well or is it GCC?

Member

dgasmith commented Jun 28, 2017

Locally was compiled with apple-clang. @loriab I think the Mac conda is compiled with clang as well or is it GCC?

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 28, 2017

Member
  • All Mac conda packages after the July–November 2016 hiatus are built with Xcode clang/clang++ and conda gfortran 4.8.5
  • 1.0-era (through July 2016) were built with conda gcc/g++/gfortran all 4.8.5
  • No Mac conda packages (though all Linux conda pkgs) built with Intel compilers
  • All Mac conda packages linked dynamically to Accelerate math libs
  • All Linux conda packages linked statically to Intel MKL math libs
Member

loriab commented Jun 28, 2017

  • All Mac conda packages after the July–November 2016 hiatus are built with Xcode clang/clang++ and conda gfortran 4.8.5
  • 1.0-era (through July 2016) were built with conda gcc/g++/gfortran all 4.8.5
  • No Mac conda packages (though all Linux conda pkgs) built with Intel compilers
  • All Mac conda packages linked dynamically to Accelerate math libs
  • All Linux conda packages linked statically to Intel MKL math libs
@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 28, 2017

Contributor

Got it. I was thinking that there might be some simple difference between @dgasmith's locally-compiled and the conda-compiled psi4s related to compiler. But it seems like that's not the case.

Contributor

rmcgibbo commented Jun 28, 2017

Got it. I was thinking that there might be some simple difference between @dgasmith's locally-compiled and the conda-compiled psi4s related to compiler. But it seems like that's not the case.

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 28, 2017

Member

Yeah, the Linux general build vs. conda build are quite different, but the Mac analogs are essentially the same (though I do have to pass -DCMAKE_CXX_FLAGS="-stdlib=libc++" explicitly to conda build whereas general build with AppleClang just does that naturally).

Member

loriab commented Jun 28, 2017

Yeah, the Linux general build vs. conda build are quite different, but the Mac analogs are essentially the same (though I do have to pass -DCMAKE_CXX_FLAGS="-stdlib=libc++" explicitly to conda build whereas general build with AppleClang just does that naturally).

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 29, 2017

Member

Ok, @andysim and @rmcgibbo, try out a mkl-built psi4 for Mac and see if it fixes both, either, or none of your problems. This is a minimal build with libint (am4) and libxc as internal dylibs. Install into a new env with

>>> conda create -n idp35 psi4test python=3 -c intel -c psi4
>>> source activate idp35
>>> otool -L lib/python3.5/site-packages/psi4/core.so 
lib/python3.5/site-packages/psi4/core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libxc.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

The env should look like this. Note that py36 is not avail. Also, do not install this into the main env of an anaconda or miniconda – subenvs only.

>>> conda list
# packages in environment at /Users/loriab/linux/miniconda3/envs/idp35:
#
icc_rt                    16.0.3                  intel_6  [intel]  intel
intelpython               2017.0.3                      4    intel
mkl                       2017.0.3                intel_6  [intel]  intel
numpy                     1.12.1             py35_intel_8  [intel]  intel
openmp                    2017.0.3                intel_8    intel
openssl                   1.0.2k                  intel_3  [intel]  intel
pip                       9.0.1              py35_intel_0  [intel]  intel
psi4test                  1.2a1.dev370+d9c89ae          py35_6    psi4
py                        1.4.34                   py35_0    defaults
pytest                    3.1.2                    py35_0    defaults
python                    3.5.3                   intel_1  [intel]  intel
setuptools                27.2.0             py35_intel_0  [intel]  intel
sqlite                    3.13.0                 intel_14  [intel]  intel
tcl                       8.6.4                  intel_16  [intel]  intel
tk                        8.6.4                  intel_26  [intel]  intel
wheel                     0.29.0             py35_intel_5  [intel]  intel
xz                        5.2.2                  intel_15  [intel]  intel
zlib                      1.2.11                  intel_2  [intel]  intel
Member

loriab commented Jun 29, 2017

Ok, @andysim and @rmcgibbo, try out a mkl-built psi4 for Mac and see if it fixes both, either, or none of your problems. This is a minimal build with libint (am4) and libxc as internal dylibs. Install into a new env with

>>> conda create -n idp35 psi4test python=3 -c intel -c psi4
>>> source activate idp35
>>> otool -L lib/python3.5/site-packages/psi4/core.so 
lib/python3.5/site-packages/psi4/core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libxc.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)

The env should look like this. Note that py36 is not avail. Also, do not install this into the main env of an anaconda or miniconda – subenvs only.

>>> conda list
# packages in environment at /Users/loriab/linux/miniconda3/envs/idp35:
#
icc_rt                    16.0.3                  intel_6  [intel]  intel
intelpython               2017.0.3                      4    intel
mkl                       2017.0.3                intel_6  [intel]  intel
numpy                     1.12.1             py35_intel_8  [intel]  intel
openmp                    2017.0.3                intel_8    intel
openssl                   1.0.2k                  intel_3  [intel]  intel
pip                       9.0.1              py35_intel_0  [intel]  intel
psi4test                  1.2a1.dev370+d9c89ae          py35_6    psi4
py                        1.4.34                   py35_0    defaults
pytest                    3.1.2                    py35_0    defaults
python                    3.5.3                   intel_1  [intel]  intel
setuptools                27.2.0             py35_intel_0  [intel]  intel
sqlite                    3.13.0                 intel_14  [intel]  intel
tcl                       8.6.4                  intel_16  [intel]  intel
tk                        8.6.4                  intel_26  [intel]  intel
wheel                     0.29.0             py35_intel_5  [intel]  intel
xz                        5.2.2                  intel_15  [intel]  intel
zlib                      1.2.11                  intel_2  [intel]  intel
@andysim

This comment has been minimized.

Show comment
Hide comment
@andysim

andysim Jun 29, 2017

Member

A familiar story by now, @loriab, but the AVX issue still persists. With DYLD_PRINT_LIBRARIES I still get

dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/lib-dynload/_struct.cpython-35m-darwin.so
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib//python3.5/site-packages/psi4/core.so
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libxc.dylib
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libderiv.dylib
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libint.dylib
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libmkl_rt.dylib
Illegal instruction: 4
Member

andysim commented Jun 29, 2017

A familiar story by now, @loriab, but the AVX issue still persists. With DYLD_PRINT_LIBRARIES I still get

dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/lib-dynload/_struct.cpython-35m-darwin.so
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib//python3.5/site-packages/psi4/core.so
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libxc.dylib
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libderiv.dylib
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libint.dylib
dyld: loaded: /Users/simmonettac/miniconda3/envs/idp35/lib/python3.5/site-packages/psi4/../../..//libmkl_rt.dylib
Illegal instruction: 4
@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 29, 2017

Contributor

@loriab: Yes, I can confirm that the mkl-built psi4 for Mac fixes my numpy linalg bug!

Contributor

rmcgibbo commented Jun 29, 2017

@loriab: Yes, I can confirm that the mkl-built psi4 for Mac fixes my numpy linalg bug!

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 29, 2017

Member

Yay, something I've done this week has worked.

I'll see if I can scrape together a default-mkl + intel-mkl-headers build that won't be so heavy.

Member

loriab commented Jun 29, 2017

Yay, something I've done this week has worked.

I'll see if I can scrape together a default-mkl + intel-mkl-headers build that won't be so heavy.

@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 29, 2017

Contributor

Is it possible that the intel-mkl could just go in the conda build: requirements:, but then use the default mkl in the run: requirements?

Contributor

rmcgibbo commented Jun 29, 2017

Is it possible that the intel-mkl could just go in the conda build: requirements:, but then use the default mkl in the run: requirements?

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 29, 2017

Member

Actually, I took the same psi4test from last night and tried it in a defaults-channel numpy & mkl via conda create -n dfl35 psi4test python=3.5 -c psi4 and it works just fine according to psi4 --test. I'll feel more assured when the full test suite is run, but it looks like the answer to your question above is yes. Now if only Intel channel had a py36.

(dfl35) >>> otool -L lib/python3.5/site-packages/psi4/core.so 
lib/python3.5/site-packages/psi4/core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libxc.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
(dfl35) >>> conda list
# packages in environment at /path/to/miniconda3/envs/dfl35:
#
mkl                       2017.0.3                      0    defaults
numpy                     1.13.0                   py35_0    defaults
openssl                   1.0.2l                        0    defaults
pip                       9.0.1                    py35_1    defaults
psi4test                  1.2a1.dev370+d9c89ae          py35_6    psi4
py                        1.4.34                   py35_0    defaults
pytest                    3.1.2                    py35_0    defaults
python                    3.5.3                         1    defaults
readline                  6.2                           2    defaults
setuptools                27.2.0                   py35_0    defaults
sqlite                    3.13.0                        0    defaults
tk                        8.5.18                        0    defaults
wheel                     0.29.0                   py35_0    defaults
xz                        5.2.2                         1    defaults
zlib                      1.2.8                         3    defaults
Member

loriab commented Jun 29, 2017

Actually, I took the same psi4test from last night and tried it in a defaults-channel numpy & mkl via conda create -n dfl35 psi4test python=3.5 -c psi4 and it works just fine according to psi4 --test. I'll feel more assured when the full test suite is run, but it looks like the answer to your question above is yes. Now if only Intel channel had a py36.

(dfl35) >>> otool -L lib/python3.5/site-packages/psi4/core.so 
lib/python3.5/site-packages/psi4/core.so:
	@rpath/core.so (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libxc.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libderiv.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libint.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libmkl_rt.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
(dfl35) >>> conda list
# packages in environment at /path/to/miniconda3/envs/dfl35:
#
mkl                       2017.0.3                      0    defaults
numpy                     1.13.0                   py35_0    defaults
openssl                   1.0.2l                        0    defaults
pip                       9.0.1                    py35_1    defaults
psi4test                  1.2a1.dev370+d9c89ae          py35_6    psi4
py                        1.4.34                   py35_0    defaults
pytest                    3.1.2                    py35_0    defaults
python                    3.5.3                         1    defaults
readline                  6.2                           2    defaults
setuptools                27.2.0                   py35_0    defaults
sqlite                    3.13.0                        0    defaults
tk                        8.5.18                        0    defaults
wheel                     0.29.0                   py35_0    defaults
xz                        5.2.2                         1    defaults
zlib                      1.2.8                         3    defaults
@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 30, 2017

Member

Observations:

  • RMcG found that psi4 Mac conda pkg (linked to native Accelerate) and numpy conda pkg (linked to in-env MKL) was fragile wrt psi4/numpy import ordering
  • DGAS found that psi4 built from source (linked to native Accelerate but managing its own RPATH) and numpy conda pkg (linked to in-env MKL) was steady wrt psi4/numpy import ordering
  • LAB found that psi4 built from source on Linux (linked to in-env OpenBlas) and numpy conda pkg (linked to in-env MKL) was fragile wrt numpy operation w/i psi4

Conclusions:

  • Linking to multiple math libs in the same conda env is risky (perhaps more so on Mac than Linux)
  • Best solution for Mac is to just link Psi4 to MKL, not Accelerate if installing psi4 into same env as numpy (which, for ease of use and distribution, we want to do)
  • Requires mix and match of default-channel MKL and intel-channel mkl-include (for mkl.h) to get full py27/py35/py36 stack
  • In practice, rolling this out along with sse4.1 "Illegal Instruction 4" fix for old Mac hardware, so expect solutions in the coming week, rather than the coming day.
Member

loriab commented Jun 30, 2017

Observations:

  • RMcG found that psi4 Mac conda pkg (linked to native Accelerate) and numpy conda pkg (linked to in-env MKL) was fragile wrt psi4/numpy import ordering
  • DGAS found that psi4 built from source (linked to native Accelerate but managing its own RPATH) and numpy conda pkg (linked to in-env MKL) was steady wrt psi4/numpy import ordering
  • LAB found that psi4 built from source on Linux (linked to in-env OpenBlas) and numpy conda pkg (linked to in-env MKL) was fragile wrt numpy operation w/i psi4

Conclusions:

  • Linking to multiple math libs in the same conda env is risky (perhaps more so on Mac than Linux)
  • Best solution for Mac is to just link Psi4 to MKL, not Accelerate if installing psi4 into same env as numpy (which, for ease of use and distribution, we want to do)
  • Requires mix and match of default-channel MKL and intel-channel mkl-include (for mkl.h) to get full py27/py35/py36 stack
  • In practice, rolling this out along with sse4.1 "Illegal Instruction 4" fix for old Mac hardware, so expect solutions in the coming week, rather than the coming day.
@rmcgibbo

This comment has been minimized.

Show comment
Hide comment
@rmcgibbo

rmcgibbo Jun 30, 2017

Contributor

@loriab: Do you recall in what sense psi4 built from source on Linux with OpenBLAS in numpy conda pkg with MKL was fragile?

Contributor

rmcgibbo commented Jun 30, 2017

@loriab: Do you recall in what sense psi4 built from source on Linux with OpenBLAS in numpy conda pkg with MKL was fragile?

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Jun 30, 2017

Member

Any test case that actually used NumPy (mostly casscf) failed. Afraid I don't remember the error trace details.

Member

loriab commented Jun 30, 2017

Any test case that actually used NumPy (mostly casscf) failed. Afraid I don't remember the error trace details.

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Oct 29, 2017

Member

Note that np.linalg.eigh (though not eig) is also susceptible to an Accelerate-linked import psi4 before a mkl-linked import numpy yielding wrong, eigh != eig, and obviously un-normalizable eigenvectors. Woe unto me for actually starting with an old branch w/o rebasing first. @dgasmith, this is the source of the accusations I was leveling against np a few weeks ago.

Member

loriab commented Oct 29, 2017

Note that np.linalg.eigh (though not eig) is also susceptible to an Accelerate-linked import psi4 before a mkl-linked import numpy yielding wrong, eigh != eig, and obviously un-normalizable eigenvectors. Woe unto me for actually starting with an old branch w/o rebasing first. @dgasmith, this is the source of the accusations I was leveling against np a few weeks ago.

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Oct 29, 2017

Member

Further note that importing numpy first won't fix everything. Accelerate-linked import psi4 can still break numpy linalg. Whether it's fixed by Accelerate to Numpy or Accelerate to mkl-rt-linked Numpy is undetermined.

Seen in evec of degen pair of hessian

       [ 0.0000,  0.0159,  0.0000, -0.0000, -0.0069,  0.0000,  0.0000, -0.0069,  0.0000],
       [ 0.0000,  0.0000,  0.1832,  0.0000,  0.0000, -0.0793,  0.0000,  0.0000, -0.0793],
       [-0.0069, -0.0000,  0.0000,  0.0030,  0.0000,  0.0000,  0.0030,  0.0000,  0.0000],
       [-0.0000, -0.0069,  0.0000,  0.0000,  0.0030,  0.0000, -0.0000,  0.0030,  0.0000],
       [ 0.0000,  0.0000, -0.0793,  0.0000,  0.0000,  0.0777,  0.0000,  0.0000, -0.0090],
       [-0.0069,  0.0000,  0.0000,  0.0030, -0.0000,  0.0000,  0.0030, -0.0000,  0.0000],
       [-0.0000, -0.0069,  0.0000,  0.0000,  0.0030,  0.0000, -0.0000,  0.0030,  0.0000],
       [ 0.0000,  0.0000, -0.0793,  0.0000,  0.0000, -0.0090,  0.0000,  0.0000,  0.0777]])

And the code turned out to be fully up-to-date. My foolishness was in not using the dev env that properly sets up MKL from conda and instead just using Mac built-ins. CMake does not have the power to select/reject dependencies based on their ldd profile.

Member

loriab commented Oct 29, 2017

Further note that importing numpy first won't fix everything. Accelerate-linked import psi4 can still break numpy linalg. Whether it's fixed by Accelerate to Numpy or Accelerate to mkl-rt-linked Numpy is undetermined.

Seen in evec of degen pair of hessian

       [ 0.0000,  0.0159,  0.0000, -0.0000, -0.0069,  0.0000,  0.0000, -0.0069,  0.0000],
       [ 0.0000,  0.0000,  0.1832,  0.0000,  0.0000, -0.0793,  0.0000,  0.0000, -0.0793],
       [-0.0069, -0.0000,  0.0000,  0.0030,  0.0000,  0.0000,  0.0030,  0.0000,  0.0000],
       [-0.0000, -0.0069,  0.0000,  0.0000,  0.0030,  0.0000, -0.0000,  0.0030,  0.0000],
       [ 0.0000,  0.0000, -0.0793,  0.0000,  0.0000,  0.0777,  0.0000,  0.0000, -0.0090],
       [-0.0069,  0.0000,  0.0000,  0.0030, -0.0000,  0.0000,  0.0030, -0.0000,  0.0000],
       [-0.0000, -0.0069,  0.0000,  0.0000,  0.0030,  0.0000, -0.0000,  0.0030,  0.0000],
       [ 0.0000,  0.0000, -0.0793,  0.0000,  0.0000, -0.0090,  0.0000,  0.0000,  0.0777]])

And the code turned out to be fully up-to-date. My foolishness was in not using the dev env that properly sets up MKL from conda and instead just using Mac built-ins. CMake does not have the power to select/reject dependencies based on their ldd profile.

@loriab

This comment has been minimized.

Show comment
Hide comment
@loriab

loriab Apr 13, 2018

Member

Conclusions here

tl;dr Thine psi4 core.so shall be linked to exactly the same LAPACK libs as thine NumPy, lest wrong arrays or damped threading ensue.

Member

loriab commented Apr 13, 2018

Conclusions here

tl;dr Thine psi4 core.so shall be linked to exactly the same LAPACK libs as thine NumPy, lest wrong arrays or damped threading ensue.

@loriab loriab closed this Apr 13, 2018

@dgasmith

This comment has been minimized.

Show comment
Hide comment
@dgasmith

dgasmith Apr 13, 2018

Member

To supplement this, if you are compiling with conda please use the psi4-path-advisor to build your base CMake script. Details found here.

Member

dgasmith commented Apr 13, 2018

To supplement this, if you are compiling with conda please use the psi4-path-advisor to build your base CMake script. Details found here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment