Installation troubles related to SymEngine

[blokhin]

Dear Richard,
Dear Brandon,

while doing pip3 install pycalphad I have some troubles:

    pycalphad/core/eqsolver.cpp:1351:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _obj;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1352:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _grad;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1353:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _hess;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1354:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _internal_cons;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1355:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _internal_jac;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1356:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _internal_cons_hess;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1357:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _multiphase_cons;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1358:14: error: ‘LLVMDoubleVisitor’ in namespace ‘SymEngine’ does not name a type
       SymEngine::LLVMDoubleVisitor _multiphase_jac;
                  ^~~~~~~~~~~~~~~~~
    pycalphad/core/eqsolver.cpp:1359:15: error: ‘LLVMDoubleVisitor’ is not a member of ‘SymEngine’
       std::vector<SymEngine::LLVMDoubleVisitor>  _masses;
                   ^~~~~~~~~
    pycalphad/core/eqsolver.cpp:1359:15: error: ‘LLVMDoubleVisitor’ is not a member of ‘SymEngine’
    pycalphad/core/eqsolver.cpp:1359:43: error: template argument 1 is invalid
       std::vector<SymEngine::LLVMDoubleVisitor>  _masses;
                                               ^
    pycalphad/core/eqsolver.cpp:1359:43: error: template argument 2 is invalid
    pycalphad/core/eqsolver.cpp:1360:15: error: ‘LLVMDoubleVisitor’ is not a member of ‘SymEngine’
       std::vector<SymEngine::LLVMDoubleVisitor>  _massgrads;
                   ^~~~~~~~~
    pycalphad/core/eqsolver.cpp:1360:15: error: ‘LLVMDoubleVisitor’ is not a member of ‘SymEngine’
    pycalphad/core/eqsolver.cpp:1360:43: error: template argument 1 is invalid
       std::vector<SymEngine::LLVMDoubleVisitor>  _massgrads;
                                               ^
    pycalphad/core/eqsolver.cpp:1360:43: error: template argument 2 is invalid
    pycalphad/core/eqsolver.cpp:1361:15: error: ‘LLVMDoubleVisitor’ is not a member of ‘SymEngine’
       std::vector<SymEngine::LLVMDoubleVisitor>  _masshessians;
                   ^~~~~~~~~
    pycalphad/core/eqsolver.cpp:1361:15: error: ‘LLVMDoubleVisitor’ is not a member of ‘SymEngine’
    pycalphad/core/eqsolver.cpp:1361:43: error: template argument 1 is invalid
       std::vector<SymEngine::LLVMDoubleVisitor>  _masshessians;
                                               ^
    pycalphad/core/eqsolver.cpp:1361:43: error: template argument 2 is invalid
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-1_nh2rra/pycalphad/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-y_mi9r8j/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-1_nh2rra/pycalphad/

Python is 3.5. NumPy, SciPy, Cython, Ipopt are present. SymEngine is of version 0.4.0. Is this a known problem?

[richardotis]

Hi Evgeny, this error means that the installer cannot find the path to the SymEngine library. There are a few ways to address this, but the easiest is to install using Anaconda. The pycalphad setup script includes hard-coded path additions for conda environments, so it should just work.

[blokhin]

Except conda, what are the ways to tell the path to the SymEngine library to the installer? In setup.py I see the functions symengine_pyx_get_include, symengine_lib_get_include, symengine_bin_get_include, and symengine_h_get_include, but they don't seem to address this properly on Debian. On the other hand, symengine Python wrapper was compiled for me just fine.

[isuruf]

That error is probably because libsymengine was compiled without LLVM support.

[richardotis]

Was the SymEngine wheel on PyPI (assuming that's where it was installed from) compiled without LLVM support? On Jul 26, 2019 2:54 PM, Isuru Fernando <notifications@github.com> wrote: That error is probably because libsymengine was compiled without LLVM support. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#232?email_source=notifications&email_token=ABQ33BW65JOXRW6W5BRVGLTQBNXBBA5CNFSM4IFTVYS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25ZRMI#issuecomment-515610801>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQ33BRGJKIKPXXZWUHFGZDQBNXBBANCNFSM4IFTVYSQ>.

[isuruf]

Was the SymEngine wheel on PyPI (assuming that's where it was installed from) compiled without LLVM support?

All the wheels are compiled with LLVM support.

[richardotis]

@isuruf For downstream Python packages needing to link to `libsymengine`, what is recommended for finding the path? What pycalphad has in setup.py currently is a bit of a hack and only works with conda environments.

[isuruf]

For downstream Python packages needing to link to libsymengine, what is recommended for finding the path? What pycalphad has in setup.py currently is a bit of a hack and only works with conda environments.

What you do for conda is fine. For pip wheels, it's hard as the pip wheel changes library names. Easiest way would be to provide pypi wheels yourself like we do at https://github.com/symengine/symengine-wheels

[isuruf]

Another way is to avoid the need to link to libsymengine. Is there a way we can improve the cython files in symengine.py to remove the need to link to libsymengine?

[richardotis]

Right now pycalphad needs access to the C type information for LLVMDoubleVisitor (and eventually also LambdaRealDoubleVisitor after symengine.py 0.4.1 ships) so that we can make native calls to their .call() method inside tight nogil loops. I think that it is these pycalphad calls to .call() that are driving the build-time link requirement for libsymengine.

I think that it would be possible for SymEngine to pass raw function pointers to pycalphad via use of PyCapsule. The SciPy LowLevelCallable support in symengine.py almost does what we need, except some pycalphad callables have array-valued output with function signatures like void (double *out, double *in). Despite the error message in https://github.com/symengine/symengine.py/blob/13cedfa1916ee007fed34ad4351e2b86fec2d31a/symengine/lib/symengine_wrapper.pyx#L4737 I think SciPy already supports this, and all that's required is a slightly different callback function signature.

With LowLevelCallable all the native type information is hidden from Python, so pycalphad wouldn't need to link to libsymengine. The only cost is a little bit of indirection overhead for each function call, but my expectation is that that is cheap. I don't fully have my head wrapped around how the callback would need to look in symengine.py, but I think a good place to start is the scipy test suite: https://github.com/scipy/scipy/blob/8dba340293fe20e62e173bdf2c10ae208286692f/scipy/_lib/tests/test_ccallback.py

[richardotis]

While putting together a PR I realized there was a much simpler solution, which I have submitted upstream: symengine/symengine.py#291

With that API we could eliminate the need for linking to libsymengine.

[richardotis]

@isuruf It looks like the solution I came up with still requires that we provide the path to the SymEngine include files, which may have some of the same difficulties with path detecton. Do you think an API like this would be possible to ship in symengine.py? Ideally something which would be robust to conda versus wheel/pip, etc.

pycalphad/setup.py

Lines 65 to 70 in c9ccc93

	def symengine_h_get_include():
	if sys.platform == 'win32':
	# Strictly only valid for recent conda installations
	return os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(symengine.__file__)))), 'Library', 'include')
	else:
	return os.path.dirname(get_paths()['include'])

[isuruf]

The wheels don't even ship the includes. See symengine/symengine.py#292

[blokhin]

@richardotis @isuruf very much appreciated 👍👍👍

[richardotis]

This should now be fixed on the develop branch, as of 05ff0cf. The next release will contain the fix.