libcantera-devel fails on Windows

[ischoegl]

While conda install -c cantera/label/dev libcantera-devel works nicely on a Linux system, it fails on Windows.

Steps taken on my system:

(base) conda create -n cantera-dev scons numpy cython ruamel_yaml libboost git ipython

and on conda-dev

conda install -c cantera/label/dev cantera
conda install -c cantera/label/dev libcantera-devel

I tried this both with and without Visual Studio (2019) environments set up. In either case, the critical error appears to be this:

C:\Users\ischoegl>pushd "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\"
The system cannot find the path specified.

Which indicates that the conda package contains remnants of a Visual 2017 Enterprise build stack. I am attaching the full logging information: libcantera-devel.log

PS: the expected location for an anaconda installation should likely be dictated by the python system path variable as returned by sysconfig.get_paths()['include']

[bryanwweber]

Can you include more of the error trace? That error might be from something conda is doing, in which case, we don't have much control over it.

[ischoegl]

Can you provide instructions on how to generate more of the error trace? Based on my reading of the log (already attached above), it appears to be that conda is trying to copy something into a path outside of conda (i.e. the original build stack). Interestingly, it doesn't attempt the same on Linux, where headers are installed in the expected location.

[bryanwweber]

I meant can you copy more of the context from that file into the post 😄

In any case, looking at the log, the error message is generated by Conda because Conda assumes that VS 2017 is present. Basically, as best I can tell, its trying to set up that command prompt to use the VS compilers. I'm not sure what to do about that, other than to ignore it and try to compile an example anyways.

[ischoegl]

That's what my interpretation is also. I tried compiling earlier today, but got stuck when I wasn't able to locate the headers (I just got the same result in another attempt). Where should I be looking?

One a separate note: the numpy conda installers work on either system, but they don't use the default sysconfig.get_paths()['include'] route.

Instead, they attach headers to their site-package and provide a custom numpy.get_include() function. On my systems, this yields

'C:\\Users\\<user>\\AppData\\Local\\Continuum\\anaconda3\\envs\\cantera-dev\\lib\\site-packages\\numpy\\core\\include'

on Windows or

'/home/<user>/.pyenv/versions/miniconda3-latest/envs/cantera-dev/lib/python3.7/site-packages/numpy/core/include'

on Linux

[bryanwweber]

I think the header files go in ${CONDA_ENV_DIR}/Library/include on Windows. They're somewhere in the environment directory 😊 The problem is there's no standard directory like on *nix platforms. I don't think following the NumPy example is the right thing to do, because they're also installing a Python package but libcantera-devel is just the C++ library.

[ischoegl]

@bryanwweber ... thanks for the suggestion. I checked the entire conda environment, i.e. ${CONDA_ENV_DIR}/Library, ${CONDA_ENV_DIR}/Lib, ${CONDA_ENV_DIR}/libs and ${CONDA_ENV_DIR}/include, and searching for header files like StFlow.h individually didn't bring up anything either. I guess this indicates that the conda installer failed. Any other suggestions?

[bryanwweber]

Nope, sounds like a bug to me! Thanks for testing 😊

[ischoegl]

No problem. Thanks for your efforts putting the package together!

[ischoegl]

@bryanwweber Any updates on bringing this to windows for 2.5.0beta? I've been using libcantera-devel extensively on linux, where it is really useful (thanks for putting this together) ... will be available for beta testing also.

[bryanwweber]

@ischoegl No updates, and unfortunately I don't think I'll have time to get to it considering we want to try to release 2.5.0 sometime in the next month-ish and I have 4 open PRs over at Cantera/cantera 😃 Fortunately, presuming the problem is with the packaging here and not with Cantera itself, we can fix it up after release and just push out new packages without having to make a full release of Cantera.

[ischoegl]

Understood. Yes, I believe it should be a simple fix that is independent from the main release. Also, good to hear that 2.5.0 is close (will be teaching combustion next semester so it may come just in time).

[ischoegl]

@bryanwweber ... just wanted to see whether there’s any update here?

[bryanwweber]

No, and unfortunately the build on Windows fails altogether at the moment. Hard to find time to get a few hours in a row to work on something these days.

[ischoegl]

Sure. I do understand ...

[ischoegl]

Regardless, if you get to it, I'd be happy to assist - I have a windows machine on hand at the moment (not just a virtual install) and testing would be pretty straight forward. My windows toolchain works (freshly rebuilt), but I probably won't be of much help when it comes to packaging.

[bryanwweber]

Thanks @ischoegl I appreciate the offer! I'll ping you when the Windows builds are functional again.

[bryanwweber]

@ischoegl There are new Conda packages available now, if you're willing to try them out.

[ischoegl]

Some progress (the package installs!), but still not successful. The old error is gone, but now I'm getting stuck at

(cantera-dev) PS C:\Users\ischo\GitHub\ctapp> python setup.py build_ext --inplace
running build_ext
building 'ctapp._ctapp' extension
creating build
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
creating build\temp.win-amd64-3.8\Release\ctapp
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ictapp -IC:\Users\ischo\anaconda3\envs\cantera-dev\lib\site-packages\numpy\core\include -IC:\Users\ischo\anaconda3\envs\cantera-dev -IC:\Users\ischo\anaconda3\envs\cantera-dev\include -IC:\Users\ischo\anaconda3\envs\cantera-dev\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tpctapp/_ctapp.cpp /Fobuild\temp.win-amd64-3.8\Release\ctapp/_ctapp.obj
_ctapp.cpp
C:\Users\ischo\anaconda3\envs\cantera-dev\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
ctapp/_ctapp.cpp(672): fatal error C1083: Cannot open include file: 'cantera/cython/funcWrapper.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

Per our previous conversation, the headers should be in ${CONDA_ENV_DIR}. Alas, searching for the offending file (funcwrapper.h) results in nothing.

I.e. no headers are installed ... I used my ctapp demo to test.

PS: I used a newly installed conda environment. For setup, see top of the page.

[bryanwweber]

@ischoegl Thanks for the quick turnaround. Just to confirm, the headers should be in ${CONDA_ENV_DIR}\include?

[ischoegl]

I think that would be the obvious location (?) to avoid interference with other environments. Within include, the path should start with cantera, I.e. same as on other platforms.

[bryanwweber]

Thanks @ischoegl. I will try to get to that before the release of 2.5.0 which we're really trying to push on, but since I expect this will be relatively low usage case, I might not be able to. Fortunately, we can always release new packages after 2.5.0 comes out 😄

If you want to take a look at this, feel free!

[ischoegl]

Np. Not sure that I am familiar enough with packaging though ...

[bryanwweber]

No problem 😄 The conda packages are pretty complicated, it's taken a long time for me to get used to them.

[ischoegl]

Yeah. Just looked at the files .. no idea what’s going on. Thanks for taking care of this for the community!

[ischoegl]

since I expect this will be relatively low usage case

I’m afraid so. It’s pretty neat that you can compile against cantera, but I’m not sure that people realize.

[ischoegl]

Final comment for the time being. This is the default path within a conda environment on my system:

In [1]: import sysconfig

In [2]: sysconfig.get_paths()['include']
Out[2]: 'C:\\Users\\ischo\\anaconda3\\envs\\cantera-dev\\Include'

Paths are not case-sensitive on Windows, so what we discussed above is confirmed.

[speth]

Conda seems to organize things slightly differently on Windows and Linux, so I'm not sure that's the right path for Cantera to install its headers.

On Windows, C:\src\conda\envs\ctdev\include is the include directory for the Python headers (i.e., Python.h is in this directory) while other conda packages have installed their headers in C:\src\conda\envs\ctdev\Library\include.

On Linux, /home/speth/.conda/envs/ctdev/include/ contains headers for various conda packages, with the Python headers in a subdirectory of that, /home/speth/.conda/envs/ctdev/include/python3.8/.

I would hope that the macOS directory structure is more like the Linux one, but one never knows.

In both cases, I have conda version 4.7.12 installed, starting from Miniconda, and I if I recall correctly, I created both of these environments in the past 6 months.

[ischoegl]

@speth ... thank you for the comment/correction (I don't use windows often, and mostly maintain it as some of my students rely on it).

So there indeed is a 'C:\\Users\\ischo\\anaconda3\\envs\\cantera-dev\\Library\\include' on my machine also, which is where other package headers (boost, etc.) end up in. Regardless, an installation of headers is currently not working on Windows.

[ischoegl]

Error still persists on 2.5.0b1 (setting up a conda environment cantera-test similar to here, but just using conda to install cantera, rather than compiling from scratch). It's a different machine than before (a conventional install rather than a virtual machine, and now using miniconda), so the error is reproducible. FWIW, here's some of the error output when I try to compile ctapp:

[...]
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ictapp -IC:\Users\ischo\miniconda3\envs\cantera-test\lib\site-packages\numpy\core\include -IC:\Users\ischo\miniconda3\envs\cantera-test -IC:\Users\ischo\miniconda3\envs\cantera-test\include -IC:\Users\ischo\miniconda3\envs\cantera-test\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /EHsc /Tpctapp/_ctapp.cpp /Fobuild\temp.win-amd64-3.8\Release\ctapp/_ctapp.obj
    _ctapp.cpp
    C:\Users\ischo\miniconda3\envs\cantera-test\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
    ctapp/_ctapp.cpp(679): fatal error C1083: Cannot open include file: 'cantera/cython/funcWrapper.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.27.29110\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\ischo\miniconda3\envs\cantera-test\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\ischo\\GitHub\\ctapp\\setup.py'"'"'; __file__='"'"'C:\\Users\\ischo\\GitHub\\ctapp\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

As before, searching for funcWrapper.h across my entire miniconda3 main folder (which should be where things get installed) yields nothing.

[ischoegl]

@bryanwweber .... now that 2.5.1 is out, - 🎉!! - I’ll have another look. I don’t expect things to have changed.

[ischoegl]

No surprises, the issue persists.

C:\Users\ischo\miniconda3\envs\cantera-test\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
ctapp/_ctapp.cpp(679): fatal error C1083: Cannot open include file: 'cantera/cython/funcWrapper.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

[ischoegl]

@bryanwweber ... I tried to piece some of this together. Looking at cantera/meta.yaml it appears that there are windows (build_devel.bat) and non-windows (build_devel.sh) versions of the build. The contents are, however, not consistent - the non-windows version appears to set prefix etc and compiles, whereas the windows version only runs an install.

I'm also not sure as to where bld.bat and build.sh enter. (they aren't referenced like build_py.bat and build_py.sh). I don't have enough background on conda to investigate further ... i.e. I don't know the values of some of the environmental variables. But it's apparent that the Windows version (which doesn't work) differs significantly from Linux (which works).

[ischoegl]

One thing I just realized is that even if the installation of headers were to work as envisioned, shared libraries on windows have limited use as they currently do not export C++ symbols (see discussion on Cantera/enhancements#97). Am I seeing this correctly?