conda package/mkl on windows

[SteveDiamond]

@enzobusseti and I have been working on making conda packages for cvxpy and all its dependencies. We are so so close, and would really appreciate help with Windows support. Here's the conda recipes we've made:
https://github.com/cvxgrp/conda-recipes
We've used these recipes to push conda binaries to https://anaconda.org/cvxgrp
They all work except for SCS on windows. The binary doesn't link against mkl for blas/lapack some reason. It does link against mkl on linux.

You should be able to run "conda build scs" inside the conda-recipes folder and have it just work. If anyone can figure out how to get a proper windows build I would be incredibly appreciative. Remember it's not enough for the package to build, it also has to link against mkl.

[bodono]

This is awesome, thanks for writing these recipes. To clarify - where does the mkl come from that should be linked against? Is it a conda distribution or must the user download mkl manually? What are the full set of steps that I should try on a clean windows install to replicate this (python version, conda install, mkl etc.). I have an old windows laptop I can test on.

[SteveDiamond]

Anaconda comes with mkl now on all platforms. The default now is for numpy, scipy, etc to link against mkl. So ideally SCS would only link against mkl as well. This happens on linux. On Mac it links against Accelerate as well, but that's fine.

On a clean windows environment you should install anaconda, run conda install conda-build, clone the cvxgrp/conda-recipes repo, then finally run conda build scs inside the conda-recipes folder.

You can test out the SCS you built by running conda install --use-local scs. Then you can import SCS in Python. The key test is whether SCS works for SDPs. It's easy to get everything working except linking to blas/lapack.

[bodono]

As far as I can tell this is an issue with the numpy conda package. SCS uses numpy.distutils.system_info function get_info to tell it where the blas/lapack libraries it should link against are located. Numpy stores that information in a site.cfg file. But it appears this is broken in the conda numpy package. On a clean windows machine I installed anaconda, then installed numpy via conda install numpy. Then in a python interpreter I run:

>>> from numpy.distutils.system_info import get_info
>>> get_info('blas_opt')
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs\_build\Library\lib is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs\_build\Library\include is invalid
.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
46: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
55: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
58: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)

This is before I've even downloaded SCS. It's looking for the libs and includes in C:\Minonda\envs\_build, which doesn't exist on my machine.

After some digging I found this part of the guide in conda about making packages relocatable, which says that the build prefix is replaced in any file that contains it with a placeholder and the file is added to the list in info/has_prefix. If you examine the packages from the conda package repo you can inspect the windows and linux numpy packages manually. In the linux one, it has an info/has_prefix file which has

/opt/anaconda1anaconda2anaconda3 text lib/python2.7/site-packages/numpy/distutils/site.cfg

and the site.cfg file has

[mkl]
library_dirs = /opt/anaconda1anaconda2anaconda3/lib
include_dirs = /opt/anaconda1anaconda2anaconda3/include
lapack_libs = mkl_lapack95_lp64
mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5

In other words the has_prefix file is telling conda to replace the placeholder string /opt/anaconda1anaconda2anaconda3 with the real prefix when it is installing it (since that's when it will be known).

However the latest windows numpy packages (e.g. numpy-1.11.1-py27_1.tar.bz2) do not have a has_prefix file, so it doesn't do the prefix swapping. The site.cfg file simply has

[mkl]
include_dirs = C:\Minonda\envs\_build\Library\include
library_dirs = C:\Minonda\envs\_build\Library\lib
lapack_libs = mkl_lapack95_lp64
mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll

But that directory doesn't exist, and the prefix is not getting replaced, so it ends up with the error above.

However, an older windows version of numpy, numpy-1.11.0-py27_2 does have an info/has_prefix file, that has

"C:\Minonda\envs\_build" text "Lib/site-packages/numpy/distutils/site.cfg"

Which is saying to swap out the C:\Minonda\envs\_build for the real prefix in the site.cfg file. So you would think that would replace the path correctly, however it still throws an error for me because the path that it replaces it with is also invalid (doesn't exist, and mixes forwards and backwards slashes in the path).

I think the message here is that the conda numpy packages for windows are untested and broken, and appear to have been broken in different ways for a long time. I guess the right thing to do is flag it with the anaconda team. I never even got to the point of installing SCS on the system!

[ghost]

can you try the same trick using another package that links to lapack/blas (e.g. scipy)? maybe that one is not broken. or really any other one. you can specify " build requirements" in a conda recipe, packages that are installed temporarily when building the package but not when installing it..

Enzo

On Aug 13, 2016, at 06:39, bodono notifications@github.com wrote:

As far as I can tell this is an issue with the numpy conda package. SCS uses numpy.distutils.system_info function get_info to tell it where the blas/lapack libraries it should link against are located. Numpy stores that information in a site.cfg file. But it appears this is broken in the conda numpy package. On a clean windows machine I installed anaconda, then installed numpy via conda install numpy. Then in a python interpreter I run:

from numpy.distutils.system_info import get_info
get_info('blas_opt')
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs_build\Library\lib is invalid.
warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:63
9: UserWarning: Specified path C:\Minonda\envs_build\Library\include is invalid
.
warnings.warn('Specified path %s is invalid.' % d)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
46: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
warnings.warn(AtlasNotFoundError.doc)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
55: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
warnings.warn(BlasNotFoundError.doc)
C:\Users\23190281\Miniconda2\lib\site-packages\numpy\distutils\system_info.py:16
58: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
warnings.warn(BlasSrcNotFoundError.doc)
This is before I've even downloaded SCS. It's looking for the libs and includes in C:\Minonda\envs_build, which doesn't exist on my machine.

After some digging I found this part of the guide in conda about making packages relocatable, which says that the build prefix is replaced in any file that contains it with a placeholder and the file is added to the list in info/has_prefix. If you examine the packages from the conda package repo you can inspect the windows and linux numpy packages manually. In the linux one, it has an info/has_prefix file which has

/opt/anaconda1anaconda2anaconda3 text lib/python2.7/site-packages/numpy/distutils/site.cfg
and the site.cfg file has

[mkl]
library_dirs = /opt/anaconda1anaconda2anaconda3/lib
include_dirs = /opt/anaconda1anaconda2anaconda3/include
lapack_libs = mkl_lapack95_lp64
mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5
In other words the has_prefix file is telling conda to replace the placeholder string /opt/anaconda1anaconda2anaconda3 with the real prefix when it is installing it (since that's when it will be known).

However the latest windows numpy packages (e.g. numpy-1.11.1-py27_1.tar.bz2) do not have a has_prefix file, so it doesn't do the prefix swapping. The site.cfg file simply has

[mkl]
include_dirs = C:\Minonda\envs_build\Library\include
library_dirs = C:\Minonda\envs_build\Library\lib
lapack_libs = mkl_lapack95_lp64
mkl_libs = mkl_core_dll, mkl_intel_lp64_dll, mkl_intel_thread_dll
But that directory doesn't exist, and the prefix is not getting replaced, so it ends up with the error above.

However, an older windows version of numpy, numpy-1.11.0-py27_2 does have an info/has_prefix file, that has

"C:\Minonda\envs_build" text "Lib/site-packages/numpy/distutils/site.cfg"
Which is saying to swap out the C:\Minonda\envs_build for the real prefix in the site.cfg file. So you would think that would replace the path correctly, however it still throws an error for me because the path that it replaces it with is also invalid (doesn't exist, and mixes forwards and backwards slashes in the path).

I think the message here is that the conda numpy packages for windows are untested and broken, and appear to have been broken in different ways for a long time. I guess the right thing to do is flag it with the anaconda team. I never even got to the point of installing SCS on the system!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[bodono]

As far as I know other packages don't expose that information like numpy does. And scipy uses the same get_info functionality from numpy to build: https://github.com/scipy/scipy/search?utf8=%E2%9C%93&q=get_info

[bodono]

One potential workaround that would work for all platforms is for the conda recipe for SCS to bypass numpy and get the lapack / blas libs directly. SCS will use the environment variables BLAS_LAPACK_LIB_PATHS and BLAS_LAPACK_LIBS to link with if they are set (before calling get_info). The conda recipe would have to install mkl first, figure out where the libraries are then set the environment variables accordingly and call setup.py. This numpy conda recipe file has all the information you need (since it's what get_info relies on).

[bodono]

To spell it out, the build.sh and bld.bat files would have to be something along the lines of

export BLAS_LAPACK_LIB_PATHS = $PREFIX/lib
export BLAS_LAPACK_LIBS = "mkl_lapack95_lp64:mkl_intel_lp64:mkl_intel_thread:mkl_core:iomp5"
python setup.py install

[clagms]

I was able to install scs on windows anaconda (python 2.7), linking to the lapack/blas libraries.
However, there are a few quirks in the setup process, which conspire to make this a nightmare.

I'll detail the steps I followed to successfully install the 32 bits version, and potential problems with installing the 64 bits version.

Steps that I followed for the 32 bits installation:

1. Install Anaconda python 2.7 (32 bits).
2. Install Visual Studio Compiler for python 2.7 from here
3. Get the LAPACK/BLAS libraries example project from here. It will become clear why we get the libraries from the examples in a minute.
4. [Optional but recommended] Open the VS solution in the example, build, and run the example. If it does not work, there is a high probability that the following steps will not work.
5. Place the libblas.dll, libblas.lib, liblapack.dll, liblapack.lib, liblapacke.dll, liblapacke.lib, somewhere like "C:\blas_lapack"
6. Edit file python\setup.py to not split the env_lib_dirs string. Concretely, correct the line starting with env_vars['library_dirs'] = [...] to env_vars['library_dirs'] = [env_lib_dirs]. The split causes problems with paths like C:\something, splitting them into "C" and "\something".
7. open a command prompt in the python directory and run:
   export BLAS_LAPACK_LIB_PATHS = C:\blas_lapack
export BLAS_LAPACK_LIBS = libblas:liblapack:liblapacke (Note that no extension is added)
   python setup.py install
8. Make sure that the log shows the linking being done correctly. That is, is should invoke the linker somewhat as follows
   [...]Visual C++ for Python\9.0\VC\Bin\link.exe [...] /LIBPATH:C:\blas_lapack [...] libblas.lib liblapack.lib liblapacke.lib [...]
   If you get errors at this stage, see the troubleshooting below.
9. If the above step was sucessfull, you should have the installation folder in your site-packages of scs, named scs-1.2[...]py2.7-win32.egg.
   Copy the libblas.dll, libblas.lib, liblapack.dll, liblapack.lib, liblapacke.dll, liblapacke.lib, files to that folder.
10. Important: go to the LAPACK/BLAS libraries example project, and copy the libgfortran-3.dll and libgcc_s_dw2-1.dll to the scs-1.2[...]py2.7-win32.egg folder. These ensure that blas and lapack can be loaded when you import scs in python.
11. Run the python\main.py to test (you may need to install guppy prior to this).

Issues with the 64 bits installation:

• Prefixes in scs_blas.h have to be played arround with. The naming conventions for the 64 bits libraries downloaded from here are different than the 32 bits version. Use "dumbin /symbols" to find the correct names and edit scs_blas.h accordingly. The errors you get if this is not correct are like error LNK2019: unresolved external symbol dsyevr_[...].
• I could not find a 64 bit version of libgfortran-3.dll and libgcc_s_dw2-1.dll. I tried multiple venues, such as cygwin, and msys64, and could not find them.
  I found similar libs (with a different name, but declaring the required symbols, such as libgfortran_64-3.dll), but these depend on other 64 bit libs that I could not find.
  The Dependency walker utility is usefull to approach this issue.

Other issues encountered:

• The --extraverbose argument does not work. The compiler throws errors like ../src\cones.c(158) : error C2065: '__func__' : undeclared identifier.

Troubleshooting:

• Linker error while running setup.py install. This can be caused by having the wrong blas_lapack libraries, having them in the wrong place, or not having corrected the setup.py.
• Compiler error '__func__' : undeclared identifier. This is caused by using the --extraverbose for the installation.
• Compiler error: stdint.h not found. See this.

I hope this helps anyone trying to install scs on windows!

[bodono]

Thanks for this very detailed guide! It will really help other people who want to install on windows anaconda. What steps do you think can we take in the SCS codebase to make this easier?

[clagms]

I'm not sure how to make the process completely automated.
However, as starting points:

• Correct python\setup.py (as done in Step 6)
• Maybe ship the blas/lapack libraries (and their dependencies) with the installation, to avoid most of the above steps. An important point is that these need to be pre-built for the appropriate platform. And I could not find a solution to 64 bits.

[bodono]

I have made the change as in step 6, and also hopefully fixed the --extraverbose issue.

[SteveDiamond]

@bodono it looks like you got installation working on windows. Could you cut a new SCS version and put it on PyPi?

[bodono]

Ok, I didn't change much but I will cut a new release to PyPi when I get a chance.

[bodono]

Just uploaded version 1.2.7.

[Boscop]

Hi, I tried to install scs via conda on Windows 10, but I get an error:

Collecting scs==2.1.2
  Downloading scs-2.1.2.tar.gz (3.5 MB)

Pip subprocess error:
The system cannot find the path specified.
The system cannot find the path specified.
    ERROR: Command errored out with exit status 1:
     command: 'D:\miniconda3\envs\ml4t\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-7l7nnm4m\\scs_148cd26a27f44bf684e2636cadf70489\\setup.py'"'"'; __file__='"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-7l7nnm4m\\scs_148cd26a27f44bf684e2636cadf70489\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j'
         cwd: C:\Users\me\AppData\Local\Temp\pip-install-7l7nnm4m\scs_148cd26a27f44bf684e2636cadf70489\
    Complete output (97 lines):
    Namespace(blas64=False, extraverbose=False, float32=False, gpu=False, int32=False, scs=False)
    running egg_info
    creating C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j\scs.egg-info
    writing C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j\scs.egg-info\PKG-INFO
    writing dependency_links to C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j\scs.egg-info\dependency_links.txt
    writing requirements to C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j\scs.egg-info\requires.txt
    writing top-level names to C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j\scs.egg-info\top_level.txt
    writing manifest file 'C:\Users\me\AppData\Local\Temp\pip-pip-egg-info-7mktfr4j\scs.egg-info\SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\me\AppData\Local\Temp\pip-install-7l7nnm4m\scs_148cd26a27f44bf684e2636cadf70489\setup.py", line 229, in <module>
        run_install()
      File "C:\Users\me\AppData\Local\Temp\pip-install-7l7nnm4m\scs_148cd26a27f44bf684e2636cadf70489\setup.py", line 226, in run_install
        install_scs()
      File "C:\Users\me\AppData\Local\Temp\pip-install-7l7nnm4m\scs_148cd26a27f44bf684e2636cadf70489\setup.py", line 201, in install_scs
        setup(
      File "D:\miniconda3\envs\ml4t\lib\site-packages\setuptools\__init__.py", line 163, in setup
        return distutils.core.setup(**attrs)
      File "D:\miniconda3\envs\ml4t\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "D:\miniconda3\envs\ml4t\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "D:\miniconda3\envs\ml4t\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\setuptools\command\egg_info.py", line 297, in run
        self.find_sources()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\setuptools\command\egg_info.py", line 304, in find_sources
        mm.run()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\setuptools\command\egg_info.py", line 535, in run
        self.add_defaults()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\setuptools\command\egg_info.py", line 571, in add_defaults
        sdist.add_defaults(self)
      File "D:\miniconda3\envs\ml4t\lib\distutils\command\sdist.py", line 228, in add_defaults
        self._add_defaults_ext()
      File "D:\miniconda3\envs\ml4t\lib\distutils\command\sdist.py", line 311, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "D:\miniconda3\envs\ml4t\lib\distutils\cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "D:\miniconda3\envs\ml4t\lib\distutils\cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "C:\Users\me\AppData\Local\Temp\pip-install-7l7nnm4m\scs_148cd26a27f44bf684e2636cadf70489\setup.py", line 112, in finalize_options
        blas_info, lapack_info = get_infos()
      File "C:\Users\me\AppData\Local\Temp\pip-install-7l7nnm4m\scs_148cd26a27f44bf684e2636cadf70489\setup.py", line 82, in get_infos
        blas_info = get_info('blas_opt')
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 584, in get_info
        return cl().get_info(notfound_action)
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 844, in get_info
        self.calc_info()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 1989, in calc_info
        if self._calc_info(blas):
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 1981, in _calc_info
        return getattr(self, '_calc_info_{}'.format(name))()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 1932, in _calc_info_openblas        info = get_info('openblas')
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 584, in get_info
        return cl().get_info(notfound_action)
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 844, in get_info
        self.calc_info()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 2195, in calc_info
        info = self._calc_info()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\system_info.py", line 2166, in _calc_info
        f = new_fcompiler(c_compiler=c)
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\fcompiler\__init__.py", line 874, in new_fcompiler        load_all_fcompiler_classes()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\fcompiler\__init__.py", line 775, in load_all_fcompiler_classes
        __import__ (module_name)
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\fcompiler\compaq.py", line 54, in <module>
        class CompaqVisualFCompiler(FCompiler):
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\fcompiler\compaq.py", line 78, in CompaqVisualFCompiler
        m.initialize()
      File "D:\miniconda3\envs\ml4t\lib\site-packages\numpy\distutils\msvccompiler.py", line 51, in initialize
        os.environ['lib'] = _merge(environ_lib, os.environ['lib'])
      File "D:\miniconda3\envs\ml4t\lib\os.py", line 675, in __getitem__
        raise KeyError(key) from None
    KeyError: 'lib'
    blas_info:
        libraries = ['cblas', 'blas', 'cblas', 'blas', 'cblas', 'blas']
        library_dirs = ['D:/miniconda3/envs/ml4t\\Library\\lib']
        include_dirs = ['D:/miniconda3/envs/ml4t\\Library\\include']
        language = f77
        define_macros = [('HAVE_CBLAS', None)]
    blas_opt_info:
        define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
        libraries = ['cblas', 'blas', 'cblas', 'blas', 'cblas', 'blas']
        library_dirs = ['D:/miniconda3/envs/ml4t\\Library\\lib']
        include_dirs = ['D:/miniconda3/envs/ml4t\\Library\\include']
        language = f77
    lapack_info:
        libraries = ['lapack', 'blas', 'lapack', 'blas']
        library_dirs = ['D:/miniconda3/envs/ml4t\\Library\\lib']
        language = f77
    lapack_opt_info:
        libraries = ['lapack', 'blas', 'lapack', 'blas', 'cblas', 'blas', 'cblas', 'blas', 'cblas', 'blas']
        library_dirs = ['D:/miniconda3/envs/ml4t\\Library\\lib']
        language = f77
        define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
        include_dirs = ['D:/miniconda3/envs/ml4t\\Library\\include']
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/1a/72/33be87cce255d4e9dbbfef547e9fd6ec7ee94d0d0910bb2b13badea3fbbe/scs-2.1.2.tar.gz#sha256=667ed6019bb4e2f925bd9291161d2c61cc0077443094437b703ea905333fd585 (from https://pypi.org/simple/scs/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement scs==2.1.2 (from versions: 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.5, 1.0.6, 1.0.7, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.7, 1.2.1, 1.2.2, 1.2.3, 1.2.5, 1.2.6, 1.2.7, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.1.post2, 2.1.2, 2.1.3, 2.1.4)
ERROR: No matching distribution found for scs==2.1.2

failed

CondaEnvException: Pip failed

I don't know why the script is looking for Visual Studio 2017, I only have 2019.
The folder C:\Program Files (x86)\Microsoft Visual Studio\2017 doesn't exist, only C:\Program Files (x86)\Microsoft Visual Studio\2019:

Apparently the script doesn't find MSVC through registry lookup but I found this registry key:

Looking at this:
https://github.com/python/cpython/blob/e14d5ae5447ae28fc4828a9cee8e9007f9c30700/Lib/distutils/msvccompiler.py#L598
if I understand it correctly, it means that my VC folder in the above screenshot should have a child folder named VC_OBJECTS_PLATFORM_INFO, but it doesn't have one, for some reason.
(Btw, when I search in the whole registry for VC_OBJECTS_PLATFORM_INFO it finds nothing.)
Maybe in MSVC 2019 the key VC_OBJECTS_PLATFORM_INFO does not exist anymore, and the MSVC detection script needs to be adapted?

(Btw, more context here: stefan-jansen/machine-learning-for-trading#177)

Any idea how to make it work? :)
I would greatly appreciate it!

[h-vetinari]

When installing with conda, you should need neither the download from PyPI, nor a compiler. Something is definitely off here.

Could you please post the outputs of (within the environment that you want to install scs in) of conda info & conda list, as well as the exact command you're trying to use to install.

Ideally, to keep things readable, please copy the outputs of conda info and conda list in something like the following (in the text-view of your GH comment):

<details>
<summary>Output of <code>conda info</code></summary>

`` ` [remove space between backticks]
[output of conda info]
`` ` [remove space between backticks]

</details>

[Boscop]

@h-vetinari Thanks for your help, I really appreciate it :)
(I'm new to the conda ecosystem, coming from Rust.)

This is the output:

Output of conda info

     active environment : base
    active env location : D:\miniconda3
            shell level : 1
       user config file : C:\Users\me\.condarc
 populated config files :
          conda version : 4.10.3
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __cuda=11.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : D:\miniconda3  (writable)
      conda av data dir : D:\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : D:\miniconda3\pkgs
                          C:\Users\me\.conda\pkgs
                          C:\Users\me\AppData\Local\conda\conda\pkgs
       envs directories : D:\miniconda3\envs
                          C:\Users\me\.conda\envs
                          C:\Users\me\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.10.3 requests/2.24.0 CPython/3.8.5 Windows/10 Windows/10.0.17763
          administrator : False
             netrc file : None
           offline mode : False

Output of conda list

# packages in environment at D:\miniconda3:
#
# Name                    Version                   Build  Channel
brotlipy                  0.7.0           py38h2bbff1b_1003
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2021.5.30            h5b45459_0    conda-forge
certifi                   2021.5.30        py38haa244fe_0    conda-forge
cffi                      1.14.3           py38hcd4344a_2
chardet                   3.0.4           py38haa95532_1003
conda                     4.10.3           py38haa244fe_0    conda-forge
conda-package-handling    1.7.2            py38h76e460a_0
console_shortcut          0.1.1                         4
cryptography              3.2.1            py38hcd4344a_1
idna                      2.10                       py_0
krb5                      1.19.1               hbae68bd_0    conda-forge
libarchive                3.5.1                hb45042f_2    conda-forge
libcurl                   7.77.0               h789b8ee_0    conda-forge
libiconv                  1.16                 he774522_0    conda-forge
libsolv                   0.7.19               h7755175_4    conda-forge
libssh2                   1.9.0                h680486a_6    conda-forge
libxml2                   2.9.12               hf5bbc77_0    conda-forge
lz4-c                     1.9.3                h8ffe710_0    conda-forge
lzo                       2.10              he774522_1000    conda-forge
mamba                     0.14.1           py38hdd88130_0    conda-forge
menuinst                  1.4.16           py38he774522_1
openssl                   1.1.1k               h8ffe710_0    conda-forge
pip                       20.2.4           py38haa95532_0
powershell_shortcut       0.0.1                         3
pycosat                   0.6.3            py38h2bbff1b_0
pycparser                 2.20                       py_2
pyopenssl                 19.1.0             pyhd3eb1b0_1
pysocks                   1.7.1            py38haa95532_0
python                    3.8.5                h5fd99cc_1
python_abi                3.8                      2_cp38    conda-forge
pywin32                   227              py38he774522_1
reproc                    14.2.1               h8ffe710_0    conda-forge
reproc-cpp                14.2.1               h0e60522_0    conda-forge
requests                  2.24.0                     py_0
ruamel_yaml               0.15.87          py38he774522_1
setuptools                50.3.1           py38haa95532_1
six                       1.15.0           py38haa95532_0
sqlite                    3.33.0               h2a8f88b_0
tk                        8.6.10               h8ffe710_1    conda-forge
tqdm                      4.51.0             pyhd3eb1b0_0
urllib3                   1.25.11                    py_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_3
wheel                     0.35.1             pyhd3eb1b0_0
win_inet_pton             1.1.0            py38haa95532_0
wincertstore              0.2                      py38_0
xz                        5.2.5                h62dcd97_1    conda-forge
yaml                      0.2.5                he774522_0
zlib                      1.2.11               h62dcd97_4
zstd                      1.5.0                h6255e5f_0    conda-forge

The exact command I'm running, that gave the scs error, is:

mamba env create -f installation/windows/ml4t.yml

as described here.
This ml4t.yml file can be found here:
https://github.com/stefan-jansen/machine-learning-for-trading/blob/2fe1120c9c71658b00a2e620f11836b8c4ecdf97/installation/windows/ml4t.yml#L474

[h-vetinari]

Well.. that environment yaml file contains several things that are not ideal, chief of which is putting scs under the pip-section, when it is easily available under conda-forge. Mixing pip & conda is heavily discouraged -- pip does not understand conda, and will trample over things and can easily bust the environment.

Same goes for osqp, ecos, cvxpy & qdldl. Please move them from the pip-section up into the main section (handled by conda), also removing the .post0.

Finally, this environment is not following best practices that there should be as few channels as possible. conda-forge & anaconda defaults are compatible with each other (at great effort to make sure everything is ABI-compatible), but other channels may for all intents and purposes contain stuff that blows up your setup. I didn't check all packages in that environment, but almost everything should be available from conda-forge, without the need for extra channels.

Aside from trying if things work with:

channels:
  - conda-forge
  - defaults
  - anaconda
  - ml4t  # assuming this is necessary; if there are custom packages that aren't in the above channels
 [and nothing more]

I would at the very least recommend ordering conda-forge & defaults first in your channel list, as they represent the highest-quality packaging you can expect for - essentially - any package (actually, conda-forge packaging even assumes conda config --set channel_priority strict for the most part, and little if any effort will be made if other packages break your environment without this setting).

[h-vetinari]

@Boscop, any success?

[Boscop]

Haven't had time to try it yet but I'll try it as soon as I'm back. Thanks :)