`pip install blosc` failing on Windows 10, Anaconda shell & MINGW64, Jupyter 5.0.0, Python 3.6.4

[karlwbaker]

pip install blosc via Anaconda prompt and git bash (MINGW64) is failing on my Windows 10, Jupyter 5.0.0, Python 3.6.4. I have never tried to install it before tonight. Text of error (and "success") messages is below.

One of the errors in Anaconda shell references "The system cannot find the path specified: 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\PlatformSDK\lib`." I uninstalled Visual Studio some months ago, but then this is not the only error flagged on that attempt.

I also tried conda install blosc which returned "All requested packages already installed." I restarted Jupyter but unfortunately, import blosc still reports "ImportError: No module named blosc."

Finally, it's worth pointing out that my final attempt in MINGW64 seemed to be successful (see second message below). Yet even after restarting my machine I get a ModuleNotFoundError.

Looking forward to using this tool, hopefully in conjunction with pickle. Alone pickle is spitting out pickles that are 3 times the size of my PNG data files. Any help is appreciated.

---

In Anaconda shell:

(py36) C:\Users\karls>pip install blosc
Collecting blosc
  Using cached https://files.pythonhosted.org/packages/f4/8f/024a70d6af0b0dd736954a16b2a4be544a2b61ef6c0f660e5221accd4f8c/blosc-1.7.0.tar.gz
Building wheels for collected packages: blosc
  Building wheel for blosc (setup.py) ... error
  Complete output from command c:\users\karls\anaconda2\envs\py36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\karls\\AppData\\Local\\Temp\\pip-install-wtukfagq\\blosc\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\karls\AppData\Local\Temp\pip-wheel-kxzq_yar --python-tag cp36:
  SSE2 detected
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.6
  creating build\lib.win-amd64-3.6\blosc
  copying blosc\test.py -> build\lib.win-amd64-3.6\blosc
  copying blosc\toplevel.py -> build\lib.win-amd64-3.6\blosc
  copying blosc\version.py -> build\lib.win-amd64-3.6\blosc
  copying blosc\__init__.py -> build\lib.win-amd64-3.6\blosc
  running build_clib
  building 'lz4' library
  error: [WinError 3] The system cannot find the path specified: 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\PlatformSDK\\lib'
  SSE2 detected
  running bdist_wheel
  running build
  running build_py
  running build_clib
  building 'lz4' library
  error: [WinError 3] The system cannot find the path specified: 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\PlatformSDK\\lib'

  ----------------------------------------
  Failed building wheel for blosc
  Running setup.py clean for blosc
Failed to build blosc
Installing collected packages: blosc
  Running setup.py install for blosc ... error
    Complete output from command c:\users\karls\anaconda2\envs\py36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\karls\\AppData\\Local\\Temp\\pip-install-wtukfagq\\blosc\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\karls\AppData\Local\Temp\pip-record-09pc3gyw\install-record.txt --single-version-externally-managed --compile:
    SSE2 detected
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.6
    creating build\lib.win-amd64-3.6\blosc
    copying blosc\test.py -> build\lib.win-amd64-3.6\blosc
    copying blosc\toplevel.py -> build\lib.win-amd64-3.6\blosc
    copying blosc\version.py -> build\lib.win-amd64-3.6\blosc
    copying blosc\__init__.py -> build\lib.win-amd64-3.6\blosc
    running build_clib
    building 'lz4' library
    error: [WinError 3] The system cannot find the path specified: 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\PlatformSDK\\lib'
    SSE2 detected
    running install
    running build
    running build_py
    running build_clib
    building 'lz4' library
    error: [WinError 3] The system cannot find the path specified: 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\PlatformSDK\\lib'

    ----------------------------------------
Command "c:\users\karls\anaconda2\envs\py36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\karls\\AppData\\Local\\Temp\\pip-install-wtukfagq\\blosc\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\karls\AppData\Local\Temp\pip-record-09pc3gyw\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\karls\AppData\Local\Temp\pip-install-wtukfagq\blosc\

---

In MINGW64:

karls@DESKTOP-VAQRJL8 MINGW64 ~ (master)
$ activate py36

karls@DESKTOP-VAQRJL8 MINGW64 ~ (master)
$ conda install blosc
Solving environment: done

## Package Plan ##

  environment location: C:\Users\karls\Anaconda2

  added / updated specs:
    - blosc


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    snappy-1.1.7               |       he46498f_3          88 KB
    blosc-1.15.0               |       hc65f11a_0         422 KB
    ------------------------------------------------------------
                                           Total:         510 KB

The following NEW packages will be INSTALLED:

    blosc:           1.15.0-hc65f11a_0
    snappy:          1.1.7-he46498f_3

The following packages will be UPDATED:

    ca-certificates: 2018.03.07-0      --> 2018.12.5-0

Proceed ([y]/n)? y


Downloading and Extracting Packages
snappy-1.1.7         | 88 KB     | ######################################################## | 100%
blosc-1.15.0         | 422 KB    | ######################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Yet I still get ModuleNotFoundError.

[esc]

Thank you very moush @karlwbaker for trying out python-blosc and for reporting the issues that you encountered.

Two things come to mind regarding you two cases:

In order to build the python-blosc package from PyPi you must have some kind of C compiler installed. Can you check this and verify that you have one? The errors seem to suggest it is trying to reference some kind of visual-studio c compiler.

I am not sure, where the conda comand find the blosc package. Could you check if you have any additional channels enabled? Perhaps conda-forge? Additionally, the version 1.15 for python-blosc doesn't make any sense. I think this might install c-blosc, which is the pure c version of blosc. This would also make sense given the fact, that you get a ModuleNotFoundError. IIRC there might be some older python-blosc packages on conda-forge, so maybe you could try conda install python-blosc?

In any case, let us know how you get on.

[karlwbaker]

Thank you, @esc, for your response. conda install python-blosc seemed to install with no issues reported, though its 44 KB size was smaller than I expected. I can't test it out right now but I'll let you know if any issues arise.

Regarding a C compiler, how would I go about verifying which I have installed?

    package                    |            build
    ---------------------------|-----------------
    python-blosc-1.7.0         |   py36he774522_0          44 KB

[esc]

@karlwbaker that sounds like it might be the right thing. Regarding c compilers on Windows, I'm afraid I can't help you there since my Windows experience is somewhat limited.

[robbmcleod]

mingw64 is not the Python compiler supported on Windows. Unless you have compiled all of Python with mingw64 it's likely to have ABI issues if you compile individual modules with it.

You don't need Visual Studio to build Python modules. Microsoft Visual C Build Tools is sufficient:

https://visualstudio.microsoft.com/visual-cpp-build-tools/

The MSVC compiler is called cl.exe.

It seems likely from your error messages that you still have environment variables pointing to your old installation of Visual Studio.

[karlwbaker]

@robbmcleod I did a search for cl.exe but didn't find it on my system. However, I have "Microsoft Visual C++ 2017 Redistributable (x86) - 14.12.25810" and "Microsoft Visual Studio Code" installed. Would either of these, if you happen to know, provide the C compiler I need?

Re the Minimalist GNU for Windows (mingw64) shell: I actually installed python-blosc using the anaconda prompt, though when that failed to properly install blosc I previously also tried mingw64.

Re your warning about mingw: Are you saying that installing some packages via conda install xxx in anaconda prompt and others (ones that won't install via the first method) via pip install xxx in mingw can cause problems? If so, that's good information as I've had to resort to both at times.

While python-blosc has installed and imports in Jupyter, I haven't been able to figure out how to run it with pickle so I can't be sure if the issues I'm having are related to the C compiler or to my code. Maybe you can help me.

I want to do something similar to the following that lets me run bz2 with pickle:

from six.moves import cPickle as pickle
import bz2
with bz2.BZ2File(set_filename, 'wb') as f:
    pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)

The result is pkl files that are 12% of the size of those produced without bz2. How would I go about implementing this with blosc?

[robbmcleod]

However, I have "Microsoft Visual C++ 2017 Redistributable (x86) - 14.12.25810" and "Microsoft Visual Studio Code" installed. Would either of these, if you happen to know, provide the C compiler I need?

No, download the build tools at the link I provided.

Re your warning about mingw: Are you saying that installing some packages via conda install xxx in anaconda prompt and others (ones that won't install via the first method) via pip install xxx in mingw can cause problems? If so, that's good information as I've had to resort to both at times.

Generally not, most libraries that have C-extensions provided pre-compiled binaries. blosc is a bit special in that the snappy codec is difficult to compile on some platforms.

An example of combining blosc with pickle or other common serialization methods:

https://github.com/robbmcleod/bloscpickle

[esc]

@karlwbaker did you get any further? Can we close this issue or is there still something open?

[karlwbaker]

The issue has been resolved with your suggestion to install python-blosc. I still haven't figured out how to get it to work with pickle, but that's another issue for another day. I've got bz2 working with pickle for now. Thanks! Karl

…

On Thu, Jan 31, 2019 at 1:31 PM Emergency Self-Construct < ***@***.***> wrote: @karlwbaker <https://github.com/karlwbaker> did you get any further? Can we close this issue or is there still something open? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#176 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AiZenTCvjsL_oIpihHgrenSPp_6_ducxks5vI2CmgaJpZM4aSxSn> .

[esc]

Thanks for the feedback! I will close the issue now. I would also recommend to take a closer look at bloscpack for serialization purposes:

https://github.com/Blosc/bloscpack

It has a pretty fresh Python-API here:

https://github.com/Blosc/bloscpack/blob/master/README.rst#python-api

And you might be able to leverage that easily for your data.