Link with the blosc shared library

[tnorth]

This patch removes the embedded blosc code and links with the existing shared library.

Not sure if my fix for setup.py is good enough. Perhaps there is a better way to explicitly require an external library ?
Fixes/feedback welcome!

Tested on Linux x86_64 only.

[esc]

I am 👎 on this PR in its current shape. However not on building against the shared library in general. Let me discuss my points.

Having a shared library is a great thing and certainly a clean way of doing things. However containing the sources like we do, has the great advantage of being able to install the extension from the PyPi using pip. If it were only possible to link against the shared lib, a potential user would have to compile the shared library first, and even use non-python tools (cmake or whatever) to do so, a IMHO significant hurdle.

So, instead I advocate for having two setup.py files. one for compiling statically and one for linking to the shared library. This way, we could still ship python-blosc from PyPi but packagers also have the option of linking against the shared library. You can even use import to share code between them. Alternatively, you could put both setup routines in the same file and control which one is uses using, for example environment variables.

We can talk about using submodules or subtree merge to manages to blosc sources in a different thread.

[tnorth]

Makes sense. I had Linux systems in mind, where package managing enables a straightforwards dependency resolution.
If time allows it, I will have a look at your suggestions, and we can indeed discuss it further. I'll close this push request.
Thanks!
Thibault

[esc]

Yeah, I am also a firm believer in the Linux way of doing everything by packages. However, much time can pass until a suitable package is built and accepted. Please do leave your branch lying around, I may try to create the things I suggested myself.

[FrancescAlted]

Thanks @tnorth for the PR. Yes, I agree with @esc that we have to devise a way to make that compatible with pip installs. This PR can be handy in the future, but for the time being, including Blosc sources is best, IMO.

[avalentino]

Hi guys, just an idea but why blosc, python-blosc and bloscpack are not merged into a single project?

[esc]

@avalentino: what would be the advantages?

[esc]

incidentally, I was working on a PR to support pulling in blosc to python-blosc using submodules today when I started experiencing hardware issues.. This may potentially simplify managing sources in the future.

[FrancescAlted]

@esc that would be great. I have heard people having issues with git subtrees. I hope submodules would work better :) Thanks for your work on this.

[esc]

@FrancescAlted : yeah? what kind of issues?

Sorry to create a prejudice, but submodules are not exactly user friendly... I see we will have to discuss some more, when the PR is ready...

[FrancescAlted]

Hmm, I think the issues where with using submodule not subtree. Apparently it is too easy (automatic) to loose your data. Sorry, I cannot give more details. Well, you are the expert, so I'll take your word on this.

[avalentino]

@avalentino: what would be the advantages?

@esc, well it feels quite natural to me. Blosc is a relatively small library that has python binfings and a command line tool as front-end.

@avalentino yes, but the three libraries are pretty different in spirit. Blosc is an standalone library written in pure C. python-blosc is simple a python wrapper on it, and I can envision other wrappers (did not know about Haskell and node.js, thanks @esc). Finally bloscpack is more than a simply command line interface to Blosc: it is a true format and chances are that it can be converted into a really flexible tool for storing large amounts of binary data.

A single project means a single tar ball, so no more problems with pip installation and no more problems for syncing with the upstream compression library.

Yes, this is the price to pay, but that's life.

[esc]

I agree with the simplicity argument. However Blosc is spreading rapidly. It is included in PyTables, Blaze and there are bindings to haskell and node.js. So I think a pure, minimal project for Blosc is really a must. But yeah, I agree the cost of pip installs and syncing with upstream is quite a drag...

[FrancescAlted]

@avalentino yes, but the three libraries are pretty different in spirit. Blosc is an standalone library written in pure C. python-blosc is simple a python wrapper on it, and I can envision other wrappers (did not know about Haskell and node.js, thanks @esc). Finally bloscpack is more than a simply command line interface to Blosc: it is a true format and chances are that it can be converted into a really flexible tool for storing large amounts of binary data.

And yes, the price to pay is that you have to pack many different packages, but that's life.

[FrancescAlted]

BTW, hope we can somewhat improve the situation by solving the shared library issue, or by using git sumdoule/subtree. Let's see.

[esc]

Great... so I tried to submit a PR and the hub tool just ate the message I spent half an hour writing:

mislav/hub#178

👎

not my day...

[FrancescAlted]

@esc sorry to hear that.