Add support for pip packages

[darkdreamingdan]

This adds support for pip packages for constructor. It roughly works the following way:

• Similar to pip support in conda itself, each line of the pip section in construct.yml is compiled into a requirements.txt, and placed into a pip/ cache directory
• pip download -r requirements.txt is executed (with some extra parameters) which downloads all relevant packages according to the spec. This is also placed in the pip/ cache directory
• The pip/ cache directory is copied into $PREFIX/pip (Linux) and NSIS does roughly the same thing for Windows (analogous to the conda pkg directory).
• After installation of conda, and pip cache directory is copied over, pip install --no-index --find-links $PREFIX/pip -r $PREFIX/pip/requirements.txt. This installs offline the previously downloaded packages.

[mingwandroid]

This is interesting work but I am not sure we want to encourage mixing pip and conda packages since their compatibility levels are somewhat dicey.

Can you not make conda packages for the software you are interested in instead?

[darkdreamingdan]

It's a fair point - I can only explain our business use case.

Due to the various advantages of Conda, we've been switching away from the official Python distribution towards Conda. Support for pip in environment.yml was one of the reasons we were able to do this. Overall, Conda has made setup for developers a whole lot easier across platforms. I had actually considered individually converting packages to conda initially, but it puts a large burden on 'meta-development' when we're just trying to rapidly build our core product with existing packages.

We're currently also building our deployment workflow around conda - beyond developers but also to package the entire installer to end-users. Ultimately, wanting pip packages is intended to ease focus on developing our deployment pathways and allow us to quickly and easily add new python packages. We currently use a combination of both conda and pip packages, mainly opting for pip when the package is either outdated or not available within existing conda repositories. Doing it ourself was just task duplication, when there's a perfectly good repository for them.

Regarding your point about mixing - I understand philosophically and I'm sure you guys want to move away from pip as far as possible. As I say, we heavily make use of the 'pip' feature within environment.yml (which has been working flawlessly for us) so this PR was a natural extension to that. Hence, in regards to this I can only suggest discouraging via other means (e.g. docs) as a middle ground rather than not providing the functionality entirely.

I'll leave it to you guys regarding your direction of travel - for us we would probably have no choice but to maintain a separate fork which we'd rather not do.

[darkdreamingdan]

Anymore thoughts on this? We're beginning our internal build processes and would like to make a decision on whether we fork or use the core codebase.

[mingwandroid]

I will raise it as an agenda item for our team meeting on Thursday.

[msarahan]

We think that this is part of improving the unification of conda and conda-env (which provided your pip support in environment.yml) generally. We are sadly too busy with preparing our own internal build system in preparation for the next Anaconda release to tackle this right now. I think you'll need to fork constructor if you need this functionality right now. One other alternative you may not be aware of is that you can have constructor be based on alternative channels. In particular, the conda-forge channel, representing a community effort towards packaging, is often more up-to-date than the packages that we are able to provide. If you can use the community packages, you may be able to avoid your fork. Conda-forge may also be an option for maintaining your development infrastructure as conda packages. It greatly reduces the workload for building packages for each platform.

[darkdreamingdan]

Thanks for explaining and added advice. We're aware of conda-forge, but have found that still the coverage is not sufficient. I look forward to the new build system, and we'll make do with forking for now.

Please feel free to give me a poke should circumstances change.

[axsauze]

+1 for this functionality - I think there are cases where PIP packages should be supported somehow in the installer, especially as the change submitted is not as significant, and the benefits are much larger, as it addresses backwards compatibility