New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pip packages #96

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@darkdreamingdan

darkdreamingdan commented Jul 20, 2017

This adds support for pip packages for constructor. It roughly works the following way:

  • Similar to pip support in conda itself, each line of the pip section in construct.yml is compiled into a requirements.txt, and placed into a pip/ cache directory
  • pip download -r requirements.txt is executed (with some extra parameters) which downloads all relevant packages according to the spec. This is also placed in the pip/ cache directory
  • The pip/ cache directory is copied into $PREFIX/pip (Linux) and NSIS does roughly the same thing for Windows (analogous to the conda pkg directory).
  • After installation of conda, and pip cache directory is copied over, pip install --no-index --find-links $PREFIX/pip -r $PREFIX/pip/requirements.txt. This installs offline the previously downloaded packages.

darkdreamingdan added some commits Jul 20, 2017

Fixed building NSIS installer from other working directories
This should be a bit more robust to running constructor in different
environments (i.e. development)
@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Jul 20, 2017

This is interesting work but I am not sure we want to encourage mixing pip and conda packages since their compatibility levels are somewhat dicey.

Can you not make conda packages for the software you are interested in instead?

@darkdreamingdan

This comment has been minimized.

darkdreamingdan commented Jul 21, 2017

It's a fair point - I can only explain our business use case.

Due to the various advantages of Conda, we've been switching away from the official Python distribution towards Conda. Support for pip in environment.yml was one of the reasons we were able to do this. Overall, Conda has made setup for developers a whole lot easier across platforms. I had actually considered individually converting packages to conda initially, but it puts a large burden on 'meta-development' when we're just trying to rapidly build our core product with existing packages.

We're currently also building our deployment workflow around conda - beyond developers but also to package the entire installer to end-users. Ultimately, wanting pip packages is intended to ease focus on developing our deployment pathways and allow us to quickly and easily add new python packages. We currently use a combination of both conda and pip packages, mainly opting for pip when the package is either outdated or not available within existing conda repositories. Doing it ourself was just task duplication, when there's a perfectly good repository for them.

Regarding your point about mixing - I understand philosophically and I'm sure you guys want to move away from pip as far as possible. As I say, we heavily make use of the 'pip' feature within environment.yml (which has been working flawlessly for us) so this PR was a natural extension to that. Hence, in regards to this I can only suggest discouraging via other means (e.g. docs) as a middle ground rather than not providing the functionality entirely.

I'll leave it to you guys regarding your direction of travel - for us we would probably have no choice but to maintain a separate fork which we'd rather not do.

@darkdreamingdan

This comment has been minimized.

darkdreamingdan commented Jul 31, 2017

Anymore thoughts on this? We're beginning our internal build processes and would like to make a decision on whether we fork or use the core codebase.

@mingwandroid

This comment has been minimized.

Collaborator

mingwandroid commented Jul 31, 2017

I will raise it as an agenda item for our team meeting on Thursday.

@msarahan

This comment has been minimized.

Contributor

msarahan commented Aug 3, 2017

We think that this is part of improving the unification of conda and conda-env (which provided your pip support in environment.yml) generally. We are sadly too busy with preparing our own internal build system in preparation for the next Anaconda release to tackle this right now. I think you'll need to fork constructor if you need this functionality right now. One other alternative you may not be aware of is that you can have constructor be based on alternative channels. In particular, the conda-forge channel, representing a community effort towards packaging, is often more up-to-date than the packages that we are able to provide. If you can use the community packages, you may be able to avoid your fork. Conda-forge may also be an option for maintaining your development infrastructure as conda packages. It greatly reduces the workload for building packages for each platform.

@darkdreamingdan

This comment has been minimized.

darkdreamingdan commented Aug 3, 2017

Thanks for explaining and added advice. We're aware of conda-forge, but have found that still the coverage is not sufficient. I look forward to the new build system, and we'll make do with forking for now.

Please feel free to give me a poke should circumstances change.

@axsauze

This comment has been minimized.

axsauze commented Sep 18, 2017

+1 for this functionality - I think there are cases where PIP packages should be supported somehow in the installer, especially as the change submitted is not as significant, and the benefits are much larger, as it addresses backwards compatibility

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment