Packaging multiple wheels in the same package

[nir0s]

So, this isn't much an issue in pip per se as much it is as an idea (yes, I've also posted it in the python-ideas google group. If this doesn't belong here, feel free to delete it.)

As part of my job, I had to provide a way for customers to install plugins in an environment where PyPI isn't accessible. These plugins are sets of Python packages which necessarily depend on one another (i.e. a regular python package with dependencies).

We decided that we want to package sets of wheels together created or downloaded by pip wheel, add relevant metadata, package them together into a single archive (tar.gz or zip) and use the same tool which packs them up to install them later on, on the destination hosts.

We came up with a tool (http://github.com/cloudify-cosmo/wagon) to do just that and that's what we currently use to create and install our plugins.

While wheel solves the problem of generating wheels, there is no single, standard method for taking an entire set of dependencies packaged in a single location and installing them in a different location.

We thought it would be a good idea to propose a PEP for that and wanted to get your feedback before we start writing the proposal.

The logic behind the PEP is that there's a standard for creating a single python package and we would like to propose a layer above it to package multiple python packages either for offline environments or any other requirement a user may have.

We're currently working on a new rearchitected version of that tool to make it more pythonic and clean but the base stays the same.

P.S. You can take a look at the repo (see the rearchitecture branch) and read the README.

I thought this might be relevant as an additional abstraction in pip for creating completely standalone packages.

Does this seem like something that would make sense? Maybe the implementation itself of our tool isn't good enough but the idea still applies? Maybe pip wagon could be a new command which uses pip wheel in the background to create those packages or maybe it should be a feature in wheel itself?

[dholth]

Ah the old wheels within wheels idea.

[dholth]

You could try distutils-Sig mailing list.

Do you store the zips uncompressed so they will compress better in the outside archive?

[nir0s]

I don't actually. Good idea. I'll open an issue for that.

[dstufft]

This would be something like Java's uber-jars yea?

[nir0s]

Yup. Exactly like an uber-jar. We also thought about allowing the addition of resources on the wagon level.

[dstufft]

I'm going to close this for now, not because I think it's a bad idea but because it's not really actionable at this point in time from pip's POV. We would need a defined standard for a Wagon via a PEP before we could do something with it. That's going to have to go through distutils-sig. Once such a thing exists then we can revisit this idea in pip.

[nir0s]

I posted on distutils-sig and would love to get feedback

https://groups.google.com/forum/#!topic/distutils-sig/Qm_1MrJrJnk