Remove pip bundle from pip2pi

[themartorana]

This uses pip install -d ... instead of pip bundle to pre-package the .tar.gz files.

[themartorana]

There are a series of commits where I was making my own pip-manifest.txt for comparison sake. I abuse branches, forgetting pull requests show the entire commit stack anyway :)

[wolever]

Interesting! Doing a quick binary search, it looks like pip install -d only started to install dependencies as of pip >= 1.1… which, I guess, is why I didn't use it initially.

Anyway, this looks awesome. I'll be doing some testing to make sure it doesn't break things, but I'm excited to get this merged.

The only thing: could you add an explicit version check for pip >= 1.1, and fail if it isn't found?

I'd like this because 1.1 isn't quite two years old yet, so I expect we'll find people in the wild using it… and the failure — that only the specified packages, and not their dependencies — will be surprising and hard to track down.

[themartorana]

That should do it. Let me know if it feels ok now. Thanks :)

[wolever]

Feelin' awesome! I'll test this over the next couple of days, then cut a release.

Also, if you're working on stuff… would you mind switching it up from using check_call(["pip", …]) to import pip; pip.main(…)? This will also knock off #5 and generally make everything awesome. Otherwise, I'll figure out how to do it.

[themartorana]

I am working on (just about done) making dir2pi create a directory structure more similar to https://pypi.python.org/simple/ - mostly because I want to mirror my repo on S3 - and thus need to get rid of symlinks. I'll submit a pull request for that separately, but it's totally up to you if you want to incorporate it or not.

In the meantime, I can totally put in the pip.main call. I'll read up on #5 and make sure it doesn't break the tests.

[wolever]

Woo! That would be incredible!

Also, I would be totally up for making dir2pi more similar to pypi.python.org/simple, and especially if there was some way to easily push it to s3.

I've actually been thinking about rejigging things there just a bit, since there's a bug right now where new package page indexes overwrite old ones… so, for example, if you run pip2pi /tmp/ fish==1.0, then run pip2pi /tmp/ fish==1.1, the index will only show fish==1.1… so just keep that in mind as you're working on stuff, in case that becomes relevant. Basically, I think the only change will be listing the remote directory before writing the index file… so it shouldn't be too bad, and I can make sure that the "backends" are pluggable, so that you could put in an S3 backend without too much drama.

[themartorana]

Shoot, hadn't thought about backends. Part of the reason I wrote this was to support multiple package versions. Basically, it goes like this:

1. Move any top-level packages in to the appropriate place
2. Crawl the package dir, recreate Simple index files

But when I submit the pull request, take a look at what's happening and see if it makes sense. This should work very well with rsync. Putting in an S3 backend might be something I tackle next.

[wolever]

Okay, will do. Ya, that's basically what I was going to do as well. Let me know if you're getting blocked on something — having someone ready to submit patches against this feature would be some good motivation to get it done ;)

[themartorana]

This should address #5 and your concerns with versions < 1.1, and take advantage of pip.main.

[themartorana]

By the way, here's an about-finished first pass at reworking Simple indexing, multiple-package-versions, index updating, and whatnot.

https://github.com/themartorana/pip2pi/tree/package_list

[wolever]

Awesome. Can you submit a pull request so we can discuss there?
On 2013-11-27, at 9:04 PM, David Martorana notifications@github.com wrote:

By the way, here's an about-finished first pass at reworking Simple indexing, multiple-package-versions, index updating, and whatnot.

https://github.com/themartorana/pip2pi/tree/package_list

—
Reply to this email directly or view it on GitHub.

(416) 906-0403
@wolever

[themartorana]

I can't submit another pull request until this one is accepted. Is that something you plan to do in the near future?

Thanks!

Dave

[wolever]

Urg! Sorry! I didn't realize you could only have one open PR at a time. I'll merge this and cut a new release today.