Raise the crate size limit?

[SimonSapin]

Servo regularly gets bugs like servo/servo#7687 filed because downloading 320 MB of git history for servo/skia takes such a long time. (This repository is used as a git dependency.)

Shallow clones (rust-lang/cargo#1171) would fix this (git clone --depth=1 fetches about 11 MB worth of objects), but according to @Manishearth libgit2 support is unlikely to happen.

We’re playing with hacks like https://github.com/servo/skia-snapshots/blob/master/update.sh , but they seem brittle and require jumping through hoops to make updates.

Another option would be to publish on crates.io, which has a well-established process to make updates. However cargo package creates a 12 MB .crate file, which is over the current 10 MB limit. This is a big pile of C++ source code. Splitting it up seems challenging at best.

Could the limit be raised a bit, say to 15 or 20 MB? Or, could we add a way for crates.io administrators to grant a derogation to individual crates?

CC @metajack @larsbergstrom

[alexcrichton]

I don't think there's a limit we can set at which point there will be no requests to raise it further, so my preferred solution to this is to add the ability for each crate to specify the maximum file size limit, and then override this on a per-crate basis. This would not be configurable by crate authors themselves, however.

[metajack]

Why does a limit exist at all? If one has to exist, it seems like it would be easy to ask for and grant crate specific exceptions, and that almost all such requests would get granted. Disk space is super cheap, and if crates are large, there's little people can do about it in some cases.

I'm not sure why your suggestion of a file size limit would be useful. Or how this is useful if crate authors can't aren't allowed to configure it.

[steveklabnik]

With Rubygems, we'd have some people accidentally screw up packaging and submit multi-gigabyte packages. Having some kind of limit is nice.

Sent from my iPhone

On Sep 22, 2015, at 16:05, Jack Moffitt notifications@github.com wrote:

Why does a limit exist at all? If one has to exist, it seems like it would be easy to ask for and grant crate specific exceptions, and that almost all such requests would get granted. Disk space is super cheap, and if crates are large, there's little people can do about it in some cases.

I'm not sure why your suggestion of a file size limit would be useful. Or how this is useful if crate authors can't aren't allowed to configure it.

―
Reply to this email directly or view it on GitHub.

[SimonSapin]

In the case of skia specifically, it looks like we’ll be able to exclude from the archive some files that are not actually used during build and get under the 10 MB limit: servo/skia#73. We’ll need to see if we can do the same for all the recursive dependencies.

[metajack]

10MB seems like a really low limit considering how many well known projects are much larger than that. It was pointed out in Servo IRC that libicu is 25MB compressed. So if there is going to be a limit to prevent accidents, I suggest it should be much higher and easier to get overridden for specific crates.

[alexcrichton]

The limit exists for what @steveklabnik said as well as to prevent abuse of maliciously uploading many gigabytes of data. It's also useful to have a per-crate limit I think instead of one global limit as there are some cases, like skia, that legitimately want a larger limit but it shouldn't have to apply to every crate in existence.