New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: way to set up a mirror #67

Open
carols10cents opened this Issue Nov 23, 2014 · 7 comments

Comments

Projects
None yet
7 participants
@carols10cents
Member

carols10cents commented Nov 23, 2014

I'm so excited crates.io has arrived!! 馃帀 Great work, Alex and everyone!!!

This is a longer-term feature request... one thing that has been an occasional yet painful experience with rubygems.org or npmjs.org is when they're down. At various points in time, my coworkers and I have tried to set up mirrors under our control of both of these and it never went very smoothly, neither on the setup or usage side of things.

I would love love love to have mirroring be something officially supported and encouraged for the health of the community, but I realize this is a difficult problem too :)

Thank you for your work!! 鉂わ笍

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler Nov 23, 2014

Member

This would also involve adding support for non-Github auth (preferably LDAP) and non-S3/Github storage.

Member

sfackler commented Nov 23, 2014

This would also involve adding support for non-Github auth (preferably LDAP) and non-S3/Github storage.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 23, 2014

Member

Architecturally I believe it's relatively easy to do. A "mirror" would essentially just be a clone of the official index along with an updated download url in its configuration. Updating a mirror would just be updating the index and perhaps downloading all the new versions that were created.

Implementing this, however, and getting cargo to think that mirrors and normal registries are the same would be somewhat difficult.

@sfackler, those may be more useful for non-official entire instances, as I would suspect that a mirror would reflect whatever the crates.io site uses. That being said, I'd love for it to be easy to spin up a local instance of a registry to use entirely independently (with a number of configuration options like auth/storage). Might be a bit harder though :)

Member

alexcrichton commented Nov 23, 2014

Architecturally I believe it's relatively easy to do. A "mirror" would essentially just be a clone of the official index along with an updated download url in its configuration. Updating a mirror would just be updating the index and perhaps downloading all the new versions that were created.

Implementing this, however, and getting cargo to think that mirrors and normal registries are the same would be somewhat difficult.

@sfackler, those may be more useful for non-official entire instances, as I would suspect that a mirror would reflect whatever the crates.io site uses. That being said, I'd love for it to be easy to spin up a local instance of a registry to use entirely independently (with a number of configuration options like auth/storage). Might be a bit harder though :)

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler Nov 23, 2014

Member

At work we use Artifactory to both serve as a repository for internal packages and to provide a transparent mirror/cache of the maven.org packages we use. New packages are transparently pulled down as needed, but we don't mirror the whole world since that'd be crazy. Something like that would probably be best served as a separate project, though it'd hopefully be able to pull in a bunch of the logic/types from crates.io.

Member

sfackler commented Nov 23, 2014

At work we use Artifactory to both serve as a repository for internal packages and to provide a transparent mirror/cache of the maven.org packages we use. New packages are transparently pulled down as needed, but we don't mirror the whole world since that'd be crazy. Something like that would probably be best served as a separate project, though it'd hopefully be able to pull in a bunch of the logic/types from crates.io.

@steveklabnik

This comment has been minimized.

Show comment
Hide comment
@weiznich

This comment has been minimized.

Show comment
Hide comment
@weiznich

weiznich Nov 9, 2016

I've written a simple caching server. Similar to Artifactory it pulls down transparently those crates that are really requested. Currently it is only a simple application without any interface or anything. Also it only mirrors an index. It's not possible to add or yank crates to the mirror.

weiznich commented Nov 9, 2016

I've written a simple caching server. Similar to Artifactory it pulls down transparently those crates that are really requested. Currently it is only a simple application without any interface or anything. Also it only mirrors an index. It's not possible to add or yank crates to the mirror.

@gilescope

This comment has been minimized.

Show comment
Hide comment
@gilescope

gilescope commented Jul 19, 2018

This feels linked to rust-lang/rust#44931

@ishitatsuyuki

This comment has been minimized.

Show comment
Hide comment
@ishitatsuyuki

ishitatsuyuki Jul 22, 2018

Member

For initial fetch, we should provide daily database dumps which is less overhead to serve and better compressed.

For dynamic syncing, the Kubernetes model is a very friendly way to organize the API.

Member

ishitatsuyuki commented Jul 22, 2018

For initial fetch, we should provide daily database dumps which is less overhead to serve and better compressed.

For dynamic syncing, the Kubernetes model is a very friendly way to organize the API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment