Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: way to set up a mirror #67

Closed
carols10cents opened this issue Nov 23, 2014 · 12 comments
Closed

Feature request: way to set up a mirror #67

carols10cents opened this issue Nov 23, 2014 · 12 comments
Labels
C-enhancement Category: Adding new behavior or a change to the way an existing feature works E-big

Comments

@carols10cents
Copy link
Member

I'm so excited crates.io has arrived!! 馃帀 Great work, Alex and everyone!!!

This is a longer-term feature request... one thing that has been an occasional yet painful experience with rubygems.org or npmjs.org is when they're down. At various points in time, my coworkers and I have tried to set up mirrors under our control of both of these and it never went very smoothly, neither on the setup or usage side of things.

I would love love love to have mirroring be something officially supported and encouraged for the health of the community, but I realize this is a difficult problem too :)

Thank you for your work!! 鉂わ笍

@sfackler
Copy link
Member

This would also involve adding support for non-Github auth (preferably LDAP) and non-S3/Github storage.

@alexcrichton
Copy link
Member

Architecturally I believe it's relatively easy to do. A "mirror" would essentially just be a clone of the official index along with an updated download url in its configuration. Updating a mirror would just be updating the index and perhaps downloading all the new versions that were created.

Implementing this, however, and getting cargo to think that mirrors and normal registries are the same would be somewhat difficult.

@sfackler, those may be more useful for non-official entire instances, as I would suspect that a mirror would reflect whatever the crates.io site uses. That being said, I'd love for it to be easy to spin up a local instance of a registry to use entirely independently (with a number of configuration options like auth/storage). Might be a bit harder though :)

@sfackler
Copy link
Member

At work we use Artifactory to both serve as a repository for internal packages and to provide a transparent mirror/cache of the maven.org packages we use. New packages are transparently pulled down as needed, but we don't mirror the whole world since that'd be crazy. Something like that would probably be best served as a separate project, though it'd hopefully be able to pull in a bunch of the logic/types from crates.io.

@steveklabnik
Copy link
Member

https://gmjosack.github.io/posts/dissecting-cratesio-minimum-mirror/

@steveklabnik steveklabnik added the C-enhancement Category: Adding new behavior or a change to the way an existing feature works label Apr 14, 2016
@weiznich
Copy link

weiznich commented Nov 9, 2016

I've written a simple caching server. Similar to Artifactory it pulls down transparently those crates that are really requested. Currently it is only a simple application without any interface or anything. Also it only mirrors an index. It's not possible to add or yank crates to the mirror.

@gilescope
Copy link

This feels linked to rust-lang/rust#44931

@ishitatsuyuki
Copy link
Contributor

For initial fetch, we should provide daily database dumps which is less overhead to serve and better compressed.

For dynamic syncing, the Kubernetes model is a very friendly way to organize the API.

bors added a commit that referenced this issue Dec 17, 2019
鈥fy-3.0.0, r=Turbo87

Bump ember-cli-uglify from 2.0.0 to 3.0.0

Bumps [ember-cli-uglify](https://github.com/ember-cli/ember-cli-uglify) from 2.0.0 to 3.0.0.
<details>
<summary>Release notes</summary>

*Sourced from [ember-cli-uglify's releases](https://github.com/ember-cli/ember-cli-uglify/releases).*

> ## v3.0.0
> #### 馃挜 Breaking Change
> * [#102](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/102) Drop support for Node.js 6 ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#64](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/64) Drop support for Node.js 4 ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### 馃彔 Internal
> * [#68](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/68) TravisCI: Remove deprecated `sudo: false` option ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#67](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/67) Remove unneeded and confusing dependency ([@&#8203;jelhan](https://github.com/jelhan))
> * [#66](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/66) CI: Run only for `master` branch, tags and PRs ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#44](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/44) Fix CI ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### Committers: 2
> - Tobias Bieniek ([@&#8203;Turbo87](https://github.com/Turbo87))
> - jelhan ([@&#8203;jelhan](https://github.com/jelhan))
</details>
<details>
<summary>Changelog</summary>

*Sourced from [ember-cli-uglify's changelog](https://github.com/ember-cli/ember-cli-uglify/blob/master/CHANGELOG.md).*

> ## v3.0.0 (2019-04-12)
>
> #### 馃挜 Breaking Change
> * [#102](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/102) Drop support for Node.js 6 ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#64](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/64) Drop support for Node.js 4 ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### 馃彔 Internal
> * [#68](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/68) TravisCI: Remove deprecated `sudo: false` option ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#67](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/67) Remove unneeded and confusing dependency ([@&#8203;jelhan](https://github.com/jelhan))
> * [#66](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/66) CI: Run only for `master` branch, tags and PRs ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#44](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/44) Fix CI ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### Committers: 2
> - Tobias Bieniek ([@&#8203;Turbo87](https://github.com/Turbo87))
> - jelhan ([@&#8203;jelhan](https://github.com/jelhan))
>
>
> ## v2.1.0 (2018-03-27)
>
> #### 馃殌 Enhancement
> * [#37](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/37) Run uglify in parallel ([@&#8203;mikrostew](https://github.com/mikrostew))
>
> #### Committers: 1
> - Michael Stewart ([@&#8203;mikrostew](https://github.com/mikrostew))
>
>
> ## v2.0.2 (2018-02-20)
>
> #### 馃殌 Enhancement
> * [#34](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/34) Updates broccoli-uglify-sourcemap to version 2.0.1 ([@&#8203;Duder-onomy](https://github.com/Duder-onomy))
>
> #### 馃彔 Internal
> * [#36](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/36) add test (basically, a default app built鈥檚 tests run even post uglifi鈥 ([@&#8203;stefanpenner](https://github.com/stefanpenner))
>
> #### Committers: 2
> - Greg Larrenaga ([@&#8203;Duder-onomy](https://github.com/Duder-onomy))
> - Stefan Penner ([@&#8203;stefanpenner](https://github.com/stefanpenner))
</details>
<details>
<summary>Commits</summary>

- [`430cb9a`](ember-cli/ember-cli-terser@430cb9a) v3.0.0
- [`9540447`](ember-cli/ember-cli-terser@9540447) Update Changelog
- [`b051655`](ember-cli/ember-cli-terser@b051655) Build(deps-dev): Bump ember-cli from 3.8.2 to 3.9.0 ([#104](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/issues/104))
- [`940606d`](ember-cli/ember-cli-terser@940606d) Build(deps-dev): Bump ember-source from 3.9.0 to 3.9.1
- [`10199ef`](ember-cli/ember-cli-terser@10199ef) Merge pull request [#102](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/issues/102) from Turbo87/node-6
- [`84fed8a`](ember-cli/ember-cli-terser@84fed8a) Build(deps-dev): Bump ember-cli from 3.8.1 to 3.8.2
- [`9cdbcf2`](ember-cli/ember-cli-terser@9cdbcf2) Drop support for Node 6
- [`9ed7828`](ember-cli/ember-cli-terser@9ed7828) Build(deps-dev): Bump ember-source from 3.8.0 to 3.9.0
- [`d778b8a`](ember-cli/ember-cli-terser@d778b8a) Build(deps-dev): Bump ember-cli-babel from 7.6.0 to 7.7.3
- [`bd30ff4`](ember-cli/ember-cli-terser@bd30ff4) Build(deps): Bump broccoli-uglify-sourcemap from 3.0.0 to 3.1.0
- Additional commits viewable in [compare view](ember-cli/ember-cli-terser@v2.0.0...v3.0.0)
</details>
<br />

[![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=ember-cli-uglify&package-manager=npm_and_yarn&previous-version=2.0.0&new-version=3.0.0)](https://dependabot.com/compatibility-score.html?dependency-name=ember-cli-uglify&package-manager=npm_and_yarn&previous-version=2.0.0&new-version=3.0.0)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

**Note:** This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

You can always request more updates by clicking `Bump now` in your [Dependabot dashboard](https://app.dependabot.com).

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)

</details>
@happyaron
Copy link

happyaron commented Oct 16, 2020

Would like to bump this issue a bit since it looks like not getting love from guys for a long time - people from certain parts of the world (in my case, China mainland) does not have decent connectivity to some most used CDNs, so that mirroring is a must for getting a decent speed of using the repository.

Almost all the other languages have some form of mirroring being widely used, for example[1] we have CPAN/CTAN/CRAN, pypi, julia, etc. There are lots of Nexus instances to partially mirror the Maven Central repository. If crates.io can be mirrored in some way, it will surely make Rust users' lives easier in those places.

[1]https://mirrors.tuna.tsinghua.edu.cn/
[2]tuna/issues#941
a bit background: TUNA mirror is the largest syncing source of other mirrors in the country, with an average rsync traffic of more than 1Gbps

@skyzh
Copy link

skyzh commented Nov 19, 2020

We have built a tool to mirror static.crates.io at https://github.com/sjtug/mirror-clone . Meanwhile, we have also set up an experimental full-mirror of static.crates.io at https://s3.jcloud.sjtu.edu.cn/899a892efef34b1b944a19981040f55b-oss01/crates.io/crates/{crate}/{crate}-{version}.crate

@Turbo87
Copy link
Member

Turbo87 commented Dec 23, 2021

seeing as e.g. Artifactory now supports Rust (see https://www.jfrog.com/confluence/display/JFROG/Cargo+Registry) I guess we can close this :)

@Turbo87 Turbo87 closed this as completed Dec 23, 2021
@happyaron
Copy link

@Turbo87 maybe a proprietary tool is not sufficient to close this issue, as a person who runs public mirror for various FLOSS, I'd like to find an open source solution, :)

@Turbo87
Copy link
Member

Turbo87 commented Dec 23, 2021

@happyaron absolutely, I would prefer that too. what I meant to say was that if Artifactory can build it then the necessary things are there for open source tools to also build something like this and there is nothing left on the crates.io side that we would need to build to enable this :)

Turbo87 pushed a commit to Turbo87/crates.io that referenced this issue Dec 7, 2022
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Turbo87 pushed a commit to Turbo87/crates.io that referenced this issue Jan 4, 2023
Co-authored-by: Renovate Bot <bot@renovateapp.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Category: Adding new behavior or a change to the way an existing feature works E-big
Projects
None yet
Development

No branches or pull requests

10 participants