Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: way to set up a mirror #67

Open
carols10cents opened this issue Nov 23, 2014 · 8 comments
Open

Feature request: way to set up a mirror #67

carols10cents opened this issue Nov 23, 2014 · 8 comments

Comments

@carols10cents
Copy link
Member

@carols10cents carols10cents commented Nov 23, 2014

I'm so excited crates.io has arrived!! 馃帀 Great work, Alex and everyone!!!

This is a longer-term feature request... one thing that has been an occasional yet painful experience with rubygems.org or npmjs.org is when they're down. At various points in time, my coworkers and I have tried to set up mirrors under our control of both of these and it never went very smoothly, neither on the setup or usage side of things.

I would love love love to have mirroring be something officially supported and encouraged for the health of the community, but I realize this is a difficult problem too :)

Thank you for your work!! 鉂わ笍

@sfackler
Copy link
Member

@sfackler sfackler commented Nov 23, 2014

This would also involve adding support for non-Github auth (preferably LDAP) and non-S3/Github storage.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Nov 23, 2014

Architecturally I believe it's relatively easy to do. A "mirror" would essentially just be a clone of the official index along with an updated download url in its configuration. Updating a mirror would just be updating the index and perhaps downloading all the new versions that were created.

Implementing this, however, and getting cargo to think that mirrors and normal registries are the same would be somewhat difficult.

@sfackler, those may be more useful for non-official entire instances, as I would suspect that a mirror would reflect whatever the crates.io site uses. That being said, I'd love for it to be easy to spin up a local instance of a registry to use entirely independently (with a number of configuration options like auth/storage). Might be a bit harder though :)

@sfackler
Copy link
Member

@sfackler sfackler commented Nov 23, 2014

At work we use Artifactory to both serve as a repository for internal packages and to provide a transparent mirror/cache of the maven.org packages we use. New packages are transparently pulled down as needed, but we don't mirror the whole world since that'd be crazy. Something like that would probably be best served as a separate project, though it'd hopefully be able to pull in a bunch of the logic/types from crates.io.

@weiznich
Copy link

@weiznich weiznich commented Nov 9, 2016

I've written a simple caching server. Similar to Artifactory it pulls down transparently those crates that are really requested. Currently it is only a simple application without any interface or anything. Also it only mirrors an index. It's not possible to add or yank crates to the mirror.

@gilescope
Copy link

@gilescope gilescope commented Jul 19, 2018

This feels linked to rust-lang/rust#44931

@ishitatsuyuki
Copy link
Contributor

@ishitatsuyuki ishitatsuyuki commented Jul 22, 2018

For initial fetch, we should provide daily database dumps which is less overhead to serve and better compressed.

For dynamic syncing, the Kubernetes model is a very friendly way to organize the API.

bors added a commit that referenced this issue Dec 17, 2019
鈥fy-3.0.0, r=Turbo87

Bump ember-cli-uglify from 2.0.0 to 3.0.0

Bumps [ember-cli-uglify](https://github.com/ember-cli/ember-cli-uglify) from 2.0.0 to 3.0.0.
<details>
<summary>Release notes</summary>

*Sourced from [ember-cli-uglify's releases](https://github.com/ember-cli/ember-cli-uglify/releases).*

> ## v3.0.0
> #### 馃挜 Breaking Change
> * [#102](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/102) Drop support for Node.js 6 ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#64](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/64) Drop support for Node.js 4 ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### 馃彔 Internal
> * [#68](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/68) TravisCI: Remove deprecated `sudo: false` option ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#67](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/67) Remove unneeded and confusing dependency ([@&#8203;jelhan](https://github.com/jelhan))
> * [#66](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/66) CI: Run only for `master` branch, tags and PRs ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#44](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/44) Fix CI ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### Committers: 2
> - Tobias Bieniek ([@&#8203;Turbo87](https://github.com/Turbo87))
> - jelhan ([@&#8203;jelhan](https://github.com/jelhan))
</details>
<details>
<summary>Changelog</summary>

*Sourced from [ember-cli-uglify's changelog](https://github.com/ember-cli/ember-cli-uglify/blob/master/CHANGELOG.md).*

> ## v3.0.0 (2019-04-12)
>
> #### 馃挜 Breaking Change
> * [#102](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/102) Drop support for Node.js 6 ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#64](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/64) Drop support for Node.js 4 ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### 馃彔 Internal
> * [#68](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/68) TravisCI: Remove deprecated `sudo: false` option ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#67](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/67) Remove unneeded and confusing dependency ([@&#8203;jelhan](https://github.com/jelhan))
> * [#66](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/66) CI: Run only for `master` branch, tags and PRs ([@&#8203;Turbo87](https://github.com/Turbo87))
> * [#44](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/44) Fix CI ([@&#8203;Turbo87](https://github.com/Turbo87))
>
> #### Committers: 2
> - Tobias Bieniek ([@&#8203;Turbo87](https://github.com/Turbo87))
> - jelhan ([@&#8203;jelhan](https://github.com/jelhan))
>
>
> ## v2.1.0 (2018-03-27)
>
> #### 馃殌 Enhancement
> * [#37](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/37) Run uglify in parallel ([@&#8203;mikrostew](https://github.com/mikrostew))
>
> #### Committers: 1
> - Michael Stewart ([@&#8203;mikrostew](https://github.com/mikrostew))
>
>
> ## v2.0.2 (2018-02-20)
>
> #### 馃殌 Enhancement
> * [#34](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/34) Updates broccoli-uglify-sourcemap to version 2.0.1 ([@&#8203;Duder-onomy](https://github.com/Duder-onomy))
>
> #### 馃彔 Internal
> * [#36](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/pull/36) add test (basically, a default app built鈥檚 tests run even post uglifi鈥 ([@&#8203;stefanpenner](https://github.com/stefanpenner))
>
> #### Committers: 2
> - Greg Larrenaga ([@&#8203;Duder-onomy](https://github.com/Duder-onomy))
> - Stefan Penner ([@&#8203;stefanpenner](https://github.com/stefanpenner))
</details>
<details>
<summary>Commits</summary>

- [`430cb9a`](ember-cli/ember-cli-terser@430cb9a) v3.0.0
- [`9540447`](ember-cli/ember-cli-terser@9540447) Update Changelog
- [`b051655`](ember-cli/ember-cli-terser@b051655) Build(deps-dev): Bump ember-cli from 3.8.2 to 3.9.0 ([#104](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/issues/104))
- [`940606d`](ember-cli/ember-cli-terser@940606d) Build(deps-dev): Bump ember-source from 3.9.0 to 3.9.1
- [`10199ef`](ember-cli/ember-cli-terser@10199ef) Merge pull request [#102](https://github-redirect.dependabot.com/ember-cli/ember-cli-uglify/issues/102) from Turbo87/node-6
- [`84fed8a`](ember-cli/ember-cli-terser@84fed8a) Build(deps-dev): Bump ember-cli from 3.8.1 to 3.8.2
- [`9cdbcf2`](ember-cli/ember-cli-terser@9cdbcf2) Drop support for Node 6
- [`9ed7828`](ember-cli/ember-cli-terser@9ed7828) Build(deps-dev): Bump ember-source from 3.8.0 to 3.9.0
- [`d778b8a`](ember-cli/ember-cli-terser@d778b8a) Build(deps-dev): Bump ember-cli-babel from 7.6.0 to 7.7.3
- [`bd30ff4`](ember-cli/ember-cli-terser@bd30ff4) Build(deps): Bump broccoli-uglify-sourcemap from 3.0.0 to 3.1.0
- Additional commits viewable in [compare view](ember-cli/ember-cli-terser@v2.0.0...v3.0.0)
</details>
<br />

[![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=ember-cli-uglify&package-manager=npm_and_yarn&previous-version=2.0.0&new-version=3.0.0)](https://dependabot.com/compatibility-score.html?dependency-name=ember-cli-uglify&package-manager=npm_and_yarn&previous-version=2.0.0&new-version=3.0.0)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

**Note:** This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

You can always request more updates by clicking `Bump now` in your [Dependabot dashboard](https://app.dependabot.com).

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)

</details>
@happyaron
Copy link

@happyaron happyaron commented Oct 16, 2020

Would like to bump this issue a bit since it looks like not getting love from guys for a long time - people from certain parts of the world (in my case, China mainland) does not have decent connectivity to some most used CDNs, so that mirroring is a must for getting a decent speed of using the repository.

Almost all the other languages have some form of mirroring being widely used, for example[1] we have CPAN/CTAN/CRAN, pypi, julia, etc. There are lots of Nexus instances to partially mirror the Maven Central repository. If crates.io can be mirrored in some way, it will surely make Rust users' lives easier in those places.

[1]https://mirrors.tuna.tsinghua.edu.cn/
[2]tuna/issues#941
a bit background: TUNA mirror is the largest syncing source of other mirrors in the country, with an average rsync traffic of more than 1Gbps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can鈥檛 perform that action at this time.