Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Support configuration of mirror repositories #271

Closed
cemerick opened this Issue · 13 comments

6 participants

Chas Emerick Phil Hagelberg André Thieme Michael Klishin Aleksander Adamowski Jean Niklas L'orange
Chas Emerick
Collaborator

For conceptual background, see http://maven.apache.org/guides/mini/guide-mirror-settings.html

While the maven ant tasks already pick up mirror configuration from ~/.m2/settings.xml, they do not pick up credentials if a mirror requires authentication. The eventual integration of aether may resolve that, but the need to futz with settings.xml for some lein use cases is unpleasant (and, aside from profiles [for which lein does not have a corollary], mirror configuration is the only settings.xml feature that lein does not yet support natively.

Proposal

Support using a map defined in ~/.lein/init.clj (leiningen-mirrors, perhaps) that associates repository names, URLs, or regexes matching either to repository names or root URLs for the corresponding mirror that should be used. This should end up being a far more flexible mechanism than what maven provides.

e.g. to use a mirror for clojars:

{"clojars" "http://internal-hostname/my/clojars/proxy"}
OR
{"http://clojars.org/repo/" "http://internal-hostname/my/clojars/proxy"}

Being able to use the repository URL is helpful in cases where specific repositories that are often defined on a project-by-project basis (e.g. releases, snapshots) can be mirrored specifically, instead of tip-toeing around the fragile repository-name concept.

Wildcards

A common use case of mirrors is to proxy all dependency resolution to a single repository; maven uses a privileged "" identifier for this purpose. We could either use a wildcard regex, or a privileged `:` keyword for this purpose:

{:* "http://internal-hostname/my/uber/repo"}
OR
{#".+" "http://internal-hostname/my/uber/repo"}

I believe I'd prefer the latter.

Match "priority"

It would seem to make sense for mirror matches to be made following a defined prioritization:

  1. exact URL match
  2. exact repository name match
  3. regex match

This would enable one to satisfy common use cases such as having a catch-all mirror, except for one or two repositories that should be accessed directly or which should be directed at a specific mirror:

{#".+" "http://internal-hostname/my/uber/repo"
 "central" false
 "http://download.java.net/maven/2/" "central"}

The above would set up central as the mirror for java.net's repo, continue to use central directly, and use the internal proxy for all other repositories.

(I'm decidedly iffy on using false to exempt a repo from being mirrored, but I'm hoping to stay away from the "*,!central" sort of notation that mvn uses.)

Phil Hagelberg
Owner

I'm going to go ahead and mark this as a 2.0 feature unless someone wants to implement it in 1.x.

André Thieme

Would be very useful. I am behind a firewall and would prefer if all requests ever would only go to the exact repos that I specify (one Artifactory in my case). Currently even simple operations take forever to finish, such as lein swank or lein midje. Before those get executed I see outputs such as:

Apr 11, 2012 2:22:13 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
Apr 11, 2012 2:22:13 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
Apr 11, 2012 2:22:13 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: Retrying request
Apr 11, 2012 2:22:13 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: Retrying request
Apr 11, 2012 2:22:34 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
Apr 11, 2012 2:22:34 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
Apr 11, 2012 2:22:34 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: Retrying request
Apr 11, 2012 2:22:34 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: Retrying request
Apr 11, 2012 2:22:55 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
Apr 11, 2012 2:22:55 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: Retrying request
Apr 11, 2012 2:22:55 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
Apr 11, 2012 2:22:55 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
Information: Retrying request
Could not transfer metadata org.clojure:clojure/maven-metadata.xml from/to snapshots (http://oss.sonatype.org/content/repositories/snapshots): Connection timed out: connect
Could not transfer metadata org.clojure:clojure/maven-metadata.xml from/to releases (http://oss.sonatype.org/content/repositories/releases): Connection timed out: connect

Only after a minute something happens. Would be great if I could block sonatype and all others.

Phil Hagelberg
Owner

@cemerick: does this require further development in pomegranate or is it just a matter of wiring the ends together?

Chas Emerick
Collaborator

It shouldn't be difficult to add, but mirrors are not yet supported in pomegranate.

Michael Klishin
Collaborator

Would it be possible to pick up mirrors from ~/.m2/settings.xml with Pomegranate? We want to try using the UK mirror of Maven central on travis-ci.org.

Chas Emerick cemerick referenced this issue in cemerick/pomegranate
Closed

Add support for mirrors #29

Chas Emerick
Collaborator

settings.xml is out of scope for pomegranate. Maybe Leiningen could support pulling mirror configuration from there, but it's (probably wisely) opted out of that for all other maven/aether-related configuration so far. Yet another source of configuration data doesn't strike me as being worthwhile (i.e. vs. ~/.lein/profiles.clj).

Michael Klishin
Collaborator

Ok, is there an example of how mirrors can be overriden using ~/.lein/profiles.clj?

Phil Hagelberg
Owner

I don't think it's appropriate to check settings.xml, but mirror support is definitely important. It requires support in pomegranate first though.

Phil Hagelberg technomancy referenced this issue in heroku/heroku-buildpack-clojure
Closed

Use s3pository mirror #15

Phil Hagelberg
Owner

Pulling in cemerick/pomegranate#34 should take care of this.

Chas Emerick cemerick referenced this issue from a commit in cemerick/pomegranate
Chas Emerick cemerick Fixes gh-37:
* All configuration options available for repositories are now available
  for mirrors, so credentials, snapshot/update policies, etc. are all supported
* Eliminates use of the mirror-matching syntax offered by 
  DefaultMirrorSelector/mvn, in exchange for simpler repository name/url
  matching using either strings or regexes, as described in
  technomancy/leiningen#271.
fb1864b
Phil Hagelberg technomancy closed this issue from a commit
Phil Hagelberg Plumb through mirror support to pomegranate. Fixes #271.
Won't actually be visible until pomegranate is bumped to 0.0.13.
a32abdf
Aleksander Adamowski

Care to add a working example in PROFILES.md of configuring the user profile that points to a company-wide Maven proxy repository that is a mirror of everything, analogous to the following setting in a Maven settings.xml?

<mirrors>
    <mirror>
      <id>internal_http_mirror</id>
      <mirrorOf>*</mirrorOf>
      <name>Nexus</name>
      <url>http://internal-nexus.example.com/content/groups/public-all/</url>
    </mirror>
  </mirrors>

I'm currently on Lein 2.2.0, struggling to get the nesting/syntax right and get a generous portion of cryptic errors like "Map literal must contain an even number of forms". And I just want to work around our corporate NTLM proxy and point Lein to our internal central Maven Nexus mirror/proxy, like my plain Maven installation is. This is such a common use case that it should be properly documented.

Aleksander Adamowski

The examples lack a case with a mirror of everything, or in general any example of using a pattern. This also seems to be undocumented anywhere except for this issue. But issue tracking systems shouldn't serve as documentation.

After some trial end error experimentation, I've figured out how to configure it in my user profile and posted an example on Stack Overflow.

I think, however, that such configuration should also be present in the sample you've linked to and the general syntax of mirror maps should be documented in official docs (https://github.com/technomancy/leiningen/tree/stable/doc).

Jean Niklas L'orange
Collaborator

@aadamowski, yeah, I see now that it isn't completely evident what the regular expressions do, and I realize that documentation would be relatively crippled if we were to put it in the sample project. I've set up #1256 which should fix the documentation whenever that issue is closed. If you have anything some comment on what should be covered (except what you've already mentioned), feel free to elaborate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.