Maven resolution fails behind proxy

[lerad]

The %maven line magic code throws an error. I guess it is due to my proxy settings. It seems as if the resolver does not take into account the proxy settings? Is there a way to give the resolver a proxy configuration?

Command:
%maven org.knowm.xchart:xchart:3.5.2
or
%maven org.apache.commons:commons-math3:3.6.1

Result:

java.lang.RuntimeException: Exception occurred while running line magic 'maven': Unable to collect/resolve dependency tree for a resolution due to: Failed to collect dependencies at org.knowm.xchart:xchart:jar:3.5.2, caused by: Connection refused: connect
at .lineMagic(#9:10)
at .(#76:1)

I tried:
System.setProperty("http.proxyHost", proxyHost);
System.setProperty("https.proxyHost", proxyHost);
System.setProperty("http.proxyPort", proxyPort);
System.setProperty("https.proxyPort", proxyPort);

With this I can successfully make network communications, i.e. the proxy configuration works. E.g.:
URL url = new URL("http://www.example.org/");
InputStream in = url.openStream();
The above code works. However the maven resolution still fails.

[SpencerPark]

Hi @lerad, thanks for raising the issue. I don't know too much when it comes to using proxies so I'm not sure how to test this myself. I would appreciate it if you could try a couple things and let me know if/how any of them work.

The kernel is using shrinkwrap resolvers to resolve dependencies and there is a section in the README about system prperties. Specifically the org.apache.maven.user-settings variable which could be a path to a maven settings.xml. More information about how to tell maven about a proxy is in the maven documentation.

The 2nd thing I would appreciate you giving me feedback on is running a cell with the following:

import org.jboss.shrinkwrap.resolver.api.maven.Maven;
import org.jboss.shrinkwrap.resolver.api.maven.strategy.TransitiveStrategy;

Maven.configureResolver()
    .withClassPathResolution(true)
    .withMavenCentralRepo(true)
    .fromFile("/path/to/settings.xml")
    .resolve("org.knowm.xchart:xchart:jar:3.5.2")
    .using(TransitiveStrategy.INSTANCE)
    .asList(File.class);

and replacing the "/path/to/settings.xml" with the path to a settings file with your proxy information in it.

I am aiming to tackle a couple issues this weekend and would appreciate the feedback by then. As this will likely require some additional documentation, any extra information you can provide would help immensely in writing it up for others that need to work through proxies. Thanks!

[lerad]

Thank you for the fast reply. I tried with a Maven configuration file, however the error persists. With the settings.xml I receive as output from the cell:

org.jboss.shrinkwrap.resolver.api.NoResolvedResultException: Unable to collect/resolve dependency tree for a resolution due to: Failed to collect dependencies at org.knowm.xchart:xchart:jar:3.5.2, caused by: Connection refused: connect
at org.jboss.shrinkwrap.resolver.impl.maven.MavenWorkingSessionImpl.wrapException(MavenWorkingSessionImpl.java:503)
at org.jboss.shrinkwrap.resolver.impl.maven.MavenWorkingSessionImpl.resolveDependencies(MavenWorkingSessionImpl.java:242)
at org.jboss.shrinkwrap.resolver.impl.maven.MavenStrategyStageBaseImpl.using(MavenStrategyStageBaseImpl.java:70)
at org.jboss.shrinkwrap.resolver.impl.maven.MavenStrategyStageBaseImpl.using(MavenStrategyStageBaseImpl.java:39)
at .(#36:1)

I tried the same code in IntelliJ and stepped through it with the java debugger. The proxies are initially loaded correctly, I can see a them in
org.jboss.shrinkwrap.resolver.impl.maven.bootstrap.MavenRepositorySystem.resolveDependencies in the repoSession parameter.

However deeper down in the call chain in org.apache.maven.wagon.providers.http.LightweightHttpWagon somehow the proxy information seem to be lost.

The exception than happens in
URL url = new URL( visitingUrl );
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection( this.proxy );
in this class.

Probably this is an issue of those libraries?

[SpencerPark]

Apologies for the silence on this issue. I have given the debugger a try and indeed the wagon being used does appear to never take note of the proxy. Feel free to ignore the ramblings below, they are more for notes but in case you are interested I have included them. Hopefully this week I can get a chance to implement something.

I've tracked it down to WagonTransporterFactory which creates a WagonTransporter from the default WagonProvider. By default this is an instance of ManualWagonProvider which will return an unconfigured wagon (the LightweightHttpsWagon we are seeing).

I believe the WagonTransporter is what should be configuring the provided wagon which is taking the proxy from the RemoteRepository and not the RepositorySystemSession! So in a sense the information is there the whole time but needs to be duplicated at some earlier point in the configuration to copy the proxy to all of the added RemoteRepositorys.

In MavenWorkingSessionImpl the repo is added to the additionalRemoteRepositories in addRemoteRepo but at this point the settings.xml is not visible so there is no proxy info. That is set later.

In ConfigurableMavenResolverSystemBaseImpl the fromFile properly updates the session with the proxy information and then bakes the resolver (ConfigurableMavenResolverSystemImpl to a MavenResolverSystemImpl). Unfortunately this may be a bit late to have a hope of catching all the places that repositories are added.

I think a custom WagonTransporterFactory that sets the correct proxy just before building the transporter might suffice although I'm not sure is a solution that should work its way upstream.

[michaelmior]

I'm experiencing the same error without a proxy. Any hints on how to debug this? The error message I get in the console specifies that an artifact failed to transfer but requesting the resource manually works fine.

[michaelmior]

Despite this failing on two completely different networks yesterday, I ran the same thing again today without issues so things are working fine for me now. I'd still be interested in hearing suggestions for debugging in the future.

[SpencerPark]

@michaelmior I'm working on a switch to use ivy (ivy-branch) which should allow using the system properties for setting up a proxy as well as having access to fix issues like these when they pop up. I should have some more time during the holidays to get some solid time in on pushing a few things out that I've been working on including this.

Failing transfer could be an https error with certificates? Could be issues with authentication of a private repo? Take a look at system variables on the working jvm and broken one and look for differences. It could also just be issues on the other side (remote is down?) or network issues? Also maven may be configured via global configurations that the magic doesn't consider? I'm sorry that's not much help, but I'm not really sure whats wrong :/

As far as it suddenly working: if you managed to download the dependencies to your maven local repo, the magic would find them there and should not require connecting to the remote repo which might be what happened.

[michaelmior]

No problem, I understand. I was just using artifacts from Maven Central, so it seems unlikely to be any configuration issues. Anyway, things are working great now! I've been looking for something like IJava for a while and other than this initial hiccup, I've been pleasantly surprised with how well it works :)