Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-46053, JENKINS-45271, JENKINS-46210, JENKINS-46148] - Update HttpClient libraries and Fix Parent POM #102

Merged
merged 18 commits into from Sep 27, 2017

Conversation

oleg-nenashev
Copy link
Member

@oleg-nenashev oleg-nenashev commented Aug 15, 2017

Downstream PRs:

@reviewbybees @jtnord @rysteboe @jglick @aheritier @olamy (seems there is a nice party in the CC list)

@ghost
Copy link

ghost commented Aug 15, 2017

This pull request originates from a CloudBees employee. At CloudBees, we require that all pull requests be reviewed by other CloudBees employees before we seek to have the change accepted. If you want to learn more about our process please see this explanation.

@oleg-nenashev oleg-nenashev changed the title [JENKINS-46053, JENKINS-45271, JENKINS-46210] - Update HttpClient libraries and Fix Parent POM [JENKINS-46053, JENKINS-45271, JENKINS-46210, JENKINS-46148] - Update HttpClient libraries and Fix Parent POM Aug 15, 2017
@recampbell recampbell requested a review from jglick August 15, 2017 15:40
Copy link
Member

@jglick jglick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK. Is there a quick summary of what this practically changes in terms of bundled libraries?

jar tf target/maven-plugin.hpi | fgrep WEB-INF/lib | cut -c13- | sort

before and after the patch and diff them.

pom.xml Outdated
@@ -44,8 +44,8 @@ THE SOFTWARE.
<properties>
<jenkins.version>1.625.3</jenkins.version>
<java.level>7</java.level>
<mavenInterceptorsVersion>1.11</mavenInterceptorsVersion>
<mavenVersion>3.1.0</mavenVersion>
<mavenInterceptorsVersion>1.12-SNAPSHOT</mavenInterceptorsVersion>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a timestamped snapshot.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree but it depends how many days the PR will leave. SNAPSHOTs are quickly removed and thus it makes the build KO each time a new SNAPSHOT is published ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will do

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SNAPSHOTs are quickly removed

Not that quickly.

pom.xml Outdated
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>3.0.0-M1</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not get a version for free from plugin-pom?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, at least maven-enforcer-plugin.version should be used

<basePackage>**</basePackage>
<includeTestCode>true</includeTestCode>
<bannedImports>
<bannedImport>org.apache.commons.httpclient.**</bannedImport>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this better than bannedDependencies?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jglick It is a transient dependency from the Jenkins core, so it will blow up if I just ban it. bannedImports rather says that this lib usage is deprecated across the Maven Plugin. jenkinsci/maven-interceptors#13

<rules>
<restrictImports implementation="de.skuzzle.enforcer.restrictimports.RestrictImports">
<basePackage>**</basePackage>
<includeTestCode>true</includeTestCode>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary.

@jglick
Copy link
Member

jglick commented Aug 15, 2017

W.r.t. JENKINS-45271, did you actually verify that a plugin may depend on maven-plugin with a new parent POM without warnings? Would be appropriate to file a downstream PR in some such plugin demonstrating this. (Generally you get this for free by solving maven-plugin’s own transitive dependency issues, but there are likely to be corner cases involving provided or optional deps, or deps also available from Jenkins core.)

pom.xml Outdated
@@ -44,8 +44,8 @@ THE SOFTWARE.
<properties>
<jenkins.version>1.625.3</jenkins.version>
<java.level>7</java.level>
<mavenInterceptorsVersion>1.11</mavenInterceptorsVersion>
<mavenVersion>3.1.0</mavenVersion>
<mavenInterceptorsVersion>1.12-SNAPSHOT</mavenInterceptorsVersion>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree but it depends how many days the PR will leave. SNAPSHOTs are quickly removed and thus it makes the build KO each time a new SNAPSHOT is published ...

pom.xml Outdated
@@ -223,7 +204,7 @@ THE SOFTWARE.
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-aether-provider</artifactId>
<version>${mavenVersion}</version>
<version>3.3.9</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will quickly have a nightmare to resolve with the 3 versions of aether (Maven 3.5.0 added a new lib) cc @olamy

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The released version of Maven plugin already depends on Maven Core 3.5.0, but Maven Plugin bundles core 3.1.0. So we are already in the YOLO mode :(

maven-aether-provider 3.5.0 does not exist. If the new library uses different package names, there should be no issue. We could bundle both. What could possibly go wrong? (c)

@oleg-nenashev
Copy link
Member Author

oleg-nenashev commented Sep 15, 2017

OK, here the fun begins....
It comes from https://github.com/apache/maven/blob/master/maven-embedder/src/main/java/org/apache/maven/cli/CLIReportingUtils.java#L162 , binary conflict

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
	at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.maven.Maven3Builder.call(Maven3Builder.java:139)
	at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: org.apache.maven.cli.CLIReportingUtils.formatDuration(J)Ljava/lang/String;
	at org.apache.maven.cli.event.ExecutionEventLogger.logReactorSummary(ExecutionEventLogger.java:170)
	at org.apache.maven.cli.event.ExecutionEventLogger.sessionEnded(ExecutionEventLogger.java:123)
	at hudson.maven.Maven3Builder$MavenExecutionListener.sessionEnded(Maven3Builder.java:457)
	at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:58)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:170)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:316)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:153)
	at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:117)
	... 21 more
ERROR: Failed to parse POMs
ha:AAAAWB+LCAAAAAAAAP9b85aBtbiIQSmjNKU4P08vOT+vOD8nVc8DzHWtSE4tKMnMz/PLL0ldFVf2c+b/lb5MDAwVRQxSaBqcITRIIQMEMIIUFgAAckCEiWAAAAA=java.io.IOException: java.lang.reflect.InvocationTargetException
	at hudson.maven.Maven3Builder.call(Maven3Builder.java:179)
	at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

@oleg-nenashev
Copy link
Member Author

oleg-nenashev commented Sep 15, 2017

So, my best guess that Maven agents uses libs from Maven (it does) and classloads the rest from Jenkins. The "rest" on Jenkins is newer than the stuff supported by Maven, hence the test blows up with flying colors.

I'd guess the first step would be to pass Maven Embedder in classpath, where it is not mentioned. If it does not help, I think. If it does not help either, I drink as @jglick suggested.

/home/jenkins/tools/hudson.model.JDK/jdk8/jre/bin/java, -Djava.awt.headless=true, -cp, /home/jenkins/.m2/repository/org/jenkins-ci/main/maven/maven3-agent/1.12-alpha-1/maven3-agent-1.12-alpha-1.jar:/home/jenkins/workspace/Plugins_maven-plugin_PR-102-WXHWYD7RSA5WGCPPNDPRGSIGISTHTVCNTG43DWU5DHIGAGDGLGUQ/target/apache-maven-3.0.1/boot/plexus-classworlds-2.4.jar, org.jvnet.hudson.maven3.agent.Maven3Main, /home/jenkins/workspace/Plugins_maven-plugin_PR-102-WXHWYD7RSA5WGCPPNDPRGSIGISTHTVCNTG43DWU5DHIGAGDGLGUQ/target/apache-maven-3.0.1, /home/jenkins/.m2/repository/org/jenkins-ci/main/remoting/2.53.2/remoting-2.53.2.jar, /home/jenkins/.m2/repository/org/jenkins-ci/main/maven/maven3-interceptor/1.12-alpha-1/maven3-interceptor-1.12-alpha-1.jar, /home/jenkins/.m2/repository/org/jenkins-ci/main/maven/maven3-interceptor-commons/1.12-alpha-1/maven3-interceptor-commons-1.12-alpha-1.jar

@oleg-nenashev
Copy link
Member Author

org.apache.maven.cli.CLIReportingUtils.formatDuration(J)Ljava/lang/String has been introduced in Maven Embedder 3.2.0 by this commit: apache/maven@be19ddb

Curious thing is that calls from Maven plugin happen within Maven Embedder:

Caused by: java.lang.NoSuchMethodError: org.apache.maven.cli.CLIReportingUtils.formatDuration(J)Ljava/lang/String;
	at org.apache.maven.cli.event.ExecutionEventLogger.logReactorSummary(ExecutionEventLogger.java:170)
	at org.apache.maven.cli.event.ExecutionEventLogger.sessionEnded(ExecutionEventLogger.java:123)
	at hudson.maven.Maven3Builder$MavenExecutionListener.sessionEnded(Maven3Builder.java:457)

@oleg-nenashev
Copy link
Member Author

OTOH I definitely see other requests like MavenExecutionRequest#getDegreeOfConcurrency() requiring Maven 3.2.2 while the default build logic bundles older Maven tools. Conclusion: We need to revert embedder to the version supported by all Maven 3.x for now (or to drop support of older Maven 3.x versions).

Maven Embedder version is required on the Agent/Master side, so we cannot just specify custom version in the classpath from what I see

@oleg-nenashev
Copy link
Member Author

CI didn't even start for the last push

@oleg-nenashev
Copy link
Member Author

I will move exclusions to the Maven Interceptors repo later.
Probably I should just make Maven Core & Co compile-time dependencies there

@oleg-nenashev
Copy link
Member Author

Look @reviewbybees @aheritier @olamy , the build is green! If everybody is fine with the approach, maybe we could integrate the PR

@oleg-nenashev
Copy link
Member Author

Current libs included into the Maven Plugin bundle...

aether-api-1.1.0.jar
aether-connector-basic-1.1.0.jar
aether-impl-1.1.0.jar
aether-spi-1.1.0.jar
aether-transport-wagon-1.1.0.jar
aether-util-1.1.0.jar
ant-1.9.2.jar
ant-launcher-1.9.2.jar
cdi-api-1.0.jar
commons-cli-1.2.jar
commons-io-2.5.jar
commons-net-3.5.jar
doxia-sink-api-1.0.jar
findbugs-annotations-1.3.9-1.jar
httpcore-4.4.6.jar
jackrabbit-webdav-2.14.1.jar
javax.inject-1.jar
jna-platform-4.1.0.jar
jsch.agentproxy.connector-factory-0.0.9.jar
jsch.agentproxy.core-0.0.9.jar
jsch.agentproxy.jsch-0.0.9.jar
jsch.agentproxy.pageant-0.0.9.jar
jsch.agentproxy.sshagent-0.0.9.jar
jsch.agentproxy.usocket-jna-0.0.9.jar
jsch.agentproxy.usocket-nc-0.0.9.jar
jsoup-1.7.2.jar
jsr250-api-1.0.jar
lib-jenkins-maven-artifact-manager-1.2.jar
lib-jenkins-maven-embedder-3.12.jar
maven-aether-provider-3.1.0.jar
maven-agent-1.12-alpha-1.jar
maven-artifact-3.1.0.jar
maven-compat-3.1.0.jar
maven-core-3.1.0.jar
maven-embedder-3.1.0.jar
maven-interceptor-1.12-alpha-1.jar
maven-model-3.1.0.jar
maven-model-builder-3.1.0.jar
maven-plugin-api-3.1.0.jar
maven-plugin.jar
maven-reporting-api-3.0.jar
maven-repository-metadata-3.5.0.jar
maven-settings-3.1.0.jar
maven-settings-builder-3.1.0.jar
maven2.1-interceptor-1.2.jar
maven3-agent-1.12-alpha-1.jar
maven3-interceptor-1.12-alpha-1.jar
maven3-interceptor-commons-1.12-alpha-1.jar
maven31-agent-1.12-alpha-1.jar
maven31-interceptor-1.12-alpha-1.jar
maven32-agent-1.12-alpha-1.jar
maven32-interceptor-1.12-alpha-1.jar
maven33-agent-1.12-alpha-1.jar
maven33-interceptor-1.12-alpha-1.jar
maven35-agent-1.12-alpha-1.jar
maven35-interceptor-1.12-alpha-1.jar
org.eclipse.sisu.inject-0.3.3.jar
org.eclipse.sisu.plexus-0.3.3.jar
plexus-cipher-1.7.jar
plexus-classworlds-2.5.2.jar
plexus-component-annotations-1.7.1.jar
plexus-interactivity-api-1.0-alpha-6.jar
plexus-interpolation-1.16.jar
plexus-sec-dispatcher-1.3.jar
plexus-utils-3.0.24.jar
wagon-file-3.0.0.jar
wagon-ftp-3.0.0.jar
wagon-http-3.0.0.jar
wagon-http-shared-3.0.0.jar
wagon-provider-api-3.0.0.jar
wagon-ssh-3.0.0.jar
wagon-ssh-common-3.0.0.jar
wagon-ssh-external-3.0.0.jar
wagon-webdav-jackrabbit-3.0.0.jar

I definitely need to exclude HTTPCore from Jackrabbit.

@oleg-nenashev
Copy link
Member Author

Deployed the build as maven-plugin-2.18-20170921.105941-1.hpi

@oleg-nenashev
Copy link
Member Author

@jglick As requested, full library diff:

MBP-onenashev-2:maven-plugin nenashev$ diff before.txt after.txt 
12,13d11
< commons-codec-1.9.jar
< commons-httpclient-3.1.jar
17,19c15,16
< httpclient-4.5.1.jar
< httpcore-4.4.3.jar
< jackrabbit-webdav-2.5.2.jar
---
> findbugs-annotations-1.3.9-1.jar
> jackrabbit-webdav-2.14.1.jar
22d18
< jsch-0.1.54.jar
33c29
< lib-jenkins-maven-embedder-3.12.1.jar
---
> lib-jenkins-maven-embedder-3.12.jar
35c31
< maven-agent-1.11.jar
---
> maven-agent-1.12-alpha-1.jar
40c36
< maven-interceptor-1.11.jar
---
> maven-interceptor-1.12-alpha-1.jar
46c42
< maven-repository-metadata-3.1.0.jar
---
> maven-repository-metadata-3.5.0.jar
50,60c46,56
< maven3-agent-1.11.jar
< maven3-interceptor-1.11.jar
< maven3-interceptor-commons-1.11.jar
< maven31-agent-1.11.jar
< maven31-interceptor-1.11.jar
< maven32-agent-1.11.jar
< maven32-interceptor-1.11.jar
< maven33-agent-1.11.jar
< maven33-interceptor-1.11.jar
< maven35-agent-1.11.jar
< maven35-interceptor-1.11.jar
---
> maven3-agent-1.12-alpha-1.jar
> maven3-interceptor-1.12-alpha-1.jar
> maven3-interceptor-commons-1.12-alpha-1.jar
> maven31-agent-1.12-alpha-1.jar
> maven31-interceptor-1.12-alpha-1.jar
> maven32-agent-1.12-alpha-1.jar
> maven32-interceptor-1.12-alpha-1.jar
> maven33-agent-1.12-alpha-1.jar
> maven33-interceptor-1.12-alpha-1.jar
> maven35-agent-1.12-alpha-1.jar
> maven35-interceptor-1.12-alpha-1.jar
64,65c60,61
< plexus-classworlds-2.5.1.jar
< plexus-component-annotations-1.5.5.jar
---
> plexus-classworlds-2.5.2.jar
> plexus-component-annotations-1.7.1.jar
69,78c65,74
< plexus-utils-3.0.17.jar
< wagon-file-2.12.jar
< wagon-ftp-2.12.jar
< wagon-http-2.12.jar
< wagon-http-shared-2.12.jar
< wagon-provider-api-2.12.jar
< wagon-ssh-2.12.jar
< wagon-ssh-common-2.12.jar
< wagon-ssh-external-2.12.jar
< wagon-webdav-jackrabbit-2.12.jar
---
> plexus-utils-3.0.24.jar
> wagon-file-3.0.0.jar
> wagon-ftp-3.0.0.jar
> wagon-http-3.0.0.jar
> wagon-http-shared-3.0.0.jar
> wagon-provider-api-3.0.0.jar
> wagon-ssh-3.0.0.jar
> wagon-ssh-common-3.0.0.jar
> wagon-ssh-external-3.0.0.jar
> wagon-webdav-jackrabbit-3.0.0.jar

@oleg-nenashev
Copy link
Member Author

oleg-nenashev commented Sep 21, 2017

Things to review:

  • maven-repository-metadata was updated to 3.5.0
    • UPD: Fine? It is a new lib included in Maven 3.5 Agent. But probably it's safe to exclude it
  • Package now includes findbugs-annotations-1.3.9-1.jar. The only source of them is Jenkins test harness from what I see

Attached current deps.txt

deps.txt

@jglick
Copy link
Member

jglick commented Sep 22, 2017

my best guess that Maven agents uses libs from Maven (it does) and classloads the rest from Jenkins

For something like this, pending JENKINS-41827 you just cannot trust JenkinsRule tests. You must run ATH locally (i.e., set LOCAL_JARS).

@jglick jglick self-requested a review September 22, 2017 13:46
@aheritier
Copy link
Member

really good job @oleg-nenashev
I didn't find any regression with my local tests (but I didn't test all combinations with recent cores ...)

screenshot 2017-09-25 20 08 40

The code review is ok too. I agree with your choices and I think that in the future we should just mark all maven dependencies as "provided" in interceptor modules.

@oleg-nenashev
Copy link
Member Author

@reviewbybees done, I'd guess. ATH passed locally as well today.
@aheritier Should we release the Maven Plugin with alpha version of Interceptors? Or should we respin a new release there?

@aheritier
Copy link
Member

aheritier commented Sep 25, 2017 via email

@oleg-nenashev
Copy link
Member Author

CC @martinda who referenced the Guava conflict in Http Request Plugin caused by the dependency on the Maven plugin. If there is an issue reported for it, I will add it to the changelog

@martinda
Copy link

martinda commented Sep 26, 2017

@oleg-nenashev
Copy link
Member Author

@martinda which particular version are required by the plugin? Ranges are available in Guava 11 provided by the core, so there may be incompatibility on the method level only.

@aheritier @olamy What about Maven 2.18-rc-1 or 3.0-rc-1 release? The current request is probably incompatible because it stops bundling libraries available in 2.17 (especially Maven 3.5.0 deps), so 3.0 may be justtified enough

@aheritier
Copy link
Member

@oleg-nenashev fine for me to do a 3.0-rc-1 cc @olamy

@oleg-nenashev
Copy link
Member Author

Works for me. Starting from tomorrow I am available to do ad-hoc fixes if the release crash-lands in production.

2.17 already feels bad due to 3.5.0 dependencies, so I doubt rc-1 may make it worse

@martinda
Copy link

@oleg-nenashev the plugin depends on com.google.guava:guava:11.0.1 which comes from org.jenkins-ci.main:jenkins-core:jar:1.609.3 (from the http-plugin mvn dependencies:tree).

@aheritier aheritier merged commit 6891d0b into jenkinsci:master Sep 27, 2017
oleg-nenashev added a commit to oleg-nenashev/maven-plugin that referenced this pull request Oct 3, 2017
The fix has been reverted in 3.0.0-rc1 by jenkinsci#102.
It was just a mistake I made when patching the dependencies, but unfortunately nobody noticed it during the code review.
@oleg-nenashev
Copy link
Member Author

Caused https://issues.jenkins-ci.org/browse/JENKINS-47233. I should have noticed it during the dependency review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants