[MSHARED-1066] - Upgrade Plexus Archiver to 4.3.0, Improve the Reproducible Builds methods #22

jorsol · 2022-04-23T09:15:55Z

Following this checklist to help us incorporate your
contribution quickly and easily:

Make sure there is a JIRA issue filed
for the change (usually before you start working on it). Trivial changes like typos do not
require a JIRA issue. Your pull request should address just this issue, without
pulling in other changes.
Each commit in the pull request should have a meaningful subject line and body.
Format the pull request title like [MSHARED-XXX] - Fixes bug in ApproximateQuantiles,
where you replace MSHARED-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean verify to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
You have run the integration tests successfully (mvn -Prun-its clean verify).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004
In any other case, please file an Apache Individual Contributor License Agreement.

src/main/java/org/apache/maven/archiver/MavenArchiver.java

cstamas · 2022-06-13T08:53:10Z

Can we split out the two thing here? Update p-archiver (and plexus-io) is unrelated to repro methods, is it?

jorsol · 2022-06-13T09:09:12Z

Can we split out the two thing here? Update p-archiver (and plexus-io) is unrelated to repro methods, is it?

Removed plexus-io update as is actually unrelated (will open a follow-up PR).

The update of plexus-archiver and reproducible methods are actually related, the reproducible methods depends on plexus-archiver 4.3.0 as plexus-archiver also contains changes to methods. It's split in two commits, and to make them two different PRs, the plexus-archiver update should be merged first and then the reproducible methods later, yet since the dependency update is a really small change I would like to keep them together.

cstamas · 2022-06-13T09:20:35Z

Kk, but it would be good to sort out maven-archiver changes, make a release soon:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20MSHARED%20AND%20fixVersion%20%3D%20maven-archiver-3.6.0

As MASSEMBLY is eagerly waiting updates
https://issues.apache.org/jira/issues/?jql=project%20%3D%20MASSEMBLY%20AND%20fixVersion%20%3D%203.4.0

jorsol · 2022-06-13T10:08:52Z

Kk, but it would be good to sort out maven-archiver changes, make a release soon: https://issues.apache.org/jira/issues/?jql=project%20%3D%20MSHARED%20AND%20fixVersion%20%3D%20maven-archiver-3.6.0

This should be ready for review, after merging this and #24 a release can be made.

jorsol · 2022-06-13T12:47:03Z

We need to move this forward to also move apache/maven-jar-plugin#43 and release maven-jar-plugin 3.3.0

cstamas · 2022-06-16T12:59:32Z

wazzup here? @michael-o

src/main/java/org/apache/maven/archiver/MavenArchiver.java

michael-o · 2022-06-21T07:21:55Z

src/main/java/org/apache/maven/archiver/MavenArchiver.java

+     * @throws IllegalArgumentException if the outputTimestamp is neither ISO 8601 nor an integer
+     */
+    public static Optional<Instant> parseBuildOutputTimestamp( String outputTimestamp )
+    {


I have the feeling that such parsing is done over and over again in many components. We need to consider to have ths at as central place.

I'm not sure what this means, are you referring to moving this out of Maven Archiver?

It is just a note that this could would be better suited in a shared util for other subcomponents as well.

src/test/java/org/apache/maven/archiver/MavenArchiverTest.java

src/main/java/org/apache/maven/archiver/MavenArchiver.java

michael-o · 2022-06-22T09:54:27Z

src/test/java/org/apache/maven/archiver/MavenArchiverTest.java

+    }
+
+    @ParameterizedTest
+    @CsvSource( { "0,0", "1,1", "9,9", "1570300662,1570300662", "2147483648,2147483648",


That source is quite insane :-D

src/test/java/org/apache/maven/archiver/MavenArchiverTest.java

src/main/java/org/apache/maven/archiver/MavenArchiver.java

michael-o · 2022-06-22T11:26:48Z

Except for the remaining comments this looks good. I will run this snapshot against MJAR, MWAR, and likely other projects just to make sure.

michael-o · 2022-06-22T12:05:37Z

Please have a look (maven-javadoc-plugin):

Generating /var/osipovmi/Projekte/maven-javadoc-plugin/target/it/MJAVADOC-137_jar/test1/target/apidocs/help-doc.html...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Maven MJAVADOC-137 test 1.0-SNAPSHOT:
[INFO]
[INFO] Maven MJAVADOC-137 test ............................ SUCCESS [  1.237 s]
[INFO] test1 .............................................. FAILURE [  1.642 s]
[INFO] test2 .............................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.131 s
[INFO] Finished at: 2022-06-22T13:45:14+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.4.1-SNAPSHOT:jar (default-cli) on project test1: Execution default-cli of goal org.apache.maven.plugins:maven-javadoc-plugin:3.4.1-S
NAPSHOT:jar failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-javadoc-plugin:3.4.1-SNAPSHOT:jar: java.lang.NoSuchMethodError: org.codehaus.plexus.archiver.jar.JarArchi
ver.configureReproducibleBuild(Ljava/nio/file/attribute/FileTime;)V
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>org.apache.maven.plugins:maven-javadoc-plugin:3.4.1-SNAPSHOT
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/org/apache/maven/plugins/maven-javadoc-plugin/3.4.1-SNAPSHOT/maven-javadoc-plugin-3.4.1-SNAPSHOT.jar
[ERROR] urls[1] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/org/eclipse/aether/aether-util/1.0.0.v20140518/aether-util-1.0.0.v20140518.jar
[ERROR] urls[2] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar
[ERROR] urls[3] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/javax/annotation/jsr250-api/1.0/jsr250-api-1.0.jar
[ERROR] urls[4] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.0.M1/org.eclipse.sisu.inject-0.3.0.M1.jar
[ERROR] urls[5] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/org/sonatype/sisu/sisu-guice/3.2.3/sisu-guice-3.2.3-no_aop.jar
[ERROR] urls[6] = file:/var/osipovmi/Projekte/maven-javadoc-plugin/target/local-repo/aopalliance/aopalliance/1.0/aopalliance-1.0.jar

michael-o · 2022-06-22T12:12:50Z

Please have a look (maven-source-plugin):

[INFO] Building: reproducible/pom.xml
[INFO] run post-build script verify.groovy
[INFO]   The post-build script did not succeed. assert buf.startsWith( "reproducible-1.0-sources.jar sha1 = 3a3687b063cfc164fbbccd1b9573b4232f540e8a" )
       |   |
       |   false
       reproducible-1.0-sources.jar sha1 = f159379802c1f0dc1083af21352286b09d364519

       encoding: UTF8
       timezone offset (minutes): -120
       M size (cmp) crc      java time     date       time           zip time   mode   name -comment; extra
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 META-INF/ ; 0
       8   25 ( 27) ee027fb2 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 META-INF/MANIFEST.MF ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-A/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-C/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-b/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-b/B2/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-b/B4/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-b/b1/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-b/b3/ ; 0
       0    0 (  0)        0 1566404932000 2019-08-21 18:28:52 +0200 1326814106  40755 dir-d/ ; 0
       8  281 (118) d793ddf2 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 META-INF/DEPENDENCIES ; 0
       8 11358 (3949) 86e2b4b4 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 META-INF/LICENSE ; 0
       8  183 (129) aeabb55c 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 META-INF/NOTICE ; 0
       8    9 ( 11) 1c933a72 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 Uppercase.txt ; 0
       8    2 (  4) 606c3d3b 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-A/A2.txt ; 0
       8    2 (  4) 890f980e 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-A/A4.txt ; 0
       8    2 (  4) 6ce14823 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-A/a1.txt ; 0
       8    2 (  4) 82ef290f 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-A/a3.txt ; 0
       8    1 (  3) 3dd7ffa7 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-C/C.txt ; 0
       8    2 (  4) 4b416ef8 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-b/B2/B2.txt ; 0
       8    2 (  4) a222cbcd 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-b/B4/B4.txt ; 0
       8    1 (  3) 71beeff9 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-b/b.txt ; 0
       8    2 (  4) 47cc1be0 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-b/b1/b1.txt ; 0
       8    2 (  4) a9c27acc 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-b/b3/b3.txt ; 0
       8    1 (  3) 98dd4acc 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 dir-d/d.txt ; 0
       8   10 ( 12) d68eda01 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 executable.txt ; 0
       8    9 ( 11) fe3d51f2 1566404932000 2019-08-21 18:28:52 +0200 1326814106 100644 lowercase.txt ; 0
[INFO]           reproducible/pom.xml ............................. FAILED (4.9 s)

jorsol · 2022-06-22T12:16:22Z

Please have a look (maven-javadoc-plugin):

The maven-javadoc-plugin declares the plexus-archiver dependency, so it needs to be updated to 4.3.0 (or removed to be used transitively) at the same time as maven-archiver.

https://github.com/apache/maven-javadoc-plugin/blob/b92897088f6fcbc5ec4f3346de591eb97a1c580f/pom.xml#L288-L292

michael-o · 2022-06-22T12:17:11Z

Users: https://github.com/search?q=org%3Aapache+maven-archiver+path%3A%2F+language%3A%22Maven+POM%22+language%3A%22Maven+POM%22&type=Code&ref=advsearch&l=Maven+POM&l=Maven+POM

michael-o · 2022-06-22T12:31:44Z

Please have a look (maven-javadoc-plugin):

The maven-javadoc-plugin declares the plexus-archiver dependency, so it needs to be updated to 4.3.0 (or removed to be used transitively) at the same time as maven-archiver.

https://github.com/apache/maven-javadoc-plugin/blob/b92897088f6fcbc5ec4f3346de591eb97a1c580f/pom.xml#L288-L292

Done. Issue resolved.

michael-o · 2022-06-22T12:37:42Z

I can confirm that maven-source-plugin passes from master, but not this PR.

jorsol · 2022-06-22T13:07:08Z

I can confirm that maven-source-plugin passes from master, but not this PR.

The IT from maven-source-plugin uses a sha1 checksum (which does not necessarily test that is reproducible).

And actually, the root cause is not even in this PR, you can make that test fail by just updating plexus-archiver to 4.2.7 (right now is at 4.2.4), plexus-archiver 4.2.7 fixes the order of META-INF/ and META-INF/MANIFEST.MF entries in a JAR file, this "reorder" of entries makes that the output (and the checksum) change.

So, essentially the "fix" is just to use the new sha1 checksum.

michael-o · 2022-06-22T13:13:49Z

I can confirm that maven-source-plugin passes from master, but not this PR.

The IT from maven-source-plugin uses a sha1 checksum (which does not necessarily test that is reproducible).

And actually, the root cause is not even in this PR, you can make that test fail by just updating plexus-archiver to 4.2.7 (right now is at 4.2.4), plexus-archiver 4.2.7 fixes the order of META-INF/ and META-INF/MANIFEST.MF entries in a JAR file, this "reorder" of entries makes that the output (and the checksum) change.

So, essentially the "fix" is just to use the new sha1 checksum.

Right, I did the fix for the order because it was broken for quite some time. Let me double check.

michael-o · 2022-06-22T13:13:56Z

I will use diffoscope.

michael-o · 2022-06-22T13:19:11Z

Here it is:

osipovmi@deblndw011x:~/var/Projekte/maven-source-plugin (master *=)
$ ~/.local/bin/diffoscope /tmp/reproducible-1.0-sources.jar ./target/it/reproducible/target/repo/org/apache/maven/its/reproducible/1.0/reproducible-1.0-sources.jar
--- /tmp/reproducible-1.0-sources.jar
+++ ./target/it/reproducible/target/repo/org/apache/maven/its/reproducible/1.0/reproducible-1.0-sources.jar
├── zipinfo {}
│ @@ -1,10 +1,10 @@
│  Zip file size: 7016 bytes, number of entries: 27
│ --rw-r--r--  2.0 unx       25 b- defN 19-Aug-21 18:28 META-INF/MANIFEST.MF
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 META-INF/
│ +-rw-r--r--  2.0 unx       25 b- defN 19-Aug-21 18:28 META-INF/MANIFEST.MF
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-A/
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-C/
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-b/
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-b/B2/
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-b/B4/
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-b/b1/
│  drwxr-xr-x  2.0 unx        0 b- stor 19-Aug-21 18:28 dir-b/b3/
├── zipnote «TEMP»/diffoscope_228psws7_10/tmptcizfwbk_.zip
│ @@ -1,11 +1,11 @@
│ -Filename: META-INF/MANIFEST.MF
│ +Filename: META-INF/
│  Comment:
│
│ -Filename: META-INF/
│ +Filename: META-INF/MANIFEST.MF
│  Comment:
│
│  Filename: dir-A/
│  Comment:
│
│  Filename: dir-C/
│  Comment:
├── filetype from file(1)
│ @@ -1 +1 @@
│ -Zip archive data, at least v2.0 to extract, compression method=deflate
│ +Zip archive data, at least v1.0 to extract, compression method=store

@jorsol Would you mind to create a PR this? Your analysis is correct.

michael-o · 2022-06-22T13:21:02Z

I will assign issues in JIRA to me and handle this PR. Muchas gracias por tu ayuda!

Signed-off-by: Jorge Solórzano <jorsol@gmail.com> This closes apache#22

jorsol force-pushed the MSHARED-1066 branch 2 times, most recently from e11bd5e to fe368cf Compare June 13, 2022 06:05

michael-o reviewed Jun 13, 2022

View reviewed changes

src/main/java/org/apache/maven/archiver/MavenArchiver.java Outdated Show resolved Hide resolved

jorsol force-pushed the MSHARED-1066 branch 2 times, most recently from c9249ff to 36b3bf9 Compare June 13, 2022 08:30

jorsol marked this pull request as ready for review June 13, 2022 08:31

jorsol force-pushed the MSHARED-1066 branch from 36b3bf9 to 6495ac9 Compare June 13, 2022 08:59

jorsol marked this pull request as draft June 13, 2022 09:16

jorsol force-pushed the MSHARED-1066 branch from 6495ac9 to 1f87b0d Compare June 13, 2022 09:54

jorsol marked this pull request as ready for review June 13, 2022 09:54

jorsol force-pushed the MSHARED-1066 branch 2 times, most recently from c3a5bf7 to eecb791 Compare June 14, 2022 10:53

cstamas requested a review from michael-o June 20, 2022 22:03

michael-o reviewed Jun 21, 2022

View reviewed changes

jorsol force-pushed the MSHARED-1066 branch from eecb791 to 0db85bb Compare June 21, 2022 08:02

jorsol requested a review from michael-o June 21, 2022 08:12

michael-o requested changes Jun 22, 2022

View reviewed changes

jorsol force-pushed the MSHARED-1066 branch from 0db85bb to 1444dca Compare June 22, 2022 11:10

michael-o requested changes Jun 22, 2022

View reviewed changes

src/main/java/org/apache/maven/archiver/MavenArchiver.java Outdated Show resolved Hide resolved

src/main/java/org/apache/maven/archiver/MavenArchiver.java Outdated Show resolved Hide resolved

jorsol force-pushed the MSHARED-1066 branch from 1444dca to 0d38b79 Compare June 22, 2022 11:29

michael-o pushed a commit to jorsol/maven-archiver that referenced this pull request Jun 22, 2022

[MSHARED-1066] Upgrade Plexus Archiver to 4.3.0

571cd2f

Signed-off-by: Jorge Solórzano <jorsol@gmail.com> This closes apache#22

michael-o pushed a commit to jorsol/maven-archiver that referenced this pull request Jun 22, 2022

[MSHARED-1067] Improve Reproducible Builds methods

2e0a452

Signed-off-by: Jorge Solórzano <jorsol@gmail.com> This closes apache#22

michael-o force-pushed the MSHARED-1066 branch from 0d38b79 to 2e0a452 Compare June 22, 2022 14:19

jorsol added 2 commits June 22, 2022 16:24

[MSHARED-1066] Upgrade Plexus Archiver to 4.3.0

cd79d39

Signed-off-by: Jorge Solórzano <jorsol@gmail.com> This closes apache#22

[MSHARED-1067] Improve Reproducible Builds methods

03153b5

Signed-off-by: Jorge Solórzano <jorsol@gmail.com> This closes apache#22

michael-o force-pushed the MSHARED-1066 branch from 2e0a452 to 03153b5 Compare June 22, 2022 14:24

asfgit merged commit 03153b5 into apache:master Jun 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MSHARED-1066] - Upgrade Plexus Archiver to 4.3.0, Improve the Reproducible Builds methods #22

[MSHARED-1066] - Upgrade Plexus Archiver to 4.3.0, Improve the Reproducible Builds methods #22

jorsol commented Apr 23, 2022 •

edited

cstamas commented Jun 13, 2022

jorsol commented Jun 13, 2022

cstamas commented Jun 13, 2022 •

edited

jorsol commented Jun 13, 2022 •

edited

jorsol commented Jun 13, 2022

cstamas commented Jun 16, 2022 •

edited

michael-o Jun 21, 2022

jorsol Jun 21, 2022

michael-o Jun 22, 2022

michael-o Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022 •

edited

michael-o commented Jun 22, 2022

jorsol commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022

jorsol commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022 •

edited

michael-o commented Jun 22, 2022

[MSHARED-1066] - Upgrade Plexus Archiver to 4.3.0, Improve the Reproducible Builds methods #22

[MSHARED-1066] - Upgrade Plexus Archiver to 4.3.0, Improve the Reproducible Builds methods #22

Conversation

jorsol commented Apr 23, 2022 • edited

cstamas commented Jun 13, 2022

jorsol commented Jun 13, 2022

cstamas commented Jun 13, 2022 • edited

jorsol commented Jun 13, 2022 • edited

jorsol commented Jun 13, 2022

cstamas commented Jun 16, 2022 • edited

michael-o Jun 21, 2022

Choose a reason for hiding this comment

jorsol Jun 21, 2022

Choose a reason for hiding this comment

michael-o Jun 22, 2022

Choose a reason for hiding this comment

michael-o Jun 22, 2022

Choose a reason for hiding this comment

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022 • edited

michael-o commented Jun 22, 2022

jorsol commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022

jorsol commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022

michael-o commented Jun 22, 2022 • edited

michael-o commented Jun 22, 2022

jorsol commented Apr 23, 2022 •

edited

cstamas commented Jun 13, 2022 •

edited

jorsol commented Jun 13, 2022 •

edited

cstamas commented Jun 16, 2022 •

edited

michael-o commented Jun 22, 2022 •

edited

michael-o commented Jun 22, 2022 •

edited