Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please properly license and sign the artifacts contributed to the release train. #763

Closed
merks opened this issue Oct 21, 2019 · 41 comments
Closed

Comments

@merks
Copy link

merks commented Oct 21, 2019

Looking at this report, specifically for Eclipse Collections contribution to SimRel:

https://download.eclipse.org/oomph/archive/simrel/collections.aggrcon/index.html

It's clear that you folks need to add a proper license to the feature and that you need to sign the feature's jars.

I've tried to get your attention via this channel:

https://www.eclipse.org/lists/cross-project-issues-dev/msg17024.html

But the latest contributions are still in bad shape.

At some point I'll ask that the planning council remove such non-conforming contributions...

If there's something I can do to help, feel free to ask.

@nikhilnanivadekar
Copy link
Contributor

@merks yes we will need help to resolve this. None of us maintainers are eclipse experts, so it would be great if you could tell us what needs to be done. I am happy to read the documentation. That said, I have tried numerous times to sign the p2 artifacts without success (there are few PRs and milestone releases performed only for the signing)
FWIW I think our jar artifacts carry the correct license and are signed. Or I am mistaken there as well?
Appreciate all the help we can get to resolve this.
Cc: @guw @ujhelyiz

@merks
Copy link
Author

merks commented Oct 27, 2019

You can compare your feature.xml's project against this example:

https://git.eclipse.org/c/emf/org.eclipse.emf.git/tree/features/org.eclipse.emf-feature

You really should include description, copyright, and most especially the license elements.

You do not need to include the *.html files in this folder:

https://github.com/eclipse/eclipse-collections/tree/master/p2-feature/org.eclipse.collections.feature

That's because the build process should copy over properties and license files from the license plugin from CBI. But the missing element in the feature.xml will result in the p2 content metadata not containing the license reference even if the actual jar does contain the copied-over license.

Your two bundles are signed, but the two artifacts for the two features are not signed. In think you need to enable something like the for the feature artifacts to be properly signed:

https://git.eclipse.org/c/emf/org.eclipse.emf.git/tree/releng/org.eclipse.emf.parent/features/pom.xml#n173

The tag is just so we could do builds without signing so you don't need that part, but you need to do something like this for the feature artifacts to be signed.

@nikhilnanivadekar
Copy link
Contributor

nikhilnanivadekar commented Oct 27, 2019

Ok. I’ll give it a try later today. Thanks for your help!

@ujhelyiz
Copy link
Member

Sorry for not answering earlier, as I was busy with other tasks - if some help is necessary, I could try to have a look at the underlying issue next week.

@nikhilnanivadekar
Copy link
Contributor

That will be awesome @ujhelyiz ! Thanks a ton for looking in to it.
@merks what are the timelines we are looking at?

@merks
Copy link
Author

merks commented Oct 28, 2019

It would be ideal if this were done for contribution to 2019-12 M2:

https://wiki.eclipse.org/Category:SimRel-2019-12#Schedule

It's likely that contributions that do not conform to these minimal requirements will be disabled soon after that point.

If you produce a new update p2 site and let me know the URL for it, I can have a quick look to see its state.

@ujhelyiz
Copy link
Member

I had a look at the issue: the latest release site was much better, as it had the required metadata included, including the EPLv2 license text, however, it was not signed.

I have opened PR #770 that ensures that EPLv1 is loaded (as far as I could see, the project is licensed under EPLv1.0) and signs the service.

A p2 repository with the signature and license entries is available for now at https://ci.eclipse.org/viatra/job/ec-feature-fix/lastSuccessfulBuild/artifact/p2-feature/org.eclipse.collections.repository/target/repository/

@merks
Copy link
Author

merks commented Nov 1, 2019

I downloaded the zipped update site and tested it locally and indeed the proper SUA 1.1 is used, all five artifacts are properly signed, and you have a *.pack.gz for the library jar (which is 15% of the size of the normal jar). That's perfect. Thanks for taking action. (I always enjoy reading the blogs about this stuff.)

@merks merks closed this as completed Nov 1, 2019
nikhilnanivadekar added a commit that referenced this issue Nov 7, 2019
Fixes generated Eclipse features #763
@ujhelyiz
Copy link
Member

@nikhilnanivadekar Is there any p2 repository available that contains the signed feature? I am asking before it should be added to the SimRel repository as soon as possible (I'd rather have a version included before M3 for the December release).

Right now the repository contains version 9.2 (added by myself about a year ago); if you point me to an appropriate repository, I can update the contribution right now.

@nikhilnanivadekar
Copy link
Contributor

The signed feature will go for 10.1.0 I think. But if we want to have the signed feature I can cherry pick your commits and do a bug fix release of 10.0.1.
I am also fine closing this release and doing a 10.1.0 release. But that will take a 2 week review period. Thoughts?

@ujhelyiz
Copy link
Member

I don't know what was the planned release date for 10.1.0; for SimRel 2019-12 the release date should be not later than the 18th December (and the final build should be ready for the 11th, the final RC build date).

To me the version number is not that important; maybe because of the short notice, a bugfix version (10.0.1) would be cleaner.

However, whichever direction you want to go, it would be important to have a pre-release build with the final version number available for next weeks M3 milestone, as that would give others (including the VIATRA project I am responsible for) to test whether everything works as expected).

@nikhilnanivadekar
Copy link
Contributor

Let me work on 10.1.0 Milestone today so that we can contribute that. I’ll schedule 10.1.0 to be released at the first available release for December

@nikhilnanivadekar
Copy link
Contributor

Reopening for tracking work for 10.1.0 and simrel contribution.

@nikhilnanivadekar
Copy link
Contributor

Created release: https://projects.eclipse.org/projects/technology.collections/releases/10.1.0
Waiting on 1 PR to be merged to kick off the review process.

@nikhilnanivadekar
Copy link
Contributor

The CI Instance of Eclipse Collections is having issues connecting to GitHub to create the artifacts: Bugzilla raised: https://bugs.eclipse.org/bugs/show_bug.cgi?id=553330

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz the p2 artifacts for 10.1.0.M1 are pushed. Build: https://ci.eclipse.org/collections/job/publish-p2-repo/
We are still having issues deploying to maven central, but sim rel is not dependent on that. Can you please test out 10.1.0.M1 p2 artifact?

@ujhelyiz
Copy link
Member

I have looked at the build job, but I have a question: where can I find the built repository? I don't see any links, etc.

Furthermore, looking into the build log the feature version 10.0.0-SNAPSHOT seems incorrect to me, considering we are aiming to have 10.1.0 milestone build here. It might be fine, but I don't know where to check.

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz you are correct 10.0.0-SNAPSHOT seems incorrect. Where can I see the feature version is setup to debug?
Also, the p2 repo is not getting published because the previously configured shell script does not work on new CI build jobs. Do you have the equivalent shell script steps for the new CI builds?
This is the current shell script:

#!/bin/bash
cd /home/data/httpd/download.eclipse.org/collections
rm -rf $RELEASE_TAG
mkdir $RELEASE_TAG
cd $RELEASE_TAG
cp -r $WORKSPACE/p2-feature/org.eclipse.collections.repository/target/repository .
cp    $WORKSPACE/p2-feature/org.eclipse.collections.repository/target/p2-repository-*.zip .
mv p2-repository-*.zip p2-repository-$RELEASE_TAG.zip

@nikhilnanivadekar
Copy link
Contributor

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz I fixed the build script. Now we have the artifacts published. Can you please take a look?

running command ssh genie.collections@projects-storage.eclipse.org ls -al /home/data/httpd/download.eclipse.org/collections/10.1.0.M1
total 32516
drwxrwsr-x  3 genie.collections technology.collections       71 Nov 22 11:33 .
drwxrwsr-x 19 genie.collections technology.collections     4096 Nov 22 11:33 ..
-rw-r--r--  1 genie.collections technology.collections 17068167 Nov 22 11:33 p2-repository-10.1.0.M1.zip
drwxr-sr-x  4 genie.collections technology.collections      171 Nov 22 11:33 repository

@ujhelyiz
Copy link
Member

Glad to hear the build works, sorry I was not able to help before.

Yes, I see that the feature is versioned 10.0.0.v20191122-1234 (the end qualifier is fine). This version is set in the maven pom.xml instances related to the build and the feature.xml file. PR #472 seems to include such a version bump.

However, executing the command mvn org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=10.1.0-SNAPSHOT -Dcollections.repository.url=file:///${project.basedir}/../p2-repository/org.eclipse.collections/target/repository in the p2-feature folder could also fix the version numbers automatically (see discussion at the PR for the reason about the property redefinition).

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz I investigated further about the 10.0.0-SNAPSHOT issue. Seems like the features folder does not get updated during the build processes with the correct qualifiers: https://github.com/eclipse/eclipse-collections/tree/master/p2-feature

Can you please tell me all the files where it needs to be updated. I will try to fix it in the build or if not fix it manually. I wish there was an automated way to do this.

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz let me try to configure that command and rerun the snapshot build.

@nikhilnanivadekar
Copy link
Contributor

nikhilnanivadekar commented Nov 22, 2019

@ujhelyiz I dont think that really worked. I am trying to set it on our Jenkins build. Any tips to do that vs the plain command line? Log: https://ci.eclipse.org/collections/job/new-version/44/consoleFull

[INFO] Making changes in /home/jenkins/agent/workspace/new-version/p2-repository
[INFO] Making changes in /home/jenkins/agent/workspace/new-version/p2-repository/org.eclipse.collections
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Eclipse Collections p2 Repository Build 10.1.0-SNAPSHOT:
[INFO] 
[INFO] Eclipse Collections p2 Repository Build ............ SUCCESS [  2.442 s]
[INFO] Eclipse Collections OSGi Bundle .................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:24 min
[INFO] Finished at: 2019-11-22T17:03:55Z
[INFO] ------------------------------------------------------------------------
[new-version] $ /opt/tools/apache-maven/latest/bin/mvn -f pom.xml -DNEXT_VERSION=10.1.0 -DRELEASE_BRANCH=master -Dmaven.repo.local=/home/jenkins/agent/workspace/new-version/.repository clean install
Picked up JAVA_TOOL_OPTIONS: -XX:+IgnoreUnrecognizedVMOptions -XX:+UseContainerSupport -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Picked up _JAVA_OPTIONS: -XX:MaxRAMPercentage=64.0
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z)
Maven home: /opt/tools/apache-maven/latest
Java version: 1.8.0_202, vendor: Oracle Corporation, runtime: /opt/tools/java/oracle/jdk-8/1.8.0_202/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-957.1.3.el7.x86_64", arch: "amd64", family: "unix"

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz I have been trying to even make it work and run in my local, but without luck. If you are able to run the command and push a commit, raise a PR, I will happily merge the PR 😄

ujhelyiz added a commit to ujhelyiz/eclipse-collections that referenced this issue Nov 22, 2019
Signed-off-by: Zoltan Ujhelyi <zoltan.ujhelyi@incquerylabs.com>
ujhelyiz added a commit to ujhelyiz/eclipse-collections that referenced this issue Nov 22, 2019
Signed-off-by: Zoltan Ujhelyi <zoltan.ujhelyi@incquerylabs.com>
@ujhelyiz
Copy link
Member

I have opened PR #776 with the fix.

I have seen two issues while trying to do this:

  1. According to the log, you seem to have executed the version change from the 'p2-repository' folder instead of the 'p2-feature'.
  2. The relative path to the p2 repository created by the main build was incorrect, I have fixed that in a separate commit (but also included in the PR mentioned above).

The version bump was done programmatically:

Ujhelyis-MacBook-Pro:p2-feature stampie$ mvn org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=10.1.0-SNAPSHOT
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T20:33:14+02:00)
Maven home: /usr/local/Cellar/maven@3.5/3.5.4/libexec
Java version: 12.0.1, vendor: AdoptOpenJDK, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-12.jdk/Contents/Home
Default locale: en_HU, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] Computing target platform for MavenProject: org.eclipse.collections:org.eclipse.collections.feature:10.1.0-SNAPSHOT @ /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.feature/pom.xml
[INFO] Fetching p2.index from http://download.eclipse.org/cbi/updates/license/
[INFO] Fetching p2.index from http://download.eclipse.org/cbi/updates/license/
[INFO] Adding repository http://download.eclipse.org/cbi/updates/license
[INFO] Fetching p2.index from http://download.eclipse.org/cbi/updates/license/2.0.2.v20181016-2210/
[INFO] Fetching p2.index from http://download.eclipse.org/cbi/updates/license/2.0.2.v20181016-2210/
[INFO] Adding repository file:/Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.feature/../../p2-repository/org.eclipse.collections/target/repository
[INFO] Resolving dependencies of MavenProject: org.eclipse.collections:org.eclipse.collections.feature:10.1.0-SNAPSHOT @ /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.feature/pom.xml
[WARNING] The following locally built units have been used to resolve project dependencies:
[WARNING]   org.eclipse.collections/10.1.0.v20191122-1602
[INFO] Resolving class path of MavenProject: org.eclipse.collections:org.eclipse.collections.feature:10.1.0-SNAPSHOT @ /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.feature/pom.xml
[INFO] Computing target platform for MavenProject: org.eclipse.collections:p2-repository:10.1.0-SNAPSHOT @ /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.repository/pom.xml
[INFO] Adding repository file:/Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.repository/../../p2-repository/org.eclipse.collections/target/repository
[INFO] Resolving dependencies of MavenProject: org.eclipse.collections:p2-repository:10.1.0-SNAPSHOT @ /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.repository/pom.xml
[WARNING] The following locally built units have been used to resolve project dependencies:
[WARNING]   org.eclipse.collections/10.1.0.v20191122-1602
[INFO] Resolving class path of MavenProject: org.eclipse.collections:p2-repository:10.1.0-SNAPSHOT @ /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.repository/pom.xml
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Eclipse Collections p2 Repository with Feature Build               [pom]
[INFO] org.eclipse.collections.feature                        [eclipse-feature]
[INFO] p2-repository                                       [eclipse-repository]
[INFO] 
[INFO] -----------------< org.eclipse.collections:p2-feature >-----------------
[INFO] Building Eclipse Collections p2 Repository with Feature Build 10.1.0-SNAPSHOT [1/3]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- tycho-versions-plugin:1.2.0:set-version (default-cli) @ p2-feature ---
[INFO] Making changes in /Users/stampie/git/eclipse-collections/p2-feature
[INFO] Making changes in /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.feature
[INFO] Making changes in /Users/stampie/git/eclipse-collections/p2-feature/org.eclipse.collections.repository
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Eclipse Collections p2 Repository with Feature Build 10.1.0-SNAPSHOT SUCCESS [  0.305 s]
[INFO] org.eclipse.collections.feature .................... SKIPPED
[INFO] p2-repository 10.1.0-SNAPSHOT ...................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.954 s
[INFO] Finished at: 2019-11-22T22:37:57+01:00
[INFO] ------------------------------------------------------------------------

ujhelyiz added a commit to ujhelyiz/eclipse-collections that referenced this issue Nov 22, 2019
Signed-off-by: Zoltan Ujhelyi <zoltan.ujhelyi@incquerylabs.com>
@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz thanks a ton for your help! I fixed it on the Jenkins build so that going forward we do not need to worry about it for every release. Screenshot of the configuration:
image

I think you can drop the second commit, if you rebase on to upstream/master then I can merge your PR.

donraab added a commit that referenced this issue Nov 22, 2019
Build fixes for release. Part of #763
@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz 10.1.0.M2 is deployed to maven central and to Eclipse p2 repository. Can you please test it out and verify? Thank you very much for your help!

@ujhelyiz
Copy link
Member

Sadly, the p2 repository is not published; I have not seen it at downlooad.eclipse.org

According to https://ci.eclipse.org/collections/job/publish-p2-repo/26/console the publishing failed.

[publish-p2-repo] $ /bin/bash /tmp/jenkins1731683002814502597.sh
/tmp/jenkins1731683002814502597.sh: line 2: cd: /home/data/httpd/download.eclipse.org/collections: No such file or directory
No mail will be sent out, as 'publish-p2-repo #26' does not have a result yet. Please make sure you set a proper result in case of pipeline/build scripts.
Finished: SUCCESS

@nikhilnanivadekar
Copy link
Contributor

I am checking what went wrong. I won’t deny that I am totally confused why build 25 worked and 26 didn’t even when there are no changes 🤷‍♂️

@nikhilnanivadekar
Copy link
Contributor

I have reverted the config from build 25. I do not remember changing the config for 26, but the config seemed to have been reverted to the one used by 24. Kicked off the build again, let us see.

@ujhelyiz
Copy link
Member

The repository https://download.eclipse.org/collections/10.1.0.M2/repository/ seems now correct. I have executed tests, and for now create a gerrit change for the simrel repository to add this change to the aggregation (will not merge it until you have reviewed it of course). This way it will be ready for this weeks milestone.

@ujhelyiz
Copy link
Member

The simrel Gerrit change is available from https://git.eclipse.org/r/c/153330/ - it seems the validation finished correctly, so it could be merged as well.

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz I have approved the code review but I can only give a +1, I think we need a ++1 correct to be merged? Please let me know if anything else needs to be done.

@ujhelyiz
Copy link
Member

@nikhilnanivadekar Yes, it needs a +2 from someone with commit rights to the repository. If nobody from the project has commit rights, you should write a Bugzilla issue to Community/Cross-project category, and the webmaster team will set up the permissions.

For now, I have added that and merged the code. This means, the next SimRel build will include this milestone build. After the final release build is ready, this contribution needs to be updated the same way.

@nikhilnanivadekar
Copy link
Contributor

Perfect! Thank you very much!

@ujhelyiz
Copy link
Member

The latest repository report shows the Eclipse Collection issues fixed.

@nikhilnanivadekar
Copy link
Contributor

Great! Thanks a ton!

@nikhilnanivadekar
Copy link
Contributor

@ujhelyiz 10.1.0 is available in the p2 repo, can you please check?

@ujhelyiz
Copy link
Member

ujhelyiz commented Dec 5, 2019

The build looks reasonable to me; I have also created a Gerrit change for SimRel: https://git.eclipse.org/r/c/153909/

@nikhilnanivadekar
Copy link
Contributor

Perfect! Thanks @ujhelyiz
Closing this issue as all problems are addressed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants