Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git-hg 3.0 #1107

Closed
karianna opened this issue May 30, 2019 · 8 comments
Closed

git-hg 3.0 #1107

karianna opened this issue May 30, 2019 · 8 comments
Assignees
Labels
Milestone

Comments

@karianna
Copy link
Member

@karianna karianna commented May 30, 2019

After a discussion with @andrew-m-leonard and @johnoliver we will attempt to do the following:

  1. For git-hg-jdk8u - Document how the script really works in the README and in the Jenkins job configuration. For example, the move-release-to-next-tag option needs to be manually executed, it creates a branch with that tag and that is what a subsequent build would need to build from.

  2. For git-hg-aarch64-jdk8u - See 1.

  3. For jdk11u, jdk12u, jdk+ - archive the existing dev branch and use a new 'merge' technique (as opposed to the 'rebase' technique). The first new run of this will recreate the dev branch.

  4. Potentially update Jenkins release build scripts to make sure folks choose release branches for the jdk8u releases as opposed to the dev or master branch.

  5. Improve git-hg diff jobs and do an actual diff between the release branch we created and what's in mercurial. The recommendation is to use bash diff <a> <b> as git's diff tries to do ancestor searches etc.

Assign List

  • - Document git-hg-jdk8u usage - Martijn
  • - Document git-hg-aarch64-jdk8u usage - Martijn
  • - Redo git-hg for 11+ - Andrew
  • - Update build scripts - Martijn / John
  • - Improve Git Diff scripts - TBD
  • - Document 11+ usage - Martijn
@karianna karianna added this to the May 2019 milestone May 30, 2019
@karianna karianna added this to TODO in openjdk-build via automation May 30, 2019
@karianna karianna moved this from TODO to In Progress in openjdk-build May 30, 2019
@karianna karianna modified the milestones: May 2019, June 2019 Jun 3, 2019
@andrew-m-leonard

This comment has been minimized.

Copy link
Contributor

@andrew-m-leonard andrew-m-leonard commented Jun 3, 2019

@karianna @johnoliver Hi guys, could you review this proposal for the new jdk11+ merge script please?

Re: our mercurial to git discussion on jdk11u, and the options of rebase or merge for the "dev" branch, i've come to the conclusion there is no "safe" way of doing either for the "dev" branch and assume then building a given jdk-11.. tag that it will include all our "dev" patches. Also using "rebase" is particularly problematic, as a) it does not merge the "tags" into the "dev" branch since it is re-writing the commits on top of our "patches", b) because it "rewrites" original commits on top of our "patches" it will cause the "dev" and "master" branches to diverge.., c) as a result it is not possible without "re-tagging" to be able to build from a "tag" as was desired.

For OpenJ9 where we have a significant number of patches, we build a release branch, that we know is based off the correct original tag with our patches merged on top...
I propose we use a "release branch" strategy similar to OpenJ9 based off a master branch that is an exact mirror inclusive of original "tags":

Initial re-build of repo:

  1. Archive the current openjdk-jdk11u "dev" branch to "dev_old" branch
  2. Create a new "master" branch as an exact mirror inclusive of "tags"
  3. Branch "master" as a new "dev" branch and apply any "Patches" from old "dev" branch

Ongoing new merge-job and Nightly builds:

  1. Fetch new mercurial HEAD updates as mirror using git fetch hg --tags ..... into master branch
  2. Merge "latest jdk-* build tag" into "dev" branch, and tag merge commit with marker tag "jdk-...._merge" (eg.jdk-11.0.4+5_merge)
  3. Then merge "HEAD" into "dev", so that "dev" branch is up to date with latest master
  4. Nightly builds build "dev" branch, ie.based on latest HEAD from openjdk

Releases:

  1. Last nightly merge job will have brought latest updates in... If OpenJDK has just "push'd" new release tag, then run job manually to update immediately...
  2. Determine what "Release tag" we need to build release against? (eg.jdk-11.0.4+6)
  3. Branch "dev" branch at "jdk-_merge" tag to a new release branch "dev_jdk-" (eg."dev_jdk-11.0.4+6")
  4. IF any new "Patches" were committed to the "dev" branch AFTER the "jdk-_dev" tag, then "Cherry Pick" those into the release branch if they are required..
  5. Release build => SCMReference="" PublishOverrideName="jdk-"

Repeat this strategy for jdk11u, jdk12u and jdk(13) repo's...
Care must be taken with Oracle managed repos as they don't have a jdkNN-dev branch and all development commits get pushed into jdkNNu, meaning the HEAD might contain "nextrelease+1" commits.

Note, on determining latest merged mercurial build, OpenJ9 uses a strategy of utilizing the "git merged jdk-*" command and performing a sort to reliably determine the latest merged build tag, ignoring "-ga" ones.

@karianna

This comment has been minimized.

Copy link
Member Author

@karianna karianna commented Jun 3, 2019

C&P from Slack so far

martijnverburg [17:52]
Thanks,  Andrew - it seems that `master` would lag as we’re only updating `dev` nightly?
We use master as a ‘this is the same as canonical upstream’ or ‘upstream without our patches applied’ sanity check

leonarda [18:17]
So master would be up to date, but "dev" would lag
does that seem reasonable? we could always merge HEAD to "dev", but that makes "releasing" more difficult as we have to release from a specifically merged tag from "dev", which if we continuously merge HEAD we may have skipped over...
it's not then impossible, just makes release branching a bit more tricky

martijnverburg [18:23]
So for our nightly builds (non-release) which branch would hold the latest from mercurial + our patch? Do we need a 3rd branch?
@andrew-m-leonard

This comment has been minimized.

Copy link
Contributor

@andrew-m-leonard andrew-m-leonard commented Jun 10, 2019

A few things I wanted to discuss about the new method:

  1. As discussed in the Issue, it basically mirrors openjdk to master (including tags) and including latest HEAD. It then merges any newly tagged builds into "dev" first and tagging the "merge commit" with "_merge", it then merges latest HEAD.
  2. This method works well and reliably, however there is some precaution (and awareness) required when working with Oracle managed releases (eg.jdk12u, and soon jdk13u..). The reason being because Oracle don't use a separate jdknn-dev branch for public development fixes, meaning the current jdknnu branch can contain commits for the release after the one coming, so since we merge HEAD into our "dev" branch then our "dev" branch contains not just commits for the next release but also release+1. Oracle's "release tag" is on their merged internal branch, so we really need to branch from that exact tag and merge any "patches" we have onto that...
    To summarize this by an example, essentially say for jdk13u:
  • When JDK13 gets branched by Oracle for the upcoming jdk-13 release jdk-updates/jdk13u will be created
  • jdk-updates/jdk13u however will also get jdk-13.0.1 commits merged into it even before jdk-13 GAs
  • Meaning our jdk13u mirror will actually be jdk-13 plus some jdk-13.0.1 commits..
    If as it stands we don't have any "patches" then this is not an issue as we just build the Oracle "jdk-.." tag, it's only when we put patches on top... this would have been an issue with the rebase method too...

In order to progress this PR, it obviously needs reviewing... but as I need to delete the current master branch, archive dev to dev_old, I need Write Authorization adding to jdk11u and jdk12u so I can do that please? Also if we can create a new jdk ("next") repo at the same time so we can mirror and build javaNext potentially as well?

@andrew-m-leonard

This comment has been minimized.

Copy link
Contributor

@andrew-m-leonard andrew-m-leonard commented Jun 10, 2019

Updated above strategy comment with latest: #1107 (comment)

karianna referenced this issue in AdoptOpenJDK/openjdk-jdk11u Jun 11, 2019
@karianna karianna pinned this issue Jun 24, 2019
@karianna

This comment has been minimized.

Copy link
Member Author

@karianna karianna commented Jun 24, 2019

In conversation with @johnoliver we discussed how we could keep the existing repo (so that we didn't lose release binaries). We thought we would be able to clone/archive the source code and run the new scripts on the existing repo.

@johnoliver

This comment has been minimized.

Copy link
Member

@johnoliver johnoliver commented Jul 3, 2019

think we are done on this? @andrew-m-leonard

@karianna karianna modified the milestones: June 2019, July 2019 Jul 5, 2019
@andrew-m-leonard

This comment has been minimized.

Copy link
Contributor

@andrew-m-leonard andrew-m-leonard commented Jul 8, 2019

git-hg mirroring for jdk11+:

  • openjdk-jdkNNu git repo mirror:
    • master = mirror of openjdk HG repo
    • dev = master(HEAD) + Adopt patches
    • release = master(latest build tag) + Adopt patches
  • Nightly builds build from "dev" branch as before
  • Release builds build from the "xx_adopt tag", which is the tagged merge of a openjdk build in the "release" branch
@karianna karianna self-assigned this Jul 18, 2019
@karianna karianna modified the milestones: July 2019, August 2019 Aug 6, 2019
@karianna karianna modified the milestones: August 2019, September 2019 Sep 2, 2019
@karianna karianna modified the milestones: September 2019, October 2019 Oct 4, 2019
@karianna karianna modified the milestones: October 2019, November 2019 Nov 3, 2019
@sxa555 sxa555 modified the milestones: November 2019, December 2019 Dec 4, 2019
@andrew-m-leonard

This comment has been minimized.

Copy link
Contributor

@andrew-m-leonard andrew-m-leonard commented Dec 4, 2019

Think this is all done

openjdk-build automation moved this from In Progress to Done Dec 4, 2019
@sxa555 sxa555 unpinned this issue Dec 19, 2019
@douph1 douph1 mentioned this issue Jan 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
openjdk-build
  
Done
4 participants
You can’t perform that action at this time.