New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[JENKINS-48061] Add new "Discover other refs" trait, GitBranchSCMHead and GitRefSCMHead #577
Conversation
And migrate old usage of plain SCMHead
|
@MarkEWaite any help is good help :) |
@rsandell while working through the pull request with debugging breakpoints set in the changes, I found a section of code that seemed to be untouched by the existing tests. Would you consider using 13882c92807ba89ea2dad694f2ec8ac6160e7f10 from my fork as ideas for additional tests? Several items are in a comment which I believe should work but do not work. I may be wrong thinking they should work, but at least the tests provide some additional coverage. |
Since it seems like the git protocol doesn't allow discovering unpublished refs anyway, we can do ahead and find the sha among all published refs instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting there
@@ -43,6 +42,11 @@ public GitBranchSCMHead(@NonNull String name) { | |||
super(name); | |||
} | |||
|
|||
@Override | |||
public String getRef() { | |||
return "refs/heads/" + getName(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Constants.R_HEADS
|
||
@Override | ||
public String getRef() { | ||
return "refs/heads/" + getName(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Constants.R_HEADS
I'm fiddling with a test for build storm |
return new GitBranchSCMRevision((GitBranchSCMHead)head, b.getSHA1String()); | ||
} | ||
} | ||
} else { //TODO change to something |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still a bit unsure what to do here.
This required a newer ish shared libraries plugin in the test scope which required a slightly newer jenkins core
I'll try to review on monday... ping me by midday in case i forget ;-) |
OK, I've been trying to come up with additional tests to write but haven't found anything yet. |
Allow more testing without the distraction of a findbugs warning
shortHashMatch); | ||
} else if (name.startsWith(Constants.R_HEADS)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this is unreachable code. I'm surprised the Java compiler didn't detect this.
Line 804 is the same condition and when that condition is satisfied, the method returns.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See my pull request which proposes to change that to Constants.R_TAGS and extends a test to reach that code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't yet performed any interactive testing of the code change, just code review and running tests with breakpoints set in a debugger. Several breakpoints in changes were not reached during my session, but I've not yet investigated further.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, this must be a copy paste error :/
public void before() throws Throwable { | ||
if (!Functions.isWindows() && "testMigrationNoBuildStorm".equals(this.getTestDescription().getMethodName())) { | ||
URL res = getClass().getResource("/jenkins/plugins/git/GitBranchSCMHeadTest/testMigrationNoBuildStorm_repositories.zip"); | ||
final File path = new File("/tmp/JENKINS-48061"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This hardcoded path ("/tmp/JENKINS-48061") embedded in the test creates the same types of problems as the JenkinsRule hardcoded path.
If different users are running Jenkins unit tests on the same Unix based computer, only one of them will be allowed to run this test successfully because other users won't have permission to remove the test directory.
See my pull request for a multi-platform proposal that worked in my simple tests.
Could we add an "after" which removes the directory with the hard coded name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, I usually don't like this way of doing it, but I couldn't figure out how to have a dynamic path without doing a configuration change and thereby voiding the test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would a relative path (tmp/JENKINS-48061
), instead of an absolute path (/tmp/JENKINS-48061
), be of any help? Just thinking out loud.
An @After
like @MarkEWaite suggested would be cool, too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ideally I'd add a "lock file" that contains a time-stamp and if the time-stamp is less than 5 minutes old then sleep until the "lock-file" is deleted... if more than 5 minutes old, wipe the temp dir.
But there should be some way to template in the path without needing to hardcode
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did we get a solution to this? If not, file a JIRA, leave a TODO comment and we can move on
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added an @After
to clean up after itself.
@MarkEWaite I merged all commits from your pr but the build storm test change into this PR. I need to have a think on how/if I can approve it somehow. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks not bad... some questions and minor niggles
GitSCMTelescope telescope = GitSCMTelescope.of(this); | ||
if (telescope != null) { | ||
String remote = getRemote(); | ||
StandardUsernameCredentials credentials = getCredentials(); | ||
telescope.validate(remote, credentials); | ||
return telescope.getRevision(remote, credentials, head); | ||
} | ||
if (head instanceof GitSCMHeadMixin) { | ||
context = context.withoutRefSpecs().withRefSpec(((GitSCMHeadMixin) head).getRef()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about somebody who has added additional ref specs in a trait... will those refspecs be removed now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes I was a bit hesitant on that myself, context.withoutRefSpecs()
does clear all refspecs from the traits.
But from what I can tell, in this context you are after a specific SCMHead
so wouldn't all other refspecs just be a waste of time and bandwidth?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A comment explaining the logic as to why it is ok to strip it out here would be ideal then
} | ||
|
||
@Override | ||
public String getRef() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
final
let's not let anyone hijack branch for a non-branch
* The ref, e.g. /refs/heads/master | ||
* @return the ref | ||
*/ | ||
String getRef(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// TODO provide a default implementation once Java 8 baseline
let's not let anyone hijack branch for a non-branch
I tried installing the updated plugin directly from the Jenkins build for this PR. I still get the following issue when attempting to load a pipeline shared library from a PR:
I'm using the refspec |
It looks like an issue that can occur if your shared pipeline doesn't fetch the refs properly, since git is trying to resolve pr/33^{commit} and pr/33 doesn't exist... Can you run "git rev-parse pr/33^{commit} in the work tree manually and see what happens? What about if you add the "--" after to force git to treat it like a revision (that should at least clarify the error message). |
@robincsmith you should not use the additional refspecs trait but instead the new "Discover other refs". |
Thanks @rsandell - I'll try that, using the latest build |
@rsandell that looks awesome. To confirm, under the new "Discover other refs" behaviour, I used Then, in my pipeline:
And the correct commit was found. |
@robincsmith yes, but with the new trait
should also work, and you can change the name mapping behind the advanced button to enable
as well. |
Yep, I confirm those work as well (with the necessary values in the 'advanced' section) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The methods that had the issues should probably all just be removed since almost all just have the default implementation. But I wanted to disturb existing code as little as possible.
@rsandell So I have not done final code review, one question for yourself and @MarkEWaite This PR adds a new "Discover other refs" trait... In my mind, the majority of users probably do not want this. If we have it there then the likelihood is that we will have lots of users using it to try and discover PRs from GitHub... something that the GitHub Branch Source is better at... similarly, we'll have users using this to integrate with Gerrit instead of driving a full Gerrit Branch Source integration, etc Would it be better to make this trait WDYT? P.S. I don't mind which you opt for, as long as you two agree and document your logic |
Putting it in a separate plugin seems a bit excessive to me, though I understand your argument for doing so. Maybe some harsh words in the description/help could suffice? Though what those harsh words would be I don't know :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hopefully from here on, I can just review the diffs ;-)
Looking good... I'm just being careful given the importance of this plugin
pom.xml
Outdated
<java.level>7</java.level> | ||
<no-test-jar>false</no-test-jar> | ||
<concurrency>1C</concurrency> | ||
<findbugs.failOnError>false</findbugs.failOnError> | ||
<workflow.version>1.14.2</workflow.version> | ||
<!--<workflow.version>1.14.2</workflow.version>--> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rainy day code?
<dependency> | ||
<groupId>org.jenkins-ci.plugins.workflow</groupId> | ||
<artifactId>workflow-step-api</artifactId> | ||
<version>2.10</version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought the workflow 2.x series required upgrading code to 2.7+ (fine if it doesn't... but we should confirm)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope, workflow 2.x is just the mono repo extraction of all the sub-plugins. All versions that I picked has a max core version of 1.642.3, that's why I bumped core dependency to that.
@@ -14,6 +14,7 @@ | |||
import org.kohsuke.stapler.QueryParameter; | |||
import org.kohsuke.stapler.StaplerRequest; | |||
|
|||
import javax.annotation.Nonnull; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we stick to the findbugs ones?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used whatever annotation that the methods extended does in an effort to try and not make too much of a mess :)
public String getDisplayName() { | ||
return "AssemblaWeb"; | ||
} | ||
|
||
@Override | ||
public AssemblaWeb newInstance(StaplerRequest req, JSONObject jsonObject) throws FormException { | ||
public AssemblaWeb newInstance(StaplerRequest req, @Nonnull JSONObject jsonObject) throws FormException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these seem like unrelated changes... ideally the core version bump would be in a separate PR... not saying to move it out now... just noting for future code archeologists
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yepp, I kept all those changes in it's own commit so that it can be reverted if it was too much in the PR, but I can't get a green build without it. I consulted with @MarkEWaite on what to do about it :)
The core and plugin bumps is needed for the built storm fix and something else I can't remember at the moment.
@@ -272,6 +317,9 @@ public final C withRemoteName(String remoteName) { | |||
@NonNull | |||
public final List<RefSpec> asRefSpecs() { | |||
List<RefSpec> result = new ArrayList<>(Math.max(refSpecs.size(), 1)); | |||
if (wantOtherRefs() && wantBranches()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what happens if only one of these is true?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well if it want other refs, then that refspec is already added to the list retrieved below. Which means that if you also want branches that refspec needs to be added, otherwise you will only fetch the other refs and not the branches.
So if we don't want other refs (and no other explicit refspec is set) then the list of refspecs is empty and so the branches will be fetched as by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I.e. this is an ugly hack to get it to work when you both want other refs and branches, if any of those are false we can go on with our normal flow. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the clarifying comment
@@ -420,6 +440,7 @@ protected void retrieve(@CheckForNull SCMSourceCriteria criteria, | |||
if (context.wantTags()) { | |||
referenceTypes.add(GitSCMTelescope.ReferenceType.TAG); | |||
} | |||
//TODO DiscoverOtherRefsTrait? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rainy day comment? (either delete it or if important but not blocking file a JIRA and update the comment with the JIRA)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a comment for you I think, I sprinkled some of those comments around where they might be needed and then implemented in those places I definitely knew they where needed and I knew what to do.
I might have missed removing some that I implemented.
@@ -537,10 +560,84 @@ public Void run(GitClient client, String remoteName) throws IOException, Interru | |||
if (context.wantTags()) { | |||
discoverTags(repository, walk, request, remoteReferences); | |||
} | |||
if (context.wantOtherRefs()) { | |||
discoverOtherRefs(repository, walk, request, remoteReferences, | |||
(Collection<GitSCMSourceContext.WantedOtherRef>)context.getOtherWantedRefs()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WantedOtherRef
is not a name I like, especially as it leaks into the public API... any suggested alternatives? (if it is the least bad, well we'll have to live with it)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AdditionalRefSpec
or something like that might be better... or worse... suggestions please... and then we pick the least ugly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RefNameMapping
aligns better with the usage that gets logged
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AdditionalRefSpec
is already used in a method somewhere.
The use of wanted
was to reflect that it is about the same as wantBranches
and wantTags
. And I guess my brain got stuck on that train :)
RefNameMapping
sounds better, I'll chew on it for a bit.
// 1. A branch name (if we have that we can return quickly) | ||
// 2. A tag name (if we have that we will need to fetch the tag to resolve the tag date) | ||
// 3. A short/full revision hash that is the head revision of a branch (if we have that we can return quickly) | ||
// 3.2 A remote refspec for example pull-requests/1/from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just renumber, no sub-numbering
tagName = StringUtils.removeStart(name, Constants.R_TAGS); | ||
context.wantBranches(false); | ||
context.wantTags(true); | ||
context.withoutRefSpecs(); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it is a ref, we save effort by returning the GitRefSCMRevision
itself, so we should set a boolean here to say "found a ref match" and then return the GitRefSCMRevision
at the end of the loop if the ref match was found. the current code will prefer a tag to a GitRefSCMRevision
despite a tag needing date resolution (i.e. more work)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes that's the way I designed it after how I interpreted it from our earlier discussions, that it's better if we could return a tag or a branch than a ref.
I guess I misunderstood?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a branch is the best
a ref should be preferred to a tag (because we do not need to resolve the time)
@@ -831,7 +977,7 @@ public SCMRevision run(GitClient client, String remoteName) throws IOException, | |||
} | |||
listener.getLogger() | |||
.printf("Selected match: %s revision %s%n", name, hash); | |||
return new SCMRevisionImpl(new SCMHead(name), hash); | |||
return new GitBranchSCMRevision(new GitBranchSCMHead(name), hash); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are missing the case where the revision does not exist on any branch but does exist in the history of an other ref. Now if that is because client.getBranchesContaining(hash, true)
doesn't allow us to query other refs, then we need to file a JIRA, add a comment and move on... otherwise we probably need to revisit the candidatePrefix
search to include other refs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, the way I think the refspec for the trait is constructed the commits will be fetched as branches on the remote, I don't like it, but it should work. I should probably add a test for that though to verify.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See AbstractGitSCMSourceTest#retrieveRevision_customRef_descendant
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just the order of resolution is retrieve(hash) needs fixing
} | ||
} else if (head instanceof GitRefSCMHead) { | ||
try { | ||
ObjectId objectId = client.revParse(((GitRefSCMHead) head).getRef()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes covered by the else... i think I was just wondering if we could merge one of the paths... no worries. fine for now
tagName = StringUtils.removeStart(name, Constants.R_TAGS); | ||
context.wantBranches(false); | ||
context.wantTags(true); | ||
context.withoutRefSpecs(); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a branch is the best
a ref should be preferred to a tag (because we do not need to resolve the time)
@@ -272,6 +317,9 @@ public final C withRemoteName(String remoteName) { | |||
@NonNull | |||
public final List<RefSpec> asRefSpecs() { | |||
List<RefSpec> result = new ArrayList<>(Math.max(refSpecs.size(), 1)); | |||
if (wantOtherRefs() && wantBranches()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the clarifying comment
So we don't have to check timestamps on the tag
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably will need refinement after it hits customers... probably good enough for now
Increment the minor version number because it requires Jenkins 1.642.3 LTS instead of the older 1.625.3 LTS.
JENKINS-48061 - Add GitBranchSCMHead and GitRefSCMHead
To be able to fetch commits not on any branch or tag
Checklist
Types of changes
What types of changes does your code introduce? Put an
x
in the boxes that apply