[JENKINS-34395] Support for tag discovery #158

stephenc · 2017-09-01T12:08:32Z

I believe this to be a full implementation, but test coverage of some aspects of this is non-trivial... OTOH given that it is mostly new functionality that would be needing testing we could just go as is

Downstream of hub4j/github-api#375

@reviewbybees

stephenc · 2017-09-01T16:19:59Z

Screenshots:

Configuring for tag discovery:

Indexing log when tags are being discovered:

Results of tag discovery:

HRMPW · 2017-09-01T17:57:12Z

How can you limit what Tags are supported, @stephenc ? If I have a repo with 500 tags I don't want 500 new projects created.

Should this be limited to annotated tags? Can it skip "Pre-release" tags?

stephenc · 2017-09-01T18:07:24Z

An extension plugin can trivially filter tags by date created

stephenc · 2017-09-01T18:13:20Z

There are several example plugins doing filtering and TagSCMHead.getDate() is the last commit for shallow tags and the tag date for annotated tags

(We could add an option to discover annotated tags only/lightweight only/signed only/etc... or maybe make that a filtering concern)

HRMPW · 2017-09-01T18:36:37Z

@stephenc We would want an Environment variable or something that can be used in a when block to control how a Tag run could be customized in the Pipeline.

@orrc creates two Environment Variables in his Git Tag Message Plugin. Something like that would help. https://wiki.jenkins.io/display/JENKINS/Git+Tag+Message+Plugin

@michaelneale @i386 @vivek @abayer Many users have been requesting this functionality. Currently, it wouldn't show up in Blue Ocean.

We have had several discussions about how this relates to creating a release or how it might work with a mvn tag. If someone creates a Pipeline with "mvn release" it will create a new tag. This will either create a new job to run if Tags are enabled or it will not show up in the Tags tab like jobs created by Tags.

stephenc · 2017-09-01T20:40:53Z

@HRMPW

We would want an Environment variable or something that can be used in a when block to control how a Tag run could be customized in the Pipeline.

you mean something like https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/BranchNameContributor.java#L88-L91 (i.e. with this change you get TAG_NAME, TAG_TIMESTAMP, TAG_DATE and TAG_UNIXTIME right now as long as you are using branch-api 2.0.11 or newer

stephenc · 2017-09-01T20:42:26Z

Currently, it wouldn't show up in Blue Ocean.

Of course if BlueOcean had actually implemented support for SCMHeadCategories, then it would magically just show up... but JENKINS-40367 hasn't had much traction

stephenc · 2017-09-01T20:49:45Z

Currently, it wouldn't show up in Blue Ocean.

I suspect given that BlueOcean is ignorant of categories, tags will show up under branches.

Lack of use of categories is also likely the root cause of slow rendering of pull requests tab in BlueOcean that I've seen JIRA for (but on phone I cannot find link for)

stephenc · 2017-09-01T20:50:34Z

There is also a similar PR for the git-plugin

michaelneale · 2017-09-01T21:03:27Z

@stephenc yeah there is some old code there. @vivek has a todo on the PR thing so he may take a look at categories etc as part of that as refresh at some point.

vivek · 2017-09-05T16:01:33Z

@stephenc

I suspect given that BlueOcean is ignorant of categories, tags will show up under branches.

blueocean (for GitHub/bitbucket) uses BranchDiscoveryTrait with strategyId 3 to take in all branches - that is all PRs and branches. Looking at this trait, tags won't show up in blueocean without extra work that is do it properly in JENKINS-40367.

stephenc · 2017-09-05T16:16:46Z

@vivek what I mean is that if the user configured the multibranch project to have the tag discovery trait then they will likely show up as branches in the blue ocean UI

…he ref/ prefix

…sing a hash rather than a ref

witokondoria · 2017-09-08T11:28:47Z

@HRMPW @stephenc im planning on submitting a aged filtering trait once #67 gets merged and released

jglick

Sounds OK to the extent I understand it.

jglick · 2017-09-08T19:16:17Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+            }
+            return result;
+        } finally {
+            Connector.release(github);


Can this not become [Auto]Closeable [sic]?

jglick · 2017-09-08T19:17:32Z

src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java

+            final GHRepository ghRepository = this.ghRepository;
+            GitHubSCMSourceContext context = new GitHubSCMSourceContext(null, SCMHeadObserver.none())
+                    .withTraits(traits);
+            Matcher prMatcher = Pattern.compile("^PR-(\\d+)(-.*)?$").matcher(headName);


Is there not already some kind of constant for this?

jglick · 2017-09-08T19:19:31Z

src/main/java/org/jenkinsci/plugins/github_branch_source/PushGHEventSubscriber.java

+                // out tags that are less than X seconds old, but as such a filter would be incompatible with events
+                // discovering tags, no harm... the key part is that a pre-filter that removes tags older than X days
+                // will not strip the tag *here* (because it will always be only a few seconds "old"), but when
+                // the fetch call actually has the real tag date the pre-filter will apply at that point in time.


Uh, OK. I guess this is all hypothetical unless and until someone writes such a filter.

stephank · 2017-09-19T09:25:57Z

I'm not well versed in Java-land, but have deployed this to our Jenkins install, and found two things so far:

The org.kohsuke.github-api dep of the github-api-plugin needs to be updated, because the current 1.86 is missing the listRefs(String) overload
I believe this fails on repos with no tags at all:

Sep 19, 2017 11:19:51 AM jenkins.branch.OrganizationFolder$OrganizationScan run
INFO: <redacted> #20170919.111643 organization scan action completed: NOT_BUILT in 3 min 8 sec
Sep 19, 2017 11:19:51 AM hudson.model.Executor finish1
SEVERE: Executor threw an exception
java.lang.Error: org.kohsuke.github.GHFileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
        at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:516)
        at org.kohsuke.github.Requester$PagingIterator.hasNext(Requester.java:481)
        at org.kohsuke.github.PagedIterator.fetch(PagedIterator.java:44)
        at org.kohsuke.github.PagedIterator.hasNext(PagedIterator.java:32)
        at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:982)
        at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:355)
        at jenkins.scm.api.SCMSource.fetch(SCMSource.java:309)
        at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263)
        at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1297)
        at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1312)
        at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:256)
        at jenkins.scm.api.trait.SCMNavigatorRequest.process(SCMNavigatorRequest.java:206)
        at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:961)
        at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:412)
        at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:276)
        at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
        at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:863)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:419)
Caused by: org.kohsuke.github.GHFileNotFoundException: {"message":"Not Found","documentation_url":"https://developer.github.com/v3"}
        at org.kohsuke.github.Requester.handleApiError(Requester.java:686)
        at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:510)
        ... 18 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/<redacted>/<redacted>/git/refs/tags
        at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:243)
        at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getInputStream(DelegatingHttpsURLConnection.java:210)
        at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:25)
        at org.kohsuke.github.Requester.parse(Requester.java:612)
        at org.kohsuke.github.Requester.parse(Requester.java:594)
        at org.kohsuke.github.Requester.access$200(Requester.java:73)
        at org.kohsuke.github.Requester$PagingIterator.fetch(Requester.java:505)
        ... 18 more

… repos without any tags

chris5287 · 2017-10-18T14:07:27Z

@stephenc thanks for the update. How easy would it be to get these changes ported over to the Bitbucket plugin as well? (https://issues.jenkins-ci.org/browse/JENKINS-47254)

stephenc · 2017-10-18T14:25:49Z

@chris5287 if you want to give it a try i’d be happy to review/mentor. I’m busy on a different project so I don’t have time to implement myself

kshultzCB · 2017-10-18T19:43:15Z

@michaelneale This is my GitHub nick. :)
I've got an ATH run going on this as we speak.
--Karl

cleverkidd · 2017-10-25T12:36:26Z

@stephenc
Hi Stephen,
I created the plugin and deployed it to jenkins. The new option appeared as expected Build everything. If I add the option and click save button, i get following exception: https://pastebin.com/nvbnShiX

can you please advise?
my jenkins-ci.plugins version is 2.36
jenkins version 2.46.3
scm-api version 2.2.3
branch-api version 2.0.14

the code is the same as you provided here

thanks in advance

ahammond · 2017-10-26T21:05:55Z

+1 the ability to trigger builds when a new tag arrives would be a huge win for us.

stephenc · 2017-10-26T21:28:12Z

@cleverkidd I created JENKINS-47678 to track that stack trace you identified. That issue is fixed in branch-api 2.0.15 (which should be available from the update centers in the next few hours)

jeff-knurek · 2017-11-02T16:26:41Z

I feel like I'm missing something on how this is suppose to work.
I've got
branch-api 2.0.15 ,
github-branch-source-plugin 2.2.5,
and https://github.com/AngryBytes/jenkins-build-everything-strategy-plugin installed

In the org folder setup, I've added the Build everything strategy. I push a tag to the repo that's being watched and nothing happens. I run Scan repository now, and the logs mention nothing about tags:


  Checking branches...

  Getting remote branches...

    Checking branch feature/publishers
      ‘Jenkinsfile’ found
    Met criteria
No changes detected: feature/publishers (still at a29373c2e692bf549f7ff114e4be6ed82f7d056d)

    Checking branch master
      ‘Jenkinsfile’ found
    Met criteria
No changes detected: master (still at a9bc55ee5d6e82111f6ca70ea6168da9289ddd12)

    Checking branch test
      ‘Jenkinsfile’ found
    Met criteria
No changes detected: test (still at b224b0276df138020efed0d1545f97c4bff294cd)

  3 branches were processed

  Checking pull-requests...

  Getting remote pull requests...

  0 pull requests were processed

Finished examining org/repo```

stephenc · 2017-11-02T17:49:00Z

@jeff-knurek

I feel like I'm missing something on how this is suppose to work.
I've got
branch-api 2.0.15 ,
github-branch-source-plugin 2.2.5,

So you would need to have 2.2.6-SNAPSHOT where you build this plugin with this PR merged.

(Hint: this plugin has not been merged yet)

jeff-knurek · 2017-11-03T15:12:48Z

I pulled the latest code from this repo, then rebased the PR code with master, built the hpi, 2.2.6-SNAPSHOT (private-ed2a6390-root)
where ed2a6390 is a reference to the last commit of this PR: ed2a6390

But still not seeing the mention of tags during a Scan??

stephenc · 2017-11-03T16:10:02Z

@jeff-knurek did you add “Discover Tags” behaviour?

jeff-knurek · 2017-11-06T11:02:18Z

@jeff-knurek did you add “Discover Tags” behaviour?

that's it!! thanks

I looked for that on my first build of the plugin, but it was still missing the change, and didn't look for it again after

kshultzCB · 2017-11-07T13:05:37Z

I've run any number of manual and automated tests against this PR, the changes look good to me.

I did observe some typical ATH flakiness, and one of the resulting issues is JENKINS-47702. This could be recreated without this PR in place.

ahammond · 2017-11-08T03:02:07Z

Yay!

ahammond · 2017-11-08T19:55:14Z

How would I get this on my jenkins server? We're running a container FROM the standard docker container that pulls plugins at build. I'm guessing I just rebuild our container?

stephenc · 2017-11-08T20:01:02Z

@ahammond I have used my psychic powers to determine exactly how your instance is configured and maintained. While I was doing that I took the liberty of upgrading the plugin for you... eh damn... seems my psychic powers are not working today

ahammond · 2017-11-09T19:43:56Z

@stephenc The correct answer turns out to be "yeah, if you're pulling plugins at build, you should get the latest version of this plugin the next time you build". Tags are now building for us. Thanks.

ashb · 2017-12-07T12:10:14Z

A question about the "build everything plugin" idea. Does the tag fetching distinguish between the initial scan (which might pull in a lot of tags) and subsequent polls/updates/events?

i.e. is it possible for me to write a plugin that will build only newly created tags, but not any tags found during the intial scan?

(i.e. like to have my Jenkins instance be re-createable, and i don't persist any of the workspace or files from instance to instance.)

As a backup I could use the date of the commit the tag is on (we use lightweight tags usually) and just ignore any tag not created in the last X*2 mins (where X is the check interval).

stephenc · 2017-12-07T12:18:24Z

@ashb this is entirely up to the person writing the BranchBuildStrategy extension... you can have multiple such ones. Build Everything is just the most basic one.

I recommend that there would be three or more strategies for building branches, tags and PRs. The tag one could be either simple or it could allow configuration to build based on the tag date (the TagSCMHead mixin has a tag date)

ashb · 2017-12-07T12:23:59Z

Sorry, I meant does a strategy plugin have enough info to distinguish those two cases? (first time populating repo vs new tag in a repo that has been scanned before)

stephenc · 2017-12-07T12:28:05Z

how could it? all it knows is there is now a tag where once there was not and the tag reports a tag date of XYZ (depending on the type of tag that could be the last commit time or the time the heavyweight tag was created)

stephenc · 2017-12-07T12:29:12Z

but you could have the strategy say "only build tags that are less than X days old and must be after today the time I am saving the form"

stephenc · 2017-12-07T12:31:28Z

This issue is merged. There is are mailing lists for discussion

stephenc force-pushed the jenkins-34395 branch from 4607dbd to 0ec2003 Compare September 1, 2017 12:08

stephenc changed the title ~~[JENKINS-34393] Support for tag discovery~~ [JENKINS-34395] Support for tag discovery Sep 1, 2017

stephenc requested review from abayer and jglick September 1, 2017 12:11

stephenc added 7 commits September 7, 2017 10:34

[JENKINS-34395] Support for tag discovery

e28cd4a

[JENKINS-34395] Updates based on manual testing

51415fa

[JENKINS-34395] Found out that github-api plays fast and loose with t…

d094402

…he ref/ prefix

[JENKINS-34395] Fix SCM build for tags

909ce64

[JENKINS-34395] Ensure that lightweight checkout works for tags

6f9ee58

[JENKINS-34395] Revert changes to test data and fix filesystem when u…

41e016f

…sing a hash rather than a ref

[JENKINS-34395] Properly handle push events for tags

51fc6ef

stephenc force-pushed the jenkins-34395 branch from c85e43c to 51fc6ef Compare September 7, 2017 09:36

jglick approved these changes Sep 8, 2017

View reviewed changes

stephenc closed this Sep 18, 2017

stephenc reopened this Sep 18, 2017

[JENKINS-34395] Update to 1.89 of github-api and guard against 404 in…

4d47488

… repos without any tags

stephenc added 2 commits October 20, 2017 13:15

Merge branch 'master' into jenkins-34395

014230c

[JENKINS-34395] Add some tests and update git dependency

e98acfa

stephenc force-pushed the jenkins-34395 branch from 8768d73 to e98acfa Compare October 23, 2017 11:09

stephenc added 2 commits November 6, 2017 21:24

Merge branch 'master' into jenkins-34395

af2923a

Merge branch 'master' into jenkins-34395

67e8211

stephenc merged commit 2fc5266 into jenkinsci:master Nov 7, 2017

stephenc deleted the jenkins-34395 branch November 7, 2017 13:07

jenkinsci locked and limited conversation to collaborators Dec 7, 2017

[JENKINS-34395] Support for tag discovery #158

[JENKINS-34395] Support for tag discovery #158

Conversation

stephenc commented Sep 1, 2017 • edited

stephenc commented Sep 1, 2017

HRMPW commented Sep 1, 2017

stephenc commented Sep 1, 2017

stephenc commented Sep 1, 2017

HRMPW commented Sep 1, 2017

stephenc commented Sep 1, 2017

stephenc commented Sep 1, 2017

stephenc commented Sep 1, 2017

stephenc commented Sep 1, 2017

michaelneale commented Sep 1, 2017

vivek commented Sep 5, 2017

stephenc commented Sep 5, 2017

witokondoria commented Sep 8, 2017 • edited

jglick left a comment

Choose a reason for hiding this comment

jglick Sep 8, 2017

Choose a reason for hiding this comment

jglick Sep 8, 2017

Choose a reason for hiding this comment

jglick Sep 8, 2017

Choose a reason for hiding this comment

stephank commented Sep 19, 2017 • edited

chris5287 commented Oct 18, 2017 • edited

stephenc commented Oct 18, 2017

kshultzCB commented Oct 18, 2017

cleverkidd commented Oct 25, 2017

ahammond commented Oct 26, 2017

stephenc commented Oct 26, 2017

jeff-knurek commented Nov 2, 2017

stephenc commented Nov 2, 2017

jeff-knurek commented Nov 3, 2017

stephenc commented Nov 3, 2017

jeff-knurek commented Nov 6, 2017 • edited

kshultzCB commented Nov 7, 2017

ahammond commented Nov 8, 2017

ahammond commented Nov 8, 2017

stephenc commented Nov 8, 2017

ahammond commented Nov 9, 2017

ashb commented Dec 7, 2017

stephenc commented Dec 7, 2017

ashb commented Dec 7, 2017

stephenc commented Dec 7, 2017

stephenc commented Dec 7, 2017

stephenc commented Dec 7, 2017

stephenc commented Sep 1, 2017 •

edited

witokondoria commented Sep 8, 2017 •

edited

stephank commented Sep 19, 2017 •

edited

chris5287 commented Oct 18, 2017 •

edited

jeff-knurek commented Nov 6, 2017 •

edited