PR status check continuous-integration/jenkins/pr-head points to unconfigured-jenkins-location sometimes

[jbw976]

PR #1582 is another example where the Jenkins build actually succeeded, but the Github UI is showing failure and pointing at http://unconfigured-jenkins-location/job/rook/job/rook/job/PR-1582/1/display/redirect. Note the unconfigured-jenkins-location in the URL (edited).

Some notes on this issue can be found in #1582 (comment), but further discussion should occur in this issue.

We really need to understand:

• why this happens intermittently
• why this can happen AFTER a green build has already been marked on the PR
• why @DoubleDensity still shows up in the continuous-integration/jenkins/pr-head status checks

/cc @ichekrygin @travisn

[galexrt]

@jbw976 Is this still an issue? I don't think I saw a PR in the last 2-3 weeks that had this issue and if they had this issue from previously, a force push fixed the issue.

[travisn]

@galexrt yes i still see it and have been restarting the builds when it happens

[jbw976]

This happened last night again, this time on #1719. I've poked around the Jenkins config and googled a bunch and can not find a cause or resolution for this very annoying issue.

[jbw976]

@travisn what do you mean "restart the builds"? In most cases, the build on Jenkin's side is actually green, but in the Github PR the check is pointing to the broken http://unconfigured-jenkins-location URL. What do you do to restart the build and have the PR notice it?

EDIT: oh, starting a new build (not replaying) for the rook/rook pipeline on the correct PR branch does actually update the PR status when it's complete. That's good to know going forward.

[travisn]

@jbw976 right, i've seen the PR status get updated after the build is restarted.

[checkaayush]

Have we tried this: https://stackoverflow.com/questions/46483609/jenkins-slack-notification-shows-unconfigured-jenkins-location?

[jbw976]

By looking at the history of statuses for a given commit in a PR, we can see the history of status updates that in turn influence the status of the PR. Take this status history for example from #1769 that can be found in full at
https://api.github.com/repos/rook/rook/statuses/100e80250f2e3f266bb80f5522477bae45abf245:

> cat pr-status.txt | jq '.[] | .creator.login + ": " + .state + ", " + .updated_at + ", " + .target_url'
"jenkins-rook: success, 2018-06-20T03:19:28Z, https://jenkins.rook.io/job/rook/job/rook/job/PR-1769/7/display/redirect"
"jenkins-rook: pending, 2018-06-20T02:37:29Z, https://jenkins.rook.io/job/rook/job/rook/job/PR-1769/7/display/redirect"
"DoubleDensity: error, 2018-06-16T07:35:45Z, http://unconfigured-jenkins-location/job/rook/job/rook/job/PR-1769/5/display/redirect"
"DoubleDensity: pending, 2018-06-16T07:35:41Z, http://unconfigured-jenkins-location/job/rook/job/rook/job/PR-1769/5/display/redirect"
"jenkins-rook: success, 2018-06-15T07:29:51Z, https://jenkins.rook.io/job/rook/job/rook/job/PR-1769/6/display/redirect"
"jenkins-rook: pending, 2018-06-15T06:32:00Z, https://jenkins.rook.io/job/rook/job/rook/job/PR-1769/6/display/redirect"

Note that build 6 is in a success state, then a day later @DoubleDensity comes along and sets it to error with the uconfigured-jenkins-location URL but for build 5 (the previous build). Then I re-ran that PR's build to get build 7 completed and successful. Very odd that the @DoubleDensity account comes along and replays an old no longer valid build.

The only way to update commit statuses according to the Github statuses docs is to have write permissions to the repo. I'm going to temporarily remove @DoubleDensity from the contributors team which should remove the write privileges. Let's see if PRs after this change no longer get error status updates with bogus URLs.

[jbw976]

I haven't seen this issue since removing the write permissions from the @DoubleDensity account. I'm going to close this issue since the massive annoyance of PRs going from green to red within a day seems to have been addressed.

[galexrt]

@jbw976 finally, thanks! Though we should look into updating the Jenkins regardless ;)