Problems updating GitHub commit status

[roidrage]

For the last couple of days, we've seen an ever-increasing amount of errors related to updating the commit status (both on .org and on .com).

The error commonly looks like this:

Could not update the PR status on https://api.github.com/repos/boto/boto/statuses/0059db720e66981336f140e0a7b16e25d15b1cdc (GH request failed 
verb:        
     :post 
url:         
      /repos/boto/boto/statuses/0059db720e66981336f140e0a7b16e25d15b1cdc
headers:     
     {"User-Agent"=>"Travis-CI/0.0.1 GH/0.11.3", "Accept"=>"application/vnd.github.v3+json,application/vnd.github.beta+json;q=0.5,application/json;q=0.1", "Accept-Charset"=>"utf-8", "Origin"=>"travis-ci.org"} 
error:       
     Faraday::Error::ClientError: the server responded with status 422 
payload:     
     nil 
response_status:  
     422 
response_body:  
     {"message":"Validation Failed","errors":[{"resource":"Status","code":"custom","f ield":"state","message":"state is not included in the list"}]}).

The first time we've seen this error in the logs was on Monday, but it really started to pick up coming in July 25 and 26.

The payload in this case seems to be empty. There was a deploy to travis-tasks on both platforms on .org and .com, but I'm not sure how/if that's related.

@drogus @rkh could you guys please look into this?

[roidrage]

I stand corrected, the first occurrence of this error on .org was on July 23rd, at 18:10, which is after the last deploy. On .com it started around the same time (18:02)

[sarahhodne]

The only change I can see is the "user rename" change, which broke something on api, probably related?

[sarahhodne]

Confirmed to be an issue with GitHub's API:

› curl -i -u henrikhodne -X POST -d '{"state":"passed","description":"Test status","target_url":"https://github.com"}' https://api.github.com/repos/henrikhodne-test/travis-test-staging/statuses/65938d910d091989a98c0db9cfb92958306fefb9
HTTP/1.1 422 Unprocessable Entity
Server: GitHub.com
Date: Fri, 26 Jul 2013 18:41:34 GMT
Content-Type: application/json; charset=utf-8
Status: 422 Unprocessable Entity
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4727
X-RateLimit-Reset: 1374867309
X-GitHub-Media-Type: github.beta
X-Content-Type-Options: nosniff
Content-Length: 197
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes
Access-Control-Allow-Origin: *

{
  "message": "Validation Failed",
  "errors": [
    {
      "resource": "Status",
      "code": "custom",
      "field": "state",
      "message": "state is not included in the list"
    }
  ]
}

Emailed GitHub Support.

[sarahhodne]

Turns out my reproducing thing above was invalid, but the issue is still there. Waiting on GitHub Support.

[sarahhodne]

This was a problem with GitHub, looks like it's been resolved now.

[sarahhodne]

This started showing up again 7-8 hours ago, notified GitHub.

[mislav]

@henrikhodne Is this still a problem? I'm experiencing missing statuses in GH Status API for finished Travis .org builds.

I'm at GitHub; can I help you to identify if we still have a bug in our API?

[sarahhodne]

@mislav Could you link me to a build where you noticed this, so I can look in our logs to see if we sent a request to the API and if there was an error?

[mislav]

@henrikhodne None of the recent ruby-build builds have any status in the Status API https://travis-ci.org/sstephenson/ruby-build/builds

[mislav]

@henrikhodne The last ruby-build build that has a recorded status in Status API is https://travis-ci.org/sstephenson/ruby-build/builds/14130802

~ 60 Travis builds after that apparently didn't record a status.

[sarahhodne]

We don't seem to have a GitHub token on file for a user with permission to update the GitHub status, so we never even send the request. Probably related to you not being able to sign in (#1766).

[mislav]

@henrikhodne I'm able to sign into Travis now. I've pushed a new branch to sstephenson/ruby-build and it got built but the Status API still isn't getting updated. What GitHub token do you have to have on file to be able to update GitHub Status? I presume it has to be sstephenson. Does he have to log in to Travis again? Please advise.

[sarahhodne]

@mislav We currently try to find a user with admin permissions to determine what GitHub token to use to update the status. It seems we no longer have sstephenson's GitHub token, so we can't find an admin on file. Having him logging in on Travis again would probably fix this, but after reading the documentation I see one only needs push access to update the repository status, so I'll work on a change that tries to find a user with push access instead of just limiting to admins.

[mislav]

Thanks. And why did you lose sstephenson's key? Do they expire or
something? Just curious how Travis does these things. Is this part of
Travis open source?

[sarahhodne]

I think the only reason for us to remove the token from the database is when the GitHub API tells us the token is invalid. The most common reason for this is if the token was revoked on GitHub.

[sarahhodne]

The niling of the token is done here, in case you're interested. The GitHub Status updating is done here.

[roidrage]

Closing this.

[mislav]

@roidrage @henrikhodne Thanks for helping me out with this. The issue is now largely gone, but I'm still seeing intermittent instances of CI statuses not being recorded in the GitHub API.

For instance, the 2 most recent open pulls to lostisland/faraday (363, 369) don't have a CI status, but the Travis builds for them ran. Here's a Travis build for the pull 369: https://travis-ci.org/lostisland/faraday/builds/23732361 — the build passed but the CI status for that commit SHA is empty and GitHub web interface doesn't display any CI status

[roidrage]

We don't have an admin user for this repository currently, that's why we can't talk to the GitHub API.

We have some fixes in the work to make this stuff better and not just look for admins.

[mislav]

The solution then, as before, is that technoweenie logs into Travis web
interface?

[roidrage]

Unfortunately yes, sorry.

[sarahhodne]

I just deployed a change that I think should fix this.