Add test to check all repos in Homu's cfg.toml are valid

[aneeshusa]

This will prevent typos in repo names as well as issues like #580.

Files:

• Make a new test file tests/sls/homu/valid_repos.py

The test should read the deployed cfg.toml file for Homu (should be at /home/servo/homu/cfg.toml) using the toml Python library. Each subkey under the repo key of the configuration should be queried for existence on GitHub as a valid repo.

Bonus points if you query the GitHub API for the webhooks of each repo to check that an appropriate webhook is set up (see https://github.com/servo/servo/wiki/Adding-a-repo-to-Homu), although I think this might require an new access token with permissions we don't necessarily want to provide.

[aneeshusa]

This can be tested locally using the servo-master1 VM in Vagrant; look at other tests under tests/sls/homu for similar examples that read the deployed configuration.

[aneeshusa]

It looks like we'd need the read:repo_hook, which only allows read and ping abilities for webhooks on public and private repos, per the GitHub v3 API docs. I think this is a reasonably small scope to provide to Travis to integrate into CI, since Travis already knows our webhook secret (that's a different secret).

[rwaweber]

Hey @aneeshusa, I think I may give this a shot tonight.

Also, mentioned it in #servo, but since I had a tough time grabbing the dependencies (at least on El Capitan) for the vagrant environment, I setup this cask.

[aneeshusa]

Great @rwaweber, ask if you have any questions.

BTW, did you try installing Vagrant 1.8.1 from the Hashicorp releases or via an older Cask URL (at least, I think you can do brew cask install <some_url>...)?

[rwaweber]

I've got a WIP here.

A couple things:

1. Just making sure that I'm not breaking anything, since I'm only developing with the single servo-master1 machine, it shouldn't be able to retrieve the builders. This error was what I was getting and led me to that conclusion.
2. I'm open to a different means than importing requests to query github for repos to avoid adding another dependency.

Reminder for myself to look into the pagination part of the github API. In the current state, a curl https://api.github.com/orgs/servo/repos will retrieve a subset of repositories, not all of them. Currently 123 repos are configured for the servo org from what I can tell.

[rwaweber]

Pagination is configured, though I ended up rate-limiting myself a couple times during testing. As a result, I implemented a check for a token variable to allow for authenticated requests. I could imagine that being rate-limited by the github api would be less than ideal 😅

@aneeshusa not sure what the best means of retrieving that could be, maybe yanking something from pillar? Is there a token thats available?

I still need to spend some time thinking about the webhook bit as well.

[jdm]

@aneeshusa Ping to reply to the previous question.

[rwaweber]

Spent a bit of time investigating the webhook check, and the earlier assumptions are correct that it would be necessary to issue an auth token with elevated access.

Also, poking around some of the configurations, I noticed that there's a line in the cfg.toml at /home/servo/homu/ that seems to be referring to a "TEST_HOMU_GH_ACCESS_TOKEN". Is there a way that I would be able to refer to that value in these tests like I attempted to do here?

[rwaweber]

We determined that a Get request to each repo url was sufficient and ultimately didn't warrant an access token.
I think that this could probably be closed now, right?

[aneeshusa]

Yep, this can be closed for now. We can open another issue for the bonus points task at another time if that becomes necessary.