Discussion: github scraping package

[willnorris]

One of the frustrations we've long had at Google is the amount of functionality that GitHub does not expose through either the REST or graphql API, for whatever reason. Most recently, this has been around things like listing the OAuth and GitHub Apps that have been approved for a GitHub organization. There are no official APIs for this (and lots of other) functionality, so you have to just do it manually through the GitHub UI, or screen scrape.

We've built a few single-purpose screen scraping tools like this using GoogleChrome/puppeteer, but that's a bit more complicated (and heavyweight) to run in our production environment than a pure Go approach.

A long time ago, I had talked about having an experimental package in go-github that provided a clean API for some of this scraping functionality, but I never actually built it until now.

I now have a proof of concept that successfully authenticates to github.com, including performing two-factor-auth, and can then scrape some of the additional data we need. The code is reasonably clean, but probably not particularly stable, given that it relies on specific class names and IDs of elements on the GitHub site.

So my question is, where should this scraping package live?

How do folks feel about having an experimental "scrape" package in go-github? It would be forever unstable, and would probably need to be exempt from any versioning policy that we have for the library as a whole. My main thought for keeping it in go-github (versus a standalone githubscrape repo, or whatever) is just visibility, but also because it would very specifically be designed to fill the gaps in the REST API, so it does actually track with the main go-github library itself pretty closely.

/cc @gauntface @gmlewis @dmitshur

[gmlewis]

SGTM!
Specifically, I like the idea of putting the scraper in this repo, with examples of how to use it.

[willnorris]

scraper added in 6b2a645. We'll continue to build it out over time, but closing this issue.

[dmitshur]

I think it's reasonable to include it in this repository, but it may make sense to keep it in a separate module. That way, its dependencies wouldn't affect the main module, and it can be versioned separately. The main module undergoes major version bumps quite often, and I understand those are not going to be useful to scrape since it's effectively at v0.

[willnorris]

@dmitshur that's a great idea (putting it into a separate module). Since the main module is defined at the root of the repository, can you define a separate module in a sub-path? I've never tried that.

[willnorris]

Okay, it seems like it "just works"? Though it looks like we might need to move the scraping example into the module as well so that goquery and friends don't get picked up as indirect dependencies in the parent module. I'll send out a second PR.

[dmitshur]

Yes, take a look at https://golang.org/wiki/Modules#faqs--multi-module-repositories for more information about it.

I would suggest moving the scraping example into scrape/example/scrape, that way it's nicely namespaced and contained in the scrape module.