Find Web-Scraping Targets

[arnavs]

[arnavs]

One idea may be to look at the SEC. I like it because it's (a) highly economic, (b) highly structured, but (c) stodgy enough to not have a simple API. See: this EDGAR service, which is exactly the kind of "have the RA do the downloads" work we were talking about. And there's plenty of data, both textual and numerical, for us to analyze.

There is also some prior art in this area:

• A paper with some example code for scraping SEC:EDGAR.
• This Python script, which I think does the same thing.

etc.

cc: @jlperla

[jlperla]

Sounds good in principle. Lets see what you guys come up with.

One thing that would be nice, but is not required, is if one of the things you scrape is an address. The reason is that I wouldn't mind showing people how to geocode as well. So SEC filing and an address for the company as a second part without be great.

[arnavs]

Makes sense. Although we'd probably get questions about why most of the companies are headquartered in Delaware :P