Web Scraping Script?

[Camiann]

@Nishnha This is great work. I was wondering if you could post the script you used to scape the public comment submissions and attachments from regulations.gov.

[Nishnha]

Hey @Camiann, I do have the script saved somewhere on another machine, but I won't be able to access it for a couple weeks.
I'll keep this issue open as a reminder to add it to the repo.

In the meantime, it may do you well to look into doing it yourself with webscraper.io. That's the tool I used. It's intuitive and it gets past the Javascript loading wall on the regulations.gov website.

[Camiann]

Thanks!

[Nishnha]

@Camiann,

Here is a Gist of the webscraper.io sitemap export that I used for web scraping the regulations.gov comments.

I had to remake the script, so it may differ slightly from the original one in terms of naming; functionality should be identical.

Hope this helps!