P7 Crawling: Office of Elementary and Secondary Education

[Daniellappv]

Description: Scrape metadata for https://oese.ed.gov/

Acceptance criteria

• We have a data dump with all the resources metadata we can get from target site

Task-list:

• Crawl the site
• Perfect the crawling to reach as many resources as possible
• Integrate with the existing pipeline rules (provide a HTML response for the parser)
• Test run with a dummy parser - it should collect datasets and dump them into JSON files
• Push the code once it checks all the above criteria

Jira card: https://open-data-ed.atlassian.net/browse/OD-507