Support data source pagination

[jzohrab]

**Description.

In coviddatascraper, PR covidatlas/coronadatascraper#835 provides support for ArcGIS data pagination. Some json result sets are too big to return in a single response, so the requests will need to manage that. Presumably, similar to GitHub API, they provide a "nextResultSet" token or similar in the response, and then clients can requery with that as a token.

We'd need to manage that for both crawls and scrapes. Presumably this could be managed with lambdas, but the cache file naming convention will need to be page-aware, and return all files.

Describe the solution you'd like

One possibility: include page number, indexed from zero, after the cache key (or name), e.g., <datetime>-<name>-<page>-<sha>.<ext>.gz. If there is only one page (which will be true in most cases), 'page' would be 0 and there won't be any other data sets, and the thing passed to scrape would just be the content.

[camjc]

JP is a good example, we hit the 10,000 limit there

[camjc]

https://services8.arcgis.com/JdxivnCyd1rvJTrY/arcgis/rest/services/v2_covid19_list_csv/FeatureServer/0/query lets you query the dataset from a UI. Here are the main settings to get the JSON we use

[camjc]

Hoping someone can advise on how the pagination works, I don't know how it does.

[ryanblock]

Agreed. I know nothing about this system. I could really use:

1. A broken source that needs this
2. Clear instructions on how it's broken (eg steps to repro)
3. If available, any ideas on how to unbreak things and get the source humming

[jzohrab]

Perhaps relevant: covidatlas/coronadatascraper#839 - "WIP: add scraper for Singapore ARCGIS" There are a few PRs for arcgis pagination open in CDS, hard to sort out what's what though. El mié., 13 de may. de 2020 a la(s) 18:02, Ryan Block ( notifications@github.com) escribió:

…

Agreed. I know nothing about this system. I could really use: 1. A broken source that needs this 2. Clear instructions on how it's broken (eg steps to repro) 3. If available, any ideas on how to unbreak things and get the source humming — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMPWDOANEH2DYRIWWXO2C3RRMKHXANCNFSM4MKGX6LA> .

[jzohrab]

Actually relevant: currently, the PA scraper fetches paginated data. It handles this itself. e.g.

$ yarn start --location PA
...
1000 records from "... url ...&resultOffset=0&resultRecordCount=50000&f=json
...
✏️  coronadatascraper-cache/2020-5-17/55c884a3dc1f7fe60c5bb08af5371500.json written
1000 records from "... url ... &resultOffset=1000&resultRecordCount=50000&f=json
...
✏️  coronadatascraper-cache/2020-5-17/de03720e752a8e6478e066e3cb308ee2.json written
1000 records from "... url ... &resultOffset=2000&resultRecordCount=50000&f=json
...

ref src/shared/scrapers/PA/
method async function TEMPfetchArcGISJSON(obj, featureURL, date) {

[jzohrab]

WIP PR will soon close this: https://github.com/covidatlas/li/pull/193/files

[jzohrab]

New PR: #218.

[jzohrab]

PR merged 🎉