This repository has been archived by the owner on Dec 5, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 33
Support data source pagination #1
Labels
enhancement
New feature or request
Comments
JP is a good example, we hit the 10,000 limit there |
https://services8.arcgis.com/JdxivnCyd1rvJTrY/arcgis/rest/services/v2_covid19_list_csv/FeatureServer/0/query lets you query the dataset from a UI. Here are the main settings to get the JSON we use |
Hoping someone can advise on how the pagination works, I don't know how it does. |
Agreed. I know nothing about this system. I could really use:
|
Perhaps relevant: covidatlas/coronadatascraper#839 -
"WIP: add scraper for Singapore ARCGIS"
There are a few PRs for arcgis pagination open in CDS, hard to sort out
what's what though.
El mié., 13 de may. de 2020 a la(s) 18:02, Ryan Block (
notifications@github.com) escribió:
… Agreed. I know nothing about this system. I could really use:
1. A broken source that needs this
2. Clear instructions on how it's broken (eg steps to repro)
3. If available, any ideas on how to unbreak things and get the source
humming
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMPWDOANEH2DYRIWWXO2C3RRMKHXANCNFSM4MKGX6LA>
.
|
Actually relevant: currently, the PA scraper fetches paginated data. It handles this itself. e.g.
ref |
WIP PR will soon close this: https://github.com/covidatlas/li/pull/193/files |
New PR: #218. |
PR merged 🎉 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
**Description.
In coviddatascraper, PR covidatlas/coronadatascraper#835 provides support for ArcGIS data pagination. Some json result sets are too big to return in a single response, so the requests will need to manage that. Presumably, similar to GitHub API, they provide a "nextResultSet" token or similar in the response, and then clients can requery with that as a token.
We'd need to manage that for both crawls and scrapes. Presumably this could be managed with lambdas, but the cache file naming convention will need to be page-aware, and return all files.
Describe the solution you'd like
One possibility: include page number, indexed from zero, after the
cache key
(orname
), e.g.,<datetime>-<name>-<page>-<sha>.<ext>.gz
. If there is only one page (which will be true in most cases), 'page' would be 0 and there won't be any other data sets, and the thing passed toscrape
would just be the content.The text was updated successfully, but these errors were encountered: