Skip to content
This repository has been archived by the owner on May 30, 2021. It is now read-only.

feat: add ArcGIS REST API fetch function #835

Closed
wants to merge 15 commits into from

Conversation

qgolsteyn
Copy link
Collaborator

This PR creates a new function to fetch a JSON from an ArcGIS dashboard's REST API. It follows the suggestion from #832 by handling pagination, and the approach is compatible. I have update the SE scraper to demonstrate intended behavior. Any non-timeseries scraper will require a more significant change to avoid creating a cache-miss due to the changed URL.

src/shared/lib/fetch/index.js Outdated Show resolved Hide resolved
src/shared/lib/fetch/index.js Outdated Show resolved Hide resolved
src/shared/lib/fetch/index.js Outdated Show resolved Hide resolved
@jzohrab
Copy link
Contributor

jzohrab commented Apr 16, 2020

This will need some work to function with the arcgis stuff currently in master. :-)

qgolsteyn and others added 7 commits April 16, 2020 16:52
Co-Authored-By: Larry Davis <lazdnet@gmail.com>
Co-Authored-By: Larry Davis <lazdnet@gmail.com>
Co-Authored-By: Emilio Graff <1@emil.io>
Co-Authored-By: Emilio Graff <1@emil.io>
Co-Authored-By: Emilio Graff <1@emil.io>
Co-Authored-By: Emilio Graff <1@emil.io>
@qgolsteyn
Copy link
Collaborator Author

Thank you for all these comments, second round of review is now open :)

Copy link
Contributor

@jzohrab jzohrab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the coronadatascraper-cache update.

I believe that @shaperilio was looking at this as well. I'm reluctant to merge this currently, perhaps we can touch base tomorrow, @qgolsteyn , @shaperilio , and me.

@shaperilio
Copy link
Collaborator

I put a PR into @qgolsteyn's branch that migrates all the scrapers that would be affected.

@shaperilio
Copy link
Collaborator

@qgolsteyn @jzohrab @camjc @lazd pinging you folks on this PR.

@jzohrab , I believe you wanted @qgolsteyn to remove a submodule update?

Note:
I migrated every scraper that this would affect in a potentially ugly way, but so that we don't get cache misses due to different URLs. Part of this ugliness is this addition to datetime:

const ARCGIS_PAGINATION_DEPLOY_DATE = '2020-04-19'

We need to set that to the date we want these sources to start using the paginated fetch. So it has to at least be the day that we merge. If scrape date is before this date, the scrapers will use their old URLs. I believe this can all go away with li's cache architecture?

Testing:
I didn't write any tests. However, for each scraper affected, I ran a timeseries across this date boundary to ensure caching and fetching in the new way worked. This was an eyeball test, quickly checking e.g. that cases didn't plummet to zero.

@shaperilio shaperilio marked this pull request as draft April 23, 2020 20:08
@shaperilio shaperilio mentioned this pull request Apr 23, 2020
@shaperilio
Copy link
Collaborator

See #889

@jzohrab
Copy link
Contributor

jzohrab commented May 28, 2020

Closing this PR, as the work is part of #889. Thank you @qgolsteyn !

@jzohrab jzohrab closed this May 28, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants