Skip to content
This repository has been archived by the owner on Dec 5, 2022. It is now read-only.

Ability for scraper to push errors down the pipeline without throwing #555

Open
lazd opened this issue Mar 20, 2020 · 2 comments
Open

Ability for scraper to push errors down the pipeline without throwing #555

lazd opened this issue Mar 20, 2020 · 2 comments

Comments

@lazd
Copy link
Contributor

lazd commented Mar 20, 2020

A couple ways this could work:

  1. Scraper called with errors array as an argument that can be pushed to (feels weird man)
  2. Scraper called with this = { errors: [] } (breaks a lot of scrapers)
  3. Scraper called with region array as an argument you can push data to, can throw at any time (i.e. throw at the end of the scraper to indicate a non-fatal error)
  4. Other ways?
@lazd
Copy link
Contributor Author

lazd commented Mar 20, 2020

@qgolsteyn @ryanblock thoughts?

@jzohrab
Copy link
Contributor

jzohrab commented Apr 2, 2020

What's an example of when this would be useful?

To me, idea 1 errors array isn't bad, it's a collector parameter. Have used such things in the past. Could even be a stream or a logger they push to.

  • Other possibility: global logging, potentially. Scrapers could instantiate a ScraperError(this, err) and give it to a global logger routine. If scrapers know they can continue despite the error, great.

@jzohrab jzohrab closed this as completed Aug 9, 2020
@jzohrab jzohrab reopened this Aug 9, 2020
@jzohrab jzohrab transferred this issue from covidatlas/coronadatascraper Aug 9, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants