Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More details on errors #78

Closed
rgaudin opened this issue Jan 14, 2021 · 2 comments
Closed

More details on errors #78

rgaudin opened this issue Jan 14, 2021 · 2 comments

Comments

@rgaudin
Copy link
Member

rgaudin commented Jan 14, 2021

Just realized that our fondamentaux ZIM was drastically smaller than it should (and previously was): 2.35 GB and 2.54 GB instead of 9.36 GB.
Those numbers were from three different runs a few days apart (Nov 4th – large one, Nov 6th and Nov 10th).

With an exit-code of 0, we had no idea those newer ZIMs were problematic.

Understanding we can't fail on every single error when scraping a generic website, we could still be a little smarter by recording and exposing the number of failed fetches so that our QA process can evaluate whether the output is OK or not.

@ikreymer, how realistic is adding a count of succeeded/failed fetches? I think the error count in stdout only regards the webpages, right ? Those runs didn't had any 1513 / 1513 (100.00%), errors: 0 (0.00%).

@stale
Copy link

stale bot commented Mar 19, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

@stale stale bot added the stale label Mar 19, 2021
@rgaudin rgaudin self-assigned this Feb 1, 2023
@rgaudin
Copy link
Member Author

rgaudin commented Feb 1, 2023

This now exists.

@rgaudin rgaudin closed this as completed Feb 1, 2023
@stale stale bot removed the stale label Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant