Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Harvester runs really slow when hit bugs #3550

Closed
jbrown-xentity opened this issue Nov 19, 2021 · 0 comments
Closed

Harvester runs really slow when hit bugs #3550

jbrown-xentity opened this issue Nov 19, 2021 · 0 comments
Assignees
Labels
bug Software defect or bug component/catalog Related to catalog component playbooks/roles

Comments

@jbrown-xentity
Copy link
Contributor

jbrown-xentity commented Nov 19, 2021

When the harvester hits a hard failure and crashes, it takes ~2.5-3 minutes to build a new machine and get the fetch process started. This use to take ~1 second. This means jobs that have a bunch of failures can tie up the harvester for hours...

How to reproduce

  1. Create a harvest source for https://www.usda.gov/sites/default/files/documents/data.json on a cloud.gov machine with harvesters.

Expected behavior

Processes the ~2K datasets in less than 15 minutes

Actual behavior

Still running after 24 hours

Sketch

We could take the "mitigation" approach, and try running the harvest job as a sub-process that gets restarted on failure (using supervisor as we do currently), or just running a bunch of fetch processes.
The long term fix is to make the harvests more robust and just report failures, don't hard fail. This will require much better error handling than is currently implemented, across upstream code and GSA code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Software defect or bug component/catalog Related to catalog component playbooks/roles
Projects
Archived in project
Development

No branches or pull requests

2 participants