Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ImportBot imports very slow #7215

Closed
2 tasks done
cdrini opened this issue Dec 5, 2022 · 0 comments · Fixed by #7210
Closed
2 tasks done

ImportBot imports very slow #7215

cdrini opened this issue Dec 5, 2022 · 0 comments · Fixed by #7210
Assignees
@cdrini
Labels
Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] Module: Import Issues related to the configuration or use of importbot and other bulk import systems. [managed] Priority: 1 Do this week, receiving emails, time sensitive, . [managed]
Milestone
Sprint 2022-10 & ...

Comments

@cdrini
Copy link
Collaborator

cdrini commented Dec 5, 2022 •
edited

  • The network requests are likely the bottle neck; run in parallel
  • Haproxy is blocking ImportBot, and limiting it to 2 open connections
@cdrini cdrini added Priority: 1 Do this week, receiving emails, time sensitive, . [managed] Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] labels Dec 5, 2022
@cdrini cdrini added this to the Active Sprint milestone Dec 5, 2022
@cdrini cdrini self-assigned this Dec 5, 2022
@cdrini cdrini added the Module: Import Issues related to the configuration or use of importbot and other bulk import systems. [managed] label Dec 5, 2022
@cdrini cdrini mentioned this issue Dec 6, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cdrini cdrini

Labels
Lead: @cdrini Issues overseen by Drini (Staff: Team Lead & Solr, Library Explorer, i18n) [managed] Module: Import Issues related to the configuration or use of importbot and other bulk import systems. [managed] Priority: 1 Do this week, receiving emails, time sensitive, . [managed]
Projects
None yet
Milestone
Sprint 2022-10 & 2022-11
Development

Successfully merging a pull request may close this issue.

Parallel process importbot imports to increase throughput + bug fixes cdrini/openlibrary
1 participant
@cdrini