Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batching of postgres updates during recon #297

Open
andrewgdotcom opened this issue Mar 14, 2024 · 0 comments
Open

Batching of postgres updates during recon #297

andrewgdotcom opened this issue Mar 14, 2024 · 0 comments

Comments

@andrewgdotcom
Copy link

Heavy recon sessions that involve lots of key updates cause postgres to spike to 100% CPU usage and are two orders of magnitude slower than restore from dump. This is most likely due to the overhead of single-update transactions, and could be mitigated by batching updates and using bulk submission (same as for restore). We could then treat each hashquery update as a single batch of (up to) 100 keys at a time.

The major disadvantage would be that backoff+retry would become much more likely, and much heavier to manage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant