Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core:purge-old-archive-data OOM's if table is too large #14836

Closed
paladox opened this issue Aug 31, 2019 · 3 comments

Comments

@paladox
Copy link
Contributor

commented Aug 31, 2019

Hi, i'm aware that if certain requests in core:archive fail, it builds up over time in the archive table. I'm trying to run core:purge-old-archive-data to see if it'll make the table smaller, but it OOMs because the table is too large. Could the script be improved to not OOM? Or could the core:archive command be improved not to fill the archive table like it does if a request fails?

For reference here's the sizes for the tables (between july and august)

-rw-rw---- 1 mysql mysql 2.4K Aug 26 18:39 piwik_archive_blob_2019_07.frm
-rw-rw---- 1 mysql mysql 652M Aug 26 18:40 piwik_archive_blob_2019_07.ibd
-rw-rw---- 1 mysql mysql 2.4K Aug 26 22:16 piwik_archive_blob_2019_08.frm
-rw-rw---- 1 mysql mysql 3.8G Aug 31 12:52 piwik_archive_blob_2019_08.ibd

-rw-rw---- 1 mysql mysql 2.8K Aug 26 18:45 piwik_archive_numeric_2019_07.frm
-rw-rw---- 1 mysql mysql 288M Aug 26 18:45 piwik_archive_numeric_2019_07.ibd
-rw-rw---- 1 mysql mysql 2.8K Aug 26 22:15 piwik_archive_numeric_2019_08.frm
-rw-rw---- 1 mysql mysql 1.4G Aug 31 12:52 piwik_archive_numeric_2019_08.ibd

@tsteur

This comment has been minimized.

Copy link
Member

commented Sep 1, 2019

What's your memory set to?

@paladox

This comment has been minimized.

Copy link
Contributor Author

commented Sep 1, 2019

We have 2gb of ram with most of it being used by redis. The memory we have set is 128M.

@tsteur

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

@paladox I recommend you set it to 256MB then. We would maybe start investigating if it was more than 512MB or so. Anything less than 256MB doesn't worry us all that much. Especially considering this is not executed very often etc. If you're keen on providing a PR that has same functionality but uses less memory we'd be happy to give the PR a review 👍

@tsteur tsteur closed this Sep 2, 2019

@tsteur tsteur added the answered label Sep 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.