Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Control Size of Query Execution Log #4155

Open
salsakran opened this issue Jan 12, 2017 · 6 comments

Comments

Projects
None yet
6 participants
@salsakran
Copy link
Contributor

commented Jan 12, 2017

Some folks are starting to report really big query execution logs, in one case > 5G.

I think we should do 2 things -

  1. Provide an optional log rotation setting that removes logs that are older than X days. This should default to a pretty high setting.
  2. Not include raw SQL queries or alternatively gzip them.
  3. Remove unused fields (result_data, result_file, version)

Also , #4154 would be nice here, as we could separate thing where we want to hold onto things for shorter or longer periods

@camsaul

This comment has been minimized.

Copy link
Member

commented Jan 12, 2017

We could do something to deduplicate the queries since the whole thing is stored in every entry

@tlrobinson

This comment has been minimized.

Copy link
Member

commented Jan 13, 2017

I think de-duplicating the queries definitions would be the biggest win. We already have query-hash we could use as the key.

Although eventually we should probably switch that hash to something cryptographically secure, depending on how we use it (e.x. caching)

@aaronbannin

This comment has been minimized.

Copy link

commented Feb 19, 2017

I'm running Metabase on the free tier of Heroku which has a 10k record limit for Postgres. It looks the query_queryexecution and view_log are taking up the majority of this allocation. Can I safely delete excess records in these tables?

From a user perspective, it might make sense to encapsulate log rotation into a hobby vs pro setting. Hobby would be much more aggressive about managing space and wouldn't require the user to first learn what the config vars mean.

@camsaul

This comment has been minimized.

Copy link
Member

commented Feb 20, 2017

@aaronbannin you can feel free to TRUNCATE those tables and get rid of all the rows until we implement some sort of solution for this issue

@pybuche

This comment has been minimized.

Copy link

commented Apr 19, 2019

Hi !
Is there a way to control this table's size now ? I am not sure this is possible :/

@flamber

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

Related to #9688

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.