Skip to content
This repository has been archived by the owner on Mar 19, 2021. It is now read-only.

feat(scripts): vacuum tables after import #23

Merged
merged 1 commit into from
Nov 8, 2016
Merged

Conversation

philbooth
Copy link
Contributor

Related to #20 (comment), adds VACUUM commands for each table after data is added to them.

Strictly speaking, in most cases we only need SORT ONLY rather than FULL. However the scripts do DELETE rows if force_reload is set, and the redshift docs state that the saving for SORT ONLY isn't normally huge, so it seems prudent to opt for FULL.

I timed some VACUUM commands earlier and they took about five seconds to complete. I didn't check the stats beforehand though to see whether they had much to do, so that may not be a fair indication of how the import scripts will be affected in practice.

You'll notice the weird extra END; that each VACUUM block starts with. This is the only way I found to work around the error VACUUM cannot run inside a transaction block. I also tried SET AUTOCOMMIT, which didn't work.

@rfk, r?

@rfk
Copy link
Contributor

rfk commented Nov 8, 2016

I'm +1 to this, and I think the vacuum should complete in a timely manner if we do it regularly.

This is the only way I found to work around the error VACUUM cannot run inside a transaction block

This flips my weirdness sensor though, I'm going to take a look and see if I can understand what's going on here before merge.

@rfk
Copy link
Contributor

rfk commented Nov 8, 2016

OK, looks like the "postgres" wrapper module runs everything in a transaction by default, and there's not an easy way to disable that. We might try moving to the lower-level "psycopg2" module but r+ on this for now.

@rfk rfk merged commit 15f9e88 into master Nov 8, 2016
@rfk rfk removed the waffle:review label Nov 8, 2016
@philbooth philbooth deleted the phil/vacuum-after branch November 8, 2016 09:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants