Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"add all tables" slows over time on many tables #111

Open
michaelcrobinson opened this issue Mar 31, 2016 · 1 comment
Open

"add all tables" slows over time on many tables #111

michaelcrobinson opened this issue Mar 31, 2016 · 1 comment

Comments

@michaelcrobinson
Copy link

I guess this is less of an issue, more of an observation, since my use case is a little weird.

I have a database with over a hundred schemas across over a hundred tables. Needless to say, this results in a lot of tables, and I knew it would take a long time to run "add all tables". Something I noticed is that over time, it slowly started taking longer to add tables. Two hours in, the process had slowed to 1 tables every >1s. It also appears to be single threaded, not that that matters.

As a workaround, I wrote a script that queries my database for a list of schemas, and for loops though the list to do all tables like so:

for SCHEMA in $SCHEMAS ; do
  bucardo add all tables db=source_db relgroup=db_group -n $SCHEMA
done

Works for add all sequences too.

turnstep added a commit to turnstep/bucardo that referenced this issue May 5, 2021
Digging into Github issue bucardo#111 showed that the slowdown was caused primarily
by the herdcheck trigger on the herdmap table. Rather than scan the whole table
after each row is added, we do a one-time check right before we commit anytime
we do inserts to the table from the 'bucardo' program.
@turnstep
Copy link
Collaborator

turnstep commented May 5, 2021

I was able to duplicate this problem: the slowdown occurs in the repeated INSERTs to the herdmap table. Specifically, the herdcheck trigger was slowing things down too much. Made a proposed fix at #234

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants