Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[YSQL] Support txn batch size for COPY to improve performance #2855

Open
ndeodhar opened this issue Nov 5, 2019 · 1 comment
Open

[YSQL] Support txn batch size for COPY to improve performance #2855

ndeodhar opened this issue Nov 5, 2019 · 1 comment
Assignees
Labels
Projects
Milestone

Comments

@ndeodhar
Copy link
Contributor

@ndeodhar ndeodhar commented Nov 5, 2019

Today for improving COPY performance, we have a non-txn-copy gflag that can be used to run copy in non transactional mode. However, this does not take effect if table requires transactions (for example, if table has an index).

We should add a copy_txn_batch_size option to improve COPY performance for such transactional tables. This will reduce the batch size for transactions instead of loading all the data in a single large transaction.

@ndeodhar ndeodhar added the area/sql label Nov 5, 2019
@ndeodhar ndeodhar added this to the v2.1 milestone Nov 5, 2019
@ndeodhar ndeodhar added this to To do in SQL Support via automation Nov 5, 2019
@mbautin

This comment has been minimized.

Copy link
Collaborator

@mbautin mbautin commented Nov 7, 2019

It might be useful to expose these COPY behavior options as GUCs, so that we could say e.g. SET yb_copy_txn_batch_size=100 in a session.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
SQL Support
  
To do
3 participants
You can’t perform that action at this time.