Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[mydumper] Combine -r and -F #525

Closed
davidducos opened this issue Dec 15, 2021 · 0 comments · Fixed by #547
Closed

[mydumper] Combine -r and -F #525

davidducos opened this issue Dec 15, 2021 · 0 comments · Fixed by #547

Comments

@davidducos
Copy link
Member

davidducos commented Dec 15, 2021

Currently you can select --rows or -F not both at the same time. What I'm thinking that could be really useful, is to have both options, but we will need to support a new file format:
<database>.<table>.<PART 1>.<PART 2>.sql

So, in cases where you have an integer primary key and you use -r and -F, we could see files like this if the table has gaps in it:

  • test_db.test_table.00000.sql
  • test_db.test_table.00000.00000.sql
  • test_db.test_table.00000.00001.sql
  • test_db.test_table.00000.00002.sql
  • test_db.test_table.00001.sql
  • test_db.test_table.00003.sql
  • test_db.test_table.00004.sql
  • test_db.test_table.00008.sql

Another alternative to implement it, is to keep an autoincrement value per table, but we need a mutex to avoid having 2 files with the same number and we need call that function every time we need to. I don't like this one, as it might cause fragmentation when we insert as we will be loosing the order.

What is the goal?
We are not going to create files larger than -F. NEVER. This is useful for the streaming use cases as large files could be a bottleneck when you have low network bandwidth.
This is the other side of #422, so, if you ask why are we trying to do both things and not just keep one way? well, there are multiple use cases and I think that we could support them.

@davidducos davidducos added this to the Release 0.11.7 milestone Dec 15, 2021
@davidducos davidducos linked a pull request Jan 17, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant