Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement] Improve the performance of incremental loading #77

Open
Lloyd-Pottiger opened this issue Oct 31, 2023 · 4 comments
Open

Comments

@Lloyd-Pottiger
Copy link
Collaborator

Now we only load incremental files one by one. However, files that have the same schema can be loaded in batch to improve the performance.

@breezewish
Copy link
Collaborator

What about updates to the same key?

@Lloyd-Pottiger
Copy link
Collaborator Author

What about updates to the same key?

Yes, it is the most directly way.

@breezewish
Copy link
Collaborator

I mean, will there be issues when multiple files contain updates to the same primary key and we concurrently load them?

@Lloyd-Pottiger
Copy link
Collaborator Author

I mean, will there be issues when multiple files contain updates to the same primary key and we concurrently load them?

Not a problem. There are rows with the same primary key in the same file. We have handled it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants