Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sample of bulk insert #23

Closed

Conversation

josejuanmontiel
Copy link

@josejuanmontiel josejuanmontiel commented Jan 30, 2023

Hi!

  • What exactly is changing? In jobber.go the writeResult change to consider interate the origin source and batch to the target database using new methods of the backend that use SQL with insert multlples rows. It hence that pass a slice to the backend method and there split acording the batch size.

  • why? Some bulks of thousands of lines could speedup if some bulk "backend" methods its used.

  • and how it helps? By configuration, could speedup the velocity of insert in target database.

  • The overall architecture.

  1. In the main loop, organize the iteration from origin to append the result to send to backend.
  2. The backend build SQL in inserts with multiple lines.

TODO:

  1. Move the batch size to configuration.
  2. If pull 21 it's merged before, add some test

@josejuanmontiel josejuanmontiel changed the title WIP Sample of bulk insert Sample of bulk insert Feb 9, 2023
@josejuanmontiel josejuanmontiel marked this pull request as ready for review February 9, 2023 22:55
@knadh
Copy link
Collaborator

knadh commented Sep 22, 2023

Sorry for missing this PR. The project is being rewritten and rebranded (#31) with a long pending to-do (replacing the Machinery lib). Once this is done in the coming couple of weeks, if you are still interested, we can re-look at this. Closing the PR for now. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants