Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize the DB Loading #23

Closed
rippinrobr opened this issue Apr 29, 2018 · 1 comment
Closed

Parallelize the DB Loading #23

rippinrobr opened this issue Apr 29, 2018 · 1 comment
Labels

Comments

@rippinrobr
Copy link
Owner

Currently, loading the different databases is done in a serial fashion, the files are read and parsed multiple times, and the data is loaded into each db type one at a time. The new process should be

  • Read the files once
  • the data should be passed to each DB type's worker who is responsible for loading the data into the DB
  • Should add a verbose flag so I only print out the progress in the way I do it now when I need it
@rippinrobr
Copy link
Owner Author

This will be resolved when the new project structure is in place. This will be backups, schemes, backups and docker files with no code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant