-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add multi-worker support for tile import. #818
Conversation
Affects both point cloud and raster tile import. Doesn't affect diffing or committing.. Resurrected existing (deprecated) option --num-processes, but made it an alias for the preferred name: --num-workers.
905111a
to
3a6ed60
Compare
kart/tile/importer.py
Outdated
|
||
def get_default_num_workers(self): | ||
num_workers = get_num_available_cores() | ||
click.echo(num_workers) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
click.echo(num_workers) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
kart/tile/importer.py
Outdated
num_workers = 1 | ||
elif num_workers > self.MAX_WORKERS: | ||
raise click.UsageError( | ||
f"Importing with more than {self.MAX_WORKERS} workers is not supported" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why? this was here for the old v2 tabular import code because each process handled one dest subtree, but it doesn't seem like it makes sense for tile-based imports
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed arbitrary MAX_WORKERS limit
@@ -524,6 +572,25 @@ def import_tiles_to_stream(self, stream, sources): | |||
|
|||
p.update(1) | |||
|
|||
def copy_multiple_files_to_lfs_cache(self, copy_and_convert_tasks): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
needs a docstring
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Affects both point cloud and raster tile import.
Doesn't affect diffing or committing..
Resurrected existing (deprecated) option --num-processes, but made it an alias for the preferred name: --num-workers.
Checklist: