Performance Optimizations for Non-Bulk Indexing #26
Conversation
ric-evans
changed the title
| try: | ||
| if is_processable_path(p): | ||
| if os.path.isfile(p): | ||
| await process_file(p, manager, fc_rc, no_patch, dryrun) | ||
| await index_file(p, manager, fc_rc, patch, dryrun) |
There was a problem hiding this comment.
If you want even more performance, you can run multiple index_file operations in parallel, which may help if it is waiting on a FC response. Example: https://github.com/WIPACrepo/iceprod/blob/master/iceprod/server/scheduled_tasks/update_task_priority.py#L66
There was a problem hiding this comment.
I'll keep that in mind if it becomes a noticeable issue. The FC turnaround-time actually isn't all that bad usually. The checksums and I3Reader reading are the most expensive ops by far.
Plus, as a consequence of multi-threading, I think parallel I/O operations would actually slow things down.
There was a problem hiding this comment.
What would be nice is to re-design the multi-processing because currently the parent process isn't doing much, which just lowers CPU utilization.
|
depends on WIPACrepo/iceprod#294 for git pips |
The indexer was designed for bulk-indexing. However, it may become necessary to point the indexer at a single file (or a small batch). Until now, that would require a not insignificant overhead.
--no-patchwith--patch--patch, skip the file--blacklist-file)--processes 1/ the default case)-n/--non-recursive)--blacklistfor shorter blacklistscloses #23
closes #24