Make parallel scan thread count configurable#1140
Conversation
Replace the hardcoded single-thread BatchScanner in ParallelSnapshotScanner with a configurable value via the new fluo.scan.num.threads property. Add setNumParallelScanThreads and getNumParallelScanThreads to FluoConfiguration (default 1 thread) with accompanying Javadocs. # Conflicts: # modules/core/src/main/java/org/apache/fluo/core/impl/ParallelSnapshotScanner.java
ctubbsii
left a comment
ctubbsii
left a comment
There was a problem hiding this comment.
'num parallel' is probably redundant, as it's implied with 'threads', so the method and property names could be shorter/more concise. I'm also not sure if this setting is only for workers. If that's the case, it could be called 'worker scan threads' and use the worker config prefix. If it's for more than just the worker, then just 'scan threads' is probably fine.
| // one thread is probably good.. going for throughput | ||
| scanner = env.getAccumuloClient().createBatchScanner(env.getTable(), this.authorizations, 1); | ||
| scanner = | ||
| env.getAccumuloClient().createBatchScanner(env.getTable(), env.getAuthorizations(), |
There was a problem hiding this comment.
This changed from this.authorizations to env.getAuthorizations(). There's a few different constructors where authorizations are passed outside of the env. This might affect a test case or something... I'd have to dig. But, probably best to keep it using this.authorizations for now.
| * {@value #PARALLEL_SCAN_THREADS_PROP} | ||
| * | ||
| * @param numThreads The number of threads to use, must be positive | ||
| * @since 2.0.0 |
There was a problem hiding this comment.
| * @since 2.0.0 | |
| * @since 2.1.0 |
| * {@value #NUM_PARALLEL_SCAN_THREADS_DEFAULT} | ||
| * | ||
| * @return The number of threads used for parallel scans. | ||
| * @since 2.0.0 |
There was a problem hiding this comment.
| * @since 2.0.0 | |
| * @since 2.1.0 |
2 participants
Replace the hardcoded single-thread BatchScanner in ParallelSnapshotScanner with a configurable value via the new fluo.scan.num.threads property. Add setNumParallelScanThreads and getNumParallelScanThreads to FluoConfiguration (default 1 thread) with accompanying Javadocs.