New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARROW-12760: [C++][Python][R] Allow setting I/O thread pool size #10316
Conversation
4485515
to
4e4fc31
Compare
python/pyarrow/io.pxi
Outdated
|
||
|
||
def set_io_thread_count(int count): | ||
"""Set the number of threads to use for I/O operations.""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe can add a "See also" reference to set_cpu_count
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, I cross-referenced the two pairs of functions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, thank you
This adds functions to change the pool size at runtime, but doesn't adjust the default size of 8. Closes apache#10316 from lidavidm/arrow-12760 Authored-by: David Li <li.davidm96@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
As noted in ARROW-15929, we have code referring to the wrong thread pool (i.e., `arrow::io_thread_count()` and `arrow::set_io_thread_count()` are actually setting the number of cores used instead of the actual thread count). This appears to have been an inadvertent addition changed in ARROW-12760 (#10316). ``` r # remotes::install_github("apache/arrow/r") library(arrow, warn.conflicts = FALSE) cpu_count() #> [1] 8 set_io_thread_count(4) cpu_count() #> [1] 4 ``` This PR switches that so that we're referring to the correct thread pool: ``` r # remotes::install_github("paleolimbot/arrow/r@r-thread-io-pool-count") library(arrow, warn.conflicts = FALSE) cpu_count() #> [1] 8 set_io_thread_count(4) cpu_count() #> [1] 8 io_thread_count() #> [1] 4 ``` Closes #12633 from paleolimbot/r-thread-io-pool-count Authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Signed-off-by: Jonathan Keane <jkeane@gmail.com>
This adds functions to change the pool size at runtime, but doesn't adjust the default size of 8.