Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Iceberg catalog config property to set iceberg.worker.num-threads #11920

Open
osscm opened this issue Apr 13, 2022 · 2 comments
Open

Add Iceberg catalog config property to set iceberg.worker.num-threads #11920

osscm opened this issue Apr 13, 2022 · 2 comments

Comments

@osscm
Copy link
Contributor

osscm commented Apr 13, 2022

related issue: #11708 comment: #11708 (comment)

TableScan::planFiles

executed in a shared Iceberg worker pool. It's shared among queries so high concurrency of metadata-heavy queries can make things worse. And this thread pool is not managed by Trino so its memory usage is not tracked.

so, we should allow this property to be passed from the catalog, so that it will not be set statically.

@electrum
Copy link
Member

I think this needs to be fixed in the Iceberg library to allow per-catalog configuration rather than relying on a global system property. I don't see a way to fix this on the Trino side.

@raunaqmorarka
Copy link
Member

There is a planWith(ExecutorService executorService) API available, which we are using in IcebergSplitSource (configurable though Make Iceberg split manager threads configurable).
We could extend it's usage to all callers of planFiles to control number of threads on Trino side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants