allow to customize the httpclient used to create S3PinotFS#10369
allow to customize the httpclient used to create S3PinotFS#10369Jackie-Jiang merged 5 commits intoapache:masterfrom
Conversation
npawar
left a comment
There was a problem hiding this comment.
lgtm.
I didn't realize we've already introduced PT notations in DateTimeFunctions. Was going to suggest we stick to the period strings we use in other places like realtime thresholds, periodic task frequencies (12h, 3m30s). But this sounds fine to me.
Please add these new props to https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3#server-config
6272a68 to
d921443
Compare
Thanks for the context. I was using PT format as Duration is what the httpClientBuilder takes. I'll make it compatible with the two formats, as it's better to keep format of timeout/period configs consistent |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #10369 +/- ##
============================================
+ Coverage 67.87% 70.44% +2.57%
- Complexity 5742 6005 +263
============================================
Files 1521 2036 +515
Lines 80305 110444 +30139
Branches 12826 16793 +3967
============================================
+ Hits 54506 77804 +23298
- Misses 21957 27202 +5245
- Partials 3842 5438 +1596
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
4ce393a to
9b002b2
Compare
...ns/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3Config.java
Outdated
Show resolved
Hide resolved
200fddc to
79270fa
Compare
79270fa to
fa41785
Compare
This PR exposes some configs to customize the httpclient used to create s3client, when to init the S3PinotFS.
Release Note
the exposed httpclient configs are
httpclient.maxConnections, e.g. 100 (this is more required for now. when we increased helixExecutorTask thread number via STATE_TRANSITION.maxThreads to load segments with more threads, some threads threw exceptions about timeout to get a http conn to download raw segments from deep store)httpclient.socketTimeout, e.g. 10shttpclient.connectionTimeout, e.g. 1m20shttpclient.connectionTimeToLive, e.g. ..httpclient.connectionAcquisitionTimeout, e.g. ..