New API to get tenant tags #10937
Merged
New API to get tenant tags #10937
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = @ApiKeyAuthDefinition(name = | ||
| HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = SWAGGER_AUTHORIZATION_KEY))) | ||
| @Path("instance") | ||
|
|
There was a problem hiding this comment.
Empty line to be removed
pom.xml
Outdated
| @@ -23,11 +23,11 @@ | |||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | |||
| <modelVersion>4.0.0</modelVersion> | |||
|
|
|||
| <parent> | |||
| <!-- parent> | |||
There was a problem hiding this comment.
This needs to be reverted.
| @@ -10,7 +10,7 @@ | |||
| * http://www.apache.org/licenses/LICENSE-2.0 | |||
| * | |||
| * Unless required by applicable law or agreed to in writing, | |||
| * software distributed under the License is distributed on an | |||
| * software distributed under the License is distributed on an | |||
There was a problem hiding this comment.
I think we can revert both changes in this file.
| HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = SWAGGER_AUTHORIZATION_KEY))) | ||
| @Path("instance") | ||
|
|
||
| public class InstanceTagFetcher { |
There was a problem hiding this comment.
I think we can call the class InstanceResource? That way it could contain APIs for any instance level info.
npuppala111
changed the title
Codecov Report
@@ Coverage Diff @@
## master #10937 +/- ##
==========================================
- Coverage 0.11% 0.11% -0.01%
==========================================
Files 2189 2203 +14
Lines 117928 118134 +206
Branches 17834 17876 +42
==========================================
Hits 137 137
- Misses 117771 117977 +206
Partials 20 20
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 141 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
chenboat
reviewed
Jun 23, 2023
| @SwaggerDefinition(securityDefinition = @SecurityDefinition(apiKeyAuthDefinitions = @ApiKeyAuthDefinition(name = | ||
| HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = SWAGGER_AUTHORIZATION_KEY))) | ||
| @Path("instance") | ||
| public class InstanceResource { |
There was a problem hiding this comment.
Please add javadoc to clearly explain the use of this new resource api
|
@npuppala111 @ankitsultana the instance tag can be retrieved from Pinot controller using the controller's instance api. The tag is stored in Zookeeper and the Pinot controller is usually the right service to got the tag info. I wonder why we can not get these info from the controller? |
For some of our observability requirements we need to retrieve tenant tags at instance level (for both brokers and servers). This is done by a separate thread in the JVM which acts as a sidecar (it's still within the same process as the instance so not exactly a sidecar per usual definition). To do this we have the following options:
Given the above I think it's best to have a separate API. |
walterddr
reviewed
Jun 27, 2023
Comment on lines
66
to
67
| InstanceConfig config = HelixHelper.getInstanceConfig(_helixManager, _instanceId); | ||
| return config.getTags(); |
There was a problem hiding this comment.
null check? zk helix admin util seems can return null here
| HttpHeaders.AUTHORIZATION, in = ApiKeyAuthDefinition.ApiKeyLocation.HEADER, key = SWAGGER_AUTHORIZATION_KEY))) | ||
| @Path("instance") | ||
| public class InstanceResource { | ||
| private final ScheduledExecutorService _executorService = Executors.newScheduledThreadPool(3); |
There was a problem hiding this comment.
Looks like _executorService is unused? Can you remove it?
| /** | ||
| * This resource API can be used to retrieve instance level information like instance tags | ||
| */ | ||
| @Api(value = "instance-tags", description = "Instance tags", tags = "instance-tags", |
There was a problem hiding this comment.
- I think we can remove the value attribute since other resources in broker and server are not using it.
- The
tagsattribute value can be changed toinstance. i.e.tags = "instance". - We can change the description to "Metadata for this instance (like tenant tags)"
| @ApiResponses(value = { | ||
| @ApiResponse(code = 200, message = "Success"), | ||
| @ApiResponse(code = 500, message = "Internal server error"), | ||
| @ApiResponse(code = 404, message = "Table not found") |
There was a problem hiding this comment.
I think we can remove the 404 message
Comment on lines
69
to
70
| InstanceConfig config = HelixHelper.getInstanceConfig(_helixManager, _instanceId); | ||
| return config.getTags(); |
There was a problem hiding this comment.
Shouldn't we return an empty list here as well if config or config.getTags() is null.
There was a problem hiding this comment.
+1. @npuppala111 can you do that here similar to how you are doing it for broker InstanceResource?
| public List<String> getInstanceTags() { | ||
| InstanceConfig config = HelixHelper.getInstanceConfig(_helixManager, _instanceId); | ||
| if (config != null && config.getTags() != null) { | ||
| return config.getTags(); |
There was a problem hiding this comment.
In case of untagged instances, are we planning to return untagged or should we return empty list?
There was a problem hiding this comment.
We will mirror InstanceConfig so as not to introduce new business logic.
chenboat
approved these changes
Jul 11, 2023
| @Produces(MediaType.APPLICATION_JSON) | ||
| public List<String> getInstanceTags() { | ||
| InstanceConfig config = HelixHelper.getInstanceConfig(_helixManager, _instanceId); | ||
| return config.getTags(); |
| import static org.apache.pinot.spi.utils.CommonConstants.SWAGGER_AUTHORIZATION_KEY; | ||
|
|
||
| /** | ||
| * This resource API can be used to retrieve instance level information like instance tags |
There was a problem hiding this comment.
Can you add a . to complete the sentence?
|
|
||
|
|
||
| /** | ||
| * This resource API can be used to retrieve instance level information like instance tags |
There was a problem hiding this comment.
Add . to finish the sentence.
tibrewalpratik17
approved these changes
Jul 11, 2023
robertzych
added a commit
to robertzych/pinot
that referenced
this pull request
Jul 26, 2023
commit 3632453a95937ab22495a09ee95478dea560e226
Merge: abc68e04f5 f793bca39d
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jul 25 15:48:08 2023 -0700
Merge branch 'apache:master' into compaction
commit f793bca39d535dda6cd05e61319abd2d8c8a6f2c
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Tue Jul 25 15:11:32 2023 -0700
In TableRebalancer, remove instance partitions only when reassigning instances (#11169)
commit 7191cc54bf823509bea1bac56dcbdf5ce9ae0a9b
Author: Sajjad Moradi <moradi.sajjad@gmail.com>
Date: Tue Jul 25 14:48:50 2023 -0700
Fix the issue of consuming segment entering ERROR state due to stream connection errors (#11166)
commit 9d55a460470d76937b55ac8c4b814a6873e5239f
Author: Shounak kulkarni <shounakmk219@gmail.com>
Date: Wed Jul 26 03:13:27 2023 +0530
Instance retag validation check api (#11077)
* API to validate safety of retag operations
* code cleanup and proper error handling
* checkstyle fixes
* handle realtime and offline server tags separately
* get all table configs at once
* logic refactor
* fixes
* test cases
* broker tenant min requirement behaviour fix
* Added docs and comments to improve readability
commit b34fad125139fcfbde5daa1f857227bb87254a64
Author: Rong Rong <rongr@apache.org>
Date: Tue Jul 25 13:20:53 2023 -0700
[multistage]fix count star handling in v1 planner (#11171)
commit d5e6a56a6a48e58c99619ad367453ce6101022ee
Author: Prachi Khobragade <pkhobragade@linkedin.com>
Date: Tue Jul 25 12:03:07 2023 -0700
deprecated(since) is not supported in jdk 8 (#11172)
commit 7231ecfee875ac80ea61b329887ab16ad65fb94f
Author: Saurabh Dubey <saurabhd336@gmail.com>
Date: Tue Jul 25 10:59:45 2023 +0530
Increment nextDocId even if geo indexing fails (#11158)
commit a7530255d4e1dfa40eb591acfbf174bc61f0e056
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Mon Jul 24 21:44:06 2023 -0700
Support sum precision (#11162)
commit abc68e04f5379c4ae129d257acdcd597998a80d9
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 21:06:59 2023 -0700
added @Encoded and URIUtils.decode
commit 8e074b9a2a321198e13dc1f5e214e7edf386dbb7
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 20:32:38 2023 -0700
removed commented and unused code
commit 6fa8cab1e0ad9b3ce1ffc2d3ac757e03c29b51f1
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Mon Jul 24 18:30:24 2023 -0700
Simplify compatible test to avoid test against itself (#11163)
commit 1d61fc28451d641aa6f0d3295f9342de446ec632
Author: Prachi Khobragade <pkhobragade@linkedin.com>
Date: Mon Jul 24 18:19:04 2023 -0700
Make interface methods non private, for java 8 compatibility (#11164)
commit 06d8540378d9df6220c0e0e47d42841b3376bd96
Author: summerhasama-stripe <134637018+summerhasama-stripe@users.noreply.github.com>
Date: Mon Jul 24 21:17:03 2023 -0400
For table rebalance, check if instances are equal for NO_OP (#11073)
commit ff6d527fbfe71cfa1afff601627b796ea59b168a
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Mon Jul 24 17:13:45 2023 -0700
[multistage] Register theta sketch aggregation functions in v2 query engine (#11143)
* Support distinctCountThetaSketch in v2
* VARIADIC variance of the ThetaSketch functions are not supported
commit 4135e072201a5c15dbe972fb12bae8ce3b349052
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Mon Jul 24 19:31:39 2023 -0400
[multistage] support physical plan in Explain queries (#11052)
* separated physical and logical plan explain.
* added SqlPhysicalExplain node and syntax support changes.
* Renamed PhysicalExplainPlanVisitor class and other minor PR comments fix.
commit ea493a253297a3e086f264787fb849e99ceb0bef
Author: Navina Ramesh <navina@apache.org>
Date: Mon Jul 24 13:10:31 2023 -0700
Extract headers for Kinesis Connector (#9713)
commit fb74572de5d2853dcf8b8f1c215b5fbf2bead87f
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 12:55:35 2023 -0700
fixed bad merge
commit ff2f8a901ec6cbdec1a2c736d00c6defc7abbb7b
Merge: 6a0a077f37 41b3c70906
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 12:51:25 2023 -0700
Merge branch 'master' into compaction
commit 41b3c70906b9d1f4550c3d3dcf9c3a8d99c65cc4
Author: Rong Rong <rongr@apache.org>
Date: Mon Jul 24 12:01:12 2023 -0700
[multistage][hotfix] should never pull project up below sort (#11161)
* [multistage][hotfix] should never pull project up below sort
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit 35fb1d0886b7757adfc6fc4d0490594d21609a3b
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Mon Jul 24 10:48:25 2023 -0700
Fix compatibility test (#11156)
commit 6a0a077f37b35672268d47eacf4d9ef1c64954bd
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 10:37:24 2023 -0700
simplified logic in getValidDocIdMetadataUrls
commit f72188430ca461d3d44e4e031765ae6eda054712
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 10:30:04 2023 -0700
restored formatting to have each ApiResponse on it's own line
commit 58e9d3405c494f980c1598d203b5bd721892c9e6
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 10:22:10 2023 -0700
getting totalDocs from validDocIdMetadata
commit 65c236c5029925471735f7e0287b7e5c23b0cd71
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 10:11:24 2023 -0700
removed getCompactionConfigs from the generator
commit b0fba83853a0bd1a0ebf4e365ee187dd995a8588
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 08:36:20 2023 -0700
removed getServerToSegmentsMap from ClusterInfoAccessor
commit d205295fad72f873ce20891dde15100111fc908a
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 24 08:24:31 2023 -0700
removed explicit task name from logs
commit 5eafe76ee89adfd12007db17de7a50c5dd7a43ac
Author: deemoliu <qiaochu@uber.com>
Date: Sun Jul 23 21:34:21 2023 -0700
[Upsert TTL] Add Watermark for each partitions for Primary key cleanup (#10915)
commit 635c2fb96e6901d0a0660e1a8878d9d493852791
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Sun Jul 23 21:32:44 2023 -0700
Fix NPE when nested query doesn't have gapfill (#11155)
commit cb0c86c17885df4c5fc42df521ecab2aabffbf65
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Sun Jul 23 21:32:29 2023 -0700
Fix the NPE when query response error stream is null (#11154)
commit c5c59972600de80d12e2bbfb6bf7b1bcf3d1e977
Author: Robert Zych <robert.zych@gmail.com>
Date: Sun Jul 23 15:07:22 2023 -0700
updated getServer to only return an online server
commit 4dd2bb5bb4df97c529dba2c259e23a12837077dd
Author: Saurabh Dubey <saurabhd336@gmail.com>
Date: Sun Jul 23 20:56:06 2023 +0530
Sanitise API inputs used as file path variables (#11132)
* Fix path traversal vulnerabilities
---------
Co-authored-by: Saurabh Dubey <saurabh.dubey@Saurabhs-MacBook-Pro.local>
commit fe2b013a657e1ad6ac508a9a37933961bc4c408b
Author: Jeff Bolle <JeffBolle@users.noreply.github.com>
Date: Fri Jul 21 20:44:33 2023 -0400
Extract record keys, headers and metadata from Pulsar sources (#10995)
commit fc5adb790cb404e00a2acd4c8d29f74058e22c6f
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jul 21 14:05:19 2023 -0700
Bump org.apache.datasketches:datasketches-java from 4.0.0 to 4.1.0 (#11148)
commit 6a10bd8acaf59c7d62af40873ad022c4858b69df
Author: soumitra-st <127247229+soumitra-st@users.noreply.github.com>
Date: Fri Jul 21 14:04:14 2023 -0700
Returning 403 status code in case of authorization failures (#11136)
commit 19558fecec4c5b6aa3857bafda161ac2332558a1
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jul 21 12:58:24 2023 -0700
removed support for segments with future end times and created new data set with only past dates
commit 1f1ffcf1a98cb38ef3e323063472b0a9d64fd19a
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jul 21 07:24:59 2023 -0700
renamed minRecordCount to invalidRecordsThresholdCount
commit 12ddd3310b65c2e3e871a8fba0c8ccc503afd980
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jul 21 01:39:22 2023 -0700
Fix a bug when use range index to solve EQ predicate (#11146)
commit 18dbadafeb0538f951d87983053a2fa0c8b30adb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jul 21 01:10:05 2023 -0700
Bump azure-storage-file-datalake from 12.14.1 to 12.16.0 (#11134)
commit d6ad12e375abeaa227193790d93f48ddec3d9211
Author: Robert Zych <robert.zych@gmail.com>
Date: Thu Jul 20 15:59:59 2023 -0700
logging after deleting segments
commit e0e2e20e590744df966824d33e7396361a2c400b
Author: Robert Zych <robert.zych@gmail.com>
Date: Thu Jul 20 15:52:28 2023 -0700
auto-formatted all changes files
commit e3f726233673f12074dde128c5991d8d9795f9b7
Author: Robert Zych <robert.zych@gmail.com>
Date: Thu Jul 20 15:31:45 2023 -0700
reusing the shared executor and connection manager
commit 723b764bc91275c0b8361d3f9135f151b6404c39
Author: Andi Miller <andi@andimiller.net>
Date: Thu Jul 20 23:05:30 2023 +0100
Allow extra aggregation types in RealtimeToOfflineSegmentsTask (#10982)
commit af35965c38ae3d15648039dddc87bbd409d8d7b0
Author: Robert Zych <robert.zych@gmail.com>
Date: Thu Jul 20 13:14:32 2023 -0700
fixed getValidDocIdMetadataUrls to handle replicated segments
commit 7e782ddd8be23e4968fc327206679f2304b42ed5
Author: Saurabh Dubey <saurabhd336@gmail.com>
Date: Fri Jul 21 01:36:12 2023 +0530
Avoid string splitting wherever possible (#11141)
Co-authored-by: Saurabh Dubey <saurabh.dubey@Saurabhs-MacBook-Pro.local>
commit ead75229af648e260c68ad991fb80a6f81af5712
Author: Shen Yu <shen@startree.ai>
Date: Thu Jul 20 10:48:29 2023 -0700
Separate BitmapBasedFilterOperator into two filter operators. (#11137)
commit 961695798ba143101b6b4bd90b0cc8f61593d35c
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Wed Jul 19 18:04:14 2023 -0700
Allow each test to specify the query engine (#11138)
commit 9cb876422c8c9a329aa5c0fdbbe0f0a8ef8711dd
Author: Vivek Iyer Vaidyanathan <vvivekiyer@gmail.com>
Date: Wed Jul 19 11:48:17 2023 -0700
Fix reload bug (#11131)
commit cfc3abcf71decbaa2f4bc25593ed2d5f1c3c187b
Author: Rong Rong <rongr@apache.org>
Date: Wed Jul 19 09:34:18 2023 -0700
[multistage][agg] support agg with literal arguments (#11105)
This is approach to work around the literal argument limitation of AggregateCall from calcite.
Approach Details
===
1. create a pre-rule that expands the LogicalAgg -- LogicalProject(with literal) plan and hoist the literals up to the LogicalAgg RelHints
- noted that at this stage we are not rewriting the project at all. only extracting the literals
- it also extract total number of operands in case some of the agg functions requires more than 1 for placeholder purposes, it doesn't actually use them for intermediate/final stage.
2. continue normal optimization
- during this time, the LogicalProject can be pivoted away from the LogicalAgg, or other things could happen to completely change the plan but LogicalAgg processed during previous step will still possess the hint with literal
3. apply the agg-exchange rule to split the hint into several splits (LEAF, INTERMEDIATE, FINAL, DIRECT)
- the literal and call signature info are still carried with the hints
4. during PhysicalPlanVisitor, it will
- attach the literals using hints to the function context
- attach placeholder identifiers where necessary
Functions added in this round
===
- FIRSTWITHTIME/LASTWITHTIME
- DISTINCTCOUNT/DISTINCTCOUNTHLL
- remaining DISTINCTCOUNTHLL variances are not added. can be done separately
- PERCENTILE
- remaining PERCENTILE variances are not added. can be done separately
- stats functions (COVARPOP, COVARSAMP, VARPOP, VARSAMP, STDDEVPOP, STDDEVSAMP)
- noted that the standard calcite notation is not yet supported, b/c putting all these functions together will result in calcite optimizing them and generates REGR_COUNT and some other non-supported AGG function
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit c39342ade872f81de830fdee4c588699db28f701
Author: Robert Zych <robert.zych@gmail.com>
Date: Wed Jul 19 08:54:08 2023 -0700
improved readability, and cleaned up getCompactionConfigs
commit 9824f603f353e86fed8b9de18e3a094ddd9dc5dd
Author: Rong Rong <rongr@apache.org>
Date: Wed Jul 19 08:26:45 2023 -0700
[multistage] add hint option for table node (#11114)
Co-authored-by: Rong Rong <rongr@startree.ai>
commit 218ecddbafe6348873f6e50d137f8c2f0db3ade3
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Jul 18 23:20:52 2023 -0700
Bump bouncycastle.version from 1.74 to 1.75 (#11125)
commit ad26013ae7f894bfeaa1ea21119ea88766df4142
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Wed Jul 19 08:18:53 2023 +0200
V2 allocation optimizations (#11112)
commit df72d1dfb1663739017212b3ea1ddc73e8455190
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Tue Jul 18 22:39:56 2023 -0700
Adding v2 map function supports (#11104)
commit 8fff7eb70597eb74c05ee788378eb8fa1ed8efaf
Author: Shen Yu <shen@startree.ai>
Date: Tue Jul 18 18:57:13 2023 -0700
Fix a flaky test OfflineClusterMemBasedBrokerQueryKillingTest.testDigestOOMMultipleQuerie. (#11121)
commit 43bcd1ea964362e152609268aa7918aaa0f39279
Author: Kartik Khare <kharekartik@gmail.com>
Date: Wed Jul 19 04:09:05 2023 +0530
Bug fix: TableUpsertMetadataManager is null (#11129)
commit 95eef85a31ca85e1134dc381f8b2e71c302f34ab
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Tue Jul 18 10:46:54 2023 -0700
Register dateTimeConverter,timeConvert,dateTrunc to v2 functions (#11097)
commit abc749774485405a75c1ec06dc6b75a1b89e68ce
Author: Erich <134291879+ege-st@users.noreply.github.com>
Date: Mon Jul 17 16:13:38 2023 -0400
revert jackson update (#11126)
commit ffedfdfcb988506b5aec004f1c01606019c0ba1e
Author: Davi S. Zucon <davizucon@users.noreply.github.com>
Date: Mon Jul 17 02:21:43 2023 -0300
Added serviceAccount to minion-stateless (#11095)
commit 7422edfc1c65f0acd7cacd8227c7110cc05412c1
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Sun Jul 16 20:59:37 2023 -0700
Bump jts-core from 1.16.1 to 1.19.0 (#11089)
Bumps jts-core from 1.16.1 to 1.19.0.
---
updated-dependencies:
- dependency-name: org.locationtech.jts:jts-core
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
commit c32318360e46a4030bfbff9d11f57b7fa0564397
Author: soumitra-st <127247229+soumitra-st@users.noreply.github.com>
Date: Sat Jul 15 15:27:06 2023 -0700
Upgrading com.google.guava:guava from 30.1.1 to 32.0.1 to stay up to date and leverage the most recent improvements (#11115)
commit abf395996dfd6d46a0f0a0115478e847838e0a6d
Author: Xuanyi Li <xuanyili@uber.com>
Date: Sat Jul 15 01:44:34 2023 -0700
quick fix to change uri (#11102)
commit 2c3ea27f3b69c39efffc49de074847188234c6a1
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Sat Jul 15 14:13:50 2023 +0530
Improve error message in case of throwables in multistage (#11103)
* Improve error message in QueryDispatcher
* address comments
* update in broker request handler
* Update pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/MultiStageBrokerRequestHandler.java
Co-authored-by: Rong Rong <rongr@apache.org>
---------
Co-authored-by: Xiang Fu <xiangfu.1024@gmail.com>
Co-authored-by: Rong Rong <rongr@apache.org>
commit 85bfeb4e17383d3552fc4beb623a3c3f9daec1e3
Author: Jiapeng Tao <jiatao@linkedin.com>
Date: Fri Jul 14 17:42:04 2023 -0700
Allow setting the default MergeRollupTask segment group manager (#11101)
commit 3d60d556c82a6b3282e740af15b30eb67267b36f
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jul 14 14:37:54 2023 -0700
Use PUT request to enable/disable table/instance (#11109)
commit ebeb0d84950bf1ab5b046ccd541a0f872b550690
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jul 14 11:01:45 2023 -0700
Bump lombok from 1.18.26 to 1.18.28 (#11090)
commit 135fa4322e77c43fe0096ea3006aade4407760c9
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jul 14 10:50:50 2023 -0700
Allow offloading multiple segments from same table in parallel (#11107)
commit 365608fbb94a0f5bdeef682a42181524ef2460b3
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Fri Jul 14 19:13:08 2023 +0200
Delete Dockerfile_mio (#11111)
commit 35d7a8faa5e969602a637a68f158c51fdc594c38
Author: Shen Yu <shen@startree.ai>
Date: Thu Jul 13 22:09:39 2023 -0700
Fix a flaky test LLCRealtimeClusterIntegrationTest.testAddRemoveDicionaryAndInvertedIndex. (#11096)
commit 6ef119005ea322071386e5451c64e2c7cf076d6d
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Thu Jul 13 23:03:34 2023 +0200
Update aws 2.20.94 (#11088)
commit 967e83b32b1180a10af650cf7cfbe5fc64e1d556
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Thu Jul 13 23:02:32 2023 +0200
Use aws bom (#11070)
commit 1622a9e7d689537881826549a9323237ca7d7e44
Author: Xuanyi Li <xuanyili@uber.com>
Date: Thu Jul 13 09:29:34 2023 -0700
Add a new controller endpoint for segment deletion with a time window (#10758)
* have an api prototyping for segment deleteion with timewindow
* fix the code style
* compose table name with type
* write unit test
* test more cases
* rephase the param explaination; force nonempty timestamp
* parameters format change
* cosmetic change on apiOperation
commit b39329966f8aff7bf3d7e7b05fad5f78e07ebc6b
Author: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Date: Wed Jul 12 19:34:16 2023 -0400
Allow using Lucene text indexes on mutable MV columns. (#11093)
commit 4e1c8fc94f405c4a2e06ac8db22f778d0a885598
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Wed Jul 12 15:10:58 2023 -0700
Adding alias for Transform function type registry (#11094)
commit d8c4d4d68e21afd3f3acdf5a6d58c52a8bfed0a8
Author: Robert Zych <robert.zych@gmail.com>
Date: Wed Jul 12 15:10:38 2023 -0700
allowing user to choose between invalidRecordsThresholdPercent or minRecordCount or both
commit 20bef67a561f4d78b72a12970547b5ca5306b8df
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Wed Jul 12 14:56:36 2023 -0400
Upgrade swagger-ui from 3.23.11 to 5.1.0 (#11091)
commit c302354dd7668e96b827b9d8e11861bd00ccaaa3
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Wed Jul 12 11:26:21 2023 -0700
Minor fix on AddTableCommand.toString() (#11082)
commit 02715141f5d344490131628fc4b37554d1648aee
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Wed Jul 12 14:20:13 2023 -0400
Upgraded commons-cli and commons-lang3 package version. (#11083)
commit 4f45907931c27713623d916ec89553530bee6fe0
Author: Erich <134291879+ege-st@users.noreply.github.com>
Date: Wed Jul 12 14:19:38 2023 -0400
Update jackson libraries to 2.13.5 (#11081)
commit 5b157dfcf572c9c94264de98f8b1fabd5dccd347
Author: Shen Yu <shen@startree.ai>
Date: Wed Jul 12 10:37:40 2023 -0700
Delete an obsolete TODO. (#11080)
commit 3f15e10ffd0bd65636c0d2653ff6d707bc6f0894
Author: Shen Yu <shen@startree.ai>
Date: Wed Jul 12 10:33:32 2023 -0700
Consistently use isNullRow in transform function tests. (#11079)
commit 8058be12f8218a69a925b7a7f2b385122cd1a740
Author: Robert Zych <robert.zych@gmail.com>
Date: Wed Jul 12 09:16:49 2023 -0700
updated testCompaction to use waitForAllDocsLoaded after the task runs
commit 0350c9b5bded49591231719c10fe074f74617cbf
Author: Saurabh Dubey <saurabhd336@gmail.com>
Date: Wed Jul 12 21:25:27 2023 +0530
Set _disableAcl appropriately when initializing directly with client (#11087)
Co-authored-by: Saurabh Dubey <saurabh.dubey@Saurabhs-MacBook-Pro.local>
commit 35750b462e67e62e8a1b46cc5ecf8545735e2672
Author: Robert Zych <robert.zych@gmail.com>
Date: Wed Jul 12 08:06:09 2023 -0700
updated testCompactionDeletesSegments to use waitForAlDocsLoaded after the task runs
commit 7f855a522026db423c6aa722ffa0907af8c8ee9b
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Wed Jul 12 06:56:31 2023 -0700
Fix grpc regression on multi-stage engine (#11086)
* Revert "upgrade protobuf-lite from 1.19.0 to 1.21.0 (#11074)"
This reverts commit c9699594c1eaf5aed70756ebc166d1b9462d7b8f.
* Adding multi-stage quickstart check
commit 437504331eba37f315cdd88c570c2cf0ccf234bc
Author: Sanskar Modi <sanskarmodi97@gmail.com>
Date: Wed Jul 12 05:24:51 2023 +0530
Support for new dataTime format in `DateTimeGranularitySpec` without explicitly setting size (#11057)
commit 1cdcf70dce5a7e145c14fd617449a431863b8fae
Author: npuppala111 <44955679+npuppala111@users.noreply.github.com>
Date: Tue Jul 11 16:05:23 2023 -0700
New API to get tenant tags (#10937)
* Added InstanceResource.java
* Added intial Executor service to InstanceResource
* Added JavaDoc
* [partial cherry-pick] Added InstanceResource for Server/Broker
* Fixed BaseResourceTest.java
* Fixed BaseResourceTest.java
* Deleted unused variable
* Minor changes to InstanceResource
* Updated InstanceResource
* Updated InstanceResource
* Fized checkstyle error in InstanceResource
* Fixed JavaDoc typo in InstanceResource
commit becb362aff8ae8c1741bb7105ac1a6c02c1b7c73
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jul 11 15:50:42 2023 -0700
fixed testValidDocIdMetadata
commit c9699594c1eaf5aed70756ebc166d1b9462d7b8f
Author: Rong Rong <rongr@apache.org>
Date: Tue Jul 11 11:47:00 2023 -0700
upgrade protobuf-lite from 1.19.0 to 1.21.0 (#11074)
Co-authored-by: Rong Rong <rongr@startree.ai>
commit f26af9018c4736eadbaf6fa2de830455e9f0b0ca
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Tue Jul 11 11:39:38 2023 -0700
[multistage] Register Pinot Transform Functions into Calcite (#11048)
* Introduce PinotSqlTransformFunction for transform function signature registration
* Add TransformFunctionType registry to calcite
- add jsonExtractScalar as return type literal inference example
- add jsonExtractScalar with optional default value example
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit 7c33dc4d9247b090ae141d0db5a2be0d881bfb3d
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Tue Jul 11 11:15:02 2023 -0700
Bump license-maven-plugin from 4.0 to 4.2 (#11071)
Bumps license-maven-plugin from 4.0 to 4.2.
---
updated-dependencies:
- dependency-name: com.mycila:license-maven-plugin
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
commit 1651d989bbc955e44796b88e5b33c4c9f6b79478
Author: Shen Yu <shen@startree.ai>
Date: Tue Jul 11 10:24:35 2023 -0700
Upgrade spark version. (#11075)
commit 9bb3cbb3d6ebf5a9fcd8a5410a5794aee02c7ee4
Author: Shen Yu <shen@startree.ai>
Date: Tue Jul 11 01:09:54 2023 -0700
Rework transform functions NULL support. (#11008)
commit 4bb7e6a7b0bf9ee502b814819c7194d4974addc1
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Tue Jul 11 00:52:51 2023 -0700
allow to preload segments with upsert snapshots to speedup table loading (#11020)
commit 19dce0e282ccd40f3da0fa2c8168efbd812f60fd
Author: Rong Rong <rongr@apache.org>
Date: Mon Jul 10 19:49:00 2023 -0700
[multistage] Fix aggregate type issues (#11068)
1. cleaned up on type issues for agg
- pinot sql agg functions are no longer pre-registered with calcite which caused type during SqlValidator
- dynamic pinot sql agg function creating is being used to avoid type inference not applying properly (such as inferring from ARG0 cannot be applied statically b/c then ARG0 refers to the input to the intermediate type, not the original DIRECT type)
2. revisit intermediate type inference when registering built-in agg (such as MIN/MAX) instead of doing raw casting from number/boolean to double/int when extracting value from input blocks
- follow up by registering explicit intermediate result type
- also follow up when `ReturnType.explicit(*)` is not possible in final result type, by working around from available validation type from RelBuilder rule call object.
3. fixed boolean handling (except bool_and/bool_or which is handled separately by #11033)
4. re-adjusted direct project rule, by using calcite AGG_PROJECT_EXPAND template
- follow up needed to ensure the direct aggregate, (see TODO#1, 2)
5. re-adjusted agg-exchange-rule
- no longer generates multiple agg now we only generate DIRECT/LEAF/FINAL (INTERMEDIATE not supported yet)
- un-generalized the rule NOT to reuse convert-agg-node and build-agg-call. They don't share logic, thus, the generic causes more problems with various nullable fields.
- removed unnecessary logic for discovering field-type/field-name/field-used, which already exists in calcite RelOptUtils
6. modified test plan generated, removed unnecessary additional agg
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit b56c7283e5907d3a185238e31b473fb9abe5b84c
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Mon Jul 10 16:24:24 2023 -0700
[cleanup] Consolidate some query and controller/broker methods in integration tests (#11064)
* Don't use static method for post queries in integration tests
* some clean up with controller/broker get member variables
* Simplify multiStage integration tests
commit e591849ce081c90ee7e9c3eb8b308d7fed8ad059
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 10 16:01:34 2023 -0700
removed record count check from getCompletedSegments() to allow for small segment deletion and potential compaction
commit 71144989066883fd5a774bd722282fce726dc50e
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jul 10 15:12:47 2023 -0700
reading invalidRecordCount as a long and comparing it to segment.fetTotalDocs() for segment deletion
commit 8171cfb000e68c40f08bd69a9c1f84e99b7c38c4
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Mon Jul 10 16:53:53 2023 -0400
Fix BOOL_AND and BOOL_OR result type (#11033)
commit e3cac6428deb891ab66ae9dc5704442b49dc4c0e
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Sun Jul 9 10:39:47 2023 -0700
pull up awssdk version to top pom.xml (#10906)
commit 10a18d55b9d27851fa2bba1f77eda3e9d600aa37
Author: Johan Adami <4760722+jadami10@users.noreply.github.com>
Date: Sun Jul 9 05:32:18 2023 -0400
retry KafkaConsumer creation in KafkaPartitionLevelConnectionHandler.java (#253) (#11040)
Co-authored-by: Johan Adami <jadami@stripe.com>
commit 12a090d90d3f339b7cec674e208dcda3b6b0bc8f
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Sun Jul 9 01:32:05 2023 -0700
Make RequestUtils always return a string array when getTableNames (#11069)
commit abfdc6161512b25c9d9035eafa4ae4b9f96b28ee
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Sat Jul 8 15:51:28 2023 -0700
Remove presto-pinot-driver and pinot-java-client-jdk8 module (#11051)
commit ed5dea97d6d277cd6ccfd16d5cd87c7cf970a5f4
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Sat Jul 8 06:58:38 2023 +0200
Replace Long attributes with primitive values to reduce boxing (#11059)
* Replace Long attributes with primitive values to reduce boxing
* Update PinotBufferIndexInput.java
---------
Co-authored-by: Xiang Fu <xiangfu.1024@gmail.com>
commit 457c2b776f6d66b0269aab34ad2ce6dd49de8e37
Author: lucifer4j <97942130+lucifer4j@users.noreply.github.com>
Date: Sat Jul 8 10:27:39 2023 +0530
Fix double-checked locking in ConnectionFactory (#11014)
The `_defaultTransport` in `ConnectionFactory` is created using
double-checked locking.
https://github.com/apache/pinot/blob/03b9d4a708e6d09838e902857ebe1f255ced4ba1/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ConnectionFactory.java#L211-L222
However, the current implementation is buggy as described in
https://rules.sonarsource.com/java/RSPEC-2168/. To fix it, make the field
`volatile`.
commit 1b81ebafc878a0a0d8cff1ecf9d828963a44449f
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Fri Jul 7 22:19:03 2023 -0400
Updated calcite version to 1.30.0 (#11067)
commit 729e86ad138a81a72e07892df2d427d8458ee2c8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jul 7 22:18:08 2023 -0400
Bump h2 from 2.1.214 to 2.2.220 (#11066)
commit 4fa9c5335d9e4798c3ad28e1b1fde7e02b01ee0d
Author: Seunghyun Lee <seunghyun@startree.ai>
Date: Fri Jul 7 16:30:44 2023 -0700
Upgrade netty from 4.1.79.Final to 4.1.94.Final (#11055)
commit 2e885e44fb32de3eaa61fc23621a810d9d5a69e8
Author: Seunghyun Lee <seunghyun@startree.ai>
Date: Fri Jul 7 16:30:37 2023 -0700
Upgrade swagger from 1.6.6 to 1.6.9 (#11054)
commit 12fb274ef79851156adaaf6c1d9c7e8adf08d625
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jul 7 16:29:59 2023 -0700
Bump bcprov-jdk15to18 (#11042)
Bumps [bcprov-jdk15to18](https://github.com/bcgit/bc-java) from 1.68 to 1.74.
- [Changelog](https://github.com/bcgit/bc-java/blob/main/docs/releasenotes.html)
- [Commits](https://github.com/bcgit/bc-java/commits)
---
updated-dependencies:
- dependency-name: org.bouncycastle:bcprov-jdk15to18
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
commit 3bc8cdf7120add44a4ce4ac89288bc7615f8b206
Author: Erich <134291879+ege-st@users.noreply.github.com>
Date: Fri Jul 7 19:29:33 2023 -0400
Update the version of jcabi log (#11047)
Co-authored-by: Erich Ess <erich.g.ess@gmail.com>
commit 022e430c212621ec8786acb8b2f360b95a66e74e
Author: Erich <134291879+ege-st@users.noreply.github.com>
Date: Fri Jul 7 19:29:04 2023 -0400
Update Jackson Databind to newer version (#11058)
* Update Jackson Databind to newer version
* Empty commit
commit ea651ffe30e49f243753b942c4aa5d821142b9be
Author: Haitao Zhang <haitao@startree.ai>
Date: Fri Jul 7 16:05:00 2023 -0700
update amazon sdk to 2.20.87 (#11056)
commit 96407ecc5fe0773abbe66c7011110468e8b0b286
Author: Ankit Sultana <ankitsultana@uber.com>
Date: Fri Jul 7 15:46:22 2023 -0700
[multistage] Add TPC-H Test for Multistage Engine (#10918)
* [multistage] Add TPC-H Test for Multistage Engine
* Remove unnecessary parquet code
* Fix test stability
* Add queries with default parameters
* Add separate query files
* Fix date/time syntax
* fix rat + fix schema
* Fix extract syntax
* Fix tests and remove h2 comparison
* Fix checkstyle
* Add symlink + revert base class changes
commit 56a78829d9a48205c116b38169f91504eef1d474
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Fri Jul 7 15:10:53 2023 -0700
upgrade parquet to more recent version (#11062)
commit 68ab6bcd8da321960cbad4715ae80dd33ffd1d60
Author: Ragesh Rajagopalan <ragesh.rajagopalan@gmail.com>
Date: Fri Jul 7 13:09:58 2023 -0700
Upgraded httpclient library version from 4.5.1 to 4.5.13 in module pinot-druid-benchmark (#11063)
commit 9b21eb94c06f880502b34b816050983809989a9f
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Fri Jul 7 09:57:16 2023 -0700
format double 0 properly to compare with h2 results (#11049)
commit 393f80bbe2e0b80f56e25a6f2c6e667b4c606df9
Author: swaminathanmanish <126024920+swaminathanmanish@users.noreply.github.com>
Date: Fri Jul 7 08:23:20 2023 -0700
Upgrading spark versions (#11037)
commit 31b01fe699abe8926b0c2aa3d816293e9055eaad
Author: Navina Ramesh <navina@apache.org>
Date: Thu Jul 6 22:27:31 2023 -0700
Update maven-core to 3.8.2 (#11043)
commit 6d99eec9977677cca1493e6ac0ecfd7d20502cdf
Author: Rong Rong <rongr@apache.org>
Date: Thu Jul 6 14:10:28 2023 -0700
[multistage] clean up aggregate v1 function for v2 planner (#11036)
* clean up internal agg hint usage and consolidate into hint options
* remove unnecessary boolean flags and replace them with enum: AggType
* refactor AggType into AggregateNode
* address comments
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit f09a91ca8171681921cceca614695e89a069b827
Author: Haitao Zhang <haitao@startree.ai>
Date: Thu Jul 6 13:16:25 2023 -0700
update freemaker to fix Server-side Template Injection (#11019)
commit 3648f182feaa36ad7d63da8388061d0ca6acb4d6
Author: Evan Galpin <egalpin@users.noreply.github.com>
Date: Thu Jul 6 13:11:40 2023 -0700
Reverts strictly skipping removeNull from comparison column (#11044)
commit c29782f895b4bee045ecc0bc814aec534293c3d5
Author: Kevin Xu <38196564+INNOCENT-BOY@users.noreply.github.com>
Date: Fri Jul 7 02:54:37 2023 +0800
[enhance]: When enable case-insensitive, don't allow to add newly column name which have the same lowercase name with existed columns. (#10991)
commit 878f61357b6868d98a3eb8c84ec1dfb414c44b6f
Author: Shounak kulkarni <shounakmk219@gmail.com>
Date: Thu Jul 6 11:51:21 2023 +0530
Instance drop operation validation check API (#10969)
* extract instance drop validity check code to a separate method
* Add javadoc
* API endpoint to expose the safety validation on instance drop operation
* lint fix
* add license header to new class
* update the endpoint path and method
* pass error list as code, message tuple
commit f3f1c3ecbe79ff23909049d09687b32b8385c44f
Author: Sanskar Modi <sanskarmodi97@gmail.com>
Date: Thu Jul 6 11:40:44 2023 +0530
Improve `DateTimeGranularitySpec` to follow the convention of new dateTimeField format (#11030)
commit a0ed20fe864a20d6d77fc005b355dc6b914b28fa
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Thu Jul 6 10:21:49 2023 +0530
Update getTenantInstances call for controller and separate POST operations on it (#10993)
commit 2fabbe40057d99afa497071f7083b4f16e40137a
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Wed Jul 5 17:23:00 2023 +0200
Apply some allocation optimizations on GrpcSendingMailbox (#11015)
* Apply some allocation optimizations on GrpcSendingMailbox
* refactor a couple of byte[] initialization
commit 6c8eb96dd8b818a6d5179a42fc669b0019950fb1
Author: Sonam Mandal <somandal@linkedin.com>
Date: Wed Jul 5 08:20:45 2023 -0700
[multistage] Initial planner changes to support the v1 aggregation functions in multistage (#10846)
* [multistage] Initial planner changes to support the v1 aggregation functions in multistage
* [multistage] Model extractFinalResult as an additional LogicalAggregate node instead of scalar function
* Address review comments
* fix merge tests
* fix pinot-segment-spi in presto test
---------
Co-authored-by: Rong Rong <rongr@apache.org>
Co-authored-by: Xiang Fu <xiangfu.1024@gmail.com>
commit f7a076496ae6e07a42207cb2c978dc74562cf0cf
Author: Shounak kulkarni <shounakmk219@gmail.com>
Date: Tue Jul 4 17:37:34 2023 +0530
Startree index build enhancement (#10905)
* StarTreeIndexSeparator to split the startrees and reuse them
* Test suite for the StarTreeIndexSeparator
* cleanup the temp dir
* Add license header
* fixed metadata.properties
* license header fix
* handle temp directory cleanup in case of exception
* rename shouldRemoveExistingStarTrees to shouldModifyExistingStarTrees
* pass index dir for deletion instead of segment dir
* Fix conditions so that star-trees are simply deleted if newer config doesn't have star-trees
* rename _indexDirectory to _indexDir
* return null for getExistingStarTrees when no existing tree is present
* minor fixes
* Moved validations to handleExistingStarTreeAddition
handleExistingStarTreeAddition now returns a boolean based on whether the existing star-tree was reused or not.
* checkstyle fix
* change logic to only separate star-trees as required
* style fix
* bug fix
* minor fixes
* handle startree cleanup locally to avoid copying index files
* bug fix
* revert the separator creation
commit ffbc5b0e52e2c8ba507f088ec1fd39996f22e8c1
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Mon Jul 3 19:19:07 2023 -0400
Ensure min/max value generation in the segment metadata. (#10891)
commit eefa8fbfe86bc5b1412f9b9085364fb8a773476f
Author: Jiapeng Tao <jiatao@linkedin.com>
Date: Mon Jul 3 16:03:58 2023 -0700
Allow custom segment grouping in MergeRollupTask based on lineage metadata (#10964)
commit 0821e3b967f442ea6e9cb3d9b5b11f95d151020d
Author: Haitao Zhang <haitao@startree.ai>
Date: Sun Jul 2 00:37:47 2023 -0700
update jersey to fix Denial of Service (DoS) (#11021)
commit a6ee2773a28b2339ac1a4fc80fba978ae69dbbde
Author: Vivek Iyer Vaidyanathan <vvivekiyer@gmail.com>
Date: Sat Jul 1 09:32:31 2023 -0700
[Multistage] Runtime changes for leveraging V1 Aggregation Functions (#10845)
* [Multistage] Runtime changes for supporting Aggregations
* Special handling for bool_and bool_or
commit 6f2bcde2e1fcee73f81cab24f4054e58d41d6b94
Author: Shen Yu <shen@startree.ai>
Date: Sat Jul 1 00:50:06 2023 -0700
Fix a flaky test OfflineGRPCServerIntegrationTest.testQueryingGrpcServer. (#11022)
commit b9cec629b74fb42bc49f196797d55a4468d30567
Author: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Date: Fri Jun 30 22:12:46 2023 -0400
Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns. (#11006)
* Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns.
* Remove unnecessary static imports
* Address code review comments.
commit 9821cee0115be999dea6d18389ad7411f8ad7c47
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Fri Jun 30 19:31:19 2023 -0400
Updated the code for showing the sql parsing exception. (#11009)
commit e1c4e2fdd165694927b06abcfeab425ea693279f
Author: Christopher Peck <27231838+itschrispeck@users.noreply.github.com>
Date: Fri Jun 30 16:05:41 2023 -0700
add support for mv cols with realtime lucene index (#11000)
commit 25173a3f2855d96abdbb2a239ec5691c15e8fb5c
Author: lucifer4j <97942130+lucifer4j@users.noreply.github.com>
Date: Sat Jul 1 04:32:46 2023 +0530
Add fromController method that accepts a PinotClientTransport (#11013)
commit fac02cf4df857fbf5f3d09ab2c3eedba0b265ce4
Author: Rekha Seethamraju <rekha.kmit@gmail.com>
Date: Fri Jun 30 15:41:19 2023 -0700
When extracting root-level field name for complex type handling, use the whole delimiter (#11005)
commit 49efcfa5875b7eb1ef2649df3a5e661860cb1fee
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jun 30 15:10:24 2023 -0700
Upgrade circe-parser from 0.13.0 to 0.14.2 (#11012)
commit 2e346d4a6a5bba818dc3261c5ec109146fe36b7d
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jun 30 15:10:06 2023 -0700
Upgrade commons-codec:commons-codec@1.11 to commons-codec:commons-codec@1.15 in pinot-pulsar (#11011)
commit bde2d01153fc996dc46d9a61c6a9dccebdb997d5
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Fri Jun 30 14:21:04 2023 -0700
Update dockerfile with recent jdk distro changes (#10963)
commit 0b097a8dab7d2953d331010c856420967fe873e1
Author: Shen Yu <shen@startree.ai>
Date: Thu Jun 29 23:31:46 2023 -0700
Fix a flaky test ADLSGen2PinotFSTest.testLastModified. (#11010)
commit 379d7610ac58724d1dcfe5968ddad8bb37a988d3
Author: swaminathanmanish <126024920+swaminathanmanish@users.noreply.github.com>
Date: Thu Jun 29 20:21:43 2023 -0700
Adding segmentRowComputer interface for SegmentProcessorFramework to use (#10987)
* Adding segmentRowComputer interface for SegmentProcessorFramework to use
* Addressing comments
commit cc5a9bec7d4341533613b87e62707a9ceabc0cd0
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Thu Jun 29 17:23:31 2023 -0700
Fix flaky MultiNodesOfflineClusterIntegrationTest.testServerHardFailure() (#10999)
commit db8793f780c72e3bbbd66450ebe53eb5cdcc6ef9
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Thu Jun 29 17:07:53 2023 -0700
Fix a typo in snapshot lock (#11007)
commit 9374964ad4b5797329333157fcbb1de56cc66657
Author: summerhasama-stripe <134637018+summerhasama-stripe@users.noreply.github.com>
Date: Thu Jun 29 12:12:42 2023 -0400
Message batch ingestion lag fix (#10983)
* add back stream message constructur with 1 arg
* fix bug with metadata being set to null in code trying to assure backwards incompatability
* Assert not null on stream message metadata
---------
Co-authored-by: Priyen Patel <priyen@stripe.com>
Co-authored-by: Rong Rong <rongr@apache.org>
commit fc26d6d8975b4cd46e26e460236a30e8b1eb2cde
Author: Shen Yu <shen@startree.ai>
Date: Wed Jun 28 19:23:44 2023 -0700
Fix the NULL ordering in TableResizer for GROUP BY. (#10939)
commit db0f3dcb73bd338d4b754329679e5f3e4d6bf7e0
Author: Shen Yu <shen@startree.ai>
Date: Wed Jun 28 19:17:18 2023 -0700
Fix a flaky test NullEnabledQueriesTest.testQueriesWithDictFloatColumnNoNullValues. (#10997)
commit ed4774c5d95b3975a72295e52268c47b3e9819fe
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Thu Jun 29 06:03:00 2023 +0530
Use pinot-query-planner utils to extract table names in pinot-controller (#10873)
Use pinot-query-planner utils to extract table names in pinot-controller
* Adding support for pushdown of explain plan queries
* cleanup extractTableNamesFromNode util from CalciteSqlParser
* Dispatch to random broker in case of table name parsing issue
commit 5c3cf01cd8b092c6e385b6a6f3427de6f54b724d
Author: Christopher Peck <27231838+itschrispeck@users.noreply.github.com>
Date: Wed Jun 28 16:20:59 2023 -0700
do not readd lucene readers to queue if segment is destroyed (#10989)
commit 82252ed84340627223258ba1aaeb1b372a86c2f9
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Wed Jun 28 13:07:25 2023 -0700
[Clean up] Do not count DISTINCT as aggregation (#10985)
commit 45a32eaaad594955754732ec18ee78abeb8e96b8
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Wed Jun 28 16:07:55 2023 +0200
Add a PrioritizedFilterOperator (#10953)
* Add a PrioritizedFilterOperator which is used when FilterOperatorUtils tries to sort a bunch of operators.
This is needed by new operators that are not included in the compilation unit. This improves FilterOperatorUtils extensibility, although a redesign may be convenient.
* Multiply most priorities by 100 in order to have more space between them
* Change default FilterOperatorUtils to do not fail when an operator is not recognized but prioritized it last.
commit 2dd3c86dd2fb5fa9fc9b7643c15246224154eccd
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 27 21:16:15 2023 -0700
returning empty SegmentConversionResult when validDocIds is empty
commit 43c4355892c906668edd4f4806833375916d0d4c
Merge: 6bfcd8f3c7 04bdb9a397
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 27 18:31:04 2023 -0700
Merge branch 'apache:master' into compaction
commit 04bdb9a39743569aaacfcb07e2cc7eaa08854791
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Tue Jun 27 17:55:17 2023 -0700
Enhance early terminate for combine operator (#10988)
commit 238629c41c551c6f42e38f817749aa445beea343
Author: Shen Yu <shen@startree.ai>
Date: Tue Jun 27 13:56:01 2023 -0700
Support NULL for multi-column GROUP BY. (#10938)
commit b001c40182047422ab40665ff6d8cc0a0db247dc
Author: Rong Rong <rongr@apache.org>
Date: Tue Jun 27 13:45:14 2023 -0700
[multistage][feature] pipeline breaker on broker reduce (#10971)
* improve broker reduce efficiency
* address comments
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit fc9f56b07bc2899fa18ac024b2a0878aedd96a5c
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Tue Jun 27 13:44:56 2023 -0700
allow to add custom context into query trace (#10977)
commit 3b07def33122b30cf3820b28ea55397862022217
Author: mingmxu <mingmin.xu@robinhood.com>
Date: Tue Jun 27 13:44:32 2023 -0700
use MultiStageRequestIdGenerator in v1 engine (#10966)
commit 6bfcd8f3c76d7622d464db038a15e843f64afe4f
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 27 11:10:48 2023 -0700
throwing a FatalException to abort the task if validDocIds is empty
commit 7606a1455131f8967dfd4c5ef66d25d0497ab8cd
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 27 07:24:23 2023 -0700
removed unused import
commit f5c1c2cedb715a76c08af90cca381d17740cdfd5
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jun 26 20:12:05 2023 -0700
deleting segments with no valid docs
commit 2d30e28bedea29d264b0d7946ebb3f0e2cf1b5cc
Author: Subbu Subramaniam <mcvsubbu@users.noreply.github.com>
Date: Mon Jun 26 19:25:26 2023 -0700
Add minion observability for segment upload/download failures (#10978)
* Add minion observability for segment upload/download failures
Currently, minions do not provide observability into upload or download
failures. Added mechanism to log errors and bump metrics when either
upload or download fails, so that operators can set alerts on these
metrics to detect the inconsistent state quickly and remediate if possible.
Issue #10973
* Style fix
commit a62cb0b2716ad35d275db6d7705b8805746111dc
Author: Shen Yu <shen@startree.ai>
Date: Mon Jun 26 19:11:02 2023 -0700
Make TransformBlockValSet::getNullBitmap return the cache. (#10967)
commit 48df78e4764e1b004ebb19b403463ac0c666d612
Author: cgenrich <cgenrich@gmail.com>
Date: Mon Jun 26 19:05:50 2023 -0700
Provide results in CompletableFuture for java clients and expose metrics (#10326)
commit 8170d435c5e8999daae9bb393e9d172aea2bdbd7
Author: Subbu Subramaniam <mcvsubbu@users.noreply.github.com>
Date: Mon Jun 26 08:38:20 2023 -0700
Changed Lists.of to ImmutableList.of for test in java8 (#10972)
commit 6a47311c03fdc020a4e05a3acf5cacb7843e3307
Author: Rong Rong <rongr@apache.org>
Date: Fri Jun 23 19:57:16 2023 -0700
[multistage][feature] add pipeline breaker stats collector (#10958)
- [multistage][pb][stats] add stats support for pipeline breaker results
- add stats test collector
- refactor the test setup format to be easier to understand
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit e8841316fa63bf5406d03f3459ae0c40985fbf10
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jun 23 15:52:01 2023 -0700
Simplify the handling for partial-upsert record update (#10970)
commit f8e783899f9a327b162c8cd62ca75cfaf410cba6
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Sat Jun 24 01:29:38 2023 +0530
Remove handle out-of-order from updateRecord method (#10968)
commit 608cb8de2bd19c8497b03be07a087591bfd1c3bd
Author: Navina Ramesh <navina@apache.org>
Date: Fri Jun 23 11:59:43 2023 -0700
Add delete support to upsert tables (#10703)
commit 7e76cac5866e5f40b734692fb082da683d2f4cab
Author: Shen Yu <shen@startree.ai>
Date: Fri Jun 23 11:58:31 2023 -0700
Use _hasNull in the object single column distinct executors. (#10914)
commit a2086072545fc67a35f5d6756c3b9efc97d15641
Author: mingmxu <mingmxus@gmail.com>
Date: Thu Jun 22 23:49:25 2023 -0700
[#10884][feature] add requestId for BrokerResponse in pinot-broker and java-client (#10943)
commit f90331464549e85da8c9c76e5ad8cb92f9cbd2aa
Author: Shen Yu <shen@startree.ai>
Date: Thu Jun 22 23:44:38 2023 -0700
Support NULL literal selection. (#10962)
commit 89ad4d80bf0604cc01672d2a0bbeb9377975fbca
Author: Shen Yu <shen@startree.ai>
Date: Thu Jun 22 23:43:23 2023 -0700
Fix TransformBlockValSet::getNullBitmap. (#10959)
commit 24c5d8fe8521b91e68dd07294a3f57b067c0d525
Author: Rong Rong <rongr@apache.org>
Date: Thu Jun 22 15:45:17 2023 -0700
[multistage][bugfix] fix pipeline breaker error block populate (#10957)
* [clean up] make pipeline breaker executor always return without exception
* [clean up] refactor server plan request util entry point
* [test] add QueryRunner test as well as PB unit-tests
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit e658f86a8ac9010079abe7960a66c8c20032d48b
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Thu Jun 22 13:49:28 2023 -0700
update libtasn1 lib to fix CVE-2022-1471 (#10775)
commit 3c92b327114c4534f354e8a8f2755197a6439fd5
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Thu Jun 22 11:59:10 2023 -0700
Take upsert snapshot when creating new consuming segment (#10928)
commit 84aa659b713c3970da41a43925427708bc72a8c2
Author: Aravind Suresh <arvindsuresh2009@gmail.com>
Date: Fri Jun 23 00:25:29 2023 +0530
Fixes SQL wildcard escaping in LIKE queries (#10897)
* Fixes SQL wildcard escaping in LIKE queries
* Updated unit tests
* Fixed lint issues
* Minor refactor
* Addressed reviewer comments
* Fixed lint
* Addressed reviewer comments
* Minor fixes
* Minor refactor
commit 3a326b06569a6b88362d27eb61995c38ac5237c1
Author: Sonam Mandal <somandal@linkedin.com>
Date: Thu Jun 22 10:25:33 2023 -0700
[multistage] Add support for RANK and DENSE_RANK ranking window functions (#10700)
* Add support for RANK and DENSE_RANK window functions
* Empty-Commit
* Fix tests after rebase
* Empty-Commit
commit aea890a5ee715bf06bf7dd1643a4e8ced3db67b9
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Thu Jun 22 21:10:00 2023 +0530
Update pinot client to not use Calcite extractTableNames (#10955)
* Update pinot client to not use Calcite extractTableNames
* addressed comments
commit 2b47dd68926dd39da05949fc204388797dfd14cb
Author: lucifer4j <97942130+lucifer4j@users.noreply.github.com>
Date: Thu Jun 22 08:09:32 2023 +0530
Make org.apache.pinot.client.BrokerResponse public (#10932)
commit 30b5da9626a774f97812b924c917fe766eba4b53
Author: Prachi Khobragade <pkhobragade@linkedin.com>
Date: Wed Jun 21 10:32:44 2023 -0700
Add a new SegmentPurger interface method to get record purger based on table configs and schema (#10946)
commit 70020005b1a878acc4efa5f813162dd8c3db80c0
Author: swaminathanmanish <126024920+swaminathanmanish@users.noreply.github.com>
Date: Wed Jun 21 10:30:36 2023 -0700
Null check before closing reader (#10954)
commit 17ee2024dd7d49ce2f5ebf00ef9d71b6ca589a00
Author: dario-liberman <130933669+dario-liberman@users.noreply.github.com>
Date: Wed Jun 21 19:26:57 2023 +0200
FUNNEL_COUNT Aggregation Function (#10867)
* New Funnel Aggregation Function
* Funnel analytics support - FUNNEL_COUNT aggregation function
* Delete FunnelAggregationFunction.java
* Simplify Tests
* Simplify Tests
* Simplify Tests
* within -> across
Fix javadoc
* Update FunnelCountAggregationFunction.java
Address comments
---------
Co-authored-by: Dario Liberman <dario.liberman@uber.com>
commit b699cdbf814d1a1a480d376faacdafd698c46ecc
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Wed Jun 21 20:54:52 2023 +0530
Push out-of-order events metrics for full upsert (#10944)
* Push out-of-order events metrics for full upsert
* add new meter for full upsert
commit 73830ae3ce94428b7e9991d1bd160bd62a14b7c1
Author: Rong Rong <rongr@apache.org>
Date: Tue Jun 20 21:28:05 2023 -0700
[multistage][cleanup] clean up PhysicalPlanContext usage (#10950)
* [clean up] server plan should not extend physical plan, it should encapsulate
* [clean up] reformat opChainExecutionContext, constructor to only take PhysicalPlanContext directly
* [clean up] remove timeoutMs as argument and member var from plan context
commit 493d4e5c1431bfd7d107efe497c9012baa9d67b0
Author: jbewing <ewing.j@northeastern.edu>
Date: Tue Jun 20 21:01:42 2023 -0400
Fix casting when prefetching mmap'd segment larger than 2GB (#10936)
commit d6152a86668eb40a2028a9942285f7c181e0d432
Author: Zhuangda Zhu <zhuangdaz@users.noreply.github.com>
Date: Tue Jun 20 20:58:15 2023 -0400
Return true when _resultSet is not null (#10899)
commit 84bc45f713daa2bdde80f30bd3e2cf0c3dc7fa95
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Tue Jun 20 15:01:29 2023 -0700
Fixing table name extraction for lateral join queries (#10933)
commit 844ca4d671ef5bb86a4bc8c90a7dc70ae7a607ad
Author: Rong Rong <rongr@apache.org>
Date: Tue Jun 20 14:00:36 2023 -0700
[hotfix] remove singleton from dynamic broadcast rule (#10949)
Co-authored-by: Rong Rong <rongr@startree.ai>
commit c00bdba936293b2d9518c0b6722c0e44ae699f33
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 20 13:45:27 2023 -0700
added precondition to check if provided minRecordCount >= 0
commit d95306040a4aad1687a98951951935b44750f3bb
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 20 13:40:11 2023 -0700
added minRecordCount config to exclude smaller segments from compaction
commit 70fc973d662ef2051dbe2c0b261d1ae4db8c4e7e
Author: soumitra-st <127247229+soumitra-st@users.noreply.github.com>
Date: Tue Jun 20 11:02:50 2023 -0700
Adding method to register components in broker API (#10942)
commit 9443c5f0bb5f9f7b8403ac42e15b8fedc4ff5a97
Author: Gaurav Joshi <68965367+joshigaurava@users.noreply.github.com>
Date: Tue Jun 20 11:02:20 2023 -0700
Updated AuthProvider to extract user identity (#10925)
commit 9aa8a12b585b8bbe84d61dbd161906622a4ed8a3
Author: Robert Zych <robert.zych@gmail.com>
Date: Tue Jun 20 09:45:23 2023 -0700
updated /validDocIdMeta to support list of segment names and updated generator to request groups of segments by server
commit 9f03d61841ee8a8ae2c44677b8fbb05fc2907147
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jun 19 13:34:41 2023 -0700
only checking the new configs if provided
commit affa2e7f9e9594ff5529066a27efd226dc7bb91c
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jun 19 13:00:59 2023 -0700
using validDocIds.getCardinality() to get totalValidDocs
commit af1d728ff011c6275055a9c686c221be7c257805
Author: Robert Zych <robert.zych@gmail.com>
Date: Mon Jun 19 10:15:58 2023 -0700
added suggestion to move getValidDocIds
commit 8b2fb03d8532afd092c0211990e61f156fb45b73
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Fri Jun 16 23:02:39 2023 -0700
[Flaky-test] Fix flaky PinotResourceManagerTest and also clean it up (#10941)
commit 4af43665ec72833af89d2ee0f5e90edcdbe1b403
Author: Ankit Sultana <ankitsultana@uber.com>
Date: Sat Jun 17 08:45:59 2023 +0530
[multistage] Do Not Log Entire Plan in QueryServer (#10930)
commit a8988dd48f0cd0bf25960271f80bd475eba79ed8
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jun 16 18:49:34 2023 -0700
Bump snappy-java from 1.1.8.2 to 1.1.10.1 (#10922)
commit ce50e71d1eaf7ad9876bf812be51f0662c6d537f
Author: Shen Yu <shen@startree.ai>
Date: Fri Jun 16 17:51:15 2023 -0700
Fix an integer overflow bug. (#10940)
commit 2baff8180f498119a7f367a94b376553b81a8333
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date: Fri Jun 16 16:37:04 2023 -0700
Bump jetty-server in /pinot-plugins/pinot-stream-ingestion/pinot-pulsar (#10908)
commit e800693dad1038c5f6a09d67ca753d71160238bd
Author: Saurabh Dubey <saurabhd336@gmail.com>
Date: Sat Jun 17 04:34:03 2023 +0530
Avoid parsing offset when invalid (#10893)
commit 9a1795c2b0e3e60b04c3d0e06bfde6d11563aa82
Author: David Cromberge <davecromberge@gmail.com>
Date: Fri Jun 16 23:49:29 2023 +0100
Upgrade to Apache Datasketches 4.0.0 (#10878)
commit b2efb8117a5f69c68e3d742031d22840fc70d172
Author: Shen Yu <shen@startree.ai>
Date: Fri Jun 16 15:16:07 2023 -0700
Properly set the records list initial size to avoid unnecessary resizing. (#10935)
commit 85bb1fab54d44e847195e35e3e0b117022985d36
Author: Vivek Iyer Vaidyanathan <vvivekiyer@gmail.com>
Date: Thu Jun 15 21:13:39 2023 -0700
Change list.of to immutableList.of to solve java 8 compilation (#10929)
commit f1966d9fa01040774ff2e153ab69f9c69903fcbe
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Thu Jun 15 12:28:28 2023 -0700
[Multi-stage] Support partition based colocated join (#10886)
commit 90dc3a3c566e5d634aa00632993ab86ce0549ba5
Author: Shen Yu <shen@startree.ai>
Date: Wed Jun 14 16:28:12 2023 -0700
Fix a bug in SELECT DISTINCT ORDER BY LIMIT. (#10887)
commit a5327e70392aad1d83d339e03d52fd9323703483
Author: Ankit Sultana <ankitsultana@uber.com>
Date: Thu Jun 15 02:11:53 2023 +0530
Add Support for Disabling Default User Agent for Http Client (#10895)
* Add Support for Disabling Default User Agent for Http Client
* set default non-null value of httpclientconfig
* Update pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClientConfig.java
Co-authored-by: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
* Update UT to ensure config is indeed picked up
---------
Co-authored-by: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
commit 81813c28fc53b7f624cd836d0d0abec20cb9d277
Author: Xiaotian (Jackie) Jiang <17555551+Jackie-Jiang@users.noreply.github.com>
Date: Wed Jun 14 12:51:22 2023 -0700
Fix MultiNodesOfflineClusterIntegrationTest.testServerHardFailure() (#10909)
commit bb4091f12f380a739ae08619c18c7874223d1be4
Author: Xiang Fu <xiangfu.1024@gmail.com>
Date: Wed Jun 14 07:43:20 2023 -0700
adding PinotLogicalExchange to add exchange type indicator (#10813)
Co-authored-by: Rong Rong <rongr@startree.ai>
commit 449c95c6b93499417a75574607d160231158e2e3
Author: swaminathanmanish <126024920+swaminathanmanish@users.noreply.github.com>
Date: Tue Jun 13 20:39:25 2023 -0700
Making segmentMapper do the init and cleanup of RecordReader (#10874)
commit 14670f49f6e43007bdeb1a06d09d049229639342
Author: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Date: Tue Jun 13 19:42:49 2023 -0400
Add clpDecode transform function for decoding CLP-encoded fields. (#10885)
* Add clpDecode transform function for decoding CLP-encoded fields.
* clpDecode transform: Add example input and output.
commit 1d9682fc67aca813d7a14b3a7415f545f9420078
Author: soumitra-st <127247229+soumitra-st@users.noreply.github.com>
Date: Tue Jun 13 11:38:57 2023 -0700
Enable case insensitivity by default (#10771)
commit 0a97ca81f3973c3e22c76bb3c4ea4d3eb7175583
Author: Abhishek Sharma <abhishek.sharma@spothero.com>
Date: Tue Jun 13 14:29:11 2023 -0400
[multistage][float-double-comparison]: updated for CAST function addition based on correct precision. (#10814)
commit 3dbdaf60b5502dd004cb213e541dbe5636fef6bd
Author: Pratik Tibrewal <tibrewalpratik@uber.com>
Date: Tue Jun 13 23:33:54 2023 +0530
Fix colocated join quickstart script (#10889)
commit 0420414fc2a5408dc7aa32fdb9d359763fc3cb34
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Tue Jun 13 10:08:00 2023 -0700
bump awssdk version for a bugfix on http conn leakage (#10898)
* bump awssdk version for a bugfix on http conn leakage
* use latest s3mock and fix S3PinotFSTest
commit 207036950ad5fc13725ee464a2d0a09f26776638
Author: Ankit Sultana <ankitsultana@uber.com>
Date: Tue Jun 13 02:08:50 2023 +0530
Close PinotFS after Data Manager Shutdowns (#10888)
commit 86b3b3e1c5bad5092f357358888ff28903ab05df
Author: Evan Galpin <egalpin@users.noreply.github.com>
Date: Fri Jun 9 18:51:18 2023 -0700
[feature] multi-value datetime transform variants (#10841)
commit b1e8655449abd903330993370dee2541869f8be9
Author: Rong Rong <rongr@apache.org>
Date: Fri Jun 9 16:37:53 2023 -0700
[multistage][bugfix] fix PB operator never returns block (#10881)
* fix PB operator never returns block thus hogging the opChainScheduler pool issue.
* also fix empty block NPE, this never happened since previously we also add no-op block to the result map.
---------
Co-authored-by: Rong Rong <rongr@startree.ai>
commit d9fb58def90d6f496cea04f841a15a3b4d466bd7
Author: Shen Yu <shen@startree.ai>
Date: Fri Jun 9 16:32:38 2023 -0700
Select distinct multi-column NULL support. (#10860)
commit 880a074c7ae71ff3937171e504f6287d48c0fa7f
Author: Shen Yu <shen@startree.ai>
Date: Fri Jun 9 15:01:19 2023 -0700
Refactor single column distinct executors (#10859)
commit 9ba7aa3481a1362eb10c3c21774b525a5b20ffba
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jun 9 14:13:32 2023 -0700
removed _finished from UpsertCompactionTaskExecutor
commit 85ad503f0bfe1ab26afe1d7ec174468570a9df7a
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jun 9 11:57:16 2023 -0700
converted /invalidRecordCount to /validDocIdMetadata
commit 479ef52e6df350d83f4342d87fb88eb120c5ea2d
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jun 9 10:05:06 2023 -0700
replaced function calls with number and checking a salary before/after compaction
commit c9ed072ffc902bda5e1e61ac8fdf30e1b22c4246
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jun 9 09:20:12 2023 -0700
increased DEFAULT_INVALID_RECORDS_THRESHOLD_PERCENT from 10 to 30
commit 869623f925cd14b2abb94d40342835af09fa8623
Author: Robert Zych <robert.zych@gmail.com>
Date: Fri Jun 9 09:17:29 2023 -0700
updated comment for INVALID_RECORDS_THRESHOLD_PERCENT
commit a75544f2c644874a88b61cea48b640d9cf05283c
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Thu Jun 8 23:51:03 2023 -0700
Fix #10713 by giving metainfo more priority than config (#10851)
commit 26e5952d75ff5accedd05c5eb9812cf900d3f413
Author: Sajjad Moradi <moradi.sajjad@gmail.com>
Date: Wed Jun 7 20:55:29 2023 -0700
Bug Fix: Segment Purger cannot purge old segments after schema evolution (#10869)
commit c704147d3e97d41bf170671cbd43ffa1064479be
Author: Gonzalo Ortiz Jaureguizar <gortiz@users.noreply.github.com>
Date: Wed Jun 7 16:48:15 2023 -0700
PinotBufferFactory and a buffer implementation that uses Unsafe (#10528)
commit 03c49e43b4f20f1248ecdbabf6359b59b388455a
Author: Rong Rong <rongr@apache.org>
Date: Wed Jun 7 14:30:18 2023 -0700
[multistage] register opChain finish callback for pipeline breaker to continue (#10870)
when the pipeline breaker returns, it is side-cart loading the result map materialized from the PipelineBreakerOperator
at this point, it will cause a race-condition for the main opChain, b/c 2 things happened:
1. the result will be returned immediately after the mailbox receives the final block and continue the main opChain process;
2. while the mailbox receives the final block, it will also return the EOS to the opChainScheduler, then OpChainScheduler de-register/close/cancel the opChain properly.
This PR ensures that once the pre-stage opChain close properly, the main-stage opChain can then be registered
Co-authored-by: Rong Rong <rongr@startree.ai>
commit 24b50ed65f5ff1405479cd4136fda85ffa99f9c1
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Wed Jun 7 13:05:10 2023 -0700
jsonPathString should return null instead of string literal "null" (#10855)
commit 0f58e48c8e82739cfffc760fcea65c8165028e9d
Author: Xiaobing <61892277+klsince@users.noreply.github.com>
Date: Wed Jun 7 13:04:40…
s0nskar
pushed a commit
to s0nskar/pinot
that referenced
this pull request
Aug 10, 2023
* Added InstanceResource.java * Added intial Executor service to InstanceResource * Added JavaDoc * [partial cherry-pick] Added InstanceResource for Server/Broker * Fixed BaseResourceTest.java * Fixed BaseResourceTest.java * Deleted unused variable * Minor changes to InstanceResource * Updated InstanceResource * Updated InstanceResource * Fized checkstyle error in InstanceResource * Fixed JavaDoc typo in InstanceResource
aidar-stripe
pushed a commit
to aidar-stripe/pinot
that referenced
this pull request
Nov 22, 2023
…pache#290) ### Notify cc stripe-private-oss-forks/pinot-reviewers r? aidar dang ### Summary Gist for pre merge diffs: https://git.corp.stripe.com/gist/47be757736cc9ba05e11fc86163e2386 Gist for post merge diffs: https://git.corp.stripe.com/gist/60045b6d760fd9270b784425fd0adb77 Gist for diff of diffs: https://git.corp.stripe.com/gist/ac1bdf895d72a87b5073d366b4fd2ab2 Gist for merge conflict resolution: https://git.corp.stripe.com/gist/89b4e01da42424654f6c3d7ef4dbf56f For upstream pulls, please list all the PRs of interest and risk - apache#10988 this should make queries with exceptions fail faster - apache#10873 offers a clear module to get table names from a query that we can use in rad-query-svc - apache#10983 summer's fix to capture correct ingestion lag - apache#10891 segment metadata should have all min/max values now - apache#10905 reuses existing startrees if you add new ones - apache#10993 makes GET `/tenant/{name}` only list. that's the only way we use it anyway. previously this could accidentally disable entire tenants - apache#11030 there's a new DateTimeGranularitySpec, but this supports our old one, too - apache#10969 adds an API to check instance state before dropping it. today, you have to just try to drop it and see if it works or not. it will fail if the instance still has tables assigned to it - apache#10991 with case-insensitive as the default, new columns can't match old columns - apache#11056 upgrades AWS sdk. will be interesting to see how this works when we pull new code into zoolander. - apache#11040 retries for kafka connections we added - apache#11020 makes server restarts faster for upsert tables - apache#10937 adds an API to get all the tags for a server/broker - apache#11107 ideally should make deleting tables, and moving segments faster - apache#11131 fixes a reload bug when removing indices - apache#11146 fixes a bug in matching docs with range index - apache#11073 summer's fix for table rebalances - apache#11166 this should make new segments that fail to talk to kproxy go offline and get retried instead of going into error state - apache#11016 rbac support is here ### Motivation this is up to the last OSS commit before Priyen's big HLL/BigDecimel that has lots of conflicts. The next OSS pull will pull just that change so we can be much more thorough about resolving conflicts. ### Testing https://docs.google.com/document/d/1pEWwr_bNN2qfdlUBbMeS_XZqUD8H5Uvy7V8bayAV7Cs/edit ### Rollout/monitoring/revert plan (Merge-committed by Merge Queue - Original PR: https://git.corp.stripe.com/stripe-private-oss-forks/pinot/pull/290)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added an API that gets tenant tags for both Broker and Server.