New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add limit pushdown for Kafka connectors, and further refactor and enhance predicate pushdown in Kafka connector #20146
base: master
Are you sure you want to change the base?
Conversation
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
hello why has it gone?
…---- Replied Message ----
| From | ***@***.***> |
| Date | 01/11/2024 01:22 |
| To | trinodb/trino ***@***.***> |
| Cc | Fred ***@***.***>,
Author ***@***.***> |
| Subject | Re: [trinodb/trino] Add limit pushdown for Kafka connectors, and further refactor and enhance predicate pushdown in Kafka connector (PR #20146) |
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @***@***.***@mosabua
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Could @wendigo @findepi @elonazoulay or anybody else with more knowledge about the Kafka connector help @xxzhky with a review please to move this PR forward. Also @xxzhky can you confirm that the build passes locally and the CI failures are wrong, or otherwise fix these issues. |
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
@xxzhky could you rebase and fix any build issues |
@mosabua yes, I confirm that the build passes locally, however, some irrational checking issues happen to me. i.g. |
I've rebased this PR locally and checked the dependency scope failure and here's the fix:
|
@wendigo OK. Thanks very much, and could you like to help me fix it. |
@xxzhky failure is in the dependency scope check, not tests. Tests are passing just fine on the CI. |
@xxzhky the diff in the comment above is the fix .. just apply that locally and amend the commit and force push.. then CI should pass |
1bea5b1
to
8917ade
Compare
refactor and enhance predicate pushdown in Kafka connector Extensive comments have been added across multiple files in the Kafka connector for better readability and understanding of the code. The commit also results in better maintainability and improved performance of the code. Significant changes have been made to improve predicate push-down support in Kafka connector. The code has been refactored to better handle intersection of old and new domains. Additionally, the functionality to override partition end and begin offsets with range has been enhanced. Lastly, an option to enable or disable predicate pushdown at runtime through session properties has been introduced. 1. By default, it supports predicate push-down. Push-down can be disabled via ``kafka.predicate-force-push-down-enabled`` config prop or ``kafka.predicate_force_push_down_enabled`` session prop. Update Kafka config in tests
…ate constraint 1.This commit introduces the ConstraintExtractor class in the Kafka connector. This class is used to extract and manipulate constraint related data and provides static functions for different operations. It is primarily used to handle different operations such as transforming constraints and expressions, managing conjuncts and converting and intersecting tuple domains. Moreover, it also handles specific cases such as timestamp casting, truncation and other function(year). 2.Replace createTestingTypeAnalyzer with new IrTypeAnalyzer The Kafka plugin test class TestConstraintExtractor has been updated, replacing `createTestingTypeAnalyzer` with a new instantiation of `IrTypeAnalyzer`. Furthermore, `trino-parser` dependency was added into `pom.xml` to ensure correct symbol translation, while enhancing the test coverage and improving the overall efficiency.
8917ade
to
472d453
Compare
plugin/trino-kafka/src/main/java/io/trino/plugin/kafka/KafkaMetadata.java
Show resolved
Hide resolved
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to split them into smaller commits like
- Add limit pushdown
- Refactor
- Enhanced pushdown
This would help to ease up the review process
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Praveen2112 ok. I'll split them into more parts. Hong long could you get the review process done?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can start reviewing it by this week.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be nice if we could have some test coverage in BaseConnectorTest
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of converting ConnectorExpression
into a TupleDomain - Can we ConnectorExpressionRewriter
which would support the same for nested expression as well.
This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua |
@bitsondatadev @colebow @mosabua I am so busy recently, and I will handle it in the near future. |
Sounds good @xxzhky .. adding stale-ignore label. |
Description
Add limit pushdown for Kafka connectors, and further refactor and enhance predicate pushdown in Kafka connector
Additional context and related issues
Add limit pushdown, and enhance predicate pushdown in Kafka connector
Significant changes have been made to improve predicate push-down support in Kafka connector. The code has been refactored to better handle intersection of old and new domains.
Additionally, the functionality to override partition end and begin offsets with range has been enhanced.
Lastly, an option to enable or disable predicate pushdown at runtime through session properties has been introduced.
By default, it supports predicate push-down. Push-down also can be disabled via kafka.predicate-force-push-down-enabled config prop or kafka.predicate_force_push_down_enabled session prop.
I try my test to make your life easier so that extensive comments have been added across multiple files in the Kafka connector for better readability and understanding of the code.
Release notes
(1) Add limit pushdown for Kafka connectors,
(2) Enhance predicate pushdown in Kafka connector.