Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TimeBasedPartitioner according to specified field #98

Closed
ofirski opened this issue Aug 2, 2016 · 6 comments

Comments

@ofirski
Copy link

commented Aug 2, 2016

Please add the ability to have time based partitioning by specific field denoting the event time (instead of ingestion time)

@xianzhen

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2016

I used fieldPartioner to handle it. But i think your idea is better!

@doriwal

This comment has been minimized.

Copy link

commented Aug 19, 2016

Agree that timeBased should have the option to use event time , for example event that send from mobile SDK at 8:58 might be pushed to kafka at 9:01 , using the ingestion time is not accurate

I also added pull request in order to support several fieldsPartitioner as in my use case I need both event time and event type in order to partition the events in parquet

@ewencp

This comment has been minimized.

Copy link
Member

commented Aug 29, 2016

@xianzhen FieldPartitioner only works for exact values, which means you can't use a timestamp field directly, you need it to be simplified to, e.g., minute, hour, or day, depending on the granularity you want.

@kerkero I think this would definitely be valuable. In particular, Kafka added timestamps natively in its format (but after this connector was originally created) so we should at least support those. Additionally, since Connect supports timestamp logical types, we should definitely have support for those fields. I think this should amount to a small additional config to specify the field and minor modifications to TimeBasedPartitioner.

@doriwal

This comment has been minimized.

Copy link

commented Aug 29, 2016

Agree that FieldPartitioner cant use timestamp , in my use case each event that i am sending to kafka contains also the following fields : day like 2016-08-23 , time like 10:00
this fields are calculated from the event timestamp (I am adding day/time fields to the event that i am sending to kafka)

@codenamelxl

This comment has been minimized.

Copy link

commented Aug 29, 2017

A little head-up, it seems that kafka-connect-hdfs has been change to use kafka-connect-storage-common which does have a very nice way to provide a timestamp field ( it can be seen here in kafka-connect-s3 which also use kafka-connec-storage-common. We now have to wait for this change come into release :) From what i see in the branch, 3.3.1-alpha2 still does not have mentioned change applied :(

@kkonstantine

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

Indeed, timestamp-based partitioners where timestamp is stored in the Kafka record or a record field have become available under kafka-connect-storage-common.

Two recent PRs have added support of such partitioners in the HDFS connector (#245 and #246).

Closing this issue, and we may reopen this one or a new one if there's an issue with these partitioners. Anyone interested in field-based timestamp partitioning is highly encouraged to try what's on master right now! Thanks everyone for highlighting this feature!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.