Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to break messages when ingesting into kafka #313

Closed
akhileshacc opened this issue Jul 2, 2020 · 5 comments
Closed

How to break messages when ingesting into kafka #313

akhileshacc opened this issue Jul 2, 2020 · 5 comments

Comments

@akhileshacc
Copy link

akhileshacc commented Jul 2, 2020

I wanted to break messages before sending to kafka endpoint.
Is it possible to do with processors ?

I am trying to send S3 files data to kafka using s3 and kafka components.

from("aws-s3://bucketname?args=....")
.to("kafka:test1?brokers=localhost:9092")

How do i break messages before sending to kafka ?

@oscerd
Copy link
Contributor

oscerd commented Jul 2, 2020

Hello.

  1. What do you mean with break messages?
  2. This is the issue tracker for camel-kafka-connector, but you're using camel. You should ask on the camel users mailing list or dev mailing list: https://camel.apache.org/community/mailing-list/ and ask there.

@akhileshacc
Copy link
Author

  1. If text file has 10mb data, can i pick 10 records with 1mb of data and then store it.
  2. Sure will do so.

@oscerd
Copy link
Contributor

oscerd commented Jul 2, 2020

You can implement a processor. Each S3ObjectInputStream can be read and you can read the data for example 1 Mb

@oscerd
Copy link
Contributor

oscerd commented Jul 2, 2020

By the way, it's better to discuss this on the mailing list to avoid confusion.

@oscerd oscerd closed this as completed Jul 2, 2020
@akhileshacc
Copy link
Author

Thanks for the help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants