Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 source connector writing duplicate messages for one S3 payload. #1541

Closed
dezka opened this issue Jun 20, 2023 · 1 comment
Closed

S3 source connector writing duplicate messages for one S3 payload. #1541

dezka opened this issue Jun 20, 2023 · 1 comment

Comments

@dezka
Copy link

dezka commented Jun 20, 2023

Version 3.20.3 of the connector, Kafka Connect 2.8, Java 11, Kafka 2.8.

I am sure this is a misconfiguration on my part, though I've looked through docs and I can't really find where I'm going wrong here.

{
  "connector.class": "org.apache.camel.kafkaconnector.awss3source.CamelAwss3sourceSourceConnector",
  "camel.kamelet.aws-s3-source.deleteAfterRead": "true",
  "camel.kamelet.aws-s3-source.bucketNameOrArn": "my-bucket",
  "topics": "mytopic",
  "tasks.max": "1",
  "name": "s3-source-test",
  "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
  "camel.kamelet.aws-s3-source.useDefaultCredentialsProvider": "true",
  "camel.kamelet.aws-s3-source.region": "us-east-1"
}

I upload one file containing the following json:

{"type":"s3","settings":{"bucket":"test123","base_path":"backups"}}

When it's processed, I see two messages show up in my topic with this same payload.

@dezka dezka changed the title S3 source connector ingesting duplicate data. S3 source connector writing duplicate messages for one S3 payload. Jun 20, 2023
@dezka
Copy link
Author

dezka commented Jun 21, 2023

Closing this due to realizing it's an issue with the publisher and not this plugin.

@dezka dezka closed this as completed Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant