Filepulse connector using on AWS MSK

[iamsamba]

Hi

I am using Filepulse 2.9.0 version on MSK to connect to s3 to read json files and push to topic.

Connector created successfully. But connector not listing files.

Starting to list object files using:
AmazonS3FileSystemListing (io.streamthoughts.kafka.connect.filepulse.fs.DefaultFileSystemMonitor:283)
Completed object files listing. '0' object files found in 72ms

I have json files in s3 path. I don't see any error in log as well.

Here is the connect config used.

connector.class=io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector
filters=ParseJSON, ExcludeFieldMessage
filters.ParseJSON.type=io.streamthoughts.kafka.connect.filepulse.filter.JSONFilter
filters.ParseJSON.source=message
filters.ParseJSON.merge=true
filters.ExcludeFieldMessage.type=io.streamthoughts.kafka.connect.filepulse.filter.ExcludeFilter
filters.ExcludeFieldMessage.fields=message
fs.cleanup.policy.class=io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy
fs.cleanup.policy.triggered.on=COMMITTED
fs.listing.class=io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing
fs.listing.filters=io.streamthoughts.kafka.connect.filepulse.fs.filter.RegexFileListFilter
fs.listing.interval.ms=10000
file.filter.regex.pattern=.\.json$
offset.policy.class=io.streamthoughts.kafka.connect.filepulse.offset.DefaultSourceOffsetPolicy
offset.attributes.string=name
skip.headers=1
topic=connect-file-s3-test-json
tasks.reader.class=io.streamthoughts.kafka.connect.filepulse.fs.reader.AmazonS3BytesArrayInputReader
tasks.file.status.storage.class=io.streamthoughts.kafka.connect.filepulse.state.KafkaFileObjectStateBackingStore
tasks.file.status.storage.bootstrap.serversbroker=********
tasks.file.status.storage.topic.partitions=10
tasks.file.status.storage.topic.replication.factor=1
tasks.max=1
aws.access.key.id=*********
aws.secret.access.key=********
aws.s3.region=******
aws.s3.bucket.name=******
aws.s3.bucket.prefix=*****/****

[fhussonnois]

Hi @iamsamba, it seems the connector does not see any S3 object due to the log Completed object files listing. '0' object files found. Maybe you should double-check the aws.s3.bucket.prefix property. In addition, make sure the configured prefix is not ending with /. In addition, make sure you have granted access to your S3 folders.

[iamsamba]

Hi @iamsamba, it seems the connector does not see any S3 object due to the log Completed object files listing. '0' object files found. Maybe you should double-check the aws.s3.bucket.prefix property. In addition, make sure the configured prefix is not ending with /. In addition, make sure you have granted access to your S3 folders.

Thanks @fhussonnois, it helps. Able to list files now.

[saurabhpolshettiwar]

@iamsamba : Are you publishing these file names to the topic?