Skip to content

S3InboundFileSynchronizer failing when processing through S3 directories #45

@riversidetraveler

Description

@riversidetraveler

I'm trying to use the S3InboundFileSynchronizer to synchronize an S3Bucket to a local directory. The bucket is organised with sub-directories such as:

bucket ->
            2016 ->
                       08 ->
                          daily-report-20160801.csv
                          daily-report-20160802.csv

etc...

Using this configuration:

    public S3InboundFileSynchronizer s3InboundFileSynchronizer() {
        S3InboundFileSynchronizer synchronizer = new S3InboundFileSynchronizer(amazonS3());
        synchronizer.setDeleteRemoteFiles(true);
        synchronizer.setPreserveTimestamp(true);
        synchronizer.setRemoteDirectory("REDACTED");
        synchronizer.setFilter(new S3RegexPatternFileListFilter(".*\\.csv$"));
        Expression expression = PARSER.parseExpression("#this.substring(#this.lastIndexOf('/')+1)");
        synchronizer.setLocalFilenameGeneratorExpression(expression);
        return synchronizer;
    }

I'm able to get as far as connecting to the bucket and listing its contents. When it comes time to read from the bucket the following exception is thrown:

org.springframework.messaging.MessagingException: Problem occurred while synchronizing remote to local directory; nested exception is org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is java.lang.IllegalStateException: 'path' must in pattern [BUCKET/KEY].
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:266)

Reviewing the code it seems that it'd be impossible to ever synchronize an S3Bucket w/ sub-directories:

     private String[] splitPathToBucketAndKey(String path) {
        Assert.hasText(path, "'path' must not be empty String.");
        String[] bucketKey = path.split("/");
        Assert.state(bucketKey.length == 2, "'path' must in pattern [BUCKET/KEY].");
        Assert.state(bucketKey[0].length() >= 3, "S3 bucket name must be at least 3 characters long.");
        bucketKey[0] = resolveBucket(bucketKey[0]);
        return bucketKey;
    }

Is there some configuration I'm missing or is this a bug?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions