Can't poll from S3 with a custom endpoint configuration

[xfrancois]

Hello,

Here is the related and more detailled issue on stackoverflow

I think there is an issue in the getHostPort() method of S3Session.java. This method doesn't look for the signing region field that an user may have filled. It only looks for amazonaws.com region, and then throws an IllegalArgumentException : S3 client with invalid S3 endpoint configured if there is no match between the user provided url and the regional pattern s3[-.]([^.]+)\.amazonaws\.com(\.[^.]*)?

spring-integration-aws - S3Session.java

@Override
	public String getHostPort() {
		Region region = this.amazonS3.getRegion().toAWSRegion();
		return String.format("%s.%s.%s:%d", AmazonS3.ENDPOINT_PREFIX, region.getName(), region.getDomain(), 443);
	}

aws-java-sdk-s3 - AmazonS3Client.java

 @Override
    public synchronized Region getRegion() {
        String authority = super.endpoint.getAuthority();
        if (Constants.S3_HOSTNAME.equals(authority)) {
            return Region.US_Standard;
        } else {
            Matcher m = Region.S3_REGIONAL_ENDPOINT_PATTERN.matcher(authority);
            if (m.matches()) {
                return Region.fromValue(m.group(1));
            } else {
                throw new IllegalStateException(
                    "S3 client with invalid S3 endpoint configured: " + authority);
            }
        }
    }

[artembilan]

@xfrancois ,

thank you for raising this concern!

I wonder if that really has to be this project responcibility to resolve that client region properly. Isn't it a part of AWS API?

What is your proposal anyway, please?

[xfrancois]

I think the good resolving of the region is indeed a aws-java-sdk-s3 problem more than a spring-integration-aws one. But spring-integration-aws fully relies on AWS region to resolve the host and port, and must succeed otherwise it throws an exception. And I think this last part is the main issue.

Maybe getHostPort() method could be designed in a way that if the info can't be retrieved, then it doesn't throw an exception, it just returns an empty String. Because it's only used to fill the REMOTE_HOST_PORT header, and the absence of this header is not very important (I guess ?). I may be wrong on that point, but in my case, I prefer that the returned value is empty than an exception.

If we don't want the value to be empty, I'm not sure there is an easy and beautiful fix. Indeed, the wanted information if located in the endpoint attribute. But endpoint is not an AmazonS3 attribute, it belongs to AmazonS3Client, which is a subclass. That force us to cast to retrieve the appropriate information.

I think we could imagine something like that, but it's clearly really ugly.

 if (this.amazonS3 instanceof AmazonS3Client && ((AmazonS3Client) this.amazonS3).getEndpoint() != null) {
            return ((AmazonS3Client) this.amazonS3).getEndpoint();
        } else {
            Region region = this.amazonS3.getRegion().toAWSRegion();
            return String.format("%s.%s.%s:%d", AmazonS3.ENDPOINT_PREFIX, region.getName(), region.getDomain(), 443);
        }

[artembilan]

That method is not intended for production use, plus it is not public:

  /**
     * For testing
     */
    synchronized URI getEndpoint() {
        return endpoint;
    }

We should consider something else...

if the info can't be retrieved, then it doesn't throw an exception

Would you mind to share with us a stack trace on the matter? Maybe there is some clue what to skip or replace to avoid an exception?

[xfrancois]

Indeed, here is the stacktrace :

2020-03-30 19:05:21,008 ERROR [scheduling-1] org.springframework.integration.handler.LoggingHandler:  org.springframework.messaging.MessagingException: nested exception is java.lang.IllegalStateException: S3 client with invalid S3 endpoint configured: localhost:9001
	at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:342)
	at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$null$1(AbstractPollingEndpoint.java:275)
	at org.springframework.integration.util.ErrorHandlingTaskExecutor.lambda$execute$0(ErrorHandlingTaskExecutor.java:57)
	at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
	at org.springframework.integration.util.ErrorHandlingTaskExecutor.execute(ErrorHandlingTaskExecutor.java:55)
	at org.springframework.integration.endpoint.AbstractPollingEndpoint.lambda$createPoller$2(AbstractPollingEndpoint.java:272)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: S3 client with invalid S3 endpoint configured: localhost:9001
	at com.amazonaws.services.s3.AmazonS3Client.getRegion(AmazonS3Client.java:4270)
	at org.springframework.integration.aws.support.S3Session.getHostPort(S3Session.java:228)
	at org.springframework.integration.file.remote.AbstractRemoteFileStreamingMessageSource.doReceive(AbstractRemoteFileStreamingMessageSource.java:214)
	at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:167)
	at org.springframework.integration.endpoint.SourcePollingChannelAdapter.receiveMessage(SourcePollingChannelAdapter.java:250)
	at org.springframework.integration.endpoint.AbstractPollingEndpoint.doPoll(AbstractPollingEndpoint.java:359)
	at org.springframework.integration.endpoint.AbstractPollingEndpoint.pollForMessage(AbstractPollingEndpoint.java:328)
	... 13 more

[artembilan]

OK, I see now!

Thanks.

So, how about something like setEndpoint(String serviceEndpoint) for the S3SessionFactory with propagation to its S3Session and returning that value if present and only after that go to the Region parsing?

I know you would need to duplicate your http://localhost:9001, but there is no other way to access to the provided custom endpoint internally.
Probably one more improvement request for AWS SDK...

[xfrancois]

Yes, I think it's a good idea, there is currently no better alternative than this solution.
I will try to raise an issue on aws-sdk side and link it here

[artembilan]

Good.
Since we are on the same page now, @xfrancois , would you mind to contribute a fix as we have discussed?
Thanks

[xfrancois]

Yes, I did it with a new constructor instead of a setter, hope it's ok.