New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3: support custom endpoint - WIP #619

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
5 participants
@takashima0411

takashima0411 commented Dec 2, 2017

#181
WIP.
get unexpected exceptions.
at use fakes3 on localhost.

at fileupload

[info]   akka.stream.alpakka.s3.impl.FailedUpload: Cannot find etag
[info]   at akka.stream.alpakka.s3.impl.S3Stream.$anonfun$completionSink$2(S3Stream.scala:237)
[info]   at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:302)
[info]   at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
[info]   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
[info]   at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
[info]   at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[info]   at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
[info]   at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
[info]   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
[info]   at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
[info]   at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[info]   at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[info]   at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[info]   at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

at download

[info] - should support custom endpoint configured by `baseUrl`2 *** FAILED *** (842 milliseconds)
[info]   akka.stream.alpakka.s3.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
[info]   at akka.stream.alpakka.s3.impl.S3Stream.$anonfun$entityForSuccess$1(S3Stream.scala:263)
[info]   at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:302)
[info]   at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
[info]   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
[info]   at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
[info]   at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[info]   at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
[info]   at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
[info]   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
[info]   at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
[info]   at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[info]   at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[info]   at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[info]   at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
@lightbend-cla-validator

This comment has been minimized.

Show comment
Hide comment
@lightbend-cla-validator

lightbend-cla-validator Dec 2, 2017

Hi @takashima0411,

Thank you for your contribution! We really value the time you've taken to put this together.

Before we proceed with reviewing this pull request, please sign the Lightbend Contributors License Agreement:

http://www.lightbend.com/contribute/cla

lightbend-cla-validator commented Dec 2, 2017

Hi @takashima0411,

Thank you for your contribution! We really value the time you've taken to put this together.

Before we proceed with reviewing this pull request, please sign the Lightbend Contributors License Agreement:

http://www.lightbend.com/contribute/cla

@ennru ennru added the p:aws-s3 label Dec 4, 2017

@ennru ennru changed the title from support custom endpoint to S3: support custom endpoint Dec 4, 2017

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Dec 5, 2017

Member

FYI team, this is an incomplete PR which aimed to resolve #181 by providing an "override" for the target Uri for the s3 requests. It's from an hakkathon we did here in Tokyo, so would be nice to try to help out here, I'm not completely up to date with Alpakka's S3 so would request some help in reviewing this :)

Member

ktoso commented Dec 5, 2017

FYI team, this is an incomplete PR which aimed to resolve #181 by providing an "override" for the target Uri for the s3 requests. It's from an hakkathon we did here in Tokyo, so would be nice to try to help out here, I'm not completely up to date with Alpakka's S3 so would request some help in reviewing this :)

@@ -17,15 +17,17 @@ final case class S3Settings(bufferType: BufferType,
proxy: Option[Proxy],
credentialsProvider: AWSCredentialsProvider,
s3Region: String,
pathStyleAccess: Boolean) {
pathStyleAccess: Boolean,
baseUrl: Option[String]) {

This comment has been minimized.

@ktoso

ktoso Dec 5, 2017

Member

In this impl this acts more like an "override s3 base url", which could be fine.
An alternative is a bit more complex, and would be alternative mappings for the s3Region strings -- and we'd by default put the global known s3 urls in there.

@ktoso

ktoso Dec 5, 2017

Member

In this impl this acts more like an "override s3 base url", which could be fine.
An alternative is a bit more complex, and would be alternative mappings for the s3Region strings -- and we'd by default put the global known s3 urls in there.

This comment has been minimized.

@easel

easel Dec 30, 2017

Contributor

I think it's probably best to go with the approach in place now. Minio, for instance, expects the region to be set to us-east-1 while using their custom endpoint.

@easel

easel Dec 30, 2017

Contributor

I think it's probably best to go with the approach in place now. Minio, for instance, expects the region to be set to us-east-1 while using their custom endpoint.

@francisdb francisdb referenced this pull request Dec 14, 2017

Merged

Adds S3 getObjectMetadata, deleteObject, putObject #563

5 of 5 tasks complete

@ennru ennru changed the title from S3: support custom endpoint to S3: support custom endpoint - WIP Dec 22, 2017

@ennru

This comment has been minimized.

Show comment
Hide comment
@ennru

ennru Jan 3, 2018

Member

#677 replaces this PR by another author.
Please chime in on that to bring this functionality to Alpakka.

Member

ennru commented Jan 3, 2018

#677 replaces this PR by another author.
Please chime in on that to bring this functionality to Alpakka.

@ennru ennru closed this Jan 3, 2018

@ennru ennru referenced this pull request Jan 3, 2018

Closed

S3: custom s3 endpoint #181

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment