Unknown Host Exception - Internal 500 Backend Error with MinIO S3

[JustMitah]

I didn't want to open a new issue seeing as this is still open, but I didn't know if it was showing up.
Is it normal that penpot throws an error when I try to change region to eu-east-1?

Execution error (ExceptionInfo) at integrant.core/spec-exception (core.cljc:398).
Spec failed on key [:app.main/assets :app.storage.s3/backend] when building system
:eu-east-1 - failed: #{:eu-central-1} in: [:region] at: [:region] spec: :app.storage.s3/region
Full report at:
/tmp/clojure-2853288006993469350.edn

It works with eu-central-1 but when I upload an image in penpot it throws an exception :

[2022-06-23 13:03:38.683] E app.http.errors - Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.

software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102) ~[penpot.jar:?]
	at software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor.modifyException(HelpfulUnknownHostExceptionInterceptor.java:59) ~[penpot.jar:?]
	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:199) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:49) ~[penpot.jar:?]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[penpot.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:181) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[penpot.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[penpot.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103) ~[penpot.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2158) ~[?:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165) ~[penpot.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479) ~[?:?]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:295) ~[?:?]
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1016) ~[?:?]
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1665) ~[?:?]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1598) ~[?:?]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) ~[?:?]

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: my-bucket-name.my.domain.name
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:102) ~[penpot.jar:?]
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[penpot.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:179) ~[penpot.jar:?]

From this previous log it's this line that bother me a bit :
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: my-bucket-name.my.domain.name
I tested with Filestash S3 so I know that my bucket is accessible with the aws key/ID and endpoint I provided.
Any ideas?

[niwinz]

About the region, this is a bug and it will be landed in the next minor release (today or monday) (PR here)

About the unknown host, I don't know, maybe the inside the docker environment the container is unable to resolve the domain you using for the s3 backend uri...

[JustMitah]

I see, guess I'll wait.
As for the host part, I don't understand your remark, does that mean I might be wrong about the URI? the only difference is the format https://s3.company.name is the link , so without any port, but I don't see how that will throw that kind of error, weird.

After googling a bit, I found this spinnaker/spinnaker#4431 (comment) (the link mentioned hal having access path parameter) Does penpot have an access path parameter that I can configure? So that it becomes https://s3.company.name/bucket_name instead of bucket_name.s3.company.name

[niwinz]

As I have posted in the previous issue, I locally use this config:

export AWS_ACCESS_KEY_ID=penpot-devenv
export AWS_SECRET_ACCESS_KEY=penpot-devenv

export PENPOT_ASSETS_STORAGE_BACKEND=assets-s3
export PENPOT_STORAGE_ASSETS_S3_ENDPOINT=http://minio:9000
export PENPOT_STORAGE_ASSETS_S3_BUCKET=penpot
# This setting will not be necessary on the next minor release (1.14.1 I guess).
export PENPOT_STORAGE_ASSETS_S3_REGION=eu-central-1

For testing penpot with minio on local devenv, and I have configured nothing for serving under different domain, so I'm pretty sure that path is used for specify the bucket.

[funkybunch]

Might not be necessary with the pending minor release, but FWIW as a workaround I named my Minio region as eu-central-1 and it worked.

Good to know I can set it back to what it was with this next release though.

[funkybunch]

As for the host part, I don't understand your remark, does that mean I might be wrong about the URI? the only difference is the format https://s3.company.name is the link , so without any port, but I don't see how that will throw that kind of error, weird.

@JustMitah I believe the port is required when using Minio since it runs the S3 service on port 9000. You could try proxying it to a standard web port, but I was only able to get the URI to resolve by specifying the port number in addition to the protocol and hostname.

[niwinz]

Might not be necessary with the pending minor release, but FWIW as a workaround I named my Minio region as eu-central-1 and it worked.

Good to know I can set it back to what it was with this next release though.

The 1.14.1 is released (already on dockerhub) with the region validation fix, now you can omit it or set whatever value you need.

[JustMitah]

Lets say that the link/storage we want to use is minio.company.name, I created a minio-test.company.name on another server...and it works just fine, so no port or region need to be specified.....
Configuration wise the two minio are the same (except one run with LDAP and has more space and the test one doesn't), minio-version wise? the test one is barely older than 9 days, so that can't be it either.
I'm still trying to figure out the issue, but thanks for the help, I'll come back with a solution once I find it.

[JustMitah]

Hello @niwinz I have a question :
https://github.com/penpot/penpot/blob/65b6d1e07bc63fae5bab2243bf2d70aeef09d568/backend/scripts/repl
In this bit of code you initialized the access key and such with admin privileges.
Like I mentioned before, the test run I did on another minio instance was done through the admin account, and uploading images worked without a problem.
But in the official company Minio, we linked it through LDAP, so I'm creating service accounts with my ldap account instead of admin.
Could it be the root of the issue? because otherwise i really cannot understand why penpot will work with an instance of minio but not another.

[JustMitah]

While I still don't know why penpot will work correctly with one instance and not with another, I fixed my problem, thanks for all the help again.

If anyone run into a similar issue working with public urls, add the following environment variable to Minio : "MINIO_DOMAIN=your_minio_reverse_proxy_domain_name" and then add to your reverse proxy the following domain "yourbucket.domain.name" this will make your bucket finally accessible to penpot as it doesn't seem the use the traditional path style.