How to set s3 region? #82

deluxghost · 2022-04-19T19:37:36Z

I'm using a local minio server with a custom region, while Send is using the default us-east-1 value. Therefore the s3 authorization header is wrong and denied by minio server

Is there any way to set the region? If not, please add a config.

The text was updated successfully, but these errors were encountered:

timvisee · 2022-04-20T14:21:12Z

You should be able to set this using these variables:

send/server/config.js

Lines 27 to 41 in b9faece

    
           s3_bucket: { 
        
             format: String, 
        
             default: '', 
        
             env: 'S3_BUCKET' 
        
           }, 
        
           s3_endpoint: { 
        
             format: String, 
        
             default: '', 
        
             env: 'S3_ENDPOINT' 
        
           }, 
        
           s3_use_path_style_endpoint: { 
        
             format: Boolean, 
        
             default: false, 
        
             env: 'S3_USE_PATH_STYLE_ENDPOINT' 
        
           },

Does that help?

deluxghost · 2022-04-20T14:56:27Z

You should be able to set this using these variables:

send/server/config.js

Lines 27 to 41 in b9faece

s3_bucket: {

format: String,

default: '',

env: 'S3_BUCKET'

},

s3_endpoint: {

format: String,

default: '',

env: 'S3_ENDPOINT'

},

s3_use_path_style_endpoint: {

format: Boolean,

default: false,

env: 'S3_USE_PATH_STYLE_ENDPOINT'

},

Does that help?

hmm, i dont see a region config, only endpoint, bucket and path style.

timvisee · 2022-04-20T14:59:58Z

I must admit that I'm not too familiar with S3. Doesn't setting a custom endpoint allow you to set the region as well (URL prefix)?

If not, I'll probably be able to take a look into this after the weekend.

deluxghost · 2022-04-20T15:14:11Z

I must admit that I'm not too familiar with S3. Doesn't setting a custom endpoint allow you to set the region as well (URL prefix)?

If not, I'll probably be able to take a look into this after the weekend.

Im not very familiar with that too, looks like there is no standard for s3 url yet

other apps like gitlab support s3 region config. If you are using a 3rdparty lib, it might be easy to implement

ihatemyisp · 2022-05-24T00:40:07Z

I'll be the first to admit my S3 knowledge is severely lacking but I use Send (docker compose) with Minio. The S3 Region I have set in Minio is not the one Send uses by default, but it works.

Do you have S3_USE_PATH_STYLE_ENDPOINT set to "true"?

I did have to change one section of the IAM policy that @timvisee has on GitLab to:

...
"Statement": [
        {
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::<s3_bucket_name>"
            ],
            "Effect": "Allow"
        },
...

NamPNQ · 2022-07-04T18:26:56Z

Use can set environment variable S3_USE_PATH_STYLE_ENDPOINT=true to fix your problems, this is not related region

boomam · 2023-05-22T18:32:15Z

Having a similar issue, the path style variable however does not solve the issue.
Are there any known workarounds?

lilas-dev · 2023-06-02T10:02:51Z

Having a similar issue, the path style variable however does not solve the issue. Are there any known workarounds?

I think you have make some mistake on your config, I just try a simple setup, its work like a charm

version: "3"

services:
  send:
    image: registry.gitlab.com/timvisee/send:latest
    ports:
      - '1234:1234'
    environment:
      - NODE_ENV=production
      - BASE_URL=http://localhost:1234
      - PORT=1234
      - REDIS_HOST=redis
      - AWS_ACCESS_KEY_ID=iloveyou
      - AWS_SECRET_ACCESS_KEY=andyoudontloveme
      - S3_BUCKET=send
      - S3_ENDPOINT=http://minio:9000
      - S3_USE_PATH_STYLE_ENDPOINT=true
    depends_on:
      minio:
        condition: service_healthy
  redis:
    image: redis
  minio:
    image: minio/minio
    command: ['server', '/data']
    environment:
      - MINIO_ROOT_USER=iloveyou
      - MINIO_ROOT_PASSWORD=andyoudontloveme
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
  mc:
    image: minio/mc
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set myminio http://minio:9000 iloveyou andyoudontloveme;
      /usr/bin/mc mb myminio/send;
      "

boomam · 2023-06-02T11:59:28Z

Having a similar issue, the path style variable however does not solve the issue. Are there any known workarounds?

I think you have make some mistake on your config, I just try a simple setup, its work like a charm

Can you confirm, what region do you have your Minio set too?
To be clear, the issue is not connecting it to Minio, but connecting to Minio where the region is not the default - as there is no variable in 'send' right now to set that.

lilas-dev · 2023-06-02T14:39:09Z

@boomam I dont see Minio support region https://min.io/docs/minio/linux/reference/minio-server/minio-server.html#environment-variables

Can you post your config and the log of error? Its hard to understand what happened without this.

boomam · 2023-06-02T14:49:52Z

@boomam I dont see Minio support region https://min.io/docs/minio/linux/reference/minio-server/minio-server.html#environment-variables

Correct, but you still have to specify it on the connection.
It's the same with other apps that can use S3, like Pydio or NextCloud.

Can you post your config and the log of error? Its hard to understand what happened without this.

I'd have to loop back in a bit to generate the error again, I ended up giving up with Send and deployed something else.

Regardless, other than the URL the config for S3 is the same as yours - just not specifying port 9000 being the only difference, as I have all my traffic go through a load balancer on 443/tcp. So the endpoint instead reads as 'https://minio.domain.com'.

To be clear on that aspect, it works with other apps - where I can specify the region in use.
So its not a LB issue or a Minio config issue. ;-)

Your other config examples, such as how you are deploying Minio, are not valid for me as they are on separate systems.

boomam · 2023-06-02T15:03:07Z

Had to tidy it up, it dumps it all on one line for some reason in STOUT -

Timestamp=1685717940242000000 
Logger=FirefoxSend 
Type=send.upload.upload 
Severity=2 
Pid=1 
EnvVersion=2.0 
Fields={"error":"UnknownEndpoint: Inaccessible 
host: `' at port `443'. This service may not be available in the `us-east-1' region.","stack":"\n    
at Request.ENOTFOUND_ERROR (/app/node_modules/aws-sdk/lib/event_listeners.js:611:46)\n    
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)\n    
at error (/app/node_modules/aws-sdk/lib/event_listeners.js:443:22)\n    
at ClientRequest.<anonymous> (/app/node_modules/aws-sdk/lib/http/node.js:99:9)\n    
at ClientRequest.emit (node:events:390:28)\n    
at ClientRequest.emit (node:domain:475:12)\n    
at TLSSocket.socketErrorListener (node:_http_client:447:9)\n    at TLSSocket.emit (node:events:390:28)"}

Connectivity between container and service is confirmed, it can get to Minio/S3.
It's the region its tripping up on.

lilas-dev · 2023-06-02T19:32:11Z

@boomam From the logs, I think the problem maybe about your config of ` S3_ENDPOINT` Le ven. 2 juin 2023 à 17:03, boomam ***@***.***> a écrit :

…

Had to tidy it up, it dumps it all on one line for some reason in STOUT - Timestamp=1685717940242000000 Logger=FirefoxSend Type=send.upload.upload Severity=2 Pid=1 EnvVersion=2.0 Fields={"error":"UnknownEndpoint: Inaccessible host: `' at port `443'. This service may not be available in the `us-east-1' region.","stack":"\n at Request.ENOTFOUND_ERROR (/app/node_modules/aws-sdk/lib/event_listeners.js:611:46)\n at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)\n at error (/app/node_modules/aws-sdk/lib/event_listeners.js:443:22)\n at ClientRequest.<anonymous> (/app/node_modules/aws-sdk/lib/http/node.js:99:9)\n at ClientRequest.emit (node:events:390:28)\n at ClientRequest.emit (node:domain:475:12)\n at TLSSocket.socketErrorListener (node:_http_client:447:9)\n at TLSSocket.emit (node:events:390:28)"} Connectivity between container and service is confirmed, it can get to Minio/S3. It's the region its tripping up on. — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQ34G6OJGQPMLMXMQVDTQZLXJH6DNANCNFSM5TZ3FNNQ> . You are receiving this because you commented.Message ID: ***@***.***>

boomam · 2023-06-02T19:37:36Z

@boomam From the logs, I think the problem maybe about your config of S3_ENDPOINT Le ven. 2 juin 2023 à 17:03, boomam @.***> a écrit :

Can you expand on this please, as it looks like your comment may have been cut off?

The S3_ENDPOINT config i am using, is in the format of -
- S3_ENDPOINT='https://minio.domain.com'

I have also tested other encapsulations, too. Along with specifying and not specifying 443.
The issue persists regardless of format.

lilas-dev · 2023-06-02T19:50:43Z

@boomam From the logs, I think the problem maybe about your config of S3_ENDPOINT Le ven. 2 juin 2023 à 17:03, boomam @.***> a écrit :

Can you expand on this please, as it looks like your comment may have been cut off? The S3_ENDPOINT config i am using, is in the format of - - S3_ENDPOINT='https://minio.domain.com'

I have also tested other encapsulations, too. Along with specifying and not specifying 443. The issue persists regardless of format.

Please try this

 S3_ENDPOINT=minio.domain.com

boomam · 2023-06-02T19:53:17Z

@boomam From the logs, I think the problem maybe about your config of S3_ENDPOINT Le ven. 2 juin 2023 à 17:03, boomam @.***> a écrit :

Can you expand on this please, as it looks like your comment may have been cut off? The S3_ENDPOINT config i am using, is in the format of - - S3_ENDPOINT='https://minio.domain.com'
I have also tested other encapsulations, too. Along with specifying and not specifying 443. The issue persists regardless of format.

Please try this
 S3_ENDPOINT=minio.domain.com

Already have, noted above. ;-)

To re-illiterate in case it was missed, the connection is fine, I can see the connection even in Minio's logs.
Its the region.

boomam · 2023-06-02T19:56:37Z

For thoroughness, re-tested regardless -
Two errors in the log, one is the same one from above, and the second is this -

Timestamp=1685735679756000000 
Logger=FirefoxSend 
Type=send.upload.upload 
Severity=2 
Pid=1 
EnvVersion=2.0 Fields={"error":"SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.","stack":"\n    
at Request.extractError (/app/node_modules/aws-sdk/lib/services/s3.js:711:35)\n    
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)\n    
at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)\n    
at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)\n    
at /app/node_modules/aws-sdk/lib/state_machine.js:26:10\n    
at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)\n    
at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:688:12)\n    
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"}

lilas-dev · 2023-06-02T20:03:06Z

Try check your config of creds, its should not wrap the string in quote

boomam · 2023-06-02T20:07:50Z

Try check your config of creds, its should not wrap the string in quote
To confirm latest tests, variables -

S3_ENDPOINT=minio.domain.com
AWS_SECRET_ACCESS_KEY=1234abc

Same issue.

boomam · 2023-06-02T20:08:43Z

lol, gotta laugh - i solved it - regenerating the key to not have special characters. :-p
Thats super-wierd, as wrapping the string in quotes, or not, regardless didnt work, but having a fully alphanumeric string, worked fine.

timvisee added question Further information is requested support labels Apr 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to set s3 region? #82

How to set s3 region? #82

deluxghost commented Apr 19, 2022

timvisee commented Apr 20, 2022

deluxghost commented Apr 20, 2022

timvisee commented Apr 20, 2022

deluxghost commented Apr 20, 2022

ihatemyisp commented May 24, 2022

NamPNQ commented Jul 4, 2022

boomam commented May 22, 2023

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023 •

edited

Loading

boomam commented Jun 2, 2023

lilas-dev commented Jun 2, 2023 via email •

edited

Loading

boomam commented Jun 2, 2023

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023

boomam commented Jun 2, 2023 •

edited

Loading

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023

boomam commented Jun 2, 2023

How to set s3 region? #82

How to set s3 region? #82

Comments

deluxghost commented Apr 19, 2022

timvisee commented Apr 20, 2022

deluxghost commented Apr 20, 2022

timvisee commented Apr 20, 2022

deluxghost commented Apr 20, 2022

ihatemyisp commented May 24, 2022

NamPNQ commented Jul 4, 2022

boomam commented May 22, 2023

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023 • edited Loading

boomam commented Jun 2, 2023

lilas-dev commented Jun 2, 2023 via email • edited Loading

boomam commented Jun 2, 2023

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023

boomam commented Jun 2, 2023 • edited Loading

lilas-dev commented Jun 2, 2023

boomam commented Jun 2, 2023

boomam commented Jun 2, 2023

boomam commented Jun 2, 2023 •

edited

Loading

lilas-dev commented Jun 2, 2023 via email •

edited

Loading

boomam commented Jun 2, 2023 •

edited

Loading