CORS 'Access-Control-Allow-Origin' not included in response

[zbychu-waligruchga]

Minio does not include the CORS header Access-Control-Allow-Origin breaking CORS functionality.

Expected Behavior

Pre-Signed URL PUT/GET endpoints to return 'Access-Control-Allow-Origin' if request contain 'Origin' header.

Current Behavior

'Access-Control-Allow-Origin' is not included when interacting with pre-signed URLs

Possible Solution

Include 'Access-Control-Allow-Origin' in every response

Steps to Reproduce (for bugs)

1. Obtain pre-signed URL for a file
2. Upload a file using PUT method with 'Origin' header included
3. Check response headers

Tested with and without preflight queries

Insomnia PUT request example

Regression

I believe this issue is a regression as it was discussed and reportedly working: #3985

Your Environment

• Version used: RELEASE.2020-12-12T08-39-07Z

[harshavardhana]

'Access-Control-Allow-Origin' is not included when interacting with pre-signed URLs

It seems like it's always set @zbychu-waligruchga - are you sure you are using the right release?

curl -i -H "Origin: http://test.com" --upload /etc/hosts https://play.minio.io:9000/testbucket/object
HTTP/2 403 
accept-ranges: bytes
access-control-allow-credentials: true
access-control-allow-origin: http://test.com
access-control-expose-headers: Date, Etag, Server, Connection, Accept-Ranges, Content-Range, Content-Encoding, Content-Length, Content-Type, Content-Disposition, Last-Modified, Content-Language, Cache-Control, Retry-After, X-Amz-Bucket-Region, Expires, X-Amz*, X-Amz*, *
content-security-policy: block-all-mixed-content
content-type: application/xml
server: MinIO
vary: Origin
x-amz-request-id: 1651110590F5E8DA
x-xss-protection: 1; mode=block
content-length: 295
date: Wed, 16 Dec 2020 02:20:57 GMT

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Key>object</Key><BucketName>testbucket</BucketName><Resource>/testbucket/object</Resource><RequestId>1651110590F5E8DA</RequestId><HostId>129c19c9-4cf6-44ff-9f2d-4cb7611be894</HostId></Error>```

[harshavardhana]

Even did presigned GET

~ curl -v -H "Origin: http://test.com" "https://play.min.io/testbucket/admin.py?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=Q3AM3
UQ867SPQQA43P2F%2F20201216%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201216T022213Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=70dd481acde780929a096e34f4261821dbbdb9084
429785f2b021d4c0b8c88d3" -o /dev/null

< HTTP/1.1 200 OK
< Server: nginx/1.14.0 (Ubuntu)
< Date: Wed, 16 Dec 2020 02:23:14 GMT
< Content-Type: application/octet-stream
< Content-Length: 8432
< Connection: keep-alive
< Accept-Ranges: bytes
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: http://test.com
< Access-Control-Expose-Headers: Date, Etag, Server, Connection, Accept-Ranges, Content-Range, Content-Encoding, Content-Length, Content-Type, Content-Disposition, Last-Modified, Content-Language, Cache-Control, Retry-After, X-Amz-Bucket-Region, Expires, X-Amz*, X-Amz*, *
< Content-Security-Policy: block-all-mixed-content
< ETag: "0b14fcb7b78d0435746c688c9eca9fcd"
< Last-Modified: Tue, 15 Dec 2020 19:36:28 GMT
< Vary: Origin
< X-Amz-Request-Id: 1651112559C80F99
< X-Xss-Protection: 1; mode=block
< x-amz-meta-mc-attrs: atime:1608060882#433680445/gid:1000/gname:harsha/mode:33204/mtime:1608060882#365680700/uid:1000/uname:harsha

[mojotalantikite]

@harshavardhana I'm actually running into something similar right now, where pre-flight checks from the browser are throwing CORS errors with CORS header ‘Access-Control-Allow-Origin’ missing. When I check via curl I see:

curl -H "Origin: http://minio:3000" \                                                                                                                       
  -H "Access-Control-Request-Method: PUT" \
  -H "Access-Control-Request-Headers: X-Requested-With" \
  -X OPTIONS --verbose http://localhost:9000/testbucket
*   Trying ::1...                                                                                                                                                                                                                             * TCP_NODELAY set
* Connected to localhost (::1) port 9000 (#0)
> OPTIONS /testbucket HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.64.1
> Accept: */*
> Origin: http://minio:3000
> Access-Control-Request-Method: PUT
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 200 OK
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Date: Thu, 17 Dec 2020 04:01:16 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0

Similarly to @zbychu-waligruchga, this is happening on pre-signed URLs from my backend application giving access to a frontend to interact with files. Requests are failing the pre-flight browser check. Using curl on the cli with the presigned urls directly work fine without the pre-flight check.

[harshavardhana]

@mojotalantikite here is what I sent to https://play.minio.io:9000

~ curl -H "Origin: http://minio:3000" -H "Access-Control-Request-Method: PUT" -H "Access-Control-Request-Headers: X-Requested-With" -X OPTIONS --verbose https://play.minio.io:9000/testbucket
...
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=play.minio.io
*  start date: Oct 26 08:38:35 2020 GMT
*  expire date: Jan 24 08:38:35 2021 GMT
*  subjectAltName: host "play.minio.io" matched cert's "play.minio.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f6c9b06e80)
> OPTIONS /testbucket HTTP/2
> Host: play.minio.io:9000
> user-agent: curl/7.68.0
> accept: */*
> origin: http://minio:3000
> access-control-request-method: PUT
> access-control-request-headers: X-Requested-With
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200 
< access-control-allow-credentials: true
< access-control-allow-headers: X-Requested-With
< access-control-allow-methods: PUT
< access-control-allow-origin: http://minio:3000
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< content-length: 0
< date: Thu, 17 Dec 2020 05:30:09 GMT
< 
* Connection #0 to host play.minio.io left intact

Don't even know what version of the software are you using here.

[mojotalantikite]

"Don't even know what version of the software are you using here."

Thanks for your response @harshavardhana! This was running in the latest docker container in my local dev environment.

So it looks like things were running fine when I was doing this from a docker container spun up on the CLI with a normal docker run -p 9000:9000 minio/minio server /data. For whatever reason when I switched over to run this in my application's docker-compose.yml file it started throwing these CORS errors.

Previously I was doing something like this:

 minio:
    restart: always
    image: minio/minio:latest
    ports:
      - "9000:9000"
    volumes:
      - s3-data:/data
    entrypoint: minio server /data
 minio_mc:
    image: minio/mc:latest
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host rm local;
      /usr/bin/mc config host add --quiet --api s3v4 local http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc rb --force local/test-bucket/;
      /usr/bin/mc mb --quiet local/test-bucket/;
      /usr/bin/mc policy set public local/test-bucket;
      "
volumes:
  s3-data

This would throw CORS errors during pre-flight from the browser. However, when I delete the volumes configurations, everything works fine. Do you know why that would be?

This seems like unexpected behavior to me, so maybe this is something to document? I've seen a lot of issues opened in this repo's history of people having issues when doing pre-signed URLs for local testing -- which I'm guessing is a common use case for minio -- and I wouldn't be surprised if it's due to some docker configuration. Thanks again!

[harshavardhana]

Will check and get back.

[harshavardhana]

So I ran this docker compose

version: '3.7'

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio:
    restart: always
    image: minio/minio:latest
    ports:
      - "9000:9000"
    volumes:
      - s3-data:/data
    entrypoint: minio server /data
  minio_mc:
    image: minio/mc:latest
    depends_on:
      - minio
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc config host rm local;
      /usr/bin/mc config host add --quiet --api s3v4 local http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc rb --force local/test-bucket/;
      /usr/bin/mc mb --quiet local/test-bucket/;
      /usr/bin/mc policy set public local/test-bucket;
      "

volumes:
  s3-data:

~ docker-compose -f /tmp/docker.yaml up
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "tmp_default" with the default driver
Creating volume "tmp_s3-data" with default driver
Creating tmp_minio_1_aaa5c2bee149 ... done
Creating tmp_minio_mc_1_b3fde86862e0 ... done
Attaching to tmp_minio_1_4b5ad847baa7, tmp_minio_mc_1_eac342a305ed
minio_1_4b5ad847baa7 | 
minio_1_4b5ad847baa7 |  You are running an older version of MinIO released 1 week ago 
minio_1_4b5ad847baa7 |  Update: Run `mc admin update` 
minio_1_4b5ad847baa7 | 
minio_1_4b5ad847baa7 | 
minio_1_4b5ad847baa7 | Endpoint:  http://172.19.0.2:9000  http://127.0.0.1:9000
minio_1_4b5ad847baa7 | 
minio_1_4b5ad847baa7 | Browser Access:
minio_1_4b5ad847baa7 |    http://172.19.0.2:9000  http://127.0.0.1:9000
minio_1_4b5ad847baa7 | 
minio_1_4b5ad847baa7 | Object API (Amazon S3 compatible):
minio_1_4b5ad847baa7 |    Go:         https://docs.min.io/docs/golang-client-quickstart-guide
minio_1_4b5ad847baa7 |    Java:       https://docs.min.io/docs/java-client-quickstart-guide
minio_1_4b5ad847baa7 |    Python:     https://docs.min.io/docs/python-client-quickstart-guide
minio_1_4b5ad847baa7 |    JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
minio_1_4b5ad847baa7 |    .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide
minio_1_4b5ad847baa7 | Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ACCESS_KEY' and 'MINIO_SECRET_KEY'
minio_mc_1_eac342a305ed | Removed `local` successfully.
minio_mc_1_eac342a305ed | Added `local` successfully.
minio_mc_1_eac342a305ed | mc: <ERROR> Unable to validate target `local/test-bucket/`. Bucket `test-bucket` does not exist.
minio_mc_1_eac342a305ed | Bucket created successfully `local/test-bucket/`.
minio_mc_1_eac342a305ed | Access permission for `local/test-bucket` is set to `public`
tmp_minio_mc_1_eac342a305ed exited with code 0

~ curl -H "Origin: http://minio:9000" -H "Access-Control-Request-Method: PUT" -H "Access-Control-Request-Headers: X-Requested-With" -X OPTIONS --verbose http://localhost:9000/testbucket 
*   Trying 127.0.0.1:9000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9000 (#0)
> OPTIONS /testbucket HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.68.0
> Accept: */*
> Origin: http://minio:9000
> Access-Control-Request-Method: PUT
> Access-Control-Request-Headers: X-Requested-With
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Headers: X-Requested-With
< Access-Control-Allow-Methods: PUT
< Access-Control-Allow-Origin: http://minio:9000
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Date: Thu, 17 Dec 2020 17:18:04 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact

[SylvainBigonneau]

I was getting the exact same issue, somehow it got fixed by simply deleting the previous volume used by my minio container, and creating a new one. As soon as I did this, my requests were working again!

[nadilas]

I was getting the exact same issue, somehow it got fixed by simply deleting the previous volume used by my minio container, and creating a new one. As soon as I did this, my requests were working again!

This is not confirmed right? I don't want to loose my existing data just to test this out. Since a few updates I am getting the following error on preflight requests when using presigned URLs:
origin 'https://...' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

[mojotalantikite]

@nadilas correct, it hasn't been confirmed and the maintainers closed this out. My fix was the same, delete the volume and re-create it and the CORS errors went away. Can't promise that it'll work for you too, but if you can back the volume up and give it a try I'd be interested to hear if it clears things up for you as well. I still have a hunch that there is a bug somewhere.

[nadilas]

@mojotalantikite just to be clear it seems very distant and unrelated that what is on the mapped persistent volume can have an effect on what kind of headers are returned for preflight requests. 🤷🏼‍♂️ I already have the data backed up, I will try it tomorrow first thing... and try to restore the old data afterwards.

[mojotalantikite]

@nadilas yeah, agreed, it seems strange, but for whatever reason it worked for me too.

[nadilas]

@mojotalantikite I have not many ideas how it is possible, but it works. Recreating the volume got rid of the error. Now the header is included. There must be some saved old state which caused the issue!?

[harshavardhana]

@mojotalantikite I have not many ideas how it is possible, but it works. Recreating the volume got rid of the error. Now the header is included. There must be some saved old state which caused the issue!?

@nadilas most probably your container was not upgraded

[nadilas]

@harshavardhana if you mean the image used in the pods, I have explicitly updated the statefulset to pull the newest image after the issue presented itself.

[mojotalantikite]

@nadilas yeah, it's likely some edge case having to do with how minio writes config state and the order in which a user sets up the system. It seems like I had the same sequence of events that is described in #11258 -- if you set MINIO_ACCESS_KEY and MINIO_SECRET_KEY the CORS policy doesn't seem to get set in the config properly. I just gave it a shot again like this out of curiosity and there was no CORS header set, but if I delete the volume and remove the env variables the CORS header is there.

Minio writes configs to /${data-volume}/.minio.sys/config/config.json, so likely when everyone that's encountered this issue deletes that volume the config gets rewritten.

I don't know what else to say. I took a look around the code base, particularly around how the CORS handler seems to be set from the globalAPIConfig, but I didn't come across anything that'd explain this behavior. It looks like these env variables mess up their tests, so maybe there is something going on that is rare and not accounted for. 🤷🏻‍♂️

At the very least this issue history might help someone in the future and save them some time.

[grubbins]

Thanks all for this writeup. We are facing the same issue. However with 50TB of data in my cluster, and a number of users around the clock, I am not very keen on deleting the volume as a fix...

I am wondering if there's another way to trigger it to work. Can the config.json file be removed on its own? Would it then be recreated if we restart?

[frjonsen]

We were also facing CORS issues. We just switched to the Bitnami Helm chart, since the official one was deprecated. We were having CORS issues using both the old and the new chart. However, we noticed that when we did a helm upgrade the chart would also generate new credentials, which we found annoying. We creating a secret manually instead, and pointing out that one in our values.yaml, in the existingSecret field. Somehow we also stopped having CORS issues when we did this. Perhaps @mojotalantikite was on the right track that it's related to how credentials are saved.

Can't really confirm that this is a waterproof workaround, but it does seem to have somehow resolved it for us.

[nadilas]

@frjonsen Do you know what existingSecret translates to after having the helm chart rendered? We are not using helm charts, but I would like to try your workaround. As this issue not only comes up during updates, but even if the whole cluster goes down at once.

[harshavardhana]

This issue has been sufficiently resolved, if there are repeated problems please open a separate GitHub issue for a newer set of discussions.