Skip to content

Path style breaks fetching INDEX_PAGE #210

Open
@HighOnMikey

Description

@HighOnMikey

loadContent will always attempt fetching the INDEX_PAGE by building the URI path using s3uri().

When using S3_STYLE = 'path', this will prepend the URI path with the S3 bucket. However, the fetch is made against nginx itself, meaning the URI path will always be incorrect for the requested index.

To Reproduce

Set the following variables:

ALLOW_DIRECTORY_LIST=true
PROVIDE_INDEX_PAGE=true
S3_STYLE=path

Expected behavior

Incorrect URI currently being built: http://127.0.0.1/<bucket_name>/<request_path>/<INDEX_PAGE>

Expected URI: http://127.0.0.1/<request_path>/<INDEX_PAGE>

Environments Tested

  • master @ c687663
  • ghcr.io/nginxinc/nginx-s3-gateway/nginx-oss-s3-gateway:latest
  • Backends: AWS, Ceph, OpenStack Swift

Workaround

I would submit a PR but I'm not sure which direction would be preferred by the project to go about building the URI path. However, here's a quick fix to demonstrate the issue:

From

const uri = s3uri(r);

To

let uri;
if (S3_STYLE === 'path') {
    uri = `${r.uri}${INDEX_PAGE}`;
} else {
    uri = s3uri(r);
}

Activity

4141done

4141done commented on Feb 8, 2024

@4141done
Collaborator

Thank you for the report @HighOnMikey , I'm a new maintainer on this project so let me take some time to understand that specific area more deeply and reply.

4141done

4141done commented on Feb 15, 2024

@4141done
Collaborator

Hi @HighOnMikey thank you for your patience.
I took a look at this and want to be sure that I understand the issue. It's possible I'm misunderstanding some part of your report:

  1. S3 path style requests expect the request in the form https://s3.region-code.amazonaws.com/bucket-name/key-name
  2. The PROVIDE_INDEX_PAGE configuration option is intended to serve a page called index.html from the requested directory. For example, if I have a bucket called test-bucket, then a request to <s3 gateway host>/ would be expected to be proxied to s3 as https://s3.region-code.amazonaws.com/test-bucket/index.html

I think that's the shared understanding we need. Testing the code I didn't see any indication that it's not doing that but I could be missing a key part of your report so just wanting to be sure.

Here are some logs from my local testing requesting /foo with the configuration you specified (I added some additional debug prints):

2024/02/15 22:00:15 [info] 75#75: *35 js: Using path style uri : /bucket-1
2024/02/15 22:00:15 [info] 75#75: *35 js: s3uri: Internal: true | uri: / | uriPath: / | basePath: /bucket-1
2024/02/15 22:00:15 [info] 75#75: *35 js: S3 Request URI: GET /bucket-1/index.html
2024/02/15 22:00:15 [info] 75#75: *37 js: Using path style uri : /bucket-1
2024/02/15 22:00:15 [info] 75#75: *37 js: s3uri: Internal: false | uri: /bucket-1/index.html | uriPath: /bucket-1/index.html | basePath: /bucket-1
2024/02/15 22:00:15 [info] 75#75: *37 js: S3 Request URI: GET /bucket-1/bucket-1/index.html
127.0.0.1 - - [15/Feb/2024:22:00:15 +0000] "GET /bucket-1/index.html HTTP/1.1" 404 146 "-" "-" "-" "/bucket-1/bucket-1/index.html" "/bucket-1/index.html"
2024/02/15 22:00:15 [info] 75#75: *35 js: Using path style uri : /bucket-1
2024/02/15 22:00:15 [info] 75#75: *35 js: s3uri: Internal: true | uri: / | uriPath: / | basePath: /bucket-1
2024/02/15 22:00:15 [info] 75#75: *35 js: S3 Request URI: GET /bucket-1?delimiter=%2F
192.168.65.1 - - [15/Feb/2024:22:00:15 +0000] "GET / HTTP/1.1" 200 1100 "-" "curl/8.4.0" "-" "/bucket-1?delimiter=%2F" "/"

I think I see the error here, when we redirect to /bucket-name/index.html the helper prepends the bucket name again:
2024/02/15 22:00:15 [info] 75#75: *37 js: S3 Request URI: GET /bucket-1/bucket-1/index.html

Is this the bug you're referring to or have I found a different bug? 😓

HighOnMikey

HighOnMikey commented on Feb 28, 2024

@HighOnMikey
Author

First of all, sorry for the delayed response!

I think that's the shared understanding we need.

Correct. The core functionality of path style requests works as intended. The bug is limited to attempting to find an index page when PROVIDE_INDEX_PAGE is enabled and S3_STYLE=path.

Is this the bug you're referring to or have I found a different bug? 😓

Yes, this is directly related to the bug. The loadContent function will use the local nginx proxy to test fetch to see if an index page exists, but it uses the s3uri function which is designed for building a URI to request from the S3 API directly.

4141done

4141done commented on Mar 14, 2024

@4141done
Collaborator

Thanks for confirming and for your patience. I'm going to try to put together a fix in the coming days. I'll tag you for a test/review when it's up.

4141done

4141done commented on Apr 19, 2024

@4141done
Collaborator

Hey @HighOnMikey sorry again for the slowness on this. Can you take a look at #230 and see if it solves your issue?

Let me know if you need a container image built for you to test.

OSobky

OSobky commented on Dec 20, 2024

@OSobky

Hello @4141done, is there a reason this MR is not merged yet? I confirm that the latest image (latest-20241216) still have the same issue if ALLOW_DIRECTORY_LIST=true & PROVIDE_INDEX_PAGE=true are enabled at the same time.

however if we have ALLOW_DIRECTORY_LIST=false & PROVIDE_INDEX_PAGE=true serving the index.html file works fine.

Note: I am using path style S3_STYLE=path

HighOnMikey

HighOnMikey commented on Apr 9, 2025

@HighOnMikey
Author

Hey @HighOnMikey sorry again for the slowness on this. Can you take a look at #230 and see if it solves your issue?

Let me know if you need a container image built for you to test.

I applied 230 to the latest commit in main, built the OSS image, and it does fix the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @4141done@HighOnMikey@OSobky

      Issue actions

        Path style breaks fetching INDEX_PAGE · Issue #210 · nginx/nginx-s3-gateway