Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix S3 service name parsing consuming the stream #9874

Merged
merged 5 commits into from Dec 22, 2023
Merged

Conversation

bentsku
Copy link
Contributor

@bentsku bentsku commented Dec 14, 2023

Motivation

As reported in #9867, we have an issue when using S3 pre-signed URL from a browser encoding the request as application/x-www-form-urlencoded and targeting a default LocalStack endpoint and not the S3 specific one. The ServiceNameParser will try to determine the AWS service, and as S3 is the "fallback" service, it will go through the whole service detection, which includes accessing request.values. This will:

  • consume the stream
  • if you send binary data, as the request is form encoded, Werkzeug will try to parse the form and fail decoding it as utf-8.
Traceback (most recent call last):
  File "/Users/benjaminsimon/Projects/localstack/localstack-ext/.venv/lib/python3.11/site-packages/werkzeug/formparser.py", line 284, in _parse_urlencoded
    stream.read().decode(),
    ^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 0: invalid start byte

Changes

When targeting S3, we have an early handler trying to determine if the request is targeting S3 by checking the bucket name and verifies that it exists. This will allow to actually skip the service name parsing, as we already set the service in the context there.

We also need to have the form parsing around a try...exception block in case the bucket does not exist, because the S3 CORS handler will not mark it as an S3 request, and the decoding of the form will fail. We will just swallow the exception and pass so that the service name parser tries the next step. This does not matter that it consumes the stream, because it will lead in an exception anyway.

@bentsku bentsku added aws:s3 Amazon Simple Storage Service area: asf semver: patch Non-breaking changes which can be included in patch releases labels Dec 14, 2023
@bentsku bentsku self-assigned this Dec 14, 2023
@bentsku bentsku linked an issue Dec 14, 2023 that may be closed by this pull request
1 task
Copy link

github-actions bot commented Dec 14, 2023

S3 Image Test Results (AMD64 / ARM64)

    2 files      2 suites   3m 8s ⏱️
383 tests 333 ✔️   50 💤 0
766 runs  666 ✔️ 100 💤 0

Results for commit 7cf1bd4.

♻️ This comment has been updated with latest results.

@coveralls
Copy link

coveralls commented Dec 14, 2023

Coverage Status

coverage: 83.986% (-0.009%) from 83.995%
when pulling 7cf1bd4 on fix-s3-too-large
into 4fe7a46 on master.

Copy link

github-actions bot commented Dec 14, 2023

LocalStack Community integration with Pro

       2 files         2 suites   1h 13m 15s ⏱️
2 416 tests 2 188 ✔️ 228 💤 0
2 417 runs  2 188 ✔️ 229 💤 0

Results for commit 7cf1bd4.

♻️ This comment has been updated with latest results.

@bentsku bentsku marked this pull request as ready for review December 14, 2023 16:42
Copy link
Member

@thrau thrau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for fixing this! as discussed offline, i think this solution is the best we can do for now!

maybe two things we can add in the future (as discussed also, not needed for this PR)

  • get service indicators not only from headers but also from query args to cover pre-signed URLs
  • instead of using request.values which triggers form parsing, first use request.args and then check if the body can be parsed as form (which would perhaps cover some rare pre-signed URL cases, but we should probably first find specific cases)

just have tiny suggestion for a better logging message

localstack/aws/protocol/service_router.py Outdated Show resolved Hide resolved
@bentsku bentsku merged commit acab579 into master Dec 22, 2023
32 checks passed
@bentsku bentsku deleted the fix-s3-too-large branch December 22, 2023 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: asf aws:s3 Amazon Simple Storage Service semver: patch Non-breaking changes which can be included in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: S3 - File not being uploaded when endpoint is localhost
3 participants