Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

POST uploads #73

Open
gaul opened this issue Jul 24, 2015 · 23 comments
Open

POST uploads #73

gaul opened this issue Jul 24, 2015 · 23 comments

Comments

@gaul
Copy link
Owner

gaul commented Jul 24, 2015

S3Proxy does not currently support POST uploads, triggering many s3-tests failures:

ERROR: s3tests.functional.test_s3.test_post_object_anonymous_request
ERROR: s3tests.functional.test_s3.test_post_object_authenticated_request_bad_access_key
ERROR: s3tests.functional.test_s3.test_post_object_set_success_code
ERROR: s3tests.functional.test_s3.test_post_object_set_invalid_success_code
FAIL: s3tests.functional.test_s3.test_post_object_authenticated_request
FAIL: s3tests.functional.test_s3.test_post_object_upload_larger_than_chunk
FAIL: s3tests.functional.test_s3.test_post_object_set_key_from_filename
FAIL: s3tests.functional.test_s3.test_post_object_ignored_header
FAIL: s3tests.functional.test_s3.test_post_object_case_insensitive_condition_fields
FAIL: s3tests.functional.test_s3.test_post_object_escaped_field_values
FAIL: s3tests.functional.test_s3.test_post_object_success_redirect_action
FAIL: s3tests.functional.test_s3.test_post_object_invalid_date_format
FAIL: s3tests.functional.test_s3.test_post_object_no_key_specified
FAIL: s3tests.functional.test_s3.test_post_object_missing_signature
FAIL: s3tests.functional.test_s3.test_post_object_user_specified_header
FAIL: s3tests.functional.test_s3.test_post_object_condition_is_case_sensitive
FAIL: s3tests.functional.test_s3.test_post_object_expires_is_case_sensitive
FAIL: s3tests.functional.test_s3.test_post_object_missing_expires_condition
FAIL: s3tests.functional.test_s3.test_post_object_missing_conditions_list
FAIL: s3tests.functional.test_s3.test_post_object_upload_size_limit_exceeded
FAIL: s3tests.functional.test_s3.test_post_object_missing_content_length_argument
FAIL: s3tests.functional.test_s3.test_post_object_invalid_content_length_argument
FAIL: s3tests.functional.test_s3.test_post_object_upload_size_below_minimum
gaul added a commit that referenced this issue Nov 15, 2015
This commit lacks support for policies.  References #73.
@gaul
Copy link
Owner Author

gaul commented Nov 15, 2015

@auguster I implemented basic support for POST uploads -- can you test to see if it resolves your issue? Note that S3Proxy still lacks support for policies.

@auguster
Copy link

Thanks you for the news !
Unfortunatly I'll be away for the whole coming week, so I might not be able to test that before the week after...

@ck86
Copy link

ck86 commented Apr 17, 2018

Is it planned to be merged?

@gaul
Copy link
Owner Author

gaul commented Apr 17, 2018

S3Proxy has partial POST upload support although there are a lot of extra bits. Do you need one feature in particular?

@ck86
Copy link

ck86 commented Apr 17, 2018

We currently migrating from S3 to Azure Blob Storage and to upload files from the client, our API creates a PostObject, which seems to work with the proxy (at least I get similar "form fields"). But when I submit the form I get a "Not Implemented" error.

@gaul
Copy link
Owner Author

gaul commented Apr 17, 2018

Could you share some example source code? Also maybe dump logs via -DLOG_LEVEL=trace although it might not capture enough information to help.

@ck86
Copy link

ck86 commented Apr 18, 2018

How I use the php aws sdk:

$options = [
    ['acl' => 'public-read'],
    ['bucket' => $bucket],
    ['starts-with', '$key', $key],
    ['content-length-range', 0, 1024 * 1024 * 1024],
];

/** @var \Aws\S3\S3Client $s3 */
$s3 = app('s3');
$postObject = new PostObjectV4($s3, $bucket, $inputs, $options);
$attributes = $postObject->getFormAttributes();
$fields = $postObject->getFormInputs();

this is an example response:

{
    "attributes": {
        "action": "http://localhost:8080",
        "method": "POST",
        "enctype": "multipart/form-data"
    },
    "fields": {
        "acl": "public-read",
        "key": "filename.ext",
        "X-Amz-Credential": "*******",
        "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
        "X-Amz-Date": "20180418T063726Z",
        "Policy": "**************",
        "X-Amz-Signature": "************"
    }
}

The above mentions field are used to submit the form with a file called filename.ext.

This is the exact error message:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
    <Code>NotImplemented</Code>
    <Message>A header you provided implies functionality that is not implemented.</Message>
    <RequestId>4442587FB7D0A2F9</RequestId>
</Error>

Unfortunately I can't or I don't know how to set the DLOG_LEVEL. I am using docker and if I am adding it after ENTRYPOINT ./target/s3proxy --properties ./s3proxy.conf the proxy will not start because of the unknown option.

@ck86
Copy link

ck86 commented May 4, 2018

any updates? :/

@gaul
Copy link
Owner Author

gaul commented Jan 4, 2019

The Docker image should take a LOG_LEVEL environment variable. I really need a self-contained test case to make progress on this; otherwise I would have to implement the entire POST handling functionality which I would like to avoid for now.

@simonbowen
Copy link

@ck86 I'm facing this issue at the moment, did you manage to find a work around?

@ck86
Copy link

ck86 commented Oct 28, 2019

@simonbowen unfortunately not. I switched to another proxy and used the signed URLs to upload a file.

@simonbowen
Copy link

@ck86 Can you tell me the name of the other proxy you used? I'm looking at using LocalStack instead at the moment.

@ck86
Copy link

ck86 commented Oct 28, 2019

minio

@simonbowen
Copy link

simonbowen commented Oct 28, 2019 via email

@gaul
Copy link
Owner Author

gaul commented Oct 29, 2019

As suggested in #73 (comment), if someone can provide a self-contained test I can add missing functionality but I don't want to commit to all possible features of POST uploads right now.

@simonbowen
Copy link

@gaul Thanks for the response. Also thanks for this great package. I'm not a Java developer and writing a test for this use case is going to be beyond me. I had dropped this into my development environment via docker and it worked great up until the HTTP POST upload.

@gaul
Copy link
Owner Author

gaul commented Oct 30, 2019

@simonbowen If you can provide a self-contained test, either via unit test, curl, or whatever, I can likely add the required functionality easily. Unfortunately figuring out how to even run something like #73 (comment) is likely more work than the fix.

@simonbowen
Copy link

@gaul I'm not sure I can just give you an example curl request, since I have to make a request to get the fields (via a request) and then use those fields in a subsequent request. All I'd be doing is converting above into a bash script. As I said already, I am not capable of writing a unit test in Java (sorry).

@gaul
Copy link
Owner Author

gaul commented Oct 31, 2019

Try turning logging up to trace level and see which parameter fails?

@liamwhite
Copy link
Contributor

Here's a simple failure

files_1          | [s3proxy] D 02-09 03:03:57.200 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:299 |::] request: Request(POST //files/philomena/avatars/2022/2/9/e9158f14-8954-11ec-a0c5-0242ac130006.png?uploads=1)@5c2e0896
files_1          | [s3proxy] D 02-09 03:03:57.201 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: Authorization: AWS4-HMAC-SHA256 Credential=local-identity/20220209/us-east-1/s3/aws4_request,SignedHeaders=content-length;host;x-amz-acl;x-amz-content-sha256;x-amz-date,Signature=d900da74357e968e29ea2cfd0f0afef09052bc8df0b4f45f9c011e3727612822
files_1          | [s3proxy] D 02-09 03:03:57.201 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
files_1          | [s3proxy] D 02-09 03:03:57.201 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: x-amz-date: 20220209T030357Z
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: x-amz-acl: public-read
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: host: files
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: User-Agent: hackney/1.18.0
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: Content-Length: 0
files_1          | [s3proxy] D 02-09 03:03:57.202 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:324 |::] header: Content-Type: application/octet-stream
files_1          | [s3proxy] E 02-09 03:03:57.204 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:769 |::] Unknown method POST with URI /philomena/avatars/2022/2/9/e9158f14-8954-11ec-a0c5-0242ac130006.png
files_1          | [s3proxy] D 02-09 03:03:57.204 S3Proxy-Jetty-41 o.gaul.s3proxy.S3ProxyHandler:2923 |::] sendSimpleErrorResponse: 501 NotImplemented A header you provided implies functionality that is not implemented. {}

because the code tests if the parameter value is empty rather than if the parameter is present

} else if ("".equals(request.getParameter("uploads"))) {

@gaul
Copy link
Owner Author

gaul commented Feb 9, 2022

@liamwhite Which S3 client do you use to access S3Proxy? The uploads=1 parameter does not exactly match the AWS documented CreateMultipartUpload RPC. I prefer to fix the client but will also accept an S3Proxy PR to be more liberal with all the parameter checks, e.g., getParameter() != null.

Note that this issue tracks browser-based POST uploads and your logs correspond to the unrelated CreateMultipartUpload RPC which confusingly has a POST HTTP verb and an uploads parameter. Let's discuss this is a separate issue or PR.

@liamwhite
Copy link
Contributor

liamwhite commented Feb 9, 2022

@jasoncodes
Copy link

Two behaviour differences I’ve noticed compared to the real S3 service when testing browser POSTs are:

  1. ${filename} in the key field is not substituted with the filename from the file field’s Content-Disposition header; and
  2. The response should be a 201 Created with a Location header pointing to the URL of the created object. The path seems to be always URL encoded. e.g. https://example.s3.amazonaws.com/uploads%2Ftest.csv. There’s also a PostResponse XML object in the body but I’ve never used it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants