@ in key for certain operations causes SignatureDoesNotMatch error #601

jmoisan21 · 2022-06-13T20:51:33Z

Describe the bug
When the @ sign is present in the key for certain operations the resulting request comes back with SignatureDoesNotMatch: The request signature we calculated does not match the signature provided. Check the Secret Access Key and signing method.

To Reproduce

let awsClient = AWSClient(
        credentialProvider: .static(accessKeyId: ..., secretAccessKey: ...),
        httpClientProvider: .createNew
    )

let myS3 = S3(client: awsClient, region: ..., partition: .aws, endpoint: ...)

let newObjectRequest = S3.PutObjectRequest(
            bucket: "bucket",
            key: "some/thing/abc@def"
)

do {
            let response = try await myS3.putObject(newObjectRequest)
} catch {
            print(error)
}

Expected behavior
The request signature would be valid.

Setup (please complete the following information):

OS: macOS & Linux
Version of soto: v5.13.1
Authentication mechanism : hard-coded credentials

Additional context
Tried with another client and it worked as expected.
Tried to identify where in the code this signature creation problem would reside so I could just PR you something but I could not figure it out.

The text was updated successfully, but these errors were encountered:

jmoisan21 · 2022-06-13T21:59:40Z

Seems like the exact same behaviour occurs with ( and/or ) as in if the above was
key: "some/thing/abc(def" or key: "some/thing/abc)def"

Which if I understand correctly https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html
then those 2 should absolutely be safe.

jmoisan21 · 2022-06-14T15:47:47Z

I now have found where this is happening and it's in soto-core. As such I will close this issue and open in the proper repo.

adam-fowler · 2022-06-14T15:52:33Z

I have a test for uploading to a S3 key with a load of special characters. For some reason I don't include @, ( or ). It might be because they need to be percent encoded.

jmoisan21 · 2022-06-14T15:54:15Z

@adam-fowler this is what I was "preparing" to add here since I did some digging around and really wanted to help you figure out if/what/how...

The signature seems to be happening in func canonicalRequest(signingData: SigningData) -> String { of Soto-core -> Sources -> SotoSignerV4 -> signer where it uses uriEncodeWithSlash().

uriEncodeWithSlash() simply uses the defined CharacterSet as followed
static let uriAllowedWithSlashCharacters = CharacterSet(charactersIn: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~/")

Looking in to the code it references https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html as the source of documentation to how this is being handled.

Point 2. is the one that applies to this specific part "Add the canonical URI parameter" which references "Normalize URI paths according to RFC 3986"

RFC 3986 defines,

Section 2.2 Reserved Characters as followed :

reserved    = gen-delims / sub-delims
      gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"
      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="

Section 2.3 Unreserved Characters as followed :

unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

** Note that uriAllowedWithSlashCharacters seems to be using only the unreserved (with slash - which is an S3 exception well defined)

Section 3 Syntax Components : for our purpose the path component is of interest which is descriptor in detail in Section 3.3.

Section 3.3 Path component defines segments as the things between the / that is composed of pchar which in turn are defined as :

pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

Thus if I understood this correctly, which I hardly want to claim to have, means that the CharacterSet to use for this part would be :
unreserved : ALPHA / DIGIT / "-" / "." / "_" / "~"
sub-delims: "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="
and the additional ones defined: ":" / "@"
The rest being pct-encoded

jmoisan21 · 2022-06-14T15:55:49Z

I am currently trying to use a local forked version that adds all the sub-delims, ":" and "@" to see if it would at least work (aside from whether it's right or not) to have a starting point...

jmoisan21 · 2022-06-14T15:58:43Z

Also would you rather I reopen here or create a net new issue in soto-core where the code actually lives?

jmoisan21 · 2022-06-14T16:29:38Z

It might be because they need to be percent encoded.

I tried manually encoding "@" with %40. It did allow the request to succeed put also resulted in the actual creation having %40 in it...

adam-fowler · 2022-06-14T16:57:02Z

It might need percent encoded in the canonical request, not the URL

…

On Tue, 14 Jun 2022 at 17:29, jmoisan21 ***@***.***> wrote: It might be because they need to be percent encoded. I tried manually encoding "@" with %40. It did allow the request to succeed put also resulted in the actual creation having %40 in it... — Reply to this email directly, view it on GitHub <#601 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHSVJ2PGJY5LVVK2CBBQWLVPCXP3ANCNFSM5YVPDKZQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

jmoisan21 · 2022-06-14T17:17:01Z

Ok so what I have so far is this is what your code does right now when the key is /a/b/c/def@hij?-(klm)~opq.rst

Printing description of canonicalPath:
"/a/b/c/def%40hij%3F-%28klm%29~opq.rst/"

Printing description of request.url:
▿ https://localhost:443/a/b/c/def@hij%3F-(klm)~opq.rst/?x-id=PutObject

Resulting in the SignatureDoesNotMatch

jmoisan21 · 2022-06-14T18:08:25Z

If I force canonicalPath to be like request.url but I get an error : InvalidArgument: Invalid Argument
In: soto-core/Sources/SotoSignerV4/signer.swift

static let uriAllowedWithSlashCharacters2 = CharacterSet(charactersIn: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~/():@")
    static let uriAllowedCharacters2 = CharacterSet(charactersIn: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~():@")

jmoisan21 · 2022-06-14T18:08:53Z

I then instead tried to match request.url to canonicalPath
(above reverted)
In: soto-core/Sources/SotoCore/Message/AWSRequest.swift

 static let pathAllowedCharacters = CharacterSet.urlPathAllowed.subtracting(.init(charactersIn: "+@()"))
  static let pathComponentAllowedCharacters = CharacterSet.urlPathAllowed.subtracting(.init(charactersIn: "+/@()"))

That had no effect at all.

jmoisan21 · 2022-06-14T18:12:44Z

Discovered that soto has a virtualAddressFixup func that makes the url back to a string and then calls it's own addingPercentEncoding at the end that does not match anything that was possibly done in soto-core...

So I change that code
In: soto/Sources/Soto/Extensions/S3/S3RequestMiddleware.swift

static let pathAllowedCharacters = CharacterSet.urlPathAllowed.subtracting(.init(charactersIn: "+@()"))

And success : definition of success is the key path was accepted, the signature was valid, the request succeeded and the resulting item server side has proper naming.

jmoisan21 · 2022-06-14T18:16:13Z

With the 3 above comments that should give you the gist of what is happening and why.
Since:

I have no clue what characters you consider important (above @,(,) that I have identified)
I am truly not certain what the proper fix is from your standpoint (soto should use the soto-core defined Charsets?)

I don't want to PR this for you. I do hope this gives you enough to hit the ground running.

Ty

adam-fowler · 2022-06-14T18:34:49Z

Discovered that soto has a virtualAddressFixup func that makes the url back to a string and then calls it's own addingPercentEncoding at the end that does not match anything that was possibly done in soto-core...

There was a change to that recently. Maybe it broke something. I'll look into it tomorrow

jmoisan21 · 2022-06-14T19:11:02Z

FWIW I reverted every changed I made, returned to to using your current soto-core and only made a change in soto for this static let pathAllowedCharacters = CharacterSet.urlPathAllowed.subtracting(.init(charactersIn: "+!$&'()*+,;=:@")) in soto/Sources/Soto/Extensions/S3/S3RequestMiddleware.swift.

This results in a/b/c/a!b$c@d&e?f(g)h~i'j*k+l,m;n.o=p:q working properly

adam-fowler · 2022-06-15T09:29:12Z

@jmoisan21 Are you working with AWS S3 or an S3 like service? I have just looked through my test code and there is a test that tests against special characters @, ( and ).

soto/Tests/SotoTests/Services/S3/S3Tests.swift

Lines 143 to 146 in ea6c998

    
           func testPutGetObjectWithSpecialName() { 
        
               let name = TestEnvironment.generateResourceName() 
        
               let filename = "test $filé+!@£$%2F%^&*()_=-[]{}\\|';:\",./?><~`.txt" 
        
               let contents = "testing S3.PutObject and S3.GetObject"

This test passes fine when run against S3

jmoisan21 · 2022-06-15T11:37:36Z

@adam-fowler S3 like service

adam-fowler · 2022-06-15T16:23:22Z

Given the special characters work ok with AWS S3, I'm guessing that service is verifying the signing incorrectly.
But I understand you said another client worked ok. Can you verify it works using the aws cli?

jmoisan21 · 2022-06-15T16:26:52Z

Can you verify it works using the aws cli?

Will report back on it

jmoisan21 · 2022-06-15T16:59:33Z

aws --version
aws-cli/2.7.7

aws s3api put-object --acl private --bucket ios --key 'a/b/c/ac@d&e?f(g)h~ij*k+l,m;n.o=p:q' --endpoint https://localhost:8080

{
    "ETag": "\"d41d8cd9{snip}ecf8427e\""
}

File is created and has proper name

Additionally... Changing the --endpoint to something that does not exist prints out what aws cli is trying to send it confirms that is does encode it all :

Could not connect to the endpoint URL: "https://localhost:1234/a/b/c/ac%40d%26e%3Ff%28g%29h~ij%2Ak%2Bl%2Cm%3Bn.o%3Dp%3Aq"

(re that message : #601 (comment))

adam-fowler · 2022-06-15T17:09:53Z

Can you simplify the key to something that still fails eg ab@cd and run

aws --debug s3api put-object --acl private --bucket ios --key 'ab@cd' --endpoint https://localhost:8080

And dump the output here.

adam-fowler · 2022-06-15T17:30:36Z

After that can you run the Soto code with the following change

let myS3 = S3(client: awsClient, region: ..., partition: .aws, endpoint: ...).with(middlewares: [AWSLoggingMiddleware()]

And add the contents below.

jmoisan21 · 2022-06-15T18:57:44Z

Can you simplify the key to something that still fails eg ab@cd and run

Please note that I sanitized the output and therefore you cannot rely on the data here to repro manually the md5/sha/etags/...

aws --debug s3api put-object --acl private --bucket test --key 'a/b/test1/ab@cd' --endpoint  https://localhost
2022-06-15 14:46:02,253 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.7.7
2022-06-15 14:46:02,253 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 's3api', 'put-object', '--acl', 'private', '--bucket', 'test', '--key', 'a/b/test1/ab@cd', '--endpoint', 'https://localhost']
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fbfa14489d0>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fbfa12a1280>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fbfa12455e0>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fbfa12505e0>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fbfa145b3a0>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fbfa12ecd30>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2022-06-15 14:46:02,268 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fbfa14505e0>
2022-06-15 14:46:02,269 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
2022-06-15 14:46:02,270 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fbfa13a0700>
2022-06-15 14:46:02,270 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fbfa13a5280>
2022-06-15 14:46:02,270 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fbfa13a51f0>
2022-06-15 14:46:02,270 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fbfa13a53a0>
2022-06-15 14:46:02,270 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fbfa13a5310>
2022-06-15 14:46:02,270 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fbfa14f6800>
2022-06-15 14:46:02,271 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.7.7 exe/x86_64 prompt/off
2022-06-15 14:46:02,271 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', 's3api', 'put-object', '--acl', 'private', '--bucket', 'test', '--key', 'a/b/test1/ab@cd', '--endpoint', 'https://localhost']
2022-06-15 14:46:02,271 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fbfa144a040>
2022-06-15 14:46:02,271 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fbfa1079790>
2022-06-15 14:46:02,271 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fbfa14bc4c0>
2022-06-15 14:46:02,271 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fbfa1073c10>
2022-06-15 14:46:02,271 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fbfa10f3310>
2022-06-15 14:46:02,283 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2022-06-15 14:46:02,291 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fbfa12ecc10>
2022-06-15 14:46:02,291 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fbfa129f160>
2022-06-15 14:46:02,305 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/service-2.json
2022-06-15 14:46:02,314 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3api: calling handler <function add_waiters at 0x7fbfa14505e0>
2022-06-15 14:46:02,327 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/waiters-2.json
2022-06-15 14:46:02,328 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('acl', <awscli.arguments.CLIArgument object at 0x7fbfc088d910>), ('body', <awscli.arguments.CLIArgument object at 0x7fbfc088d940>), ('bucket', <awscli.arguments.CLIArgument object at 0x7fbfc088d970>), ('cache-control', <awscli.arguments.CLIArgument object at 0x7fbfc088d9a0>), ('content-disposition', <awscli.arguments.CLIArgument object at 0x7fbfc088d9d0>), ('content-encoding', <awscli.arguments.CLIArgument object at 0x7fbfc088da00>), ('content-language', <awscli.arguments.CLIArgument object at 0x7fbfc088da30>), ('content-length', <awscli.arguments.CLIArgument object at 0x7fbfc088da60>), ('content-md5', <awscli.arguments.CLIArgument object at 0x7fbfc088da90>), ('content-type', <awscli.arguments.CLIArgument object at 0x7fbfc088dac0>), ('checksum-algorithm', <awscli.arguments.CLIArgument object at 0x7fbfc088daf0>), ('checksum-crc32', <awscli.arguments.CLIArgument object at 0x7fbfc088db20>), ('checksum-crc32-c', <awscli.arguments.CLIArgument object at 0x7fbfc088db50>), ('checksum-sha1', <awscli.arguments.CLIArgument object at 0x7fbfc088db80>), ('checksum-sha256', <awscli.arguments.CLIArgument object at 0x7fbfc088dbb0>), ('expires', <awscli.arguments.CLIArgument object at 0x7fbfc088dbe0>), ('grant-full-control', <awscli.arguments.CLIArgument object at 0x7fbfc088dc10>), ('grant-read', <awscli.arguments.CLIArgument object at 0x7fbfc088dc40>), ('grant-read-acp', <awscli.arguments.CLIArgument object at 0x7fbfc088dc70>), ('grant-write-acp', <awscli.arguments.CLIArgument object at 0x7fbfc088dca0>), ('key', <awscli.arguments.CLIArgument object at 0x7fbfc088dcd0>), ('metadata', <awscli.arguments.CLIArgument object at 0x7fbfc088dd00>), ('server-side-encryption', <awscli.arguments.CLIArgument object at 0x7fbfc088dd30>), ('storage-class', <awscli.arguments.CLIArgument object at 0x7fbfc088dd60>), ('website-redirect-location', <awscli.arguments.CLIArgument object at 0x7fbfc088dd90>), ('sse-customer-algorithm', <awscli.arguments.CLIArgument object at 0x7fbfc088ddc0>), ('sse-customer-key', <awscli.arguments.CLIArgument object at 0x7fbfc088ddf0>), ('sse-customer-key-md5', <awscli.arguments.CLIArgument object at 0x7fbfc088de20>), ('ssekms-key-id', <awscli.arguments.CLIArgument object at 0x7fbfc088de50>), ('ssekms-encryption-context', <awscli.arguments.CLIArgument object at 0x7fbfc088de80>), ('bucket-key-enabled', <awscli.arguments.BooleanArgument object at 0x7fbfc088deb0>), ('no-bucket-key-enabled', <awscli.arguments.BooleanArgument object at 0x7fbfc088dee0>), ('request-payer', <awscli.arguments.CLIArgument object at 0x7fbfc088df10>), ('tagging', <awscli.arguments.CLIArgument object at 0x7fbfc088df40>), ('object-lock-mode', <awscli.arguments.CLIArgument object at 0x7fbfc088df70>), ('object-lock-retain-until-date', <awscli.arguments.CLIArgument object at 0x7fbfc088dfa0>), ('object-lock-legal-hold-status', <awscli.arguments.CLIArgument object at 0x7fbfc088dfd0>), ('expected-bucket-owner', <awscli.arguments.CLIArgument object at 0x7fbfc0897040>)])
2022-06-15 14:46:02,328 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.put-object: calling handler <function add_streaming_output_arg at 0x7fbfa144a310>
2022-06-15 14:46:02,329 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.put-object: calling handler <function add_cli_input_json at 0x7fbfa10f3af0>
2022-06-15 14:46:02,329 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.put-object: calling handler <function add_cli_input_yaml at 0x7fbfa10f3d30>
2022-06-15 14:46:02,329 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.put-object: calling handler <function unify_paging_params at 0x7fbfa12a1790>
2022-06-15 14:46:02,342 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/s3/2006-03-01/paginators-1.json
2022-06-15 14:46:02,342 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.s3api.put-object: calling handler <function add_generate_skeleton at 0x7fbfa1393ca0>
2022-06-15 14:46:02,342 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.put-object: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fbfc088d190>>
2022-06-15 14:46:02,342 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.put-object: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fbfc08970d0>>
2022-06-15 14:46:02,342 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.s3api.put-object: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fbfc0897160>>
2022-06-15 14:46:02,343 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.acl: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,343 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.put-object: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fbfa10bd1c0>
2022-06-15 14:46:02,343 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'private' for parameter "acl": 'private'
2022-06-15 14:46:02,343 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.body: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,343 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.bucket: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,343 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.put-object: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fbfa10bd1c0>
2022-06-15 14:46:02,344 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'test' for parameter "bucket": 'test'
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.cache-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.content-disposition: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.content-encoding: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.content-language: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.content-length: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.content-md5: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.content-type: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.checksum-algorithm: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.checksum-crc32: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.checksum-crc32-c: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.checksum-sha1: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.checksum-sha256: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.expires: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.grant-full-control: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.grant-read: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.grant-read-acp: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.grant-write-acp: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.s3.put-object: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fbfa10bd1c0>
2022-06-15 14:46:02,344 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'a/b/test1/ab@cd' for parameter "key": 'a/b/test1/ab@cd'
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.metadata: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.server-side-encryption: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.storage-class: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.website-redirect-location: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.sse-customer-algorithm: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.sse-customer-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,344 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.sse-customer-key-md5: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.ssekms-key-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.ssekms-encryption-context: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.bucket-key-enabled: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.tagging: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.object-lock-mode: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.object-lock-retain-until-date: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.object-lock-legal-hold-status: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.expected-bucket-owner: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.s3.put-object.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fbfc0463610>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.put-object: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fbfc088d190>>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.put-object: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fbfc08970d0>>
2022-06-15 14:46:02,345 - MainThread - botocore.hooks - DEBUG - Event calling-command.s3api.put-object: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fbfc0897160>>
2022-06-15 14:46:02,345 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2022-06-15 14:46:02,345 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2022-06-15 14:46:02,345 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2022-06-15 14:46:02,345 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2022-06-15 14:46:02,345 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2022-06-15 14:46:02,346 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2022-06-15 14:46:02,346 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json
2022-06-15 14:46:02,354 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fbfa08074c0>
2022-06-15 14:46:02,356 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fbfa078cee0>
2022-06-15 14:46:02,356 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fbfa078cca0>
2022-06-15 14:46:02,356 - MainThread - botocore.regions - DEBUG - Creating a regex based endpoint for s3, my-test
2022-06-15 14:46:02,358 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.PutObject: calling handler <function base64_decode_input_blobs at 0x7fbfa14bcc10>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x7fbfa0827310>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x7fbfa0822700>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x7fbfa0827c10>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x7fbfa0822670>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fbfa155b3a0>>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7fbfa155b430>>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x7fbfa08224c0>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <function conditionally_calculate_md5 at 0x7fbfa069ab80>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x7fbfa08229d0>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fbfa155b3a0>>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x7fbfa0827d30>
2022-06-15 14:46:02,359 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=PutObject) with params: {'url_path': '/test/a/b/test1/ab%40cd', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-acl': 'private', 'User-Agent': 'aws-cli/2.7.7 exe/x86_64 prompt/off command/s3api.put-object', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg=='}, 'body': b'', 'url': 'https://localhost/test/a/b/test1/ab%40cd', 'context': {'client_region': 'my-test', 'client_config': <botocore.config.Config object at 0x7fbfa1a4d9a0>, 'has_streaming_input': True, 'auth_type': None, 'signing': {'bucket': 'test'}}}
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fbfa1a4d820>>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.PutObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fbfa155b490>>
2022-06-15 14:46:02,359 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7fbfa08223a0>
2022-06-15 14:46:02,360 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fbfa155b490>>
2022-06-15 14:46:02,360 - MainThread - botocore.utils - DEBUG - Using S3 path style addressing.
2022-06-15 14:46:02,360 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-06-15 14:46:02,360 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/test/a/b/test1/ab%40cd

content-md5:1B2M2Y8AsgTpgAmY7PhCfg==
host:localhost
x-amz-acl:private
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20220615T170622Z

content-md5;host;x-amz-acl;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
2022-06-15 14:46:02,360 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220615T170622Z
20220615/my-test/s3/aws4_request
176794116ea96d087fc50bb15696bb54945247fc95273fa1f3a6af8222a1c349
2022-06-15 14:46:02,360 - MainThread - botocore.auth - DEBUG - Signature:
f74cb491ea251fa40ddd64c9f4237cb1ad937cf26d248c1005cac12ae40f90ee
2022-06-15 14:46:02,360 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://localhost/test/a/b/test1/ab%40cd, headers={'x-amz-acl': b'private', 'User-Agent': b'aws-cli/2.7.7 exe/x86_64 prompt/off command/s3api.put-object', 'Content-MD5': b'1B2M2Y8AsgTpgAmY7PhCfg==', 'X-Amz-Date': b'20220615T170622Z', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'Authorization': b'AWS4-HMAC-SHA256 Credential=secret/20220615/my-test/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-acl;x-amz-content-sha256;x-amz-date, Signature=f74cb491ea251fa40ddd64c9f4237cb1ad937cf26d248c1005cac12ae40f90ee', 'Content-Length': '0'}>
2022-06-15 14:46:02,361 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): localhost
2022-06-15 14:46:02,998 - MainThread - urllib3.connectionpool - DEBUG - https://localhost "PUT /test/a/b/test1/ab%40cd HTTP/1.1" 200 0
2022-06-15 14:46:02,999 - MainThread - botocore.parsers - DEBUG - Response headers: {'Connection': 'keep-alive', 'ETag': '"d41d8cd98f00b204e9800998ecf8427e"', 'x-amz-request-id': '1851281095', 'Content-Length': '0'}
2022-06-15 14:46:02,999 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2022-06-15 14:46:03,000 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fbfa155b310>>
2022-06-15 14:46:03,001 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-06-15 14:46:03,001 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fbfa155b3a0>>
2022-06-15 14:46:03,001 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <function enhance_error_msg at 0x7fbfa1448c10>
2022-06-15 14:46:03,001 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fbfa1a4de80>>
2022-06-15 14:46:03,002 - MainThread - awscli.formatter - DEBUG - RequestId: 1851281095
{
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\""
}

jmoisan21 · 2022-06-15T19:19:11Z

After that can you run the Soto code with the following change

Please note that I sanitized the output and therefore you cannot rely on the data here to repro manually the md5/sha/etags/...

Your soto code unchanged:

Request:
  PutObject
  PUT https://localhost/test/a/b/test1/ab@cd?x-id=PutObject
  Headers: [
    x-amz-acl : private
    user-agent : Soto/6.0
    content-type : application/octet-stream
  ]
  Body: empty
Response:
  Status : 403
  Headers: [
    Connection : keep-alive
    x-amz-request-id : 1851115345
    Content-Length : 273
    Content-Type : application/xml
  ]
  Body: 
  <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature provided. Check the Secret Access Key and signing method.</Message><Resource></Resource><RequestId>1851115345</RequestId></Error>

jmoisan21 · 2022-06-15T19:20:29Z

And in case you wondered, my soto change from #601 (comment)

Request:
  PutObject
  PUT https://localhost/test/a/b/test1/ab%40cd?x-id=PutObject
  Headers: [
    x-amz-acl : private
    user-agent : Soto/6.0
    content-type : application/octet-stream
  ]
  Body: empty
Response:
  Status : 200
  Headers: [
    Connection : keep-alive
    ETag : "d41d8cd98f00b204e9800998ecf8427e"
    x-amz-request-id : 1851281097
    Content-Length : 0
  ]
  Body: empty

PutObjectOutput(bucketKeyEnabled: nil, checksumCRC32: nil, checksumCRC32C: nil, checksumSHA1: nil, checksumSHA256: nil, eTag: Optional(\"\\\"d41d8cd98f00b204e9800998ecf8427e\\\"\"), expiration: nil, requestCharged: nil, serverSideEncryption: nil, sseCustomerAlgorithm: nil, sseCustomerKeyMD5: nil, ssekmsEncryptionContext: nil, ssekmsKeyId: nil, versionId: nil)

adam-fowler · 2022-06-17T13:42:42Z

Can you verify #604 fixes your issue

jmoisan21 · 2022-06-17T13:49:20Z

Will fetch the package using branch "s3-path-characters" and test. Should have results in the next hour.

jmoisan21 · 2022-06-17T14:32:42Z

@adam-fowler I can confirm that it solves for exactly "@()" characters.

But since you are there and doing the work... I tried you test string "test $filé+!@£$%2F%^&*()_=-[]{}\\|';:\",./?><~.txt"` and it fails.

What was cli sends when using that string is : "test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%5C%3B%3A%22%2C./%3F%3E%3C~%60.txt"

What you send (with the current branch fix) is : test%20$fil%C3%A9%2B!%40%C2%A3$%252F%25%5E&*%28%29_=-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt

Short version : $!&*=-':, should probably be added to your CharacterSet.urlPathAllowed.subtracting(.init(charactersIn: "+@()")) change

adam-fowler · 2022-06-17T14:52:02Z

That was my concern. That there would be other characters. But I'm overly keen on just adding a whole load of additional characters, the signing stuff is so frustrating and what we have works for AWS S3, it is bloody frustrating it doesn't work on other S3 like services. I'm not sure this is even a bug with Soto, is it not more a bug with the S3 service you are using. It should be able to deal with the URLs I am passing it.

You included - in your list. Are you sure that needs percent encoded? I'll add the characters from the special handling list in https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html.

jmoisan21 · 2022-06-17T15:16:10Z

Argh I am sorry I took as much as my time as I could to make sure I would not make that mistake when comparing the two strings.

test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C%27%3B%3A%22%2C./%3F%3E%3C~%60.txt <- AWS cli
test%20$fil%C3%A9%2B!%40%C2%A3$%252F%25%5E&*%28%29_=-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt <- current result

Change the chars in current result one by one

$ => test%20%24fil%C3%A9%2B!%40%C2%A3%24%252F%25%5E&*%28%29_=-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt
! => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E&*%28%29_=-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt
& => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26*%28%29_=-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt
* => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_=-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt
= => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C'%3B:%22,./%3F%3E%3C~%60.txt
' => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C%27%3B:%22,./%3F%3E%3C~%60.txt
: => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C%27%3B%3A%22,./%3F%3E%3C~%60.txt
, => test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C%27%3B%3A%22%2C./%3F%3E%3C~%60.txt

Compare again

test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C%27%3B%3A%22%2C./%3F%3E%3C~%60.txt <- result
test%20%24fil%C3%A9%2B%21%40%C2%A3%24%252F%25%5E%26%2A%28%29_%3D-%5B%5D%7B%7D%5C%7C%27%3B%3A%22%2C./%3F%3E%3C~%60.txt <- AWS cli

Add spaces to help the eyes

test %20%24 fil %C3%A9%2B%21 %40%C2%A3%24 %252F%25%5E %26%2A%28%29 _%3D-%5B%5D %7B%7D%5C%7C %27%3B%3A%22 %2C./%3F%3E %3C~%60.txt <- result
test %20%24 fil %C3%A9%2B%21 %40%C2%A3%24 %252F%25%5E %26%2A%28%29 _%3D-%5B%5D %7B%7D%5C%7C %27%3B%3A%22 %2C./%3F%3E %3C~%60.txt <- AWS cli

Resulting list would be : $!&*=':,

jmoisan21 · 2022-06-17T15:18:35Z

I'm not sure this is even a bug with Soto, is it not more a bug with the S3 service you are using. It should be able to deal with the URLs I am passing it.

I would tend to agree with you. The only thing that makes me side on caution here is that the official aws cli outputs something different then yours and this may be a ticking time bomb for your code...

adam-fowler · 2022-06-17T15:40:41Z

I've added those characters in as well if you want to test it
Out of interest what S3 like service are you using?

jmoisan21 · 2022-06-17T15:45:10Z

Updated the package and can see the new change static let s3PathAllowedCharacters = CharacterSet.urlPathAllowed.subtracting(.init(charactersIn: "+@()&$=:,'!*"))

Testing...

jmoisan21 · 2022-06-17T15:48:18Z

Successfully created test1/test $filé+!@£$%2F%^&*()_=-[]{}\|';:",. and confirmed server side that the name is exactly that one (no percent encoding).

I call this one solved! Aligned with aws cli for your test string.

Thank you very much

jmoisan21 · 2022-06-17T15:48:51Z

Out of interest what S3 like service are you using?

Sorry cannot say that here.

adam-fowler · 2022-06-18T08:24:26Z

I'm about to do a v6 release of Soto, in the next couple of weeks. Would you be happy to upgrade to v6 (You have been using what will be v6 when testing the changes)? I could back port the changes to v5 but would prefer if I could avoid that.

You can find a list of the main changes in v6 here. The other major addition, not listed, is v6 also includes Sendable conformance.

jmoisan21 · 2022-06-18T12:22:47Z

The tests I have been performing were in really controlled environment. I am happy to run the current v6 you through a real set of test to verify.

Based on the fact that you merged #601 in main I assume that means that setting my package to pull from main would effectively include your current work for v6?

adam-fowler · 2022-06-18T12:41:39Z

I am happy to run the current v6 you through a real set of test to verify.

That'd be great if you could do that. Yes main is what the 6.0 release will be.

jmoisan21 · 2022-06-20T21:08:59Z

Been running main through testing all day and will probably have it run all day tomorrow also. So far so good with nothing to report.

I almost feel confident answering the question :

Would you be happy to upgrade to v6 (You have been using what will be v6 when testing the changes)?

But I will give it one more day of load testing before calling it.

adam-fowler · 2022-06-21T12:53:11Z

v6 has been released today, so you'll be able to upgrade to a full version if you are happy with your tests

jmoisan21 · 2022-06-21T14:01:38Z

Yes ty I saw right away since it includes my other PR ;-)

Also the fact that this is a release will make things much easier for me since I will be able to pin to a version and not just main

(I do hope to close this issue by EOD)

jmoisan21 · 2022-06-22T12:28:10Z

So far I have not encountered any issues with v6. Therefore I am calling it. This fix is not to be back ported for me.

If anything comes up I will look forward to getting it fixed in the v6 series!

Thanks

jmoisan21 closed this as completed Jun 14, 2022

jmoisan21 reopened this Jun 14, 2022

jmoisan21 closed this as completed Jun 22, 2022

@ in key for certain operations causes SignatureDoesNotMatch error #601

@ in key for certain operations causes SignatureDoesNotMatch error #601

Comments

jmoisan21 commented Jun 13, 2022 • edited Loading

jmoisan21 commented Jun 13, 2022

jmoisan21 commented Jun 14, 2022

adam-fowler commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

adam-fowler commented Jun 14, 2022 via email

jmoisan21 commented Jun 14, 2022 • edited Loading

jmoisan21 commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022 • edited Loading

adam-fowler commented Jun 14, 2022

jmoisan21 commented Jun 14, 2022

adam-fowler commented Jun 15, 2022

jmoisan21 commented Jun 15, 2022

adam-fowler commented Jun 15, 2022

jmoisan21 commented Jun 15, 2022

jmoisan21 commented Jun 15, 2022

adam-fowler commented Jun 15, 2022

adam-fowler commented Jun 15, 2022

jmoisan21 commented Jun 15, 2022 • edited Loading

jmoisan21 commented Jun 15, 2022

jmoisan21 commented Jun 15, 2022

adam-fowler commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022

adam-fowler commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022

adam-fowler commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022

jmoisan21 commented Jun 17, 2022 • edited Loading

adam-fowler commented Jun 18, 2022

jmoisan21 commented Jun 18, 2022

adam-fowler commented Jun 18, 2022

jmoisan21 commented Jun 20, 2022

adam-fowler commented Jun 21, 2022

jmoisan21 commented Jun 21, 2022

jmoisan21 commented Jun 22, 2022

jmoisan21 commented Jun 13, 2022 •

edited

Loading

jmoisan21 commented Jun 14, 2022 •

edited

Loading

jmoisan21 commented Jun 14, 2022 •

edited

Loading

jmoisan21 commented Jun 15, 2022 •

edited

Loading

jmoisan21 commented Jun 17, 2022 •

edited

Loading