-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dealing with Escaped Characters in S3 Key Names #3584
Comments
Hi @hossimo thanks for reaching out. Regarding this point:
When I tried to pass a tag value containing a "%" to the put_object_tagging command, it fails with the error: So the failure is due to an invalid tag rather than an invalid key. If you passed a valid tag but invalid key, then you would get an error like: So I think the PutObjectTagging API is behaving as expected. This is the S3 documentation I found on using special characters in key names: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html. I couldn't find anything related to special characters and tagging so maybe that needs to be clarified somewhere in the docs. |
Hey Tim, thanks for getting back on this. Using It's not what was getting passed to the Hopefully a more concise example: I have S3 passing events to SNS and SNS trigging the Lambda. The user uploads a file to S3 called This comes to Lambda as an SNS event containing the following {
"Records": [
{
"s3": {
"bucket": {
"name": "bucket"
},
"object": {
"key": "this+is+a+test.txt"
}
}
}
]
} Now notice that the The issue is now, I want to apply some tags to this bucket/key I just received, but in order to do this you have to un-escape it first. So here is an example using this above simplified SNS/Message block: import boto3
import json
def lambda_handler(event, context):
s3 = boto3.client("s3")
event = json.loads(event)
for record in event["Records"]:
bucket = record["s3"]["bucket"]["name"]
objectName = record["s3"]["object"]["key"] # this+is+a+test.txt
tags = []
if "Pizza" in objectName:
tags.append({"Key" : "Project", "Value" : "Great"})
if "Hamburger" in objectName:
tags.append({"Key" : "Project", "Value" : "Good"})
if "Liver" in objectName:
tags.append({"Key" : "Project", "Value" : "Yuck"})
### This will fail
try:
s3.put_object_tagging(
Bucket=bucket,
Key=objectName, # this+is+a+test.txt
Tagging={
"TagSet" : tags
}
)
except Exception as e:
print("FAIL")
print(e)
## Un-Escape the Key
objectName = format_path(objectName)
### This will pass
try:
s3.put_object_tagging(
Bucket=bucket,
Key=objectName, # this is a test.txt
Tagging={
"TagSet" : tags
}
)
except Exception as e:
print("PASS")
print(e)
def format_path(path):
path = path.replace("+", " ")
path = path.replace("%2B", "+")
path = path.replace("%21", "!")
path = path.replace("%27", "'")
path = path.replace("%28", "(")
path = path.replace("%29", ")")
path = path.replace("%26", "&")
path = path.replace("%24", "$")
path = path.replace("%40", "@")
path = path.replace("%3D", "=")
path = path.replace("%3B", ";")
path = path.replace("%3A", ":")
path = path.replace("%2C", ",")
path = path.replace("%3F", "?")
return path
if __name__ == "__main__":
event = """{
\"Records\": [
{
\"s3\": {
\"bucket\": {
\"name\": \"bucket\"
},
\"object\": {
\"key\": \"this+is+a+test.txt\"
}
}
}
]
}"""
lambda_handler (event, None) So now as long as you have a bucket called
This is all expected behaviour. the issue is I need to build and maintain the What if a user uploads a valid windows file name I hope this makes it clearer. Thanks again for your time. |
Hi @hossimo thanks for following up — have you tried using |
Ok well now I feel a little dumb I honestly didn't think of url encoded strings, in my head I was thinking of some special AWS sauce, maybe it was the pluses that threw me off. Anyway that's a simple fix for my problem with. Thanks. |
Ok I'm glad that fixes the issue — and I'm glad you opened this in case others run into it as well. I don't have much direct experience with S3->SNS->Lambda workflows but it might be worth noting this information somewhere in the service documentation. |
Sorry, just one more thought. While I agree there is a simple way around this and it is likely in a manual somewhere. Wouldn't this be a safer (less likely someone would make a bonehead mistake like me) solution: s3.put_object_tagging(
Bucket=bucket,
Key=objectName, # this+is+a+test.txt
Decode=True,
Tagging={
"TagSet" : tags
}
) This way I'm shifting the blame, but it keeps this managed by the API. Sure on the backend it could just be calling |
@hossimo we would have to forward requests for adding API parameters to the appropriate service team (we also recommend doing this directly through AWS Support if you have a support plan.) I could forward your suggestion for the PutObjectTagging API to the S3 team but maybe instead it should be a request for the SNS team if it is an SNS API that is adding the special characters? |
Ok got it. I do have a support plan, I wasn't sure they were related to this as well. I'll close this up for now. Thanks again |
Ok thanks for letting me know, if there are any updates from the support ticket that you'd like to share here in the future please feel free to do so. Otherwise I can go ahead and close this. |
Describe the feature
I've asked this over on SO with no reply, and I apologize if this is located somewhere in the Docs and I haven't found it.
Often a file gets uploaded into my buckets that contain characters that need to be escaped. I have no control of what my users will call their file based on their workflows. This is all fine and good until you want to process that file and need to escape. This isn't hard but seems like there should be somewhere in the API to do this for you safely.
As an example, the below in the Use Case section with
Pizza-Is-Better-Than-Liver.txt
but fails withPizza Is+AmazeBalls.txt
Obviously, when
Pizza Is+AmazeBalls.txt
fails it's because the key returned is actuallyPizza+Is%2BAmazeBalls.txt
but passing that directly toput_object_tagging
fails since that's not the actual key of the object.If it doesnt exist I would suggest something like the below proposed solution packed into some comon part of the library where it makes sense like
boto3.client.escape(input: str)
(sorry I dont fully understand the bakend, just an eample)Use Case
Proposed Solution
Other Information
No response
Acknowledgements
SDK version used
through boto
Environment details (OS name and version, etc.)
boto3 1.26.41
The text was updated successfully, but these errors were encountered: