URL rewrite not working for us - adding unsafe to the url?

[adam91holt]

Hi,

We are using the serverless image resizer in a new project and also looking to bring it into an old project. However, we want to use a similar URL to how we used to do image resizing on the server which was like.

/AUS/image001.jpg?h=400&w=400

In the image resizer for the serverless URL this needs to be...

/fit-in/400x400/AUS/image001.jpg

We have come up with the following Regexp for Python as per the docs https://docs.aws.amazon.com/solutions/latest/serverless-image-handler/appendix-b.html

[(r'^([\w\./]+)\?h=([\d]+)&w=([\d]+)$', r'/fit-in/\2x\3\1'),(r'^([\w\./]+)\?w=([\d]+)&h=([\d]+)$', r'/fit-in/\3x\2\1')]

However, this is never resizing the image and in the cloudwatch logs we are getting logs like the following.

[INFO] 2018-09-20T08:40:47.149Z dd22ee1e-bcb0-11e8-8346-6fbe28a7c414 200 GET /unsafe/AUS/image001.jpg (0.0.0.0) 961.66ms

The URL in the log obviously doesn't have the '/fit-in/400x400' and '/unsafe' is added to the URL.

If anyone could help here that would be fantastic. The lambda function was deployed using the cloud formation template and has not been modified. We have specified in the environment variables for the lambda function that rewrite is enabled and the rewrite pattern above.

Thanks

[mc962]

unsafe is added on by default to urls when ALLOW_UNSAFE_URLS is set to True in the conf file.

From what I remember when experimenting with thumbor alone, the url shape needs to have this prepended to it, or it will be seen as an incorrect url by thumbor. This is why this lambda will prepend it to the url for you when you make the request.
https://github.com/awslabs/serverless-image-handler/blob/master/source/image-handler/lambda_function.py#L117

When set to False, you need to prepend an hmac/sha1 token to the url you send to the lambda (forwarded to thumbor), in this case in place of unsafe you get some secure token to prevent url tampering: https://github.com/thumbor/thumbor/wiki/security.

I never did anything with url rewrite feature, so you might be on your own with that, although the code for the rewrite I believe is in the lambda_function link listed above.

[razor-x]

I think this may be a new issue in v3. In v2 both safe and unsafe urls worked, although I was never able to disable the unsafe url. On v3 with the same config the safe urls do not work.

I am not using any rewrite rules, so I wouldn't expect things to work with the rewrite rules added on.

See the notes in my PR here: #55

Does anyone have the safe urls working on v3 with unsafe URLs disabled?

[mc962]

I haven't had a chance to test v3 yet, so I might update this when I am able to do so in a few weeks.

In my setup I managed to get safe urls working while unsafe urls would not work (as expected since they should have been disabled). But I didn't use any environment variables set in cloudformation (which it sounds like is what you were doing, correct me if I am wrong), and instead made the changes to the settings directly in the thumbor.conf file and bundled that change into a new lambda deploy. No rewrite rules were used.

[razor-x]

Good info @mc962, thanks.

Ideally env vars would work, since they are in the official documentation and secure URLs are required for most production deploys. Having to build a custom bundle just to adjust ENV level config options is not ideal for most users and significantly raises the advertised barrier to entry.

I would really like an official response from the AWS maintainers as they have just released v3 and this functionality appears broken, yet all the related issued were closed as fixed.

[gsingh04]

@mc962 @razor-x
for safe url implementation either you update env variables or update the lambda config, both should have the same effect, preferred being updating lambda env variables.

v3.0.0
Safe url is implemented by generating a hash for the options and image URL. The mechanism implemented uses the SECURITY_KEY that you provide as part of the URL for hash generation. The solution takes care of generating hash based on your security key. See here:
https://github.com/awslabs/serverless-image-handler/blob/50856dc17ea7671829f3f940ae27f3a35807b59f/source/image-handler/lambda_function.py#L139
You would notice how we are replacing your security key from the URL with the hashed value before sending it to thumbor to process. thumbor verifies if it matches the hash that it generates using the security key from thumbor.conf.
You can use the demo UI to see it working, as well as refer here.

v2.2
In this version the solution was not calculating the hash and you were supposed to provide the hash as part of the request URL.
You can try this Link which uses the above mentioned mechanism from v2.2 for safe URL implementation.

I believe the only difference in above 2 methods being hashing offloaded to lambda. Please feel free to share your thoughts/concerns.

[razor-x]

@gsingh04 Thank you for the response, it's clear there are two separate issues now:

1. Currently setting ALLOW_UNSAFE_URLS=No on the lambda does not disable the unsafe URLs.

   • The expected behaviour is that setting this to any value supported by strtobool would disable safe urls. Instead they remain enabled.

2. The new behaviour of v3.0.0 that changes the safe URLs from /<hash>/<path> to /<security_key>/<path> is incorrect and completely defeats the purpose of the safe URL. The security key cannot appear in the public url to an image, only the hash can appear in a public url.

   • The purpose of the safe URL is to prevent un-trusted 3rd parties ("the world") from "guessing" image urls.
   • This works by hiding the SECURITY_KEY from the world and using it on the backend to generate the hashed urls to send to the world.
   • Putting the SECURITY_KEY in url and generating the hash means the key is now exposed to the world.
   • Effectively, the new behaviour functions no differently from the unsafe urls, just with the value unsafe reamed to SECURITY_KEY.
   • Please refer to the official docs for more info on this: https://github.com/thumbor/thumbor/wiki/security

[gsingh04]

@razor-x
thanks for raising the concern, please allow us sometime to address it.

[nils-acumen]

I'm confused. My expectation would be that the returned url would be something along the lines of /unsafe/fit-in/400x400/AUS/image001.jpg rather than /unsafe/AUS/image001.jpg.

Can anyone shed some light as to why this isn't the case?

[gsingh04]

@nils-acumen you are right with your expectation, issue being discussed with @razor-x is separate and related to safe url implementation.
@adam91holt I was able to replicate your issue. Issue is not with your re-write pattern but you need to take into account the query strings and CloudFront behavior. Please enable DEBUG log level on lambda before moving any further.
1. You need to enable CloudFront to forward query strings, by default the solution deployed CF distribution doesn’t have it enabled. You can follow these steps to achieve it:

• Go to CloudFront console https://console.aws.amazon.com/cloudfront/home
• Select the distribution deployed by the solution
• Go to ‘Behaviors’ tab, you should see 1 behavior created as part of the solution
• Click Edit and scroll down to ‘Query String Forwarding and Caching’. Select ‘Forward All, cache based on all’ and click ‘Yes, Edit’ at the bottom of the page.
• Go back to distributions, your distribution ‘Status’ would be in-progress. It takes about 10 minutes for the distribution status to change to ‘Deployed’

Now query strings will be forwarded by the CF distribution to microservice. It can be observed in the lambda debug level logs as well.

2. You need to re-construct the URL on lambda with the query strings and then pass the re-constructed URL for rewrite. See below sample code, you would need to make similar change to request_thumbor method

logging.debug('original_request: %s' % (json.dumps(original_request)))
http_path = original_request['path']
#testing request with query strings
queryStringParameters = original_request['queryStringParameters']
url_parts = list(urlparse.urlparse(http_path))
query = dict(urlparse.parse_qsl(url_parts[4]))
query.update(queryStringParameters)
url_parts[4] = urlencode(query)
#updated http_path
http_path=urlparse.urlunparse(url_parts)
logging.debug('reconstructed path: %s' % (http_path))
http_path = rewrite(http_path);
logging.debug('path after rewrite: %s' % (http_path))

Once I followed these steps I was able to make the URL work with similar rewrite pattern. See below log entry for the relevant event.

[DEBUG] 2018-10-04T17:39:54.824Z 812ff60d-c7fc-11e8-b0f3-a18b1e9e7d34 original path "/serverless-image-handler-ui/img/face2.jpg?h=500&w=500", applyed pattern "^([\w+-./]+)?h=([\d]+)&w=([\d]+)$", result "/fit-in/500x500/serverless-image-handler-ui/img/face2.jpg"
[DEBUG] 2018-10-04T17:39:54.824Z 812ff60d-c7fc-11e8-b0f3-a18b1e9e7d34 path after rewrite: /fit-in/500x500/serverless-image-handler-ui/img/face2.jpg
[DEBUG] 2018-10-04T17:39:54.824Z 812ff60d-c7fc-11e8-b0f3-a18b1e9e7d34 http_path: /unsafe/fit-in/500x500/serverless-image-handler-ui/img/face2.jpg

[razor-x]

@gsingh04 Would you like me to create separate issues for 1 and 2 in my comment above so this one can go back on topic to address the original submitter's issue?

[gsingh04]

@razor-x that would be better, this issue is not related to how we implement safe url.
However, I do not believe 1 is an issue. I verified with ALLOW_UNSAFE_URL=No and I was able to disable unsafe url. I noticed in your above correspondence you mentioned ALLOW_UNSAFE_URLS=No, which has a typo. Can you make sure you are using the correct parameter. We can open a separate issue for 2. Were you able to use and verify the link that I shared with you if it was working as desired.

@adam91holt we will leave this issue for you to close.

[razor-x]

@gsingh04

1. I've submitted a new issue here which clearly demonstrates the issues I have raised about v2 and v3, hopefully all cases are reproducible on your end, at least for v3. Secure and unsafe URLs do not work as expected  #58

2. You are right about the typo in my comment which I copied from URL rewrite not working for us - adding unsafe to the url? #54 (comment), however in my actual deployment I use the correct env var name (see the screenshots).

3. Please consider merging my 2 PRs or noting why they cannot be merged yet. Even with the issues around the URLs, I believe they are still valid fixes both now and after the URL issues are resolved.

[gsingh04]

@adam91holt please close the issue if above instructions helped resolve it.