URL Decoding on server is not correct

[olivercoad]

In order to handle any user input correctly (see #4), it has to be URL encoded before placing in the URL.
For example, in javascript, encodeURIComponent("asdf ") returns asdf%20%20.
This generates the following face.

Unfortunately, that's not the only way spaces (and probably other things) can be URL encoded.
For example, an HTML form request or System.Web.HttpUtility.UrlEncode will encode spaces as + rather than %20: System.Web.HttpUtility.UrlEncode("asdf ") returns asdf++.
This generates the following face, noticably different to above.

Furthermore, it returns the same face for asdf%2B%2B as asdf++; %2B is the url encoding of +.

I expect the problem is to do with how flask automatically decodes parts of the path to pass as the hash parameter

checkface/src/server/checkface.py

Lines 131 to 132 in 594152d

	@app.route('/api/<path:hash>', methods=['GET'])
	def image_generation(hash):

Expected Behaviour:
The generated face for the url encodings as asdf%20%20 and asdf++ should both be the same (should both be the first image), corresponding to the user input of asdf .
The generated face for the url encoding as asdf%2B%2B should be different (it should be the second image), and correspond to the user input of asdf++.

[olivercoad]

Apparently github's issue markdown won't preserve multiple spaces. The above examples are for
"asdf "
without the quotation marks and with two spaces at the end. This is inconsequetial to the issue though - it doesn't matter where or how many spaces are used.

[cdilga]

I would have thought that any client bears responsibility of encoding correctly - if it's encoded differently as ++, then we should just respect whatever is sent. If there are spaces sent by the client as UrlEncoded %20%20 then I don't see a problem.

I'm not sure why we should expect asdf%20%20 and asdf++ to both be the same.

I was wondering what the consensus was on handling this, as I presume it would come up often, and this stack overflow post summarises that they have distinct purposes depending on where they are placed

[olivercoad]

Now in the query part, spaces may be encoded to either "+" (for backwards compatibility: do not try to search for it in the URI standard) or "%20" while the "+" character (as a result of this ambiguity) has to be escaped to "%2B".

This is the problem, they are two different but both valid encodings of the same user input.

I'm not sure why we should expect asdf%20%20 and asdf++ to both be the same.

I partly agree because the url encoding is an implementation detail and from the user's point of view it makes no difference (we could just as easily base64 encode it or use any other encoding instead). However we do need the same user input to produce the same image regardless of the client, and currently that's not the case.

[olivercoad]

Actually even javascript's decodeURIComponent doesn't decode + to ... https://stackoverflow.com/a/20247435

[cdilga]

Surely we can just advise a url encoding scheme for all clients in an API doc, which should resolve this issue from a dev perspective on new clients using our API, including our own client

[olivercoad]

It gets worse...

We really do need to actually fix it on the server side, otherwise we can't start with a slash.
pallets/flask#900
Luckily it will be pretty easy to fix. I realised that of course flask isn't decoding a plus to escape because it's part of the path not the query parameter (I was thinking of it like encoding a query parameter because that's what we do for the frontend site). So we just need to put the encoded request in a query parameter instead of the path and it will work correctly.
Google Colab notebook for testing

We can still leave the original endpoint path working for legacy purposes (and it's a little more elegant for handwriting in img tags 😄)