Allow non utf-8 char in uri

[martinos]

I am using Phoenix as a SMS gateway. My app is receiving request from another SMS Gateway (https://www.kannel.org/) which sends a GET request to my app. The uri of the request contains a escaped binary that cannot be converted back to utf-8. This gives me the following msg in my logs:

[debug] ** (Plug.Conn.InvalidQueryError) invalid UTF-8 on urlencoded params, got byte 159

This was working fine on v1.8.3.

In order to reproduce this you can add %9f in any query string. The web server will return silently a 400

This is causes a major issue on my side since we can't change the query string from the sender.

I would be great if we could add an option to validate or not the incoming urlencoded params.

[josevalim]

Hi @martinos, can you please send a PR that allows validation to be disabled? You want to look at Plug.Conn.fetch_query_params/2, probably to add an option that says validate_utf8: false. IIRC, we already have such an option for Plug.Parsers.

[martinos]

Hi @josevalim, the validation is made deep in the Query.decode function (https://github.com/elixir-plug/plug/blob/master/lib/plug/conn/query.ex#L99). If I want to put the condition there, it would imply passing the option down to the Query.decode_www_form function, passing through about 5 functions.

Another solution would be to do the validation before doing the actual query decoding. It would imply to unescape the full query string then do the utf8 validation. We would then decode the query string. This might increase the request time since the escaping would be done twice (once before decoding the query and once when parsing the individual parameters). However the validation function would only be called only once.

I think that the second solution would be best codewise but I would like to have your feedback.

[josevalim]

I wouldn't want to make query handling more expensive for an optional case, so unfortunately I think option 1 is the best way to go. In Plug sometimes we have to write "ugly" code, so everyone else doesn't have to. :)

[jn-jairo]

I am using phoenix, where do I add the option to disable the validation? I tried to add it in the get but it is not working.

# router.ex

  scope "/", MyAppWeb do
    pipe_through :api

    get "/foo", FooController, :foo, validate_utf8: false
  end

Edit: ChatGPT helped me, that was the answer:

To disable URL decoding validation for UTF-8 in the Phoenix framework for Elixir, you can add the option validate_utf8: false to the plug configuration in your endpoint.ex file. Look for the plug responsible for handling URL decoding, typically named Plug.Parsers. Then, add the option within the configuration for this plug. Here's how you can do it:

# In your endpoint.ex file
plug Plug.Parsers,
  parsers: [:urlencoded, :multipart, :json],
  pass: ["*/*"],
  validate_utf8: false # Add this line to disable UTF-8 validation

By setting validate_utf8 to false, you're telling Phoenix not to perform UTF-8 validation during URL decoding. This can be useful in certain scenarios where you're dealing with non-standard URL encoding.