URL fragment encode/decode of special characters

[Jand42]

Summary

Some special characters occurring in a string that is inside an Endpoint value will map to URLs that are denied by default by ASP.NET settings and possibly by other hosting environments.

For example : is encoded to /%3A fails with default requestPathInvalidCharacters setting, a newline character encoded to /%0A fails even with an empty setting.

Also, strings sent as FormData (either through a form submit or using Router.Ajax) will be encoded/decoded separately, the extra decode in a Router<string> might eliminate occurrences of %xx which are meant to be sent as-is.

Details

The implementation for WebSharper 4.1's Router.rString and similarly Router.Infer<string> uses standard URL fragment decode/encode for parsing/writing URLs. This is conceptually wrong, as a Router<string> value may be used for a FormData value, which is already decoded through another mechanism on the server. So decoding/encoding URL fragments should be separate from just string handling, but done on only real URL parts when converting a request to the Path record interpreted by a Router.Parse.

Also this makes it possible to customize the URL fragment decode/encode mechanism on a global level.

Most URL format restrictions can be lifted in IIS by having

  <system.web>
    <httpRuntime requestValidationMode="2.0" requestPathInvalidCharacters="" />
    <pages validateRequest="false" />
  </system.web>
  <system.webServer>
    <security>
      <requestFiltering allowDoubleEscaping="true" />
    </security>
  </system.webServer>

but not all, for example the : and ? characters are still failing.

WebSharper 3.0/4.0-s custom URL fragment encoding could be revived and made client-side compatible for an alternative option, for perfect back-compatibility and also tested if that covers more special characters.

Proposal

• Make encoding/decoding URL fragments a separate concern from reading string fragments, done in one step on the whole URL before running parsing logic or after writing the segments by a Router object.
• As IRouter (type used inside a Sitelet) deals with System.URIs directly instead of a broken down representation, this conversion must happen where Router type (and the optimized, internal server-only version) implements IRouter.
• First, revive the exact WebSharper 4.0 implementation make it client-side compatible, and test it for the same array of special characters. Then there are 2 possible way of fragment encode/decode but no dynamic way of switching between them.
• If we decide that a dynamic switch is needed because neither are not fully covering wanted use cases, it can be either a mutable global value, or an additional field on Router, containing two string->string. functions.

[Jand42]

String encoding for routed URLs was reverted to WebSharper 4.0-s custom transformation (any non-alphanumeric character replaced to ~xx or ~uxxxx). Support was added for this to work on the client-side too. This ensures that string values are recovered properly and are passing standard URL correctness checks.

This logic is still tied to strings, not any URL fragment, so a custom-defined Router value would not be affected.