Google API clients are improperly escaping URLs

[chrsmith]

Google API clients are improperly escaping URLs

This issue has been causing problems for users of Google APIs for years. For example, Issue #454 , Issue #534, Issue #636, and I'll bet there are others. This is the issue to track work for making this just work for API clients, regardless of .NET framework target and without needing to modify code or config.

Problem

There are two different behaviors of System.Uri depending on the version of the .NET Framework you are using. @mmdriley put together a great repro in Issue #636. See [Microsoft Connect 511010](https://connect.microsoft.com/VisualStudio/feedback/details/511010/erroneous-uri-parsing-for-encoded-reserved-characters-according-to-rfc-3986 Known deviation from the URI spec), Microsoft Connect 94109.

Whereas System.Uri in .NET 4.0 will convert "%2F" into "/" when encoded into a URL. .NET 4.5 will leave the "%2F" as-is. The problem is that certain APIs require a "%2F" and will fail if there is a "/" instead.

This behavior is especially problematic when using the Google Cloud Storage API. Storage objects usually have slashes in their name, so when URLs get generated for API calls the URL is invalid. For example, to get the GCS object with name "test/file1.txt" you need to generate a URL such as:

GET https://www.googleapis.com/storage/v1/b/gcs-bucket-name/o/test%2Ffile1.tmp?key={API_KEY}

So with the .NET 4.0 behavior, the request will be invalid:

GET https://www.googleapis.com/storage/v1/b/gcs-bucket-name/o/test/file1.tmp?key={API_KEY}

Note that while .NET 4.0 is nearly out of its lifespan, .NET 4.5 will emulate the 4.0 behaviour for assemblies that are targeting .NET 4.0. (e.g. from an assembly-level attribute.)

Work Around

There are a couple of easy work-arounds for this issue.

First, the easiest thing to do is to have your assembly explicitly target the .NET Framework version 4.5. This can be done within Visual Studio and/or by specifying an assembly-level attribute.

This option isn't ideal because using a Google API shouldn't force you to target a specific version of the framework.

Second, you can modify your app.config or web.config file to modify the behavior. Just add this stanza to the file:

<uri>
  <schemeSettings>
    <add name="http" genericUriParserOptions="DontUnescapePathDotsAndSlashes" />
  </schemeSettings>
</uri>

This option isn't ideal because some scenarios don't have an appropriate config file. For example, when the assembly is hosted such as in PowerShell modules. Also, again, using a Google API shouldn't force you to set application-wide level policy.

Possible Solutions

This SO post has a hack that used reflection to poke at the private fields in System.Uri to get consistent behavior.

In fact, this project used to use the reflection hack but it was removed later.

Alternatively, we can be hoist the Uri generation into a separate AppDomain, which is forced to run with the 4.5 behavior.

Alternatively, we can be use some heuristics and double-encode the data as needed and trick the 4.0 System.Uri.

There might even be a better solution.

I'd like to welcome suggestions for how to proceed.

[jskeet]

Alternatively, we can be use some heuristics

It looks like this wouldn't so much be "heuristic" as "try with a known URL, and see what the AbsoluteUri property comes back as" - as per the repro in #636. So we should at least be pretty confident in it.

[mmdriley]

here's my first draft:
https://gist.github.com/mmdriley/a219d139fa4a56da483c

see also, similar code in the AWS .NET SDK:
https://github.com/aws/aws-sdk-net/blob/e1a0c61f0d852b9ae9050c8794d7476e21a2bc88/sdk/src/Core/Amazon.Runtime/AmazonServiceClient.cs#L410

[mmdriley]

Passing thought: can .NET Native applications claim to target 4.0? If so, we might look for reflection data that's not there.

[jtattermusch]

Possibly related: I also spotted this problem when I was trying to make unit tests pass on mono (at that time I thought this is mono vs .NET incompatibility issue):
https://github.com/google/google-api-dotnet-client/pull/616/files#diff-58a277240f102603523e70fe06e2a816R958

[mmdriley]

Fixed by #698 and released in v1.11.