[iOS] Links get double-%-encoded, breaking downloads

[gnprice]

The effect of this is that a similar symptom to #3303 is still live on iOS. But the cause is unrelated, so making this a separate issue.

Originally described at #4089 (comment) and #4089 (comment) , just before I merged #4089 fixing #3303 .

To reproduce:

• Went to message list, found a message with an upload that wasn't an image. (To do that: in the webapp searched for has:attachment, and scrolled through history to find one that wasn't an image.) Specifically the file happened to be a PDF, with a .pdf extension on its name.

• Hit the link. Got a browser view. But after a few seconds of loading, the result was an error:
  

  That appears to be from S3 directly, because that's in the location bar at the top. The error message begins:

  SignatureDoesNotMatch The request signature we calculated does not match the signature you provided. Check your key and signing method. [... then a bunch of details ...]

• Ditto on a second try. I watched the timing closely this time, and it was only about a second between me tapping the link and the error message appearing. So it's not an expiration issue -- there's something else wrong.

On further investigation, I have a partial diagnosis:

• From that error page, I hit the "share" icon and chose "Copy", to get the URL onto the clipboard. Then went and pasted it elsewhere (the compose box, as the handiest place.) Here it is:
  https://zulip-uploads.s3.amazonaws.com/1230/-Opc2L055IYelpraPb1oRDeU/sagas.pdf?Signature=2mpNGb0ysKFpVN8bkkMVsulQSVE%253D&Expires=1591317724&AWSAccessKeyId=AKIAIEVMBCAT2WD3M5KQ

• I went and pulled up the same upload in the webapp, to compare. (It works fine there.) Here's the URL I find in the location bar there:
  https://zulip-uploads.s3.amazonaws.com/1230/-Opc2L055IYelpraPb1oRDeU/sagas.pdf?Signature=xipp%2FD69nk89xmkKha3cx6K%2FSSg%3D&Expires=1591317779&AWSAccessKeyId=AKIAIEVMBCAT2WD3M5KQ

• I think the problem is that %253D. That's the percent-encoding of %3D, which is itself the percent-encoding of =. Note there's a %3D at the end of the Signature query-parameter in the successful URL. Both signatures look like base64, which very often ends with a = as padding.

• So it seems like we're double-encoding the URL, and as a result the decoding of it has a signature ending in %3D instead of in = and the signature doesn't validate.

Looking at the code, it's clear where that's happening -- in src/utils/openLink.js, just in the iOS branch, we call encodeURI on the URL. That sure will turn a %3D into a %253D.

Unfortunately it's going to be a bit trickier than just removing that call, because it was put there to fix another bug: 66a9e9d (#3507) fixed #3315.

So it seems like we need to, in openLink on iOS before passing the URL to SafariView.show:

• %-encode non-ASCII characters -- that's Exception in SafariView on non-ASCII URLs ("unsupported scheme") #3315;
• but not %-encode % itself, instead leave it alone;
• and presumably also leave alone all the other characters that encodeURI leaves alone, like a and Z and /;
• and it's not clear if we should %-encode the remaining characters that encodeURI affects, like and " and a bunch of other punctuation.

A key step in fixing this is going to be just end-to-end testing: make a URL filled with a ton of these characters, post it in a Zulip message, try following that link, and see what URL actually comes through.

[strifel]

Same problem when trying to open a link to a big blue button video call.
URL gets converted to
localhost:9991/calls/bigbluebutton/join?meeting_id=%2522zulip-553592035814%2522&password=%2522SEVK77GAJK%2522&checksum=%2522096c3a04e5e26106ac4a1df1f8fb1bfa2f464ed9%2522
but should be
localhost:9991/calls/bigbluebutton/join?meeting_id=%22zulip-553592035814%22&password=%22SRVK14GAJK%22&checksum=%22090c3a04e5e26104ac4a1df1f8fb17fa2f464ed9%22

[baxterdmutt]

But if you are in the iOS app and tap the link and cut and paste into Safari on the same device, the file is downloaded fine. So there is nothing wrong with the link.

[gnprice]

@baxterdmutt Yes, if you long-press the link in order to copy it, that doesn't involve the code that has this bug. This bug applies when you directly open the link within the app.

[baxterdmutt]

Is there a timeline on when this issue will be fixed. It’s a real difficulty for a few of our staff that are not super tech savvy.

[gnprice]

We've just heard another report of this issue, from a user by email.

[fuzunspm]

We have the same issue, if we try to open file via in app browser it won’t open with an error message of

{"result":"error","msg":"Invalid filename"}

but if we copy link and paste in to browser then it loads the file successfully. The url is different between copy and touch to open

[dpo]

Same issue trying to open a PDF file dropped into the conversation.

[chrisbobbe]

We've just heard another report of this issue, from a user by email.

[chrisbobbe]

Looking at the code, it's clear where that's happening -- in src/utils/openLink.js, just in the iOS branch, we call encodeURI on the URL. That sure will turn a %3D into a %253D.

Unfortunately it's going to be a bit trickier than just removing that call, because it was put there to fix another bug: 66a9e9d (#3507) fixed #3315.

Quoting from the linked commit:

The SafariView library apparently crashes when passed a url
parameter that isn't all ASCII. Strictly such a thing isn't
a URL (or URI), in fact, but rather an "IRI":
https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier

But since 74e1418, we're not using SafariView. If we remove the encodeURI call in the current code—

/** Open a URL in the in-app browser. */
// TODO(#4146): Take a URL object, not a string.
export function openLinkEmbedded(url: string): void {
  if (Platform.OS === 'ios') {
    WebBrowser.openBrowserAsync(encodeURI(url));
  } else {
    NativeModules.CustomTabsAndroid.openURL(url);
  }
}

—and if we don't crash on a url that has a non-ASCII character, would we have a good fix?

[chrisbobbe]

But since 74e1418, we're not using SafariView. If we remove the encodeURI call in the current code—

[…]

—and if we don't crash on a url that has a non-ASCII character, would we have a good fix?

Hmm, well I didn't get a crash but there's still a bug, so this won't work out.

With this change:

- WebBrowser.openBrowserAsync(encodeURI(url));
+ WebBrowser.openBrowserAsync(url);

, when I tapped a link with href "https://en.wiktionary.org/wiki/þ", the in-app browser didn't open, and a second later I got this in the console:

Possible Unhandled Promise Rejection (id: 0):
Error: An exception was thrown while calling `ExpoWebBrowser.openBrowserAsync` with arguments `(
    "https://en.wiktionary.org/wiki/\U00fe",
        {
    }
)`: The specified URL has an unsupported scheme. Only HTTP and HTTPS URLs are supported.
Error: An exception was thrown while calling `ExpoWebBrowser.openBrowserAsync` with arguments `(
    "https://en.wiktionary.org/wiki/\U00fe",
        {
    }
)`: The specified URL has an unsupported scheme. Only HTTP and HTTPS URLs are supported.
    at Object.promiseMethodWrapper [as callMethod] (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:5424:36)
    at NativeModulesProxy.<computed>.<computed> [as openBrowserAsync] (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:160179:32)
    at openBrowserAsync$ (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:331035:71)
    at tryCatch (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:24476:19)
    at Generator._invoke (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:24456:26)
    at Generator.next (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:24507:23)
    at tryCatch (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:24476:19)
    at invoke (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:24514:22)
    at http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:24538:13
    at tryCallTwo (http://192.168.43.23:8081/index.bundle?platform=ios&dev=true&minify=false&modulesOnly=false&runModule=true&app=org.zulip.Zulip:28933:7)

which is a match for the issue, #3315, that the encodeURI was meant to fix.

[chrisbobbe]

So we're back to this as the thing to try:

So it seems like we need to, in openLink on iOS before passing the URL to SafariView.show:

• %-encode non-ASCII characters -- that's Exception in SafariView on non-ASCII URLs ("unsupported scheme") #3315;
• but not %-encode % itself, instead leave it alone;
• and presumably also leave alone all the other characters that encodeURI leaves alone, like a and Z and /;
• and it's not clear if we should %-encode the remaining characters that encodeURI affects, like and " and a bunch of other punctuation.

A key step in fixing this is going to be just end-to-end testing: make a URL filled with a ton of these characters, post it in a Zulip message, try following that link, and see what URL actually comes through.

I'm nervous about selectively encoding some things and not others, and it looks like I had the same thought in a comment a few years ago. It seems like that could introduce bugs of its own.

[gnprice]

I'm nervous about selectively encoding some things and not others, and it looks like I had the same thought in a comment a few years ago. It seems like that could introduce bugs of its own.

While following up on a comment you made on another issue thread (#5518 (comment)), I think I found a good clean solution for this concern!

Specifically, I think the right way to implement that selective encoding probably is to round-trip the URL string through new URL. It turns out this

• percent-encodes all non-ASCII characters;
• leaves % alone;
• also leaves alone characters like a and Z and (usually (*)) /;
• and has a very complicated answer for just which ASCII characters it does percent-encode -- an answer that varies from one part of a URL to another.

---

Here's a quick demo, contrasting the URL parser with encodeURI:

> new URL('https://example.com/?a=b%3D&c=þ').href
'https://example.com/?a=b%3D&c=%C3%BE'
> encodeURI('https://example.com/?a=b%3D&c=þ')
'https://example.com/?a=b%253D&c=%C3%BE'

Note that both of them percent-encode þ, addressing #3315; but new URL(…).href refrains from percent-encoding the % in %3D, addressing this issue.

---

In terms of the URL spec, a note there (https://url.spec.whatwg.org/#url-code-points) explains:

Code points greater than U+007F DELETE will be converted to percent-encoded bytes by the URL parser.

Concretely, what's happening in that example is that in query state, the input gets encoded using the special-query percent-encode set; and that contains all code points above U+007E, but does not contain U+0025 %. The same two points are true of each of the other percent-encode sets defined in the standard, which are variously used when processing different parts of the URL.

(*) The "usually" on not processing U+002F / is that the userinfo percent-encode set, but none of the other sets, does include that character. This is the percent-encode set that gets applied to the username and password (!) found before the @ in the authority of a URL like http://johndoe:p4ssw0rd@example.com/. As one might hope, that syntax is not permitted in a "valid URL string" as defined in the current standard. 😛

[gnprice]

(*) The "usually" on not processing U+002F / is that the userinfo percent-encode set, but none of the other sets, does include that character. This is the percent-encode set that gets applied to the username and password (!) found before the @ in the authority of a URL like http://johndoe:p4ssw0rd@example.com/.

(Hmm, as I try to construct a demo, I find that it's actually not possible for a / to reach that point in the actual parsing of a URL. But that same percent-encode set also gets used when setting the username or password on a URL object, so that's where its inclusion of / comes into play. Demo:

> u = new URL('http://johndoe:p4ssw%30rð@example.com/'); u.href
'http://johndoe:p4ssw%30r%C3%B0@example.com/'
> u.password = 'p4ssw/rð'; u.href
'http://johndoe:p4ssw%2Fr%C3%B0@example.com/'

Note the / in the assigned value, and the %2F in the resulting output.

Anyway, the specifics of / and especially of URL "userinfo" is a fun diversion but not something we need to actually care about. 🙂 )