macOS `open` fails to locate file: URL percent-encoding not compliant with RFC 3986 (brackets not percent-encoded)

[taylrfnt]

Hello!

One of the members of the Ghostty community reached out to get help with an issue they were facing in Ghostty on their local machine. After some triage, we discovered that there is an issue with select characters that results in macOS's open command failing when attempting to open a hyperlink output by eza --hyperlink / OSC8 content. This issue is observable in any terminal that uses open for macOS url handling (e.g. Ghostty, Kitty, Alacritty, and likely others I have not mentioned).

TL;DR

TL;DR: This issue stems from a technical disagreement on the characters which should be encoded in a URL. The technical disagreement on URL specifications is discussed in servo/rust-url#541.

Background

The sections below have more details on the resulting issue being observed as a result of the current URL standard being applied, which results in open failing to find the file in select cases.

URL encoding in eza

eza currently uses functions from the rust-url crate to encode urls:

eza/src/output/file_name.rs

Line 426 in 0e40702

let abs_path = utf8_percent_encode(abs_path, CONTROLS).to_string();

rust-urldoes not comply with RFC 3986, and instead aligns with the WHATWG URL Living Standard, where URL reserved characters like square brackets ([ and ]) are not required to be encoded.

Paths using non-ASCII characters

Some users do not use English/ASCII characters and instead use their native language in their filesystem. This means that files and folders in the path may contain unsafe ASCII characters.

For example, the path brought to our attention by a Ghostty user included the following unsafe ASCII: 日本.

rust-url does percent-encode these characters as expected:

> eza --hyperlink
�]8;;file://test/%E6%97%A5%E6%9C%AC�\日本�]8;;�\

If we try to use open with this URL, it will accept the percent-encoded path without complaint, which is great!

I mention this type of file naming/path for reference to how a user typically sees the hyperlink feature, and what is happening in the code itself.

Issue Reproduction & Observations

Paths with URL reserved characters
While it is not my preference, users are permitted to (and do) use URL reserved characters in their filesystems.

If we combine the example provided above with square brackets (a set of URL control characters) into a directory called [日本], we can observe an issue resulting from the differing URL encoding standards that renders hyperlinks unusable in macOS. Note that other unsafe ASCII characters enclosed in square brackets will produce the same result, I just chose the same non-ASCII characters for consistency's sake.

If a user were to use eza to create a hyperlink that contains this directory in the path, the output would be as follows (again, displaying the literal output here, but terminals which support OSC8 would display a hyperlink):

> eza --hyperlink
�]8;;file://test/[%E6%97%A5%E6%9C%AC]�\[日本]�]8;;�\

If the resulting encoded URL is passed to open, the following error is returned:

> open file://test/[%E6%97%A5%E6%9C%AC]
The file file://test/[%E6%97%A5%E6%9C%AC] does not exist.

• Most terminals which support OSC8 end up invoking open on a user's click of the hyperlink, but that can mask the error from open depending on the specific terminal's error handling. I have invoked open manually here to make it apparent, rather than just a click with no response.

Clearly, the file exists - the user just used eza to create a hyperlink for it. So, there must be something going on with open.

If we replace the encoding of the file path with something that complies with RFC 3986, such as jq, we see the path is encoded differently:

> printf %s '[日本]'|jq -sRr @uri
%5B%E6%97%A5%E6%9C%AC%5D

Note

The left square bracket ([) is now encoded to %5B, and the right square bracket (]) is encoded to %5D.

If a user were to provide this version of the encoded path to open, it will work as expected.

While there is no mention of RFC 3986 in the man pages for open, Apple's implementation appears to be a strict interpretation whenever percent-encoded characters are present in the URI contents.

Additional Information

Interestingly, here's a few other observations that are not immediately relevant to the issue, but may be relevant to further discussion about how this could be addressed (either here or upstream):

• open does accept brackets in plain text form (not encoded) when there are no other unsafe ASCII characters in the URL. (e.g. [test] would result in file:///test/[test], which open allows and will successfully spawn a Finder window)
• This issue is limited to macOS. On Linux systems, the xdg-open implementation is less strict and it handles the hyperlinks with plain square brackets that are returned by eza --hyperlink without issue.

---

I do not have a stance on which URL standard is better or which should be implemented in eza or the upstream rust_url. This reports serves to document the issue and present accompanying information in the event this is not a known or intended limitation of hyperlinks in eza. Cheers!