netrw: Opening a URI-encoded URL by `gx` will open corrupted URL

[KSR-Yasuda]

Steps to reproduce

1. Write some URL-encoded URL such as https://www.example.com/#:~:text=%E3%81%82 in some buffer.
2. Put cursor on the URL, and run gx shortcut (= NetrwBrowseX) in normal mode.
3. It tries to open the target URL in the web browser, but the opening page URL is corrupted
   like https://www.example.com/#:~:text=/%E3/%81/%82.

Here it is calling extra escape() that causes this.
Due to this, it would fails with paths including # or %, the characters escaped here.

The comment for it says they've added the escape() since shellescape(..., 1) in netrw#Open() is insufficient.

Is the escape() really required now?

Expected behaviour

It opens the correct URL.

Version of Vim

9.1.866

Environment

• Operating system: Windows 11 Pro 64bit 24H2 (Version 10.0.26100.2605)

  • MSYS2 (msys2-runtime 3.5.4-8)

• Terminal: mintty '3.7.6' 2024-09-24_05:10 (x86_64-pc-msys)

  • Value of $TERM: xterm-256color
  • Shell: GNU bash, version 5.2.37(1)-release (x86_64-pc-msys)

• Vim g:netrw_browsex_viewer value: start

• Browser: Google Chrome v131.0.6778.140

Logs and stack traces

No response

[chrisbra]

I remember when that comment was made, I wondered myself about it whether this is true. At least I cannot reproduce it:

:exe '!echo ' .. shellescape('one#two%three four',1)

seems to work just fine. So perhaps we can just get rid of the double escape here? @Konfekt

[Konfekt]

I wondered myself about it whether this is true

Everybody is wondering about what's happening.

Testing was not exhaustive, but it made common URLs with hashes such ashttps://github.com/vim/vim/issues/16252/#issue-2749168683 work, so I assumed % (current buffer) needed similar treatment to # (alternate buffer).
On Ubuntu 22.04 with Vim 9.1.919, removing that escaping of # breaks it.

It may be that only # is causing trouble, as with

" runtime/autoload/netrw.vim (line 5202)
call netrw#Open(escape(fname, '#'))

gx opens the URL https://www.example.com/#:~:text=%E3%81%82 fine.

What's curious is that @KSR-Yasuda 's URL is escaped with forward instead of backward slashes, though: https://www.example.com/#:~:text=/%E3/%81/%82.
On Ubuntu 22.04 with Vim 9.1.919 backward slashes are added at % but a forward slash at #: https://www.example.com//#:~:text=\%E3\%81\%82.

[Konfekt]

shellescape(..., 1)

Since @KSR-Yasuda uses MSYS on Microsoft Windows, it also looks like shellescape(..., 1) does not work as intended here? Maybe there are two issues at play.

[KSR-Yasuda]

What's curious is that @KSR-Yasuda 's URL is escaped with forward instead of backward slashes, though: https://www.example.com/#:~:text=/%E3/%81/%82. On Ubuntu 22.04 with Vim 9.1.919 backward slashes are added at % but a forward slash at #: https://www.example.com//#:~:text=\%E3\%81\%82.

Probably it is because of path conversion of MSYS2 env.

Manually executing start 'https://www.example.com/\#:~:text=\%E3\%81\%82' opens
https://www.example.com//#:~:text=/%E3/%81/%82.
Every \ chars are replaced with /.

Even if you drop %-escape, #-escape may cause the problem in MSYS2.
For https://www.example.com/#:~:text=%E3%81%82#a,
it opens the URL https://www.example.com/#:~:text=%E3%81%82/#a (extra / for the latter #).

[Konfekt]

Even if you drop %-escape, #-escape may cause the problem in MSYS2.

Maybe "" //b are not required here if you use the Git Bash version of MSYS2

" autoload/netrw.vim (line 5032)
exe 'silent !start "" //b' a:args s:redir() | redraw!

or whether cygpath comes installed with your MSYS2

Probably it is because of path conversion of MSYS2 env.

One could prepend MSYS_NO_PATHCONV=1on Git Bash according to https://github.com/git-for-windows/build-extra/blob/main/ReleaseNotes.md#known-issues or MSYS2_ARG_CONV_EXCL='*' on MSYS2 according to https://www.msys2.org/docs/filesystem-paths/

I wonder if the authors @vim-jp of https://github.com/vim-jp/vital.vim maybe already worked out all the edge cases long ago and could give definite replies.

[Konfekt]

@KSR-Yasuda If you could find a combination of above knobs that suits you?

[KSR-Yasuda]

shellescape(..., 1)

Since @KSR-Yasuda uses MSYS on Microsoft Windows, it also looks like shellescape(..., 1) does not work as intended here?

shellescape(..., 1) itself looks fine in MSYS2 env.

shellescape('https://www.example.com#:~:text=%E3%81%82#a', 1)

" => 'https://www.example.com\#:~:text=\%E3\%81\%82\#a'

Probably it runs
the vim command like :exe 'silent !start ' . 'https://www.example.com\#:~:text=\%E3\%81\%82\#a',
that implies the bash command start https://www.example.com#:~:text=%E3%81%82#a.

Rather Ubuntu env. is exceptional to need extra escape(), isn't it?

There defines netrw#Launch() for each env.
If still extra escape() is needed, shouldn't you call it there?
(The one for Ubuntu would be this?)

Otherwise, instead of escape(), you might had better quote the path.

Even if you drop %-escape, #-escape may cause the problem in MSYS2.

Maybe "" //b are not required here if you use the Git Bash version of MSYS2

" autoload/netrw.vim (line 5032)
exe 'silent !start "" //b' a:args s:redir() | redraw!

No, without "" //b, it opened command prompt... (start command would have went wrong.)

My env is pure MSYS2, if you mean Git for Windows.

[Konfekt]

Rather Ubuntu env. is exceptional to need extra escape(), isn't it?
... If still extra escape() is needed, shouldn't you call it there?

Not so sure how special an issue of Ubuntu that is. Same phenomenon on Debian 12 and thus likely its forks such as Linux Mint and others.

It would help if Apple and Microsoft Windows users could confirm that they did not need it, and it is maybe rather a Linux phenomenon (though I suspect that Mac OS behaves similar).

No, without "" //b, it opened command prompt...

Thank you, that's good to know.

My env is pure MSYS2

Okay, does it come with cygpath? The above link https://www.msys2.org/docs/filesystem-paths/ claimed so, but I think mine didn't. So you are using the start script?

[chrisbra]

Testing was not exhaustive, but it made common URLs with hashes such as #16252 work, so I assumed % (current buffer) needed similar treatment to # (alternate buffer).
On Ubuntu 22.04 with Vim 9.1.919, removing that escaping of # breaks it.

I don't think this is true:

:exe '!echo ' .. shellescape('https://github.com/vim/vim/issues/16252/#issue-2749168683',1)
https://github.com/vim/vim/issues/16252/#issue-2749168683

Press ENTER or type command to continue

so I assume we can get rid of the double escape() ?

The path converstion thingy is tricky however. Not sure we should try to open that can of worms.

[Konfekt]

so I assume we can get rid of the double escape() ?

Strange, but we can try and time will tell

The path converstion thingy is tricky however. Not sure we should try to open that can of worms.

One could prepend MSYS_NO_PATHCONV=1on Git Bash according to git-for-windows/build-extra@main/ReleaseNotes.md#known-issues or MSYS2_ARG_CONV_EXCL='*' on MSYS2 according to msys2.org/docs/filesystem-paths

If @KSR-Yasuda has success with either of them, why not? Git Bash and MSYS2 can be distinguished by environment variables

[KSR-Yasuda]

One could prepend MSYS_NO_PATHCONV=1on Git Bash according to git-for-windows/build-extra@main/ReleaseNotes.md#known-issues or MSYS2_ARG_CONV_EXCL='*' on MSYS2 according to msys2.org/docs/filesystem-paths

If @KSR-Yasuda has success with either of them, why not? Git Bash and MSYS2 can be distinguished by environment variables

Anyway, I think it is not the point.
Appearance of extra / or \ itself would be wrong already.

MSYS_NO_PATHCONV is just an option to stop path separator conversion, isn't it?

Even if you drop %-escape, #-escape may cause the problem in MSYS2. For https://www.example.com/#:~:text=%E3%81%82#a, it opens the URL https://www.example.com/#:~:text=%E3%81%82/#a (extra / for the latter #).

As I said, even escape(fname, '#') introduces extra / before #.

For the first #, it does not make a problem.
But the latter #, the anchor part #ANCHOR#a and #ANCHOR/#a are completely different things.

[Konfekt]

Anyway, I think it is not the point.
Appearance of extra / or \ itself would be wrong already.
MSYS_NO_PATHCONV is just an option to stop path separator conversion, isn't it?

Yes, that's the second issue, that eventually should also be fixed if additional escaping has to be done.

I think what caused confusion is that despite

" autoload/netrw.vim (lines 5112-5114)
fun! netrw#Open(file) abort
  call netrw#Launch(s:viewer() .. ' ' .. shellescape(a:file, 1))
endf

:Open https://github.com/vim/vim/issues/16252#issuecomment-2556009926 still does not work, but requires escaping #. However, Itt's better to force the user to escape it if this causes issues elsewhere.