Expanding special characters anywhere in a command string

[yegappan]

The expand() function can be used to expand special characters. If the special
characters appears anywhere in the command string, then this cannot be used.

Vim supports expanding special characters anywhere in a command string
for internal commands. For example, the 'makeprg' option accepts a command
string where special characters can occur anywhere in the string.
But a Vim plugin cannot support this. So plugins like dispatch.vim use a complicated
function to expand the special characters (look at the dispatch#expand() function in
https://github.com/tpope/vim-dispatch/blob/master/autoload/dispatch.vim).

This patch adds a expandcmd() function that can be used to expand all the
special characters in a command string.

[codecov-io]

Codecov Report

Merging #4514 into master will decrease coverage by 0.22%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4514      +/-   ##
==========================================
- Coverage   80.63%    80.4%   -0.23%     
==========================================
  Files         110      110              
  Lines      143701   141088    -2613     
==========================================
- Hits       115874   113448    -2426     
+ Misses      27827    27640     -187

Impacted Files	Coverage Δ
src/evalfunc.c	89.19% <100%> (-0.1%)	⬇️
src/version.c	56.75% <0%> (-32.19%)	⬇️
src/libvterm/src/unicode.c	87.17% <0%> (-7.7%)	⬇️
src/if_cscope.c	75.5% <0%> (-0.93%)	⬇️
src/gui.c	57.2% <0%> (-0.91%)	⬇️
src/search.c	81.22% <0%> (-0.84%)	⬇️
src/move.c	86.58% <0%> (-0.72%)	⬇️
src/ui.c	54.98% <0%> (-0.7%)	⬇️
src/tag.c	77.82% <0%> (-0.69%)	⬇️
src/gui_gtk_x11.c	48.5% <0%> (-0.62%)	⬇️
... and 85 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 260addf...d708abb. Read the comment docs.

[brammool]

Yegappan wrote:

The expand() function can be used to expand special characters. If the special characters appears anywhere in the command string, then this cannot be used. Vim supports expanding special characters anywhere in a command string for internal commands. For example, the 'makeprg' option accepts a command string where special characters can occur anywhere in the string. But a Vim plugin cannot support this. So plugins like dispatch.vim use a complicated function to expand the special characters (look at the dispatch#expand() function in https://github.com/tpope/vim-dispatch/blob/master/autoload/dispatch.vim). This patch adds a expandcmd() function that can be used to expand all the special characters in a command string.

Thanks. I updated the docs to mention that the expansion happens like for the ":edit" command. Other commands, such as ":next" work slightly differently. I also reported the error, when there is one. There is still the case it returns with an error, but without an error message. Perhaps we should also fail then with some generic message?

…

-- hundred-and-one symptoms of being an internet addict: 134. You consider bandwidth to be more important than carats. /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[vim-ml]

Hi Bram,

On Sun, Jun 9, 2019 at 9:05 AM Bram Moolenaar ***@***.***> wrote: Yegappan wrote: > The expand() function can be used to expand special characters. If the > special characters appears anywhere in the command string, then this > cannot be used. > > Vim supports expanding special characters anywhere in a command string > for internal commands. For example, the 'makeprg' option accepts a command > string where special characters can occur anywhere in the string. > But a Vim plugin cannot support this. So plugins like dispatch.vim use a complicated > function to expand the special characters (look at the dispatch#expand() function in > https://github.com/tpope/vim-dispatch/blob/master/autoload/dispatch.vim). > > This patch adds a expandcmd() function that can be used to expand all the > special characters in a command string. Thanks. I updated the docs to mention that the expansion happens like for the ":edit" command. Other commands, such as ":next" work slightly differently.

Thanks for merging the patch. Hopefully this is sufficient for plugins to expand options like 'makeprg', 'grepprg', 'equalprg', 'formatprg', 'keywordprg', etc.

I also reported the error, when there is one. There is still the case it returns with an error, but without an error message. Perhaps we should also fail then with some generic message?

Are you referring to the case where expand_filename() returns an empty error message string? Regards, Yegappan

[vim-ml]

Yegappan wrote:

> > The expand() function can be used to expand special characters. If the > > special characters appears anywhere in the command string, then this > > cannot be used. > > > > Vim supports expanding special characters anywhere in a command string > > for internal commands. For example, the 'makeprg' option accepts a command > > string where special characters can occur anywhere in the string. > > But a Vim plugin cannot support this. So plugins like dispatch.vim use a complicated > > function to expand the special characters (look at the dispatch#expand() function in > > https://github.com/tpope/vim-dispatch/blob/master/autoload/dispatch.vim). > > > > This patch adds a expandcmd() function that can be used to expand all the > > special characters in a command string. > > Thanks. I updated the docs to mention that the expansion happens like > for the ":edit" command. Other commands, such as ":next" work slightly > differently. > Thanks for merging the patch. Hopefully this is sufficient for plugins to expand options like 'makeprg', 'grepprg', 'equalprg', 'formatprg', 'keywordprg', etc. > > I also reported the error, when there is one. There is still the case > it returns with an error, but without an error message. Perhaps we > should also fail then with some generic message? > Are you referring to the case where expand_filename() returns an empty error message string?

Yes.

…

-- hundred-and-one symptoms of being an internet addict: 139. You down your lunch in five minutes, at your desk, so you can spend the rest of the hour surfing the Net. /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[vim-ml]

Hi,

On Sun, Jun 9, 2019 at 9:15 AM Yegappan Lakshmanan ***@***.***> wrote: > > Yegappan wrote: > > > The expand() function can be used to expand special characters. If the > > special characters appears anywhere in the command string, then this > > cannot be used. > > > > Vim supports expanding special characters anywhere in a command string > > for internal commands. For example, the 'makeprg' option accepts a command > > string where special characters can occur anywhere in the string. > > But a Vim plugin cannot support this. So plugins like dispatch.vim use a complicated > > function to expand the special characters (look at the dispatch#expand() function in > > https://github.com/tpope/vim-dispatch/blob/master/autoload/dispatch.vim). > > > > This patch adds a expandcmd() function that can be used to expand all the > > special characters in a command string. > > Thanks. I updated the docs to mention that the expansion happens like > for the ":edit" command. Other commands, such as ":next" work slightly > differently. > Thanks for merging the patch. Hopefully this is sufficient for plugins to expand options like 'makeprg', 'grepprg', 'equalprg', 'formatprg', 'keywordprg', etc.

This will be useful for plugins which parse the above options and start a job asynchronously using job_start(). To do this, these plugins need to expand the special items in the above options and then pass the resulting strings as arguments to job_start(). - Yegappan

[vim-ml]

Hi Bram,

On Sun, Jun 9, 2019 at 11:19 AM Bram Moolenaar ***@***.***> wrote: > > > This patch adds a expandcmd() function that can be used to expand all the > > > special characters in a command string. > > > > Thanks. I updated the docs to mention that the expansion happens like > > for the ":edit" command. Other commands, such as ":next" work slightly > > differently. > > > > I also reported the error, when there is one. There is still the case > > it returns with an error, but without an error message. Perhaps we > > should also fail then with some generic message? > > > > Are you referring to the case where expand_filename() returns an > empty error message string? Yes.

I went through all the code paths in expand_filename() and the called functions where an empty error message string is returned. In all these cases, an error message is already printed. I couldn't find a case where an empty error message string is returned without printing an error message. Are you aware of any of theses cases that should be handled in f_expandcmd()? Thanks, Yegappan

[vim-ml]

Yegappan wrote:

On Sun, Jun 9, 2019 at 11:19 AM Bram Moolenaar ***@***.***> wrote: > > > > > This patch adds a expandcmd() function that can be used to expand all the > > > > special characters in a command string. > > > > > > Thanks. I updated the docs to mention that the expansion happens like > > > for the ":edit" command. Other commands, such as ":next" work slightly > > > differently. > > > > > > I also reported the error, when there is one. There is still the case > > > it returns with an error, but without an error message. Perhaps we > > > should also fail then with some generic message? > > > > > > > Are you referring to the case where expand_filename() returns an > > empty error message string? > > Yes. > I went through all the code paths in expand_filename() and the called functions where an empty error message string is returned. In all these cases, an error message is already printed. I couldn't find a case where an empty error message string is returned without printing an error message. Are you aware of any of theses cases that should be handled in f_expandcmd()?

Thanks for checking. I was just wondering if this can be improved. If it looks like it already works properly, there is no need to spend more time on this.

…

-- hundred-and-one symptoms of being an internet addict: 191. You rate eating establishments not by the quality of the food, but by the availability of electrical outlets for your PowerBook. /// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \\\ /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\ \\\ an exciting new programming language -- http://www.Zimbu.org /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[lacygoill]

@yegappan Thank you for implementing expandcmd(). I would like to use it but I have 2 questions:

Would it make sense for the function to also expand // into /last search pattern/?

vim/runtime/doc/quickfix.txt

Lines 1026 to 1027 in 7a3330f

	If {pattern} is empty (e.g. // is specified), the last
	used search pattern is used. |last-pattern|

As an example, it could perform this expansion:

/my pattern
:echo expandcmd('vim //gj %')
vim /my pattern/gj current_file

---

Also, expandcmd() does not escape special characters when they're generated by an expansion (e.g. a filename containing the literal character %). For %, #, !, I guess it's not an issue, because we can use fnameescape(); but for spaces, it's different:

$ vim /tmp/foo\ bar/baz
:echo expandcmd('vim /pat/ %')

vim /pat/ /tmp/foo bar/baz

In the expansion, how can we know whether the original command was grepping into the file /tmp/foo bar/baz, or in the files /tmp/foo and $PWD/bar/baz?

Shouldn't expandcmd() at least escape a space when it's used in a filename?

vim /pat/ /tmp/foo\ bar/baz
                  ^

[vim-ml]

Hi,

On Fri, Aug 28, 2020 at 6:49 AM lacygoill ***@***.***> wrote: @yegappan <https://github.com/yegappan> Thank you for implementing expandcmd(). I would like to use it but I have 2 questions: Would it make sense for the function to also expand // into /last search pattern/? https://github.com/vim/vim/blob/7a3330fc578033f06a94c23de61a23edcc59f95e/runtime/doc/quickfix.txt#L1026-L1027 As an example, it could perform this expansion: /my pattern :echo expandcmd('vim //gj %') vim /my pattern/gj current_file

The expandcmd() function expands the special characters listed under ":h cmdline-special". As the empty search pattern (//) is not a special character, it won't be expanded. Just like how commands like 'vimgrep' handle the empty search pattern and replace it with the last search pattern, the command itself should handle the argument.

------------------------------

Also, expandcmd() does not escape special characters when they're generated by an expansion (e.g. a filename containing the literal character %). For %, #, !, I guess it's not an issue, because we can use fnameescape(); but for spaces, it's different: $ vim /tmp/foo\ bar/baz :echo expandcmd('vim /pat/ %') vim /pat/ /tmp/foo bar/baz In the expansion, how can we know whether the original command was grepping into the file /tmp/foo bar/baz, or in the files /tmp/foo and $PWD/bar/baz? Shouldn't expandcmd() at least escape a space when it's used in a filename? vim /pat/ /tmp/foo\ bar/baz ^

I think you can use the ":S" filename modifier to escape the special characters in the file name. - Yegappan

[lacygoill]

I think you can use the ":S" filename modifier to escape the special characters in the file name.

Ok, but I don't think it works when expanding ## into the filenames of the arglist.

$ cd /tmp
$ vim a\ b c d\ e
:echo expandcmd('vim /pat/ ##:S')

vim /pat/ a b c d e:S

I guess we need to handle ## specially:

:echo expandcmd('vim /pat/ ' .. argv()->map({_, v -> shellescape(v, 1)})->join())