Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bash>=4.4: bad substitution error on non-word characters #22

Closed
iakremnev opened this issue Sep 29, 2020 · 17 comments · Fixed by #23
Closed

bash>=4.4: bad substitution error on non-word characters #22

iakremnev opened this issue Sep 29, 2020 · 17 comments · Fixed by #23
Assignees
Labels
p0-critical Max priority (ASAP) shell-bash

Comments

@iakremnev
Copy link

Bug Report

When I try to tab-complete dvc add [targets...] command, it can fail in the following scenario:

  1. multiple targets are specified
  2. the first target contains path separator

Example:

$ dvc add data/r[TAB]  # --> dvc add data/raw  # Completes OK
$ dvc add data/raw da[TAB]  # --> dvc add data/raw datbash: _shtab_dvc_add_data/raw_COMPGEN: bad substitution

Please provide information about your setup

Output of dvc version:

$ dvc version
DVC version: 1.7.9 (pip)
---------------------------------
Platform: Python 3.6.9 on Linux-5.4.0-47-generic-x86_64-with-Ubuntu-18.04-bionic
Supports: http, https, s3
Cache types: hardlink, symlink
Cache directory: ext3 on /dev/sda1
Workspace directory: ext3 on /dev/sda1
Repo: dvc, git
@efiop
Copy link
Contributor

efiop commented Sep 29, 2020

Thanks for the report @iakremnev ! Transfering to shtab.

@efiop efiop transferred this issue from iterative/dvc Sep 29, 2020
@efiop efiop added the p0-critical Max priority (ASAP) label Sep 29, 2020
@casperdcl
Copy link
Collaborator

can't reproduce this. can you post the output of shtab --version?

@iakremnev
Copy link
Author

shtab 1.3.1

@iakremnev
Copy link
Author

iakremnev commented Sep 29, 2020

Here's how I reproduce this bug:

mkdir test && cd test
dvc init --no-scm
mkdir a
touch a/b.txt a/c.txt
dvc add a/b.txt a[TAB]
dvc add a/b.txt abash: _shtab_dvc_add_a/b.txt_COMPGEN: bad substitution

@casperdcl
Copy link
Collaborator

casperdcl commented Sep 29, 2020

Hmm I've followed exactly your code and get:

dvc add a/b.txt a[TAB]
a   a/

Could you try re-initialising completion?

Try doing this:

eval "$(dvc completion -s bash)"
dvc add a/b.txt a[TAB]

If that works then you probably just need to re-install completions (https://dvc.org/doc/install/completion)

@casperdcl casperdcl self-assigned this Sep 29, 2020
@iakremnev
Copy link
Author

Re-installed with apt install --reinstall, still no luck.

Here's some stack trace with set -x:

$ dvc add data/raw d[TAB]
+ local word=d
+ COMPREPLY=()
+ '[' 3 -eq 1 ']'
+ '[' 3 -eq 2 ']'
+ '[' 3 -ge 3 ']'
+ _shtab_dvc_compgen_subcommand_ add data/raw
++ _shtab_replace_hyphen add
++ echo add
++ sed s/-/_/g
++ _shtab_replace_hyphen data/raw
++ echo data/raw
++ sed s/-/_/g
+ local flags_list=_shtab_dvc_add_data/raw
+ local args_gen=_shtab_dvc_add_data/raw_COMPGEN
bash: _shtab_dvc_add_data/raw_COMPGEN: bad substitution

@iakremnev
Copy link
Author

The executed code from /etc/bash_completion.d/dvc is:

# Notes:
# `COMPREPLY` contains what will be rendered after completion is triggered
# `word` refers to the current typed word
# `${!var}` is to evaluate the content of `var`
# and expand its content as a variable
#       hello="world"
#       x="hello"
#       ${!x} ->  ${hello} ->  "world"
_shtab_dvc() {
  local word="${COMP_WORDS[COMP_CWORD]}"

  COMPREPLY=()

  if [ "${COMP_CWORD}" -eq 1 ]; then
    _shtab_dvc_compgen_root_ ${COMP_WORDS[1]}
  elif [ "${COMP_CWORD}" -eq 2 ]; then
    _shtab_dvc_compgen_command_ ${COMP_WORDS[1]}
  elif [ "${COMP_CWORD}" -ge 3 ]; then
    _shtab_dvc_compgen_subcommand_ ${COMP_WORDS[1]} ${COMP_WORDS[2]}
  fi

  return 0
}

It seems like ${COMP_CWORD} should be equal to 2, not 3, and _shtab_dvc_compgen_command_ should be called instead of _shtab_dvc_compgen_subcommand_. My reasoning is that dvc add is a command and not a subcommand.

@iakremnev
Copy link
Author

iakremnev commented Sep 29, 2020

According to bash man page, COMP_CWORD is exactly 3 at position dvc add data/raw da[TAB]. There's clearly a bug in these branching statements.

@casperdcl
Copy link
Collaborator

COMP_CWORD isn't the issue here. This error doesn't make sense:

+ local args_gen=_shtab_dvc_add_data/raw_COMPGEN
bash: _shtab_dvc_add_data/raw_COMPGEN: bad substitution

can you run this?

foo(){
  local args_gen=_shtab_dvc_add_data/raw_COMPGEN
}
foo

there's no "substitution" occurring in this command so it should just work without errors. What bash --version are you using?

@iakremnev
Copy link
Author

Actually, it's not this line causing the error but the next one:

_shtab_dvc_compgen_subcommand_() {
  local flags_list="_shtab_dvc_$(_shtab_replace_hyphen $1)_$(_shtab_replace_hyphen $2)"
  local args_gen="${flags_list}_COMPGEN"   # <-- this is _shtab_dvc_add_data/raw_COMPGEN now
  [ -n "${!args_gen}" ] && local opts_more="$(${!args_gen} "$word")"  # fails on ${!args_gen}
  local opts="${!flags_list}"
  if [ -z "$opts$opts_more" ]; then
    _shtab_dvc_compgen_command_ $1
  else
    COMPREPLY=( $(compgen -W "$opts" -- "$word"; [ -n "$opts_more" ] && echo "$opts_more") )
  fi
}

Even smaller example:

$ foo() {
>  local x=a/b
>  [ ${!x} ]
>}
$ foo
bash: a/b: bad substitution

@iakremnev
Copy link
Author

iakremnev commented Sep 29, 2020

Let me explain once again: when I type dvc add data/raw da and press TAB, COMP_WORDS=(dvc add data/raw da) and COMP_CWORD=3.
Therefore

elif [ "${COMP_CWORD}" -ge 3 ]; then
    _shtab_dvc_compgen_subcommand_ ${COMP_WORDS[1]} ${COMP_WORDS[2]}

is called as if dvc add data/raw is a valid subcommand.
Are you sure it's the intended behavior?

@casperdcl
Copy link
Collaborator

casperdcl commented Sep 29, 2020

$ foo(){
>  local x=a/b
>  [ ${!x} ]  # command exits non-zero but shouldn't cause a crash
>  echo "all fine"
> }
$ foo
all fine

is called as if dvc add data/raw is a valid subcommand.

yes that's expected. [ -n "${!args_gen}" ] will exit non-zero but not crash.

@iakremnev
Copy link
Author

iakremnev commented Sep 29, 2020

Hmm, this is interesting. This function doesn't print "all fine" for me.
Bash version is 4.4.20(1)-release.

BASHOPTS=checkwinsize:cmdhist:complete_fullquote:expand_aliases:extglob:extquote:force_fignore:histappend:interactive_comments:progcomp:promptvars:sourcepath
SHELLOPTS=braceexpand:emacs:hashall:histexpand:history:interactive-comments:monitor

@casperdcl
Copy link
Collaborator

Very interesting. I'm on GNU bash, version 4.3.48(1)-release with the same BASHOPTS and SHELLOPTS.

@iakremnev
Copy link
Author

I used docker images to try this scenario in different bash versions. I found that for bash<=4.3.48 there's no failure and foo return with error code 0, and for bash>=4.4.23 there's an immediate exit with error code 1.

@casperdcl casperdcl changed the title Tab completion fails in bash bash>=4.4: bad substitution error on non-word characters Sep 30, 2020
casperdcl added a commit that referenced this issue Sep 30, 2020
casperdcl added a commit that referenced this issue Sep 30, 2020
@casperdcl
Copy link
Collaborator

casperdcl commented Sep 30, 2020

could you try #23?

pip install 'shtab>=1.3.2'
eval "$(dvc completion -s bash)"
dvc add a/b.txt a[TAB]

@iakremnev
Copy link
Author

I confirm, the problem is solved! Thanks @casperdcl!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p0-critical Max priority (ASAP) shell-bash
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants