argcomplete still doesn't work for me :-/ please help

[yarikoptic]

#4477 fixed it up but I still can't get it working, looking for advice and comparison on how to troubleshoot.

(git)lena:~datalad/datalad-master[master]git
$> source tools/cmdline-completion 
I: Enabling support of bash completions for zsh

$> git describe              
0.13.0rc1-48-gbe1ad7e79

$> datalad clone --help | tail -n 2
                        'ephemeral'), or value must start with 'shared-'

$> datalad cl<TAB>
no matches for: `file' or `corrections'

$> _ARGCOMPLETE=1 datalad -l 1 --help
...
[DEBUG  ] Generating detailed description for the parser 
[Level 5] Finished setup_parser 
(dev3) 1 28915 ->1

so that one exits with -1 -- is that what you see too? is that expected?

my argcomplete:

$> PYTHON=python3 welp argcomplete  
PATH       : /usr/lib/python3/dist-packages/argcomplete/__init__.py
SRC PATH   : /usr/lib/python3/dist-packages/argcomplete/__init__.py
VERSION    : 1.8.1
__version__: No __version__
PACKAGE    : python3-argcomplete
ii  python3-argcomplete 1.8.1-1.3    all          bash tab completion for argparse (for Python 3)
python3-argcomplete:
  Installed: 1.8.1-1.3
  Candidate: 1.8.1-1.3
  Version table:
 *** 1.8.1-1.3 900
        900 http://deb.debian.org/debian bullseye/main amd64 Packages
        900 http://deb.debian.org/debian bullseye/main i386 Packages
        600 http://http.debian.net/debian sid/main amd64 Packages
        100 /var/lib/dpkg/status
     1.8.1-1 100
        100 http://deb.debian.org/debian buster/main amd64 Packages
        100 http://deb.debian.org/debian buster/main i386 Packages

[kyleam]

You example works for me:

% source ./tools/cmdline-completion
I: Enabling support of bash completions for zsh
% datalad cl
clean  clone

That is with argcomplete 1.11.1 in a virtualenv and on the same datalad commit as you report.

so that one exits with -1 -- is that what you see too? is that expected?

I don't know offhand what's expected, but I see an exit of 1:

% _ARGCOMPLETE=1 datalad -l 1 --help >/dev/null 2>&1 
% echo $?
1

[mih]

FTR: It works for me too

[yarikoptic]

_ARGCOMPLETE=1 datalad -l 1 --help >/dev/null 2>&1

I have obscured my post with not showing run with out -l 1 -- it is totally blank.

$> _ARGCOMPLETE=1 datalad --help     
(dev3) 1 28927 ->1 

and in your case I guess it produces some valuable output, correct?

[yarikoptic]

FWIW the same with argcomplete 1.11.1

[kyleam]

_ARGCOMPLETE=1 datalad --help

[...] and in your case I guess it produces some valuable output, correct?

It's empty on my end. (edit: That was not a clear way to say that. I see no output.)

[kyleam]

How about when you run with _ARC_DEBUG=1? If I export that, I see a lot of text dumped and the final bit is

[...]
active options: ['clean', 'clone']
display completions: {'': 'show this help message.  --help-np forcefully disables\n                 the use of a pager for displaying the help message'}

Returning completions: ['clean', 'clone']

[yarikoptic]

aha -- that reveals something!:

(git)lena:~datalad/datalad-maint[maint]git
$> source tools/cmdline-completion
I: Enabling support of bash completions for zsh
$> _ARC_DEBUG=1 _ARGCOMPLETE=1 datalad --help
Unable to open fd 8 for writing, quitting

Let's proceed on kislyuk/argcomplete#303

[yarikoptic]

thanks to the guidance in aforementioned issue, I found the rabbit hole:

$> datalad installTraceback (most recent call last):
  File "/home/yoh/proj/datalad/datalad-maint/venvs/dev3/bin/datalad", line 7, in <module>
    from datalad.cmdline.main import main
  File "/home/yoh/proj/datalad/datalad-maint/datalad/__init__.py", line 47, in <module>
    cfg = ConfigManager()
  File "/home/yoh/proj/datalad/datalad-maint/datalad/config.py", line 259, in __init__
    self.reload(force=True)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/config.py", line 316, in reload
    stdout, stderr = self._run(run_args, log_stderr=True)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/config.py", line 606, in _run
    out = self._runner.run(self._config_cmd + args, **kwargs)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/cmd.py", line 1138, in run
    out, err = super().run(
  File "/home/yoh/proj/datalad/datalad-maint/datalad/cmd.py", line 957, in run
    raise exc
datalad.support.exceptions.CommandError: CommandError: 'git --git-dir= config -z -l' failed with exitcode -11

and reached to the bottom of it in a painful succession with

the following helper diff on cmd.py (let it never come useful again!)

diff --git a/datalad/cmd.py b/datalad/cmd.py
index b34c5f219..be564a49e 100644
--- a/datalad/cmd.py
+++ b/datalad/cmd.py
@@ -946,6 +946,33 @@ class Runner(object):
                         self._log_err(out[1], expected=expect_stderr)
 
                 if status not in [0, None]:
+                    """Something in popen_env flips git to segfault!"""
+                    def doit(keys):
+                        kw = dict(
+                        stdout = outputstream, stderr = errstream,
+                        shell = shell, cwd = popen_cwd,
+                        stdin = stdin
+                        )
+
+                        _p = subprocess.Popen(
+                            cmd,
+                            env={k: popen_env[k] for k in keys},
+                            **kw);
+                        _out = _p.communicate();
+                        _status = _p.poll();
+                        return _status
+
+                    all_keys = list(popen_env.keys())
+                    _status = doit(all_keys)
+                    assert _status < 0
+                    for _i in range(len(popen_env)):
+                        _m = -(1+_i)
+                        if doit(all_keys[:_m]) == 0 :
+                            import epdb; epdb.serve()
+                            print(_m, "good! next one was %s" % all_keys[:_m+1][-1])
+                        else:
+                            print(_m, "still bad")
+
                     exc = CommandError(
                         cmd=cmd,
                         code=status,

which resulted in a bug report against git-annex: https://git-annex.branchable.com/bugs/standalone_shim_is_segfaulting_if_IFS__61____36____39____92__013__39__/ which can be summarized as

$> IFS=$'\013' /usr/lib/git-annex.linux/git-annex version
[1]    1040489 segmentation fault (core dumped)  IFS=$'\013' /usr/lib/git-annex.linux/git-annex version
$> IFS=$'\013' /usr/lib/git-annex.linux/git version      
[1]    1040532 segmentation fault (core dumped)  IFS=$'\013' /usr/lib/git-annex.linux/git version

and which explains why you could not reproduce my pain (nobody used standalone git annex).

I now wonder: shouldn't we filter out IFS (and may be some other env vars) from the "outside" environment (still allowing them to be defined explicit) which we pass into underlying commands?

[yarikoptic]

it works for me now (that standalone build is fixed) and nobody got excited about IFS, so let's close