doit based dev interface garbles pdb command history (in some cases)

[ev-br]

There is a little glitch with the new doit dev.py interface, and I'm not exactly sure where to even start looking from. The bottom line is that in pdb, the uparrow does not work for getting a previous command. Roughly like this:

Things work in a regular python session:

$ python dev.py --shell
<the stuff is building, a new shell appears>

$ python
Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def func(x):
...     a, b = 1, 2; import pdb; pdb.set_trace()
... 
>>> func(2)
--Return--
> <stdin>(2)func()->None
(Pdb) p a
1
<press the up arrow : the previous command appears>
(Pdb) p a

Then, throw a breakpoint somewhere, e.g. with this trivial patch:

$ git diff
diff --git a/scipy/linalg/tests/test_basic.py b/scipy/linalg/tests/test_basic.py
index faa3666839..198ab65e5d 100644
--- a/scipy/linalg/tests/test_basic.py
+++ b/scipy/linalg/tests/test_basic.py
@@ -63,6 +63,9 @@ class TestSolveBanded:
             x = solve_banded((l, u), ab, b)
             assert_array_almost_equal(dot(a, x), b)
 
+        import pdb; pdb.set_trace()
+
+
     def test_complex(self):
         a = array([[1.0, 20, 0, 0],
                    [-30, 4, 6, 0],

and run tests:

$ python dev.py test -s linalg
💻  ninja -C /home/br/repos/scipy/scipy/build
ninja: Entering directory `/home/br/repos/scipy/scipy/build'
[2/2] Generating scipy/generate-config with a custom command
Build OK
💻  meson install -C build
Installing, see meson-install.log...
Installation OK
SciPy from development installed path at: /home/br/repos/scipy/scipy/build-install/lib/python3.8/site-packages
Running tests for scipy version:1.10.0.dev0+1303.c432856, installed at:/home/br/repos/scipy/scipy/build-install/lib/python3.8/site-packages/scipy
================================= test session starts ==================================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/br/repos/scipy/scipy, configfile: pytest.ini
collected 3621 items / 7 deselected / 3614 selected                                    

scipy/linalg/tests/test_basic.py 
>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>
--Return--
> /home/br/repos/scipy/scipy/build-install/lib/python3.8/site-packages/scipy/linalg/tests/test_basic.py(66)test_real()->None
-> import pdb; pdb.set_trace()
(Pdb) p a
array([[  1.,  20.,   0.,   0.],
       [-30.,   4.,   6.,   0.],
       [  2.,   1.,  20.,   2.],
       [  0.,  -1.,   7.,  14.]])
<press the up arrow: garbage>
(Pdb) ^[[A

@tupui did you mention something about the .bash_profile not being properly sourced?

[tupui]

cc @sayantikabanik

I am not sure what is happening, the profile seems to be loaded but the interaction with conda is somehow fragile. I have the following when I am in a conda env, and nothing if I am outside anything:

❯ python do.py shell
💻  ninja -C .../scipy/build
ninja: Entering directory `.../scipy/build'
[2/2] Generating scipy/generate-config with a custom command
Build OK
💻  meson install -C build
Installing, see meson-install.log...
Installation OK
Spawning a Unix shell...

[WARNING]: Console output during zsh initialization detected.

When using Powerlevel10k with instant prompt, console output during zsh
initialization may indicate issues.

You can:

  - Recommended: Change ~/.zshrc so that it does not perform console I/O
    after the instant prompt preamble. See the link below for details.

    * You will not see this error message again.
    * Zsh will start quickly and prompt will update smoothly.

  - Suppress this warning either by running p10k configure or by manually
    defining the following parameter:

      typeset -g POWERLEVEL9K_INSTANT_PROMPT=quiet

    * You will not see this error message again.
    * Zsh will start quickly but prompt will jump down after initialization.

  - Disable instant prompt either by running p10k configure or by manually
    defining the following parameter:

      typeset -g POWERLEVEL9K_INSTANT_PROMPT=off

    * You will not see this error message again.
    * Zsh will start slowly.

  - Do nothing.

    * You will see this error message every time you start zsh.
    * Zsh will start quickly but prompt will jump down after initialization.

For details, see:
https://github.com/romkatv/powerlevel10k/blob/master/README.md#instant-prompt

-- console output produced during zsh initialization follows --

Traceback (most recent call last):
  File ".../.poetry/bin/poetry", line 17, in <module>
    from poetry.console import main
  File ".../.poetry/lib/poetry/console/__init__.py", line 1, in <module>
    from .application import Application
  File ".../.poetry/lib/poetry/console/application.py", line 3, in <module>
    from cleo import Application as BaseApplication
ModuleNotFoundError: No module named 'cleo'

And since we advice to use a conda env, this could be an issue. Note that working in this shell does work just fine.

[tupui]

While we are at it, ipython also has a strange behaviour where there are no colours.

[ev-br]

As a possible lead, there's something called readline: https://stackoverflow.com/questions/10140198/python-debugger-pdb-stopped-handlying-up-down-arrows-shows-a-instead

Also, the cpython docs have a cryptic comment, which may or may not be related, about Cmd.raw_input attribute:

(https://docs.python.org/3/library/cmd.html#cmd.Cmd.use_rawinput)
A flag, defaulting to true. If true, cmdloop() uses input() to display a prompt and read the next command; if false, sys.stdout.write() and sys.stdin.readline() are used. (This means that by importing readline, on systems that support it, the interpreter will automatically support Emacs-like line editing and command-history keystrokes.)

https://docs.python.org/3/library/cmd.html

[ev-br]

I actually asked on the python discourse: https://discuss.python.org/t/what-controls-the-pdb-history-the-up-arrow-key/21780/2

So ISTM the issue is somewhere in pydevtool, which goes at some lengths at emulating the cmd. @schettino72 would this make any sense to you maybe?

[schettino72]

The problem is that doit (and pytest as well) capture sys.stdout before actually sending the output to the terminal. this mess up with readline. Both pytest and doit provides a --pdb option to handle this.

doit also provides a custom set_trace() [1] invocation, pytest used to have one but needed any more [2].

Actions:

1. Confirm the problem is really with doit (repeat same test case above but using doit.tool.set_trace(), instead of pdb.set_trace().
2. fix doit to install a hook so pdb.set_trace() works fine (as pytest have done years ago)
3. add a --pdb option dev.py, so it automatically drops on a pdb session when an exception occurs.

makes sense?

[1] https://pydoit.org/tools.html#set-trace
[2] https://docs.pytest.org/en/7.1.x/historical-notes.html?highlight=set_trace#pytest-set-trace

[ev-br]

Tried it now, doesn't seem to work:

In [3]: doit.tools.set_trace()
--Call--
> /home/br/.virtualenvs/scipy-dev/lib/python3.8/site-packages/IPython/core/displayhook.py(252)__call__()
-> def __call__(self, result=None):
(Pdb) kjkj
*** NameError: name 'kjkj' is not defined
(Pdb) ^[[A         # <<<<<  press up arrow
*** SyntaxError: invalid syntax

In [5]: doit.__version__
Out[5]: (0, 36, 0)