Make sage.interfaces.expect compatible with ptyprocess >= 0.5.2 #32147

mkoeppe · 2021-07-06T19:17:32Z

(from #31280 comment:42)

CC: @antonio-rojas @isuruf @kiwifb @dimpase @orlitzky

Component: interfaces

Issue created by migration from https://trac.sagemath.org/ticket/32147

mkoeppe · 2021-07-18T02:51:27Z

comment:1

(#31280 comment:46):
If we are lucky, this is just pexpect/ptyprocess@f673a54

mkoeppe · 2021-07-18T04:52:16Z

comment:2

Just catching PtyProcessError in addition to ExceptionPexpect on our side does not seem to fix the problem.

tornaria · 2022-01-19T17:31:46Z

comment:6

Just to mention that I've been using system ptyprocess-0.7.0 (via #29665) for a few weeks without any issue.

What's the exact issue? I haven't seen any report of what goes wrong on macos when using an updated ptyprocess.

mkoeppe · 2022-01-19T18:10:07Z

comment:7

It looks like this:

$ ./sage -pip install -U ptyprocess
Requirement already satisfied: ptyprocess in ./local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages (0.5.1)
Collecting ptyprocess
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Installing collected packages: ptyprocess
  Attempting uninstall: ptyprocess
    Found existing installation: ptyprocess 0.5.1
    Uninstalling ptyprocess-0.5.1:
      Successfully uninstalled ptyprocess-0.5.1
Successfully installed ptyprocess-0.7.0
egret:~/s/sage/sage-rebasing/worktree-gcc11 (mkoeppe-9.5.beta9+updates $%>)$ ./sage -tp src/sage/interfaces/
too many failed tests, not using stored timings
Running doctests with ID 2022-01-19-10-06-35-c84011cc.
Using --optional=build,ccache,cryptominisat,debugpy,e_antic,homebrew,igraph,jupymake,normaliz,pip,pycryptosat,pynormaliz,sage,sage_spkg
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_jones_numfield,database_knotinfo,dvipng,ffmpeg,graphviz,imagemagick,jupymake,kenzo,latte_int,lrslib,mcqd,meataxe,pandoc,pdf2svg,plantri,pynormaliz,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.plot,sage.rings.number_field,sage.rings.real_double,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,sphinx,tdlib
Sorting sources by runtime so that slower doctests are run first....
Doctesting 61 files using 8 threads.
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/gap.py
**********************************************************************
File "src/sage/interfaces/gap.py", line 644, in sage.interfaces.gap.Gap_generic._eval_line
Failed example:
    a = gap(3)
Exception raised:
    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 511, in _execute_line
        E.sendline(line)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 578, in sendline
        return self.send(s + self.linesep)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 569, in send
        return os.write(self.child_fd, b)
    OSError: [Errno 5] Input/output error

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 658, in _eval_line
        (normal, error) = self._execute_line(line, wait_for_prompt=wait_for_prompt,
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 513, in _execute_line
        raise RuntimeError("Error evaluating %s in %s"%(line, self))
    RuntimeError: Error evaluating \$sage7:=3;; in Gap

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 1166, in _start
        Expect._start(self, "Failed to start GAP.")
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 444, in _start
        self.quit()  # in case one is already running
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 626, in quit
        self._close()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 558, in _close
        self._expect.close(force=force)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 328, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 204, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:3268)
        self.fileobj.close()
    OSError: [Errno 5] Input/output error

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 1470, in __init__
        self._name = parent._create(value, name=name)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 513, in _create
        self.set(name, value)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 1328, in set
        self._eval_line(cmd, allow_use_file=True)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 701, in _eval_line
        self._start()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/gap.py", line 1170, in _start
        self.quit()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 626, in quit
        self._close()
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 558, in _close
        self._expect.close(force=force)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pexpect/pty_spawn.py", line 328, in close
        self.ptyproc.close(force=force)
      File "sage/interfaces/sagespawn.pyx", line 201, in sage.interfaces.sagespawn.SagePtyProcess.close (build/cythonized/sage/interfaces/sagespawn.c:3177)
        self.write(str_to_bytes(self.quit_string))
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/ptyprocess/ptyprocess.py", line 568, in write
        return self._writeb(s, flush=flush)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/ptyprocess/ptyprocess.py", line 558, in _writeb
        n = self.fileobj.write(b)
    ValueError: write to closed file

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 694, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1088, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.gap.Gap_generic._eval_line[8]>", line 1, in <module>
        a = gap(Integer(3))
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 302, in __call__
        result = self._coerce_from_special_method(x)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 330, in _coerce_from_special_method
        return (x.__getattribute__(s))(self)
      File "sage/structure/sage_object.pyx", line 718, in sage.structure.sage_object.SageObject._gap_ (build/cythonized/sage/structure/sage_object.c:6249)
        return self._interface_(G)
      File "sage/structure/sage_object.pyx", line 694, in sage.structure.sage_object.SageObject._interface_ (build/cythonized/sage/structure/sage_object.c:5783)
        X = I(s)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/interface.py", line 295, in __call__
        return cls(self, x, name=name)
      File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-gcc11/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage/interfaces/expect.py", line 1475, in __init__
        raise TypeError(*x.args)
    TypeError: write to closed file
[......]
----------------------------------------------------------------------
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/gap.py  # 69 doctests failed
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/mwrank.py  # 6 doctests failed
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/expect.py  # 40 doctests failed
sage -t --random-seed=87883514352856014297393615604236705289 src/sage/interfaces/quit.py  # 4 doctests failed
----------------------------------------------------------------------
Total time for all tests: 39.7 seconds
    cpu time: 42.7 seconds
    cumulative wall time: 130.5 seconds
Features detected for doctesting:

mkoeppe · 2022-03-05T01:55:27Z

comment:8

Attachment: ptyprocess-failures.txt

dimpase · 2023-09-26T10:59:41Z

I cannot reproduce some of this with python 3.11.5 on macOS and system-wide installed ptyprocess 0.7.0 (and pexpect 0.4.8), used via --enable-system-site-packages.

dima@oucl13243 sage % ./sage --pip freeze | grep ptypr     
ptyprocess==0.7.0
dima@oucl13243 sage % uname -a
Darwin oucl13243.lan 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64
dima@oucl13243 sage % ./sage -tp src/sage/interfaces/

----------------------------------------------------------------------
sage -t --warn-long 89.9 --random-seed=78948596107961109861595569669537602143 src/sage/interfaces/mwrank.py  # 2 doctests failed
sage -t --warn-long 89.9 --random-seed=78948596107961109861595569669537602143 src/sage/interfaces/expect.py  # 38 doctests failed
sage -t --warn-long 89.9 --random-seed=78948596107961109861595569669537602143 src/sage/interfaces/quit.py  # 4 doctests failed

some of it depends upon the random seed. E.g. the following reliably passes. Or, perhaps, it only matters that it's not
run in parallel with other tests?

dima@oucl13243 sage % ./sage -tp --random-seed=192325368743415462395845369501584124267 src/sage/interfaces/quit.py
Running doctests with ID 2023-09-26-11-57-44-882b3f70.
Git branch: HEAD
Git ref: 9.2.rc2-27519-gf00562ee4e
Running with SAGE_LOCAL='/Users/dima/software/sage/local' and SAGE_VENV='/Users/dima/software/sage/local/var/lib/sage/venv-python3.11'
Using --optional=homebrew,ipympl,jupymake,notedown,pip,pycosat,pysingular,rst2ipynb,sage,sage_spkg,texttable
Features to be detected: 4ti2,benzene,bliss,buckygen,conway_polynomials,csdp,cvxopt,cvxopt,database_cremona_ellcurve,database_cremona_mini_ellcurve,database_cubic_hecke,database_jones_numfield,database_knotinfo,dvipng,fpylll,gap_package_atlasrep,gap_package_design,gap_package_grape,gap_package_guava,gap_package_hap,gap_package_polycyclic,gap_package_qpa,gap_package_quagroup,gfan,graphviz,imagemagick,ipython,jupymake,kenzo,latte_int,lrcalc_python,lrslib,mcqd,meataxe,mpmath,msolve,nauty,networkx,numpy,palp,pandoc,pdf2svg,pdftocairo,pexpect,phitigra,pillow,plantri,polytopes_db,polytopes_db_4d,pplpy,primecountpy,ptyprocess,pynormaliz,pyparsing,python_igraph,requests,rubiks,sage.combinat,sage.geometry.polyhedron,sage.graphs,sage.groups,sage.libs.ecl,sage.libs.flint,sage.libs.gap,sage.libs.linbox,sage.libs.m4ri,sage.libs.ntl,sage.libs.pari,sage.libs.singular,sage.misc.cython,sage.modular,sage.modules,sage.numerical.mip,sage.plot,sage.rings.complex_double,sage.rings.finite_rings,sage.rings.function_field,sage.rings.number_field,sage.rings.padics,sage.rings.polynomial.pbori,sage.rings.real_double,sage.rings.real_mpfr,sage.sat,sage.schemes,sage.symbolic,sage_numerical_backends_coin,sagemath_doc_html,scipy,singular,sphinx,sympy,tdlib
Doctesting 1 file using 2 threads.
sage -t --warn-long 88.6 --random-seed=192325368743415462395845369501584124267 src/sage/interfaces/quit.py
    [21 tests, 1.44 s]
----------------------------------------------------------------------
All tests passed!

See sagemath#32147 for details

mkoeppe · 2023-09-27T02:54:19Z

Yes, repro of this bug is difficult. Didn't succeed when I tried just now.

dimpase · 2023-09-27T04:18:47Z

try

gap.eval("quit;")
gap.quit()

this reproduces the bug every time on my Intel macOS.

mkoeppe · 2023-09-27T04:30:40Z

Perfect, thanks. Meanwhile I was also able to provoke the doctester into reproducing the failures.

This will close sagemath#32147 ptyprocess 0.5.2+ included a change that broke our expect.py on macOS Here we guard a bit more around thrown errors, and this appears to suffice URL: sagemath#36339 Reported by: Dima Pasechnik Reviewer(s): Dima Pasechnik, Gonzalo Tornaría, Matthias Köppe

mkoeppe added this to the sage-9.4 milestone Jul 6, 2021

mkoeppe added c: interfaces labels Jul 6, 2021

mkoeppe modified the milestones: sage-9.4, sage-9.5 Jul 19, 2021

mkoeppe modified the milestones: sage-9.5, sage-9.6 Dec 14, 2021

mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 5, 2022

mkoeppe modified the milestones: sage-9.7, sage-9.8 Aug 31, 2022

mkoeppe modified the milestones: sage-9.8, sage-9.9 Jan 7, 2023

mkoeppe removed this from the sage-10.0 milestone Mar 16, 2023

mkoeppe mentioned this issue Sep 25, 2023

allow use of ptyprocess>0.5.1 #36331

Closed

dimpase mentioned this issue Sep 26, 2023

IPython/Jupyter upgrade for Sage 9.5 #31280

Closed

dimpase added a commit to dimpase/sage that referenced this issue Sep 26, 2023

fix expect interface for newer ptyprocess

69daf99

See sagemath#32147 for details

dimpase mentioned this issue Sep 26, 2023

fix expect interface for newer ptyprocess #36339

Merged

dimpase added a commit to dimpase/sage that referenced this issue Sep 26, 2023

fix expect interface for newer ptyprocess

d4cc09d

See sagemath#32147 for details

vbraun closed this as completed in #36339 Sep 27, 2023

mkoeppe added this to the sage-10.2 milestone Sep 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make sage.interfaces.expect compatible with ptyprocess >= 0.5.2 #32147

Make sage.interfaces.expect compatible with ptyprocess >= 0.5.2 #32147

mkoeppe commented Jul 6, 2021

mkoeppe commented Jul 18, 2021

mkoeppe commented Jul 18, 2021

tornaria commented Jan 19, 2022

mkoeppe commented Jan 19, 2022

mkoeppe commented Mar 5, 2022

dimpase commented Sep 26, 2023 •

edited

mkoeppe commented Sep 27, 2023

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

Make sage.interfaces.expect compatible with ptyprocess >= 0.5.2 #32147

Make sage.interfaces.expect compatible with ptyprocess >= 0.5.2 #32147

Comments

mkoeppe commented Jul 6, 2021

mkoeppe commented Jul 18, 2021

mkoeppe commented Jul 18, 2021

tornaria commented Jan 19, 2022

mkoeppe commented Jan 19, 2022

mkoeppe commented Mar 5, 2022

dimpase commented Sep 26, 2023 • edited

mkoeppe commented Sep 27, 2023

dimpase commented Sep 27, 2023

mkoeppe commented Sep 27, 2023

dimpase commented Sep 26, 2023 •

edited