intermittent async pexpect errors #21

Closed
escapewindow opened this Issue Oct 28, 2016 · 0 comments

Projects

None yet

1 participant

@escapewindow
Member

11:00 <aki> async pexpect appears to have an intermittent hang, making async gpg sign_keys unreliable. i may have to tear out gpg homedir creation into a separate process =\

We basically get errors like

DEBUG:scriptworker.gpg:gpg --homedir /tmp/tmpxtfienl7 --no-default-keyring --secret-keyring /tmp/tmpxtfienl7/secring.gpg --keyring /tmp/tmpxtfienl7/pubring.gpg -u C4F19E3EBD445E29E680B26A1E47FDB1FD63EC52 --lsign-key 83A4B550BC682F0B060157B04654904BB484B6B2
ERROR:asyncio:Exception in callback BaseSelectorEventLoop.add_reader(11, <bound method...d=11 polling>>)
handle: <Handle BaseSelectorEventLoop.add_reader(11, <bound method...d=11 polling>>)>
Traceback (most recent call last):
  File "/tools/python35/lib/python3.5/asyncio/selector_events.py", line 234, in add_reader
    key = self._selector.get_key(fd)
  File "/tools/python35/lib/python3.5/selectors.py", line 191, in get_key
    raise KeyError("{!r} is not registered".format(fileobj)) from None
KeyError: '11 is not registered'

This appears to be pexpect/pexpect#347; a patch is pending in pexpect/pexpect#376 .

We can either try to get the upstream issue fixed, or tear out gpg homedir creation into its own parallel process.

@escapewindow escapewindow added a commit to escapewindow/scriptworker that referenced this issue Oct 28, 2016
@escapewindow escapewindow Pull gpg homedir creation out of the main scriptworker process.
Originally, gpg homedir creation happened before scriptworker launched,
and then a background async task polled git and rebuilt the gpg homedirs
if it found a new valid git commit.  In order to sign the gpg keys in
the background, we used async pexpect.

However, as noted in #21, async pexpect isn't reliable currently.  Our
options included fixing the upstream pexpect bug or pulling the gpg
homedir creation into a separate process that we can call via cron.

This patch allows us to use that latter approach.
28ce227
@escapewindow escapewindow added a commit to escapewindow/scriptworker that referenced this issue Oct 29, 2016
@escapewindow escapewindow Pull gpg homedir creation out of the main scriptworker process.
Originally, gpg homedir creation happened before scriptworker launched,
and then a background async task polled git and rebuilt the gpg homedirs
if it found a new valid git commit.  In order to sign the gpg keys in
the background, we used async pexpect.

However, as noted in #21, async pexpect isn't reliable currently.  Our
options included fixing the upstream pexpect bug or pulling the gpg
homedir creation into a separate process that we can call via cron.

This patch allows us to use that latter approach.
1456ab7
@escapewindow escapewindow added a commit to escapewindow/scriptworker that referenced this issue Oct 29, 2016
@escapewindow escapewindow Pull gpg homedir creation out of the main scriptworker process.
Originally, gpg homedir creation happened before scriptworker launched,
and then a background async task polled git and rebuilt the gpg homedirs
if it found a new valid git commit.  In order to sign the gpg keys in
the background, we used async pexpect.

However, as noted in #21, async pexpect isn't reliable currently.  Our
options included fixing the upstream pexpect bug or pulling the gpg
homedir creation into a separate process that we can call via cron.

This patch allows us to use that latter approach.
c71530d
@escapewindow escapewindow closed this in #22 Nov 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment