New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IncompleteRead while Composer polls for repomd files #2758

Closed
bowlofeggs opened this Issue Nov 20, 2018 · 3 comments

Comments

Projects
None yet
1 participant
@bowlofeggs
Member

bowlofeggs commented Nov 20, 2018

It seems that Python's http library can raise a new Exception in Python 3 that we don't catch:

[2018-11-20 14:42:34][fedmsg.consumers    INFO]epel7-testing Polling http://download01.phx2.fedoraproject.org/pub/epel/testing/7/aarch64/repodata/repomd.xml
[2018-11-20 14:42:34][fedmsg.consumers   ERROR]epel7-testing Exception in ComposerThread(epel7-testing)
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 410, in work
    self._compose_updates()
  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 959, in _compose_updates
    self._wait_for_sync()
  File "/usr/lib/python3.7/site-packages/bodhi/server/consumers/masher.py", line 1327, in _wait_for_sync
    newsum = hashlib.sha1(masterrepomd.read()).hexdigest()
  File "/usr/lib64/python3.7/http/client.py", line 460, in read
    s = self._safe_read(self.length)
  File "/usr/lib64/python3.7/http/client.py", line 612, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(3295 bytes read, 1 more expected)

This one should be pretty easy to fix - we just need to catch this new Exception and retry, similar to what we do if a URLError is raised.

@bowlofeggs bowlofeggs self-assigned this Nov 21, 2018

bowlofeggs added a commit that referenced this issue Nov 21, 2018

Handle IncompleteRead in the composer when on Python 3.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>

@mergify mergify bot closed this in #2769 Nov 21, 2018

mergify bot added a commit that referenced this issue Nov 21, 2018

Handle IncompleteRead in the composer when on Python 3.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>

mergify bot pushed a commit that referenced this issue Nov 21, 2018

Handle IncompleteRead in the composer when on Python 3.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>
(cherry picked from commit be7fa17)

@bowlofeggs bowlofeggs added the Hotfixed label Nov 21, 2018

bowlofeggs added a commit that referenced this issue Nov 26, 2018

Handle IncompleteRead in the composer when on Python 3.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>
(cherry picked from commit be7fa17)

mergify bot added a commit that referenced this issue Nov 26, 2018

Handle IncompleteRead in the composer when on Python 3.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>
(cherry picked from commit be7fa17)
@bowlofeggs

This comment has been minimized.

Member

bowlofeggs commented Nov 28, 2018

The fix for this issue is now released with Bodhi 3.11.1 and deployed to Fedora's production instance.

@bowlofeggs

This comment has been minimized.

Member

bowlofeggs commented Nov 30, 2018

The fix I made for this mistakenly handled IncompleteRead on urlopen() instead of on read(), so the error still happens.

@bowlofeggs bowlofeggs reopened this Nov 30, 2018

@bowlofeggs

This comment has been minimized.

Member

bowlofeggs commented Nov 30, 2018

I once again hotfixed this by moving the read() statement into the Exception handler.

bowlofeggs added a commit to bowlofeggs/bodhi that referenced this issue Nov 30, 2018

Correctly handle IncompleteRead in the Composer.
fixes fedora-infra#2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>

bowlofeggs added a commit to bowlofeggs/bodhi that referenced this issue Nov 30, 2018

Correctly handle IncompleteRead in the Composer.
fixes fedora-infra#2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>

bowlofeggs added a commit to bowlofeggs/bodhi that referenced this issue Dec 3, 2018

Correctly handle IncompleteRead in the Composer.
fixes fedora-infra#2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>

mergify bot pushed a commit that referenced this issue Dec 3, 2018

Correctly handle IncompleteRead in the Composer.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>
(cherry picked from commit 38cd7df)

mergify bot added a commit that referenced this issue Dec 3, 2018

Correctly handle IncompleteRead in the Composer.
fixes #2758

Signed-off-by: Randy Barlow <randy@electronsweatshop.com>
(cherry picked from commit 38cd7df)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment