bodhi API sometimes returns responses with empty bodies instead of JSON

[decathorpe]

As the issue title states, sometimes, but it seems especially when bodhi is "under load", or when setting rows_per_page parameter to a bigger value than the default (e.g. 100), querying the bodhi REST API yields server responses with empty HTTP message bodies, but a "success" HTTP code.

This leads to all kinds of problems (and it's something that I have to actively work around in my bodhi rust client bindings by having a special failure case for "successful" responses with empty response bodies instead of valid JSON).

Since this is a transient / random issue, I don't have exact steps to reproduce it, except for doing some API requests until you get a response with an empty HTTP body instead of JSON.

[mattiaverga]

Can you post an example query for which you see the problem? What model are you quering? Updates?

[decathorpe]

It happens intermittently for almost all endpoints, including comments, updates, and overrides. Querying packages and users seem to be mostly unaffected, and releases returns too few values to be a good data point.

I've tried to reproduce it today, and running this python3 script in two terminals side by side eventually triggered the bug (after a minute or so):

import requests

while True:
    requests.get("https://bodhi.fedoraproject.org/overrides/?rows_per_page=100&page=10&releases=EPEL-8").json()

This crashes with an exception as soon as an empty response is encountered, because an empty string is not valid JSON.

An empty response was returned from bodhi in the first terminal (simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)), and invalid JSON in the second terminal (simplejson.errors.JSONDecodeError: Unterminated string starting at: line 1 column 67028577 (char 67028576)).

[mattiaverga]

Oh, well, the problem seems to be in the nearly 300 buildroot overrides for EPEL-8 from tdawson.
For example https://bodhi.fedoraproject.org/overrides/xdg-desktop-portal-kde-5.18.4-1.el8.1
Trying to load a hundred of them will cause a timeout and the above error.

All of these had their notes updated a million times and at every update the new text was appended to the existing, causing them to become gigantic...
I don't know how that user submitted and updated all those BRs, but I suppose they made something really wrong. From our side I suppose we could just set a maximum char limit to BR notes field to prevent that.

@cverna what do you think? Since this will bring a db migration, how should we handle existing records with more than the new limit? Should we just discard everything beyond the limit?
And what about new entries? Since resubmitting a BR will append the new notes, should we raise an error, or discard the existing and add the new notes?

[decathorpe]

I don't think this is a timeout issue. The returned HTTP response has a 200 code, just an empty response body. I don't see how that's possible from looking at the bodhi server code.

[mattiaverga]

yeah, I misspoke. I mean that the object serialization routine hits some timeouts and sends out data before finishing the object when we try to access those overrides.

I've looked into the code and that happened because every time a BRO is submitted as new, but another BRO with the same build already exists, the old one is used and the new notes are added. So, if one repeatedly create a BRO for the same build nvr, we will end up with gigantic notes, as the example I posted.

I'm going to submit a PR to avoid that, by limiting override notes to a maximum of XXX chars (500? 1.000?) and dropping the older lines in a situation like this.
That's quite ready, I only need to write unit tests (and decide the amount of chars, suggestions welcome).

And we will need to truncate the notes of those problematic overrides manually with a direct query on the database, but that's beyond my privileges... I can prepare the appropriate query to reduce work to admins, if needed.

[decathorpe]

That should certainly help. But unless today is a day where I should not think about programming, then this will not address the underlying issue where sometimes faulty responses are sent?

[mattiaverga]

OK, I've posted a PR that in future should prevent a situation like this.
About the existing overrides that kills the JSON response, I will open a request on Fedora-infra asking to manually fix those, a simple query like
UPDATE buildroot_overrides SET notes = LEFT(notes, 2000) WHERE LENGTH(notes) > 2000;
should be enough. But I'll wait to ask until the datacenter move is done and the busy guys have some time to look at it.

I'll leave this ticket open as a remainder for the faulty responses, but I can't say what closes the connection while the data is still flowing, if it's on to the server or the client side.