Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bodhi web UI unusable for large updates after 2.2.0 update #951

Closed
kalev opened this issue Sep 21, 2016 · 11 comments

Comments

@kalev
Copy link
Contributor

commented Sep 21, 2016

After latest Bodhi update, the sheer number of taskatron test results seems to cause most web browsers to hang. Try any larger update, such as https://bodhi.fedoraproject.org/updates/FEDORA-2016-78efe5595f -- Firefox locks up here after clicking on this link and Chrome as well.

@frieben

This comment has been minimized.

Copy link

commented Sep 21, 2016

Firefox fills all available real memory and swap space after opening https://bodhi.fedoraproject.org/updates/FEDORA-2016-78efe5595f. I killed the process at 8 GB and counting .. .

@bowlofeggs

This comment has been minimized.

Copy link
Member

commented Sep 22, 2016

Thanks for the bug report! A quick fix would probably be to limit the number of taskotron results that are rendered in the template to some arbitrary but small-enough number. I think we could get a fix like that in soon so that the pages at least render.

Something perhaps a little more sohpisticated would be to sort the tasks so that the failing ones sort highest, limit them to some arbitary number, and if there are more than our limit we can just link to the full taskotron output (something like this https://taskotron.fedoraproject.org/artifacts/all/8ac4e78e-7e6a-11e6-bab6-525400120b80/task_output/ ?)

What do you think?

@kalev

This comment has been minimized.

Copy link
Contributor Author

commented Sep 22, 2016

Sure, makes sense to me to limit the results for now, thanks!

@kparal, any comments about sorting taskotron tasks output?

@kparal

This comment has been minimized.

Copy link
Contributor

commented Sep 22, 2016

There are multiple issues, I think. I'll try to describe them in detail, sorry for a long post.

First, tasks IDs look like namespace.task_name.any.number.of.subresults. We currently have just a single important namespace, dist. So some of results are reported simply as dist.depcheck. Other tasks, like rpmgrill, decided to report an overall result as dist.rpmgrill, but also individual subresults as dist.rpmgrill.subresult (see here). Publishing the overall result as dist.rpmgrill is not mandatory, some tasks might not have that (only the subresults). The tasks can simply do what they like under namespace.task_name. In the future the namespaces might be structured themselves, like group.cloud, but the list of namespaces will be available through API.

So, for the moment, I'd probably only display results with one level under dist, so e.g. dist.depcheck and dist.rpmgrill, but not dist.rpmgrill.multilib. The subresults for dist.rpmgrill could then either be expanded with an expander in a tree-like structure (loaded only after you click on it), or there could be simply just a link to resultsdb (resultsdb v2 will have support for querying subresults of a result, @rajcze can tell us the details).

Second, all our tasks publish results per koji build. Some of them also publish per-bodhi-update results (like depcheck), but the new ones don't (rpmgrill, abicheck) and we'd like to move away from per-bodhi-update results (once bodhi is ready) even for the old ones. We've had big issues in keeping those per-update results in sync with bodhi state (when results are edited) and figured it would be much easier to aggregate the results on bodhi side (which always knows the current state of the update, and doesn't need to do slow network queries). So, bodhi should aggregate per-build results into an overall result for multi-build updates. (Well, it's not strictly necessary, but then updates with >5 builds will get enormous list of results that will be tedious to read through).

So, I imagine this could be again solved with tree-like structure. The top item would be an overall depcheck result, and if you expanded that, you would see depcheck results for individual builds. It could even show an "incomplete" icon next to the overall result, if bodhi detected there are still results missing for certain builds.

Overall I imagine it could be like this:

Taskotron results
-----------------
dist.depcheck        PASS
 (expanded on demand)
 |- foo-1.0-1.fc25   PASS
 |- bar-2.0-1.fc25   PASS
dist.upgradepath     FAIL
 |- foo-1.0-1.fc25   PASS
 |- bar-2.0-1.fc25   FAIL
dist.rpmlint         INCOMPLETE
 |- foo-1.0-1.fc25   PASS
dist.abicheck        FAIL INCOMPLETE
 |- foo-1.0-1.fc25   FAIL
dist.rpmgrill        NEEDS_INSPECTION
 |- foo-1.0-1.fc25   PASS <see subresults>
 |- bar-2.0-1.fc25   NEEDS_INSPECTION <see subresults>

Of course, if you can then sort this list first according to outcome importance (FAIL > NEEDS_INSPECTION > PASS), and then according to task name, that would be also helpful. But sorting is probably the last thing to do here, first we need to trim the current extra long list of individual results into some shorter, easier to read structure (e.g. like the one proposed above).

Sounds reasonable? I understand this is not trivial to implement. If you need any help with resultsdb queries or even support for some new queries (we're working on resultsdb v2, now is the time to ask for compatibility-breaking features), we'll try to do our best.

@rajcze

This comment has been minimized.

Copy link

commented Sep 22, 2016

@kparal I'll just clean up some terminology, that might lead to false expectancies:

... (resultsdb v2 will have support for querying subresults of a result, @rajcze can tell us the details).

This is not technically true, as we don't really store/provide definitive relations between results, on the Taskotron/ResultsDB side. So there is no way to say "give me all 'subresults' of a this result with ID X". What you could do, though (and what @kparal probably meant) is that you could do a query like this: .../results?item=foo&testcases=dist.rpmgrill to get the "overall result" (if the task provides it), and then .../results?item=foo&testcases=dist.rpmgrill.* to get the results for the "subchecks". But bare in mind, that this is for the item, and even though technically "rpmgrill.*" (for that item) are "subresults" of "rpmgrill", there is no actual tree structure in the ResultsDB's results. You need to use the semantics of the testcase names, to infer it (should you want to do it).

... figured it would be much easier to aggregate the results on bodhi side (which always knows the current state of the update, and doesn't need to do slow network queries).

Absolutely. This IMHO also goes a bit with what you said about the tasks - that each and every task can store the results in a different way. Bodhi should be the place where the results get aggregated to "can/can not be pushed". This of course means, that Bodhi should know which of the results are "blocking", but this is a policy isuue, basically. What I wanted to say here is, that we strive to keep ResultsDB as dumb as possible, so even if the "decision" would not be made in Bodhi, some kind of middleware would have to be created to "aggregate" the results for (istead of) it. ResultsDB will happily provide the data, but it won't ever hold/provide any meaning for it.

One more thing that I wanted to mention - ResultsDB v2 will have an additional endpoint (created with Bodhi in mind, to be honest), that will provide "latest results" for all relevant testcases & filter. What it means is that instead of grabbing a list of testcases first, and then querying ResultsDB multiple times, you'll be able to do .../results/latest?item=foo (look here for full docs: http://docs.resultsdb20.apiary.io/#reference/0/results/get-a-list-of-current-results-for-a-specified-filter-_fixme:-reword-to-make-more-sense_ ) to get the latest results for that item, and all submitted testcases. At the moment, the list does not contain "missing" results (i.e. if a result for depcheck is not yet submitted, it is not returned) - this could be changed, if you wanted, but I don't see much of a benefit there. Most of the testcases won't have results for a specified item anyway, and if you want to show "missing results", you need to know which are relevant anyway (showing many irrelevant testcases as "missing results" would imho be bad ux).

Hope that helps. And as @kparal said - we are about to roll out new version, so if you want any functionality, now is the time to ask.

@bowlofeggs

This comment has been minimized.

Copy link
Member

commented Sep 27, 2016

I wanted to make updates visible in Bodhi as quickly as possible, so I've worked out a "dirty hack" for this problem and have it deployed to staging right now:

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2016-78efe5595f

Basically, if there are more than 16 builds in an update (this one had 60-something IIRC) Bodhi begins to filter out all passing test results to reduce the number of elements in the page. You will notice that the update still takes a long while to load, but it should load eventually without going too crazy on RAM. The number 16 is completely arbitrary (but a pleasing power of 2) - I just played around with different numbers of updates until my browser seemed near the limit. It also inserts a note at the top of the results table to indicate that the results are being filtered.

I plan to submit a PR with this change today and will hopefully get a 2.2.3 release out with it soon if we are allowed to do so during the infrastructure freeze. I don't intend for this patch to be the long term fix to this problem, but just a band-aid so the pages at least load. If we find that 16 is still too many we can adjust it later.

bowlofeggs added a commit to bowlofeggs/bodhi that referenced this issue Sep 27, 2016
Only show non-passing taskotron results when more than 16 builds.
Taskotron recently began to return a large number of test results
per-package. For updates with many packages, this was causing
browsers to use a whole lot of memory rendering all the results.
This commit adjusts the logic so that only non-passing results are
displayed on updates with more than 16 builds.

re fedora-infra#951
bowlofeggs added a commit that referenced this issue Sep 27, 2016
Only show non-passing taskotron results when more than 16 builds.
Taskotron recently began to return a large number of test results
per-package. For updates with many packages, this was causing
browsers to use a whole lot of memory rendering all the results.
This commit adjusts the logic so that only non-passing results are
displayed on updates with more than 16 builds.

re #951
@bowlofeggs

This comment has been minimized.

Copy link
Member

commented Sep 27, 2016

I am dropping the critical tag on this since we have a short-term fix that at least allows large updates to be viewed.

@bowlofeggs bowlofeggs removed the Critical label Sep 27, 2016

@bowlofeggs bowlofeggs changed the title Bodhi web UI unusable for large updates after 2.2.0 update Rework how taskotron results are displayed Sep 27, 2016

@bowlofeggs bowlofeggs changed the title Rework how taskotron results are displayed Bodhi web UI unusable for large updates after 2.2.0 update Sep 27, 2016

@bowlofeggs

This comment has been minimized.

Copy link
Member

commented Sep 27, 2016

On second thought, I'm going to close this issue since I did fix what the original report is about. I'll file a new issue to rethink how we display taskotron results, and I'll link that here. Please feel free to carry on the discussion there!

@bowlofeggs

This comment has been minimized.

Copy link
Member

commented Sep 27, 2016

I've filed #983 as a place for us to continue the discussion about how Bodhi should be displaying taskotron data. Thanks for the comments so far!

@kalev

This comment has been minimized.

Copy link
Contributor Author

commented Sep 27, 2016

Great, thanks!

@bowlofeggs

This comment has been minimized.

Copy link
Member

commented Sep 27, 2016

bodhi-2.2.3 is now deployed to production!

amolkahat added a commit to amolkahat/bodhi that referenced this issue Feb 2, 2017
Only show non-passing taskotron results when more than 16 builds.
Taskotron recently began to return a large number of test results
per-package. For updates with many packages, this was causing
browsers to use a whole lot of memory rendering all the results.
This commit adjusts the logic so that only non-passing results are
displayed on updates with more than 16 builds.

re fedora-infra#951
ryanlerch added a commit to ryanlerch/bodhi that referenced this issue Mar 29, 2017
Optimize the ajax calls to resultsdb
previously, when presenting the automated test results,
bodhi queried every test case possible for each build
in an update. Consequently, if a large update was encountered,
this resulted in a large amount of calls to results db.
This patch utilises the results/recent/ endpoint of resultsdb
to just give us the most recent results for a build.

Additionaly, this patch re-organizes the output of the automated
test results to order them by packagename in the automatedtests
tab.

Related: fedora-infra#951 fedora-infra#983
ryanlerch added a commit to ryanlerch/bodhi that referenced this issue Mar 29, 2017
Optimize the ajax calls to resultsdb
previously, when presenting the automated test results,
bodhi queried every test case possible for each build
in an update. Consequently, if a large update was encountered,
this resulted in a large amount of calls to results db.
This patch utilises the results/recent/ endpoint of resultsdb
to just give us the most recent results for a build.

Additionaly, this patch re-organizes the output of the automated
test results to order them by packagename in the automatedtests
tab.

Related: fedora-infra#951 fedora-infra#983
ryanlerch added a commit to ryanlerch/bodhi that referenced this issue Mar 29, 2017
Optimize the ajax calls to resultsdb
previously, when presenting the automated test results,
bodhi queried every test case possible for each build
in an update. Consequently, if a large update was encountered,
this resulted in a large amount of calls to results db.
This patch utilises the results/recent/ endpoint of resultsdb
to just give us the most recent results for a build.

Additionaly, this patch re-organizes the output of the automated
test results to order them by packagename in the automatedtests
tab.

Related: fedora-infra#951 fedora-infra#983
crungehottman added a commit to crungehottman/bodhi that referenced this issue Apr 5, 2017
Optimize the ajax calls to resultsdb
previously, when presenting the automated test results,
bodhi queried every test case possible for each build
in an update. Consequently, if a large update was encountered,
this resulted in a large amount of calls to results db.
This patch utilises the results/recent/ endpoint of resultsdb
to just give us the most recent results for a build.

Additionaly, this patch re-organizes the output of the automated
test results to order them by packagename in the automatedtests
tab.

Related: fedora-infra#951 fedora-infra#983
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.