Fix rest api filtering by jangmarker · Pull Request #3526 · buildbot/buildbot

jangmarker · 2017-08-15T14:34:02Z

Fixes queries like:
/api/v2/builds?builderid__eq=1&builderid__eq=2

This would have yielded an empty result with the previous code.

Contributor Checklist:

I have updated the unit tests
I have created a file in the master/buildbot/newsfragment directory (and read the README.txt in that directory)
I have updated the appropriate documentation
code is fixed to match the documentation

codecov · 2017-08-15T14:43:58Z

Codecov Report

Merging #3526 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3526      +/-   ##
==========================================
+ Coverage   88.29%   88.36%   +0.06%     
==========================================
  Files         323      323              
  Lines       33878    34047     +169     
==========================================
+ Hits        29912    30085     +173     
+ Misses       3966     3962       -4

Impacted Files	Coverage Δ
master/buildbot/data/resultspec.py	`91.2% <100%> (+0.21%)`	⬆️
master/buildbot/process/builder.py	`84.54% <0%> (+1.36%)`	⬆️
master/buildbot/process/build.py	`94.12% <0%> (+1.37%)`	⬆️
master/buildbot/worker/openstack.py	`97.4% <0%> (+1.45%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1cbfd9...624ec02. Read the comment docs.

tardyp · 2017-08-15T15:38:46Z

good catch. This needs unit tests!

does that also fix #3527 ?

jangmarker · 2017-08-16T03:08:55Z

I've added unit tests, however, I haven't found a usage of the database contains operator - the tags property in /builders/ is not queried from the database, it seems. Maybe I've been overlooking something.

It doesn't fix #3527, in fact I discovered #3527 while working on this. For a given list property of a database object the contains operator requires all values given in the REST call to be in that list. This is a sensible query, but a query where at least one of the values given in the REST call have to be in the list property is not possible at the moment (at least I haven't found a way).

tardyp · 2017-08-16T06:53:15Z

Hi @jangmarker the Rest API filtering is featured around the needs for the UI.
I think nobody actually looked at the plural case for contains.

If you have usecases, I think it makes more sense to have the contains operator have a default 'or' behaviour. It better matches the behaviour for 'eq'.
We could add a xcontains operator, which has a 'and' plural behaviour.

tardyp · 2017-08-16T07:03:32Z

master/buildbot/data/resultspec.py

+    plural_operators_sql = {
+        'eq': lambda d, v: d.in_(v),
+        'ne': lambda d, v: d.notin_(v),
+        'contains': lambda d, vs: (v.in_(d) for v in vs),


needs a plural test for contains. I am not sure how this code would work

tardyp · 2017-08-16T07:04:52Z

master/buildbot/data/resultspec.py

@@ -49,17 +49,26 @@ class FieldBase(object):
        'contains': lambda d, v: set(v) <= set(d),


looks like this 'old' code should provide a OR behaviour at data API level.
It is just the sql mode which was not implemented

Currently the old code does not provide OR behavior. For instance:

https://nine.buildbot.net/api/v2/builders?tags__contains=buildbot&order=builderid
yields multiple builders, including builderid: 1 (first one)

https://nine.buildbot.net/api/v2/builders?tags__contains=try&order=builderid
yields one builder builderid: 1

https://nine.buildbot.net/api/v2/builders?tags__contains=try&tags__contains=buildbot&order=builderid (combination of 1+2)
yields only one builder (AND behavior), but with OR it would yield all builders shown in the queries 1 and 2

I'll create a new pull request that changes the contains behavior to OR and then will adapt this request accordingly.

tardyp

fix the contain behaviour

tardyp · 2017-08-21T16:36:43Z

@jangmarker I think you wanted to rebase this PR?

jangmarker · 2017-08-23T02:02:28Z

Yes, sorry, I didn't make it before vacation. My response times will be rather slow until September.

Fixes queries like: /api/v2/builds?builderid__eq=1&builderid__eq=2 This would have yielded an empty result with the previous code.

jangmarker force-pushed the fix-rest-api-filtering branch from 7f437da to 45092b7 Compare August 15, 2017 14:43

jangmarker force-pushed the fix-rest-api-filtering branch from ae99f21 to f4d0848 Compare August 16, 2017 02:59

jangmarker force-pushed the fix-rest-api-filtering branch from f4d0848 to 23e237d Compare August 16, 2017 03:10

tardyp reviewed Aug 16, 2017

View reviewed changes

tardyp requested changes Aug 16, 2017

View reviewed changes

jangmarker mentioned this pull request Aug 16, 2017

Change REST API's __contains to OR behavior #3531

Merged

3 tasks

REST API: Fix single __contains for SQL queries

28ad845

jangmarker force-pushed the fix-rest-api-filtering branch from 23e237d to e26cf36 Compare August 23, 2017 02:00

jangmarker added 3 commits September 4, 2017 11:50

REST API: Make plural filter operators work with Alchemy

c580a04

Fixes queries like: /api/v2/builds?builderid__eq=1&builderid__eq=2 This would have yielded an empty result with the previous code.

Add release note

7513c10

Unit tests for REST filters if SQL is involved

6c88fd1

jangmarker force-pushed the fix-rest-api-filtering branch from e26cf36 to 313c3d1 Compare September 4, 2017 03:51

REST: Separate single SQL operators from python operators

624ec02

jangmarker force-pushed the fix-rest-api-filtering branch from 313c3d1 to 624ec02 Compare September 4, 2017 03:52

tardyp approved these changes Sep 5, 2017

View reviewed changes

tardyp merged commit d4218cc into buildbot:master Sep 5, 2017

jangmarker mentioned this pull request Sep 7, 2017

Builders UI: Load builds based on tags filter #3590

Merged

1 task

jangmarker deleted the fix-rest-api-filtering branch September 7, 2017 09:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix rest api filtering#3526

Fix rest api filtering#3526
tardyp merged 5 commits intobuildbot:masterfrom
jangmarker:fix-rest-api-filtering

jangmarker commented Aug 15, 2017 •

edited

Loading

Uh oh!

codecov bot commented Aug 15, 2017 •

edited

Loading

Uh oh!

tardyp commented Aug 15, 2017

Uh oh!

jangmarker commented Aug 16, 2017 •

edited

Loading

Uh oh!

tardyp commented Aug 16, 2017

Uh oh!

tardyp Aug 16, 2017

Uh oh!

tardyp Aug 16, 2017

Uh oh!

jangmarker Aug 16, 2017

Uh oh!

tardyp left a comment

Uh oh!

tardyp commented Aug 21, 2017

Uh oh!

jangmarker commented Aug 23, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -49,17 +49,26 @@ class FieldBase(object):
		'contains': lambda d, v: set(v) <= set(d),

Conversation

jangmarker commented Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contributor Checklist:

Uh oh!

codecov bot commented Aug 15, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tardyp commented Aug 15, 2017

Uh oh!

jangmarker commented Aug 16, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tardyp commented Aug 16, 2017

Uh oh!

tardyp Aug 16, 2017

Choose a reason for hiding this comment

Uh oh!

tardyp Aug 16, 2017

Choose a reason for hiding this comment

Uh oh!

jangmarker Aug 16, 2017

Choose a reason for hiding this comment

Uh oh!

tardyp left a comment

Choose a reason for hiding this comment

Uh oh!

tardyp commented Aug 21, 2017

Uh oh!

jangmarker commented Aug 23, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jangmarker commented Aug 15, 2017 •

edited

Loading

codecov bot commented Aug 15, 2017 •

edited

Loading

jangmarker commented Aug 16, 2017 •

edited

Loading