Incorrect results returned from search query when logged-in

[judell]

blog.jonudell.net has 5 annotations with these group properties:

        "group": "__world__",
        "group": "__world__",
        "group": "__world__",
        "group": "MEpy2d",
        "group": "__world__",

I'm logged into 0.7.5.70 (g562d537.dirty), it reports 5 as I reckon it should because I'm a member of MEpy2d.

Badge query: GET /api/search?limit=0&uri=http://blog.jonudell.net/
Result: {"rows": [], "total": 5}

But no annotations anchor because the extension makes this query:

https://stage.hypothes.is/api/search?group=__world__&limit=200&offset=0&order=asc&sort=created&uri=http:%2F%2Fblog.jonudell.net%2F

And gets no results: {"rows": [], "total": 0}

[nickstenning]

I think the issue here is that searches as logged-in users aren't finding all the annotations they should find. Here's a gist showing the results of identical queries to the search endpoint when logged-out vs logged-in (on stage). The logged-in query shows fewer annotations, which can't possibly be correct.

Bizarrely, removing the group=__world__ parameter from the query, when logged-in, results in the correct annotations being returned.

Either way, the invariant is this: a logged-in query should always return the same annotations or more annotations than a logged-out query. I'll look into this a bit later on today.

[seanh]

I wonder if these are pre-groups annotations that have no group field? I'll look into this anyway

[nickstenning]

Ahhh, that might explain it. I might've been confused by the fact that they get rewritten on the way out.

[nickstenning]

And that would explain why it works when you're logged out -- the groups feature is toggled off so the group param is ignored.

[seanh]

Oh, actually now that I look my server-side code to insert group:'__world__' if the client sends no group was one of the bits that got removed in the end, so you can totally create annotations with no group field using the API now. It shouldn't happen though as our client does send the group field, so in Jon's case these will be pre-groups annotations

[seanh]

Yep, I can reproduce this locally:

• Use the API to create an annotation of a URI with no group field
• Chrome extension badge counts that annotation
• Sidebar doesn't show that annotation if groups feature is on, does show it if groups is off

[seanh]

Possible solutions:

1. Original groups design was that annotations in the public group (i.e. in no group) would have no groups field, making them the same as pre-groups annotations and annotations created via the API without a group field. Possibly too late to go back to this now though.
2. Have the sidebar not send any group filter when the public group is selected
3. Have the server transform searches for the public group into searches for public or no group
4. Put back the server-side code that silently inserts group __world__ if the client sends no group, and also do a migration for pre-group annotations
5. As 4 but instead of inserting group __world__ just reject annotations with no group field, give a validation error

[seanh]

My vote is for 4

[nickstenning]

Yep. 4. That requires changing only h.api.search.transform.prepare.

[seanh]

And doing an Es migration. Ok, I'll do that after lunch then

[nickstenning]

My point was that if you change prepare, we can run a migration by doing hypothesis annotool conf/production.ini prepare.

[nickstenning]

I confirm this fixed on stage.