SPIKE: Research Annotations not created in DB despite "successful" POST /api/annotations request

[robertknight]

@nlisgo and myself have both experienced situations where, when creating a large number (eg. 500 or 1000) of annotations with a sequence of requests, a small number of annotations are subsequently missing from the DB despite all of the POST /api/annotations requests returning a 200 response with a valid payload.

So far I have only been able to reproduce the issue in production, not locally.

Steps to reproduce

1. Fill in the API_KEY constant below with a valid developer token and run the script against prod.

#!/usr/bin/env python

import requests
import json

API_KEY = '{redacted}'

ids = []
n_anns = 500
for i in range(0, n_anns):
    print('Creating annotation {}'.format(i))
    ann = {'uri': 'https://example.org/robert-is-testing-stuff',
           'text': 'A bulk annotation creation test'}
    headers = {'Authorization': 'Bearer {}'.format(API_KEY)}
    create_rsp = requests.post('https://hypothes.is/api/annotations',
                               data=json.dumps(ann), headers=headers)
    if create_rsp.status_code >= 400:
        print('Request failed: ', create_rsp.json())
        continue
    id_ = create_rsp.json()['id']
    ids.append(id_)

fetched_ok = []
for id_ in ids:
    headers = {'Authorization': 'Bearer {}'.format(API_KEY)}
    read_rsp = requests.get('https://hypothes.is/api/annotations/{}'.format(id_),
                            headers=headers)
    if read_rsp.status_code != 200:
        print('Failed to fetch annotation {}'.format(id_), read_rsp.json())
    else:
        fetched_ok.append(id_)

print('Fetched {} of {} annotations'.format(len(fetched_ok), len(ids)))

Expected result

All 500 annotations are created successfully

Actual result

I saw a small number of annotations which were "created" successfully yet do not subsequently appear in the DB or in Elasticsearch.

Creating annotation 0
Creating annotation 1
...
Creating annotation 499
('Failed to fetch annotation WhvyKNOFEeeFPw9UxPBc5A', {u'status': u'failure', u'reason': u"Either the resource you requested doesn't exist, or you are not currently authorized to see it."})
('Failed to fetch annotation bDpi0tOFEeewg0sezoe5qA', {u'status': u'failure', u'reason': u"Either the resource you requested doesn't exist, or you are not currently authorized to see it."})
('Failed to fetch annotation cV2h1NOFEeeJPo9ASdDObw', {u'status': u'failure', u'reason': u"Either the resource you requested doesn't exist, or you are not currently authorized to see it."})
Fetched 497 of 500 annotations

Notes

• My working hypothesis at the moment is that the transaction which created the annotation was rolled back after the HTTP response had been sent.
• This issue originally came up when @nlisgo attempted to perform a bulk import of >1000 annotations from Disqus into Hypothesis.

[seanh]

The fact that you were only able to reproduce it in production, not locally, suggests that it might have something to do with concurrent transactions that are happening at the same time as the transaction that fails. This is what it turned out to be with a previous session rollback issue we had. Really hard to debug unfortunately

[tgbugs]

I can report that I encountered this issue multiple times when doing a bulk POSTs of scibot annotations (40,000) to __world__. I have a strange thought that I did not experience this issue when doing a similarly large set of POSTs to a private group, but I think that that is because I was keeping a cached local copy of the responses and so never noticed the issue until I posted to __world__ where I was not keeping a copy.

[robertknight]

Some relevant chatter from the #eng-sr channel back in July 2017: https://hypothes-is.slack.com/archives/C074BUPEG/p1499245083385744

"closed with a dirty session" errors are still being reported frequently in Sentry: https://sentry.io/hypothesis/h/issues/232281261/

[seanh]

Thanks for the report @tgbugs . Damn, we (or at least I) had no idea this bug existed. We need to fix this!

@tgbugs I don't suppose you remember roughly how long ago it was, when you encountered this issue?

[tgbugs]

It was on November 9th. If you run an api query to fetch all the annotations that show up here: https://hypothes.is/users/scibot?q=group%3A__world__ you should be able to get the timestamps for all the POSTs. They were running consecutively, so you could probably narrow down to +/- 100ms the times when one was missed (the issue occured) by looking for gaps in the submissions (those were done in 3 batches so there will be bigger gaps as well).

[seanh]

Good to know. So I guess it's possible this bug was introduced recently, though I've got a feeling it has been here for a while and we never noticed.

[dwhly]

This possibly also happened recently to @heatherstaines who tried to reply to an annotation she had just made.

She also says it happened to her a few weeks back.

If this is the case it may mean that it's more serious than just batch imports.

[robertknight]

A slightly more useful script for reproducing the issue that does not leave hundreds of annotations around requiring cleanup afterwards: https://gist.github.com/robertknight/24e9a408472e62c910b8d6c522edda64

[judell]

Tonight I was running a test that led to me interactively create and then immediately delete annotations. Once out of a few dozen attempts, a delete provoked a 404.

[heatherstaines]

That's very interesting, as I make and delete annotations all the time in connections with the demos. I can't recall getting a 404--unless perhaps I had deleted the group in another tab, then went to try to delete an annotation in that group on another tab. Still, it doesn't stand out in my mind as something that I've experienced often. Heather

…

On Fri, Jan 5, 2018 at 12:55 AM, Jon Udell ***@***.***> wrote: Tonight I was running a test that led to me interactively creating and then immediately deleting annotations. Once out of a few dozen attempts, a delete provoked a 404. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4704 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXoiFxjuDwgqEL1skv4JwupYEqWYGv85ks5tHbk0gaJpZM4QsBso> .

-- Heather Ruland Staines Director, Partnerships Hypothes.is M: 203-400-1716 Skype: heather.staines

[robertknight]

Based on additional logging added in #4728 we've learned a few things:

1. The issue affects more than just creating annotations. It has been observed happening on DELETE /api/annotations and PATCH /api/annotations requests. My guess is that it potentially affects all requests that modify the DB. The most common issue is failure to update auth tickets when making a GET /api/badge call.
2. The issue definitely appears to happen less frequently shortly after a deploy, requiring hundreds of attempts to reproduce. Half an hour or so after a deploy, a repro happens much quicker (eg. typically only 50 or so attempts required)
3. Looking at an example of a failed PATCH /api/annotations call looking at the breadcrumbs we can see that a txn.{id} commit event is usually preceded by a txn.{id} commit <zope.sqlalchemy.datamanager.SessionDataManager> event which is a log message produced by the "transaction" package when it commits a resource manager. In the case when the warning is produced, the "zope" commit event is missing just before the txn.{id} commit event.

Taking (2) and (3) together, that strongly suggests this is related to the earlier issues with zope.sqlalchemy's internal _SESSION_STATE_ map.