Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use job queue to ensure eventual synchronization of deleted annotations from Postgres to Elasticsearch #7840

Closed
robertknight opened this issue Feb 6, 2023 · 1 comment · Fixed by #8564
Closed

Use job queue to ensure eventual synchronization of deleted annotations from Postgres to Elasticsearch #7840

robertknight opened this issue Feb 6, 2023 · 1 comment · Fixed by #8564
Assignees
@seanh
Labels
bug S4

Comments

@robertknight
Copy link
Member

We have found several instances (hypothesis/client#5219, #7796) of annotation deletions not being synchronized from Postgres to Elasticsearch. So far the issue seems to happen infrequently, but when it does, it causes confusion due to mismatches between the search result count and actual number of annotations, and in API clients it can cause them to fail to paginate properly through results (see hypothesis/client#5219, although we need a separate solution for this).

We have a system in place to ensure that annotations are guaranteed to be eventually from Postgres => Elasticsearch after being created/updated, even in the event of an outage of Elasticsearch or RabbitMQ. This ticket is about using that system to improve reliability of deletion sync as well.

Original context: https://hypothes-is.slack.com/archives/C4K6M7P5E/p1675687184055039
@robertknight robertknight mentioned this issue Feb 6, 2023
Open
robertknight added a commit to hypothesis/client that referenced this issue Feb 7, 2023
@robertknight
Make search client paging more robust
b111d5c 
There is currently an issue in h where the search API call may sometimes return
pages that are incomplete, even if there are more pages of results to fetch
afterwards [1]. Until this is resolved in the backend, change the client to
continue paging through results as long as the current page has at least one
entry, which we need to get the cursor for the next page, and we expect more
results based on the `total` value.

Fixes #5219

[1] hypothesis/h#7840
robertknight added a commit to hypothesis/client that referenced this issue Feb 7, 2023
@robertknight
Make search client paging more robust
ebda9be 
There is currently an issue in h where the search API call may sometimes return
pages that are incomplete, even if there are more pages of results to fetch
afterwards [1]. A non-full page caused the client to stop fetching more pages as
it incorrectly assumed it had reached the end. Ultimately this needs to be
resolved in the API [2] by adding an explicit next-page link. Until that is done,
change the client to continue paging through results as long as the current page
has at least one entry (needed to construct the cursor) and we expect more
results based on the `total` value in the response.

Fixes #5219

[1] hypothesis/h#7840
[2] hypothesis/h#7841
@robertknight robertknight mentioned this issue Feb 7, 2023
Merged
robertknight added a commit to hypothesis/client that referenced this issue Feb 7, 2023
@robertknight
Make search client paging more robust
a62690d 
There is currently an issue in h where the search API call may sometimes return
pages that are incomplete, even if there are more pages of results to fetch
afterwards [1]. A non-full page caused the client to stop fetching more pages as
it incorrectly assumed it had reached the end. Ultimately this needs to be
resolved in the API [2] by adding an explicit next-page link. Until that is done,
change the client to continue paging through results as long as the current page
has at least one entry (needed to construct the cursor) and we expect more
results based on the `total` value in the response.

Fixes #5219

[1] hypothesis/h#7840
[2] hypothesis/h#7841
robertknight added a commit to hypothesis/client that referenced this issue Feb 8, 2023
@robertknight
Make search client paging more robust
73b1652 
There is currently an issue in h where the search API call may sometimes return
pages that are incomplete, even if there are more pages of results to fetch
afterwards [1]. A non-full page caused the client to stop fetching more pages as
it incorrectly assumed it had reached the end. Ultimately this needs to be
resolved in the API [2] by adding an explicit next-page link. Until that is done,
change the client to continue paging through results as long as the current page
has at least one entry (needed to construct the cursor) and we expect more
results based on the `total` value in the response.

Fixes #5219

[1] hypothesis/h#7840
[2] hypothesis/h#7841
@leedenison leedenison added the bug label May 19, 2023
@nairiboo nairiboo added the S4 label Jul 7, 2023
@seanh
Copy link
Contributor

seanh commented Feb 11, 2024

I'll also add that when a user asks us to delete their data (in this case: their annotation) we should be sure to actually delete that data from our systems, and this issue means that we sometimes fail to actually delete the data from Elasticsearch.

@seanh seanh self-assigned this Feb 16, 2024
seanh added a commit that referenced this issue Feb 16, 2024
@seanh
Use the job queue to delete annotations from Elasticsearch
508322c 
Fixes #7840
@seanh seanh mentioned this issue Feb 16, 2024
Closed
seanh added a commit that referenced this issue Feb 16, 2024
@seanh
Use the job queue to delete annotations from Elasticsearch
6137205 
Fixes #7840
seanh added a commit that referenced this issue Feb 16, 2024
@seanh
Use the job queue to delete annotations from Elasticsearch
390762d 
Fixes #7840
seanh added a commit that referenced this issue Feb 19, 2024
@seanh
Use the job queue to delete annotations from Elasticsearch
45f3410 
Fixes #7840
seanh added a commit that referenced this issue Feb 19, 2024
@seanh
Use job queue to delete annotations from Es
9de067b 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 19, 2024
@seanh
Use job queue to delete annotations from Es
ea316f8 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 26, 2024
@seanh
Use job queue to delete annotations from Es
0be03da 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 27, 2024
@seanh
Use job queue to delete annotations from Es
05b3f1a 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 28, 2024
@seanh
Use job queue to delete annotations from Es
8744a34 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
@seanh seanh mentioned this issue Feb 28, 2024
Merged
seanh added a commit that referenced this issue Feb 28, 2024
@seanh
Use job queue to delete annotations from Es
87251b3 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 28, 2024
@seanh
Use job queue to delete annotations from Es
e2f3edc 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 28, 2024
@seanh
Use job queue to delete annotations from Es
39c1854 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 29, 2024
@seanh
Use job queue to delete annotations from Es
fd1fc3d 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
seanh added a commit that referenced this issue Feb 29, 2024
@seanh
Use job queue to delete annotations from Es
97e4bcc 
Use the job queue to ensure that annotations get deleted from
Elasticsearch.

Fixes #7840
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@seanh seanh

Labels
bug S4
Projects
None yet
Milestone
No milestone
4 participants
@robertknight @seanh @leedenison @nairiboo