Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out" #1703

Closed
thjaeckle opened this issue Jul 21, 2023 · 0 comments · Fixed by #1751
Assignees
Labels
Milestone

Comments

@thjaeckle
Copy link
Member

We encountered in the following case:

  • a policy is used by lots of things
  • a change it made to that policy
  • as a result a lot of search index updates are triggered for all things which used that policy

If now some search "ThingUpdaters" may not be able to retrieve the policy

  • then following WARNings can be seen:
Got AskTimeout during ask for message <SudoRetrievePolicy> and entityId <org.eclipse.ditto:shared-policy / policy> - retrying.. : <Recipient [Actor[akka://ditto-cluster/system/sharding/policyProxy#-1059358312]] had already been terminated. Message of type [org.eclipse.ditto.policies.api.commands.sudo.SudoRetrievePolicy].>
  • followed by INFO logs:
Computed - due to missing enforcer - 'emptied out' ThingWriteModel for metadata <Metadata [thingId=org.eclipse.ditto:my-thing, thingRevision=491, thingPolicy=PolicyTag [id=org.eclipse.ditto:shared-policy, revision=4], allReferencedPolicies=[PolicyTag [id=org.eclipse.ditto:shared-policy, revision=4]], modified=null, events=[], timers=[0 timers], ackRecipients=[], invalidateThing=false, invalidatePolicy=true, updateReasons=[UNKNOWN, POLICY_UPDATE]]> and thing <...
  • and then the thing in the search index basically is "emptied" - it can no longer be found
  • this situation does also not "heal itself" - so the eventual consistency is broken here

This is obviously a bug.
How to fix it however I do not yet know..

@thjaeckle thjaeckle added the bug label Jul 21, 2023
thjaeckle added a commit that referenced this issue Sep 15, 2023
…r due to internal timeouts

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
thjaeckle added a commit that referenced this issue Sep 15, 2023
…r due to internal timeouts

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>
thjaeckle added a commit that referenced this issue Sep 20, 2023
…arch-index-on-policy-failed-fetch

#1703 fixed wrongly emptied out search index on policy retrieval error due to internal timeouts
@thjaeckle thjaeckle self-assigned this Oct 6, 2023
@thjaeckle thjaeckle added this to the 3.4.0 milestone Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
1 participant