Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new event type for tag updates which fan-out to content updates #24

Closed
wants to merge 1 commit into from

Conversation

regiskuckaertz
Copy link

Currently tag updates/removals trickle down into N content updates, all managed via tagmanager and perhaps something else behind the scenes. This is problematic for several reasons:

  1. for each article, the whole article content has to be pulled from the repository and wired on the kinesis stream
  2. ... which has limited capacity anyway, and it often caused problems in the past
  3. when Porter receives a content update, it has no way of knowing whether that update has been triggered by a bona fide change in the article, or an update to one of its tags: therefore, it cannot make an informed decision whether to prioritise that update or not. Admittedly, these updates are much less important than changes to the content of the article.

With this change, 1 and 2 can entirely go away. In Porter, when an FANOUTUPDATE comes in, we can handle it as a partial reindex, i.e. stream the list of content IDs into bulk updates that will automatically be assigned a lower priority. The same treatment can be done for DELETE events, which it isn't at the moment 🤔

@regiskuckaertz
Copy link
Author

Closing as I think there's a better design:

  • when a tag is updated, the UPDATE event is sent as usual on the tag index stream
  • that event is caught by Porter, doing its job as usual
  • it is also caught by Composer (or via a different stream, that's an implementation detail), which will do two things:
  1. update the PG repository if it has to
  2. send a TAGUPDATE event on the flexible index stream
  • that event is caught by Porter, it contains the tag, thanks to which we can perform a reindex of all the articles attached to that tag, with a script to increase the revision field by one.

That way each source keeps responsibility for changes of the content it owns. You're good with that @itsibitzi @mbarton @akash1810 ? (Please include anyone I forgot to mention)

@regiskuckaertz regiskuckaertz deleted the rk-fanout branch March 28, 2019 11:36
@akash1810
Copy link
Member

cc @guardian/digital-cms

@regiskuckaertz
Copy link
Author

Moved the conversation over there

@mchv
Copy link
Member

mchv commented Apr 2, 2019

The revision field should not be increased in Content API outside of Composer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants