[Discovery] Handle multiple lifecycle events on single event bus topic

[robrap]

This is a bit of a discovery ticket.

Acceptance Criteria:

• ADR for handling of multiple lifecycle events (created, updated, deleted) on a single topic.
• Update how-to as appropriate.
• Create follow-up ticket (if needed) for any work that would enable onboarding new multi-event type topics sooner rather than later.

Notes:

• Confluent recommends keeping multiple related event types on the same topic for ordering purposes:
  • See https://www.confluent.io/blog/put-several-event-types-kafka-topic/
  • And more recent article: https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/

Questions:

• Will we follow Confluent’s recommendation to use a single topic for multiple related events?
  • How will we get a partition key from our current event definition?
  • How will we handle the schema used for the topic to allow for the multiple event types?
    • Note: the second article (https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/) offers two options with different pros/cons.
    • [Robert] I'm thinking the Union solution is probably more complicated than what we need. However, I'd like to understand how schema evolution works with the other solution.
• Will this solution work for other event bus technologies other than Kafka?
• What are good naming conventions for the event_type, event definition (signal), data class, etc. for the proposed solution?

[robrap]

Moved from original private ticket: https://2u-internal.atlassian.net/browse/ARCHBOM-2008

[robrap]

Move some of these notes into the doc ticket, and closing this.

[robrap]

Note: This was closed without completing, and will remain closed (for now). However, I just updated the how-to to refer to this closed ticket: https://openedx.atlassian.net/wiki/spaces/AC/pages/3508699151/How+to+start+using+the+Event+Bus#Event-design.

Other considerations for ordering will include Redis Streams capabilities, which don't have the same partiton ordering guarantees as Kafka.

Here are docs that I moved from the how-to back to this ticket here:

Confluent, a hosted solution for Kafka, recommends keeping multiple related event types on the same topic for ordering purposes. If going for the approach of multiple lifecycle events in one topic, you will want to consider the partition key carefully. See Publishing a signal for more.

See https://www.confluent.io/blog/put-several-event-types-kafka-topic/
And more recent article: https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/

[robrap]

I'm re-opening this ticket because I want to make progress on it, and it is difficult to find as a closed ticket. :)

[robrap]

Also, we would need to update docs on naming conventions for topics if someone plans to add multiple events, since the topic name would no longer match 1-to-1. This should possibly planned for even if only the first of multiple events are being published?

[robrap]

It may be that some lifecycle events all match the same schema(???), and in this case, they all might be able to be written to the same topic with no change (i.e. just allowing the single schema).

[rgraber]

If you look at

openedx-events/openedx_events/learning/signals.py

Line 86 in 143ffd7

CERTIFICATE_CREATED = OpenEdxPublicSignal(

, you can see that the CERTIFICATE_CREATED, _CHANGED, and _REVOKED all have the same schema. COURSE_ENROLLMENT_CREATED and _CHANGED also take the same data structure.

[robrap]

@rgraber: You are welcome to test that sharing the same data structure is enough to make this work when that is the case. At the same time, I would like to understanding how this works for signals that don't share the same data structure. Note that the first events of this type will likely be these XBLOCK_ events, and one has different data.

[robrap]

• Docs include two options: 1) changing the naming strategy to any event type, but one schema per event type or 2) using an Avro union of the specific event types.
• @rgraber learned that option 2 is blocked on a potential Confluent bug, and can add more context.
• Option 1 we learned we can do at the cluster level, but possibly not on a per-topic basis, because the per-topic settings may not be exposed by Confluent Cloud. See topic settings for Confluent Cloud.
  • TODO: Do a timeboxed test of setting this setting via terraform, or ticket this separately.
  • Based on the results of the test, either set by-topic or set for cluster, and potentially document the consequences.

[rgraber]

ADRs are here (openedx-events) and here (edx-event-bus-kafka]

[rgraber]

Closing this. Updating documentation will be added as part of openedx/event-bus-kafka#164