Platform Event Bus

[e0d]

Quick links

• Event Bus tasks in GitHub project
• OEP-52: Event Bus Architecture
• Confluence project docs

Abstract

The already accepted OEP-41: Asynchronous Server Event Message Format details the general format and conventions the Open edX platform should use for asynchronous event messaging across services. It also provides background on a set of "Event Messaging Architectural Goals" for the Open edX platform.

The under-review-provisional OEP-52: Event Bus Architecture provides additional details around how the Open edX platform will be enhanced with an event bus technology that enables a reliable way to send events to multiple consumers across services, following the publish-subscribe (pub/sub) messaging pattern.

Ultimately, one could envision that most cross service communication would take advantage of the event bus to provide a loosely-coupled architecture.

This roadmap item is about making the event bus available to the Open edX platform, as well as seeing through some initial use cases in a Production environment.

Approach

2U's Arch-BOM team has created an architectural runway for cross-service communication using an event bus. The team implemented an event for catalog data from Studio to Discovery using an event bus implemented with Apache Kafka. This is a partial implementation of the OEP, which includes links to a variety of provisional decisions documented in ADRs.

The OEP also discusses introducing an abstraction layer that will enable alternative technology options for the event bus for the Open edX community. The Arch-BOM team created some foundations for the abstraction layer, and tCRIL is working on a Redis Stream implementation, and completing the abstraction layer.

Status

Initial use case production ready

This work has been completed. 2U is using the catalog changed event from Studio to Discovery in Production on edx.org.

Next up

Note: The Future epic is doubling as a placeholder for tasks that may move to other epics above as they are created in GitHub.

• tCRIL is working on a Redis Streams implementation of the event bus.
  • BUS view of Open edX Data Working Group Project.
  • Also see https://github.com/openedx/event-bus-redis/issues
  • Updating and accepting the OEP should be a part of this work.
• Work with 2U squads to implement additional use cases.
  • Use cases must be behind toggles until the event bus is required for a named release.
• Deferred discovery/maintenance on an as-needed bases.

Event Bus Project in GitHub:

• Event Bus project

Training and docs

Note: The section may move once we have a proper home page for docs.

• Event Bus Intro, Open edX Conference presentation recording
  • Conference presentation slides
• OEP-52: Event Bus Architecture
• Confluence project docs

[e0d]

@robrap could you build out the content here a little to make this item more approachable? We'll have a template eventually. I think what David did here is good for now: #27

[robrap]

@e0d: I updated the description and I am in process of moving more of the work from Jira to GitHub issues. Let me know if you have any other specific questions you want addressed before removing the "needs definition" label. Thank you.

[robrap]

@jmakowski1123: Maybe you are the right person to ask about the process for getting the "needs definition" label removed? What are the requirements that need to be met? Is this done? Thanks.

[jmakowski1123]

@jmakowski1123: Maybe you are the right person to ask about the process for getting the "needs definition" label removed? What are the requirements that need to be met? Is this done? Thanks.

Thanks @robrap , the details in the abstract and approach look good to me! Would it be possible to share links to the use cases and any other related design/specs here in this ticket as well? (once they are ready)

[robrap]

Update as of Nov 15, 2022:

• The Happy Path epic is nearly closed. Going to Production in the coming days. All QA in Stage has passed.
• The Sad Path epic is also nearly closed. We just need some doc updates.
• The Production Ready epic is close. Working out details of minimal observability and alerting.

[robrap]

Changes to main description:

• Updated language in the "Approach" section to better align with the current status.
• Updated "Status" subsection "Initial use case production ready" by marking closed and removing all the details.
• Updated "Status" subsection "Next Steps" to note tCRIL work on Redis and the Abstraction Layer.

@bmtcril: If you could add a link for the Redis work and/or update status on this issue as appropriate, that would be great. Thank you!

[robrap]

@jmakowski1123: Maybe this ticket should move past "Backlog" status? :)

[sarina]

Closing as this is marked "Shipped" on the PR board 🎉