Skip to content

XDB Proposal & Design

Jump to bottom
Quanzheng Long edited this page Sep 21, 2023 · 56 revisions

Overview

This docs will describe the overall design of the xdb project. It will starts with the goal of the project, and then the technologies that it will be building on top of as tech stack. Then it will go to the high level design of how all the components work together. Finally it comes with a background section, and a FAQ section.

Goal

What is XDB

XDB is not a new database. It's "eXtending a database with a new async programming model".

This means nothing will be dropped from the database that it extended from -- all the existing functionalities of a database will be kept. With this extension, a database will provide both synchronous and asynchronous programming experience.

Internally, xdb utilize database CDC(change data capture) + message queue as implementation, which is a popular design pattern for async programming in the industry. Therefore, from another point of view, xdb creates an abstraction of this design pattern, so that users don't need to implement this pattern themselves(which requires to deal with a lot of low level details of a database).

Start with the problem

As of today, there are many excellent database products -- from SQL to NoSQL, and to even "NewSQL". They all provide "synchronous" APIs for applications. For example, a request is made from application to insert a record/row/document, then a database will process and respond back the result synchronously.

This is great but far not enough. The real-world applications often have to deal with "asynchronous process". For example, placing an order means inserting some database records first, then call external service for sending emails, reminders, or tracking order status. Those operations after inserting the database records are "asynchronous":

  • There is no builtin/easy way to atomically inserting database records and also call external service
  • Communicating with other services often need backoff retry for resiliency (eventual consistency)
  • It may need to wait for some duration (like sending reminder often need to use a "durable timer")
  • The operations need to be orchestrated with some sequence based on business need

Additionally, those async operations are usually coupling with databases, because databases are usually the "source of truth" for read/write. For example, user may want to update a record if user click the link in a reminder email, or update the record when the order status moved from "shipped" to "delivered".

Today to build an reliable and scalable async process:

  • People put queues to represent different steps, and using consumer to executing the steps asynchronously
  • People build their own "durable timer services" to persist a timer, or use cron job to run periodically to hack/work around
  • People use "workflow" services or programming model like "Step Functions/Conductor/iWF" to orchestrate the API for databases/services
  • People use database CDC(change data capture) + message queue to build async logic after database changes

Those solutions are complicated:

  • Business logic are scattered across different places, hard to maintain
  • Have to deal with a lot of low level APIs(like the CDC message schema)
  • May require a lot of work to sync/transport data between different services and database

The new solution: async programming framework as part of a database

The "iWF async programming model" could work seamlessly with any database, to be the same thing to users.

With xdb, users will not need to depend/learn many technologies -- the development/maintenance/operation will be significantly simplified.

Some renamings from iWF

  • Workflow --> Process
  • WorkflowState --> AsyncState
  • ObjectWorkflow --> ObjectProcess (the interface/base class of iWF/xdb SDKs)

Tech Stack and high-level design

More detailed design with example

Background

FAQ

What will be more "extension" in the future

Clone this wiki locally