XDB Proposal & Design

Overview

This docs will describe the overall design of the xdb project. It will starts with the goal of the project, and then the technologies that it will be building on top of as tech stack. Then it will go to the high level design of how all the components work together. Finally it comes with a background section, and a FAQ section.

Goal

What is XDB

XDB is not a new database. It's "eXtending a database with a new async programming model".

This means nothing will be dropped from the database that it extended from -- all the existing functionalities of a database will be kept. With this extension, a database will provide both synchronous and asynchronous programming experience.

Internally, xdb utilize database CDC(change data capture) + message queue as implementation, which is a popular design pattern for async programming in the industry. Therefore, from another point of view, xdb creates an abstraction of this design pattern, so that users don't need to implement this pattern themselves(which requires to deal with a lot of low level details of a database).

Start with the problem

As of today, there are many excellent database products -- from SQL to NoSQL, and to even "NewSQL". They all provide "synchronous" APIs for applications. For example, a request is made from application to insert a record/row/document, then a database will process and respond back the result synchronously.

This is great but far not enough. The real-world applications often have to deal with "asynchronous process". For example, placing an order means inserting some database records first, then call external service for sending emails, reminders, or tracking order status. Those operations after inserting the database records are "asynchronous":

• There is no builtin/easy way to atomically inserting database records and also call external service
• Communicating with other services often need backoff retry for resiliency (eventual consistency)
• It may need to wait for some duration (like sending reminder often need to use a "durable timer")
• The operations need to be orchestrated with some sequence based on business need

Additionally, those async operations are usually coupling with databases, because databases are usually the "source of truth" for read/write. For example, user may want to update a record if user click the link in a reminder email, or update the record when the order status moved from "shipped" to "delivered".

Today to build an reliable and scalable async process:

• People put queues to represent different steps, and using consumer to executing the steps asynchronously
• People build their own "durable timer services" to persist a timer, or use cron job to run periodically to hack/work around
• People use "workflow" services or programming model like "Step Functions/Conductor/iWF" to orchestrate the API for databases/services
• People use database CDC(change data capture) + message queue to build async logic after database changes

Those solutions are complicated:

• Business logic are scattered across different places, hard to maintain
• Have to deal with a lot of low level APIs(like the CDC message schema)
• May require a lot of work to sync/transport data between different services and database

The new solution: async programming framework as part of a database

What if a database can provide async programming model out of the box?

• Users will not need different dependencies to build applications
• No need to sync/transport data between databases and others

The iWF programming model is a perfect fit for this idea. It could work seamlessly with any database.

In iWF framework, users will define a "persistence schema" for storing custom data as "data attributes". Then other two basic components of an iWF workflow: "WorkflowState" and "RPC", user will write code to read and write the data. In the python SDK example(because python is shorter. Java and Golang SDKs are also available in iWF):

class SubmitState(WorkflowState[Form]):
    def execute(self, ctx: WorkflowContext, input: Form, command_results: CommandResults, persistence: Persistence,
                communication: Communication,
                ) -> StateDecision:
        persistence.set_data_attribute(data_attribute_form, input)
        persistence.set_data_attribute(data_attribute_status, "waiting")
        print(f"API to send verification email to {input.email}")
        return StateDecision.single_next_state(VerifyState)

class VerifyState(WorkflowState[None]):
    def wait_until(self, ctx: WorkflowContext, input: T, persistence: Persistence, communication: Communication,
...

    def execute(self, ctx: WorkflowContext, input: T, command_results: CommandResults, persistence: Persistence,
...

class UserSignupWorkflow(ObjectWorkflow):
    def get_workflow_states(self) -> StateSchema:
        return StateSchema.with_starting_state(SubmitState(), VerifyState())

    def get_persistence_schema(self) -> PersistenceSchema:
        return PersistenceSchema.create(
            PersistenceField.data_attribute_def(data_attribute_form, Form),
            PersistenceField.data_attribute_def(data_attribute_status, str),
            PersistenceField.data_attribute_def(data_attribute_verified_source, str),
        )

    def get_communication_schema(self) -> CommunicationSchema:
...

    @rpc()
    def verify(
            self, source: str, persistence: Persistence, communication: Communication
    ) -> str:
        status = persistence.get_data_attribute(data_attribute_status)
        if status == "verified":
            return "already verified"
        persistence.set_data_attribute(data_attribute_status, "verified")
        persistence.set_data_attribute(data_attribute_verified_source, source)
        communication.publish_to_internal_channel(verify_channel)
        return "done"

You can read more details in the iWF wiki.

It would be much nicer if the persistence schema of the above workflow can be mapped directly to a database schema like this:

class UserSignupWorkflow(ObjectWorkflow):
    def get_workflow_states(self) -> StateSchema:
...

    def get_persistence_schema(self) -> PersistenceSchema:
        return PersistenceSchema.create(
            table_name = "user",
            primary_key = "id",
            PersistenceField.data_attribute_def(col_name=data_attribute_form, Form),
            PersistenceField.data_attribute_def(col_name=data_attribute_status, str),
            PersistenceField.data_attribute_def(col_name=data_attribute_verified_source, str),
        )
...

This means that that the same workflow code will be operating directly on the database:

• Each workflow execution will be mapped to a row of table "user"
• To map to a specific row, the workflowId is the "primary_key" of the table "id"
• Attributes "data_attribute_form", "data_attribute_status" and "data_attribute_verified_source" are mapped to the columns of the table

And the rest of the code is exactly the same:

• Any persistence.get_data_attribute(...) is reading from the table row
• Any persistence.set_data_attribute(...) is writing into the table row
• The persistence read/write in a WorkflowState/RPC can be atomic if using locking for persistence policy

Some difference from iWF

• Workflow --> Process
• WorkflowState --> AsyncState
• ObjectWorkflow --> ObjectProcess (the interface/baseClass in iWF/xdb SDKs)

Tech Stack and high-level design

More detailed design with example

Background

FAQ

What will be more "extension" in the future

Support multiple tables of a single process

Support additional read using options

Support additional write using API