Skip to content
main
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
img
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Amazon QLDB Ledger Loader

This is a framework for asynchronous loading of data into an Amazon QLDB ledger. The framework receives events from a delivery channel and writes them to the target ledger, performing idempotency checks and ensuring that document revisions are written in order. The supported delivery channels are listed below.

Loader programs (not included in this project) read source data and write each record as an event into the delivery channel in a specified message event format (see below). The loader program is de-coupled from the work of validating events and writing them to the ledger. It simply needs to read data from the source as fast as it can and put events into the channel.

Overview

The project includes a default implementation that is useful for loading a QLDB export into a new ledger, but is easily extensible to suit other data loading scenarios, including ongoing cross-region replication. The qldb-export-processor project provides the loader program for loading exports.

Event Format

The loader framework expects events as JSON or Ion strings with the following fields:

Field Required (Y/N) Description
op Y Identifies the data load operation for this record. Valid values are INSERT, UPDATE, DELETE, ANY. A value of ANY causes the framework to decide which operation is needed based on the event and the current state of the ledger.
table Y Identifies the table this record belongs to in the target ledger.
id N The unique identifier for this record. Used for idempotence checks. If omitted, no idempotence check is conducted.
version N Identifies the sequential revision number of this record. Used to ensure that revisions of documents are written in the correct order in the target ledger. If omitted, no version check is conducted.
data N The document (record) to be written to the ledger. Must be a JSON or Ion string. If omitted, this event will be treated as a deletion.

For example:

{
  "op": "UPDATE",
  "table": "Person",
  "id": "JuvoLmXrENBEFUDWacVuOb",
  "version": 1,
  "data": {
    "FirstName": "Brent",
    "LastName": "Logan",
    "DOB": "1967-07-03",
    "GovId": "LOGANB486CG",
    "GovIdType": "Driver License",
    "Address": "43 Stockert Hollow Road, Everett, WA, 98203"
  }
} 

Supported Delivery Channels

The following event delivery channels are supported.

Amazon SQS

Amazon SQS is useful for delivering events to a single destination.

Amazon SQS

Amazon SNS

Amazon SNS is useful for delivering events to multiple subscribers, even across regions. If ordering of events is required, use the Amazon SNS to Amazon SQS approach below. AWS Lambda does not currently support subscribing to an SNS FIFO topic, but it will subscribe to an SQS FIFO queue.

A Lambda function receives events from the SNS topic and loads the event into the target ledger. The Lambda service provides a limited number of event retries because it receives events from SNS asynchronously. Error-and-retry is the mechanism for preserving ordering of revisions of a document. Therefore, SNS should only be used as delivery channel for loads consisting of new records to deliver to the ledger, not for updates. Otherwise, an excessive number of events may end up in the dead-letter queue.

Single-region approach: Amazon SNS

Multi-region approach: Amazon SNS Multi-Region

Amazon SNS to Amazon SQS

Events can be written to an Amazon SNS topic in a fan-out architecture to distribute events to SQS queues in multiple regions to accomplish simultaneous multi-region writes. This is useful over a straight SNS-only fanout approach if FIFO is required because Lambda does not currently subscribe to SNS FIFO topics but will subscribe to SQS FIFO queues.

Amazon SNS to SQS

Amazon Kinesis

An Amazon Kinesis Data Stream can also be used as the delivery channel.

Amazon Kinesis

This framework can be used with QLDB Streaming to send data from one ledger to another. For example, cross-region replication.

Amazon Kinesis Cross-Region Replication

To use the framework to consume events from a QLDB stream, set the LoaderHandlerName parameter in the CloudFormation stack to software.amazon.qldb.load.receiver.QldbLedgerStreamEventReceiver::handleRequest.

Amazon Managed Streaming for Apache Kafka (MSK)

Amazon Managed Streaming for Apache Kafka

Amazon EventBridge

This framework can be used with Amazon EventBridge as the delivery channel. A Lambda function receives events from an EventBridge bus rule and loads the event into the target ledger. The Lambda service provides a limited number of event retries because it receives events from EventBridge asynchronously. Error-and-retry is the mechanism for preserving ordering of revisions of a document. Therefore, EventBridge should only be used as delivery channel for loads consisting of new records to deliver to the ledger, not for updates. Otherwise, an excessive number of events may end up in the dead-letter queue.

Amazon EventBridge

Requirements

  1. Sign up for AWS — Before you begin, you need an AWS account. For more information about creating an AWS account and retrieving your AWS credentials, see Accessing Amazon QLDB in the Amazon QLDB Developer Guide.
  2. A QLDB ledger — See Basic operations for ledgers.
  3. An S3 bucket — The bucket will be used to store code for deployment to Lambda. See Creating a bucket.

Setup

Build the project with the following command.

mvnw install

The build creates a JAR file in the target folder called amazon-qldb-ledger-load-VERSION-deploy-pkg.jar. Upload the file to an S3 bucket, where it will be used by one of the CloudFormation templates below.

Deploy the loader project by creating a stack with one of the templates in the src/main/cfn folder of the project. Select one of the following:

Template Description
qldb-ledger-load-via-sqs.yml SQS
qldb-ledger-load-via-sns.yml SNS
qldb-ledger-load-via-sns-to-sqs.yml SNS to SQS
qldb-ledger-load-via-kinesis.yml Kinesis
qldb-ledger-load-via-msk.yml MSK
qldb-ledger-load-via-eventbridge.yml Amazon EventBridge

Getting Help

Please use these community resources for getting help.

  • Ask a question on StackOverflow and tag it with the amazon-qldb tag.
  • Open a support ticket with AWS Support.
  • Make a new thread at AWS QLDB Forum.
  • If you think you may have found a bug, please open an issue.

License

This library is licensed under the MIT-0 license.

About

No description, website, or topics provided.

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages