Skip to content

Latest commit

 

History

History
28 lines (21 loc) · 977 Bytes

s3.md

File metadata and controls

28 lines (21 loc) · 977 Bytes

transport: s3

The S3 transporter puts logical WAL messages as gzipped objects in an S3 bucket partitioned by year, month and day in increasing order of time. This time is at the time of the PUT request creation, and not when the data was written to Postgres nor read from the pg-bifrost client.

Key Construction

The basic construction of an S3 key for this transporter is defined as:

<bucket name>/<key space>/<year as YYYY>/<month as MM>/<day as DD>/<datetime>_<first wal record id>.gz

Example

For a given configuration:

BIFROST_S3_BUCKET=test_bucket
BIFROST_S3_KEY_SPACE=test_files

A batch written at 2000:01:02 12:03:04 with the WAL record ID of the first record of 1336674354448 will be put at:

test_bucket/test_files/2000/01/02/2000_01_32_12_03_04_1336674354448.gz

You may also omit BIFROST_S3_KEY_SPACE which will result in an object being put at:

test_bucket/2000/01/02/2000_01_32_12_03_04_1336674354448.gz