Skip to content

Writable local stream that rotates and syncs to AWS S3 based on max file size or age. Optimized for speed and concurrency.

License

Notifications You must be signed in to change notification settings

littlstar/rotating-s3-stream

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

27 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

rotating-s3-stream

This module creates a local write stream that periodically syncs and rotates to S3 based on file size and/or file duration. A write stream is created locally, then when the file exceeds maxFileSize or maxFileDuration, it is copied to S3, deleted locally, and a new write stream is created. By default, the files are ISO 8601 DateTime format, but you can also supply a custom naming function with the createFileName option.

Install

rotating-s3-stream has only been tested on MacOS and Linux. For maximum efficienty, it uses child_process along with the native AWS CLI, so that must be installed and configured separately. Instructions on how to do that can be found here.

Stream Options

  • localPrefix - {String} Required. Used as the local prefix for write stream.
  • s3Prefix - {String} Required. Used as the S3 prefix for syncing the write stream.
  • maxFileSize - {Number} Required. The max file size (bytes) the write stream can reach before rotating.
  • maxFileAge - {Number} Required. The max file age (seconds) the write stream can be before rotating.
  • createFileName - {Function} Optional. You can provide your own file-naming function. Defaults to ISO 8601.
  • checkRotationSchedule - {String} Optional. Provide a cron-style string that is the schedule for checking if a sync is necessary. Defaults to * * * * *

Example usage

const RotatingS3Stream = require('rotating-s3-stream')

const streamOptions = {
  localPrefix: '/tmp/rotating-stream-logs',
  s3Prefix: 's3://my-bucket/logs',
  maxFileSize: 1000000,                       // 1MB
  maxFileAge: 3600,                           // 1 hour
  createFileName: () => Date.now(),           // Use unix timestamps
  checkRotationSchedule: '* * * * * */5'      // Every 5 seconds
}

const stream = new RotatingS3Stream(streamOptions)

const logData = {
  user: 1,
  action: 'log-in'
}

// Will appear locally at: /tmp/rotating-stream-logs/<Current timestamp>
// And will be synced to S3://my-bucket/logs/<Current timestamp>
stream.write(JSON.stringify(logData))

About

Writable local stream that rotates and syncs to AWS S3 based on max file size or age. Optimized for speed and concurrency.

Resources

License

Stars

Watchers

Forks

Packages

No packages published