AWS S3 helpers for Node.js, as fast as a Lambo.

Table of Contents

• Table of Contents
• Presentation
• Installation
• Technical information
  • Node.js
  • Tests
    • Linting
    • Unit
• Usage
  • Environment variables
  • Import
  • async getObjectContent(params)
  • async getObjectHash(params)
  • async listKeys(params[, opts])
  • async upload(params[, options])
  • async uploadFile({ path, params[, options] })
  • async uploadDirectory({ path, params[, options, rootKey, ignore] })
  • Errors
    • Object structure
    • Codes
• Code of Conduct
• Contributing
• Support
• Security
• License

Presentation

AWS S3 helpers, as fast as a Lambo.

params and options of each helper are the same as in the AWS documentation.

aws-sdk module will automatically check for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, so there is no manipulation of any kind of vulnerable credentials.

Go Fast or Go Home.

Installation

npm install s3-lambo

npm i -S s3-lambo

Technical information

Node.js

• Language: JavaScript ES6/ES7
• VM: Node.js >= Dubnium (10.22.1)

Tests

In order to run unit tests, you'll need to set AWS_BUCKET, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

Command to run all tests:

npm test

Full example:

AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE" \
AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
AWS_BUCKET="test-bucket" \
npm test

Linting

ESLint with Airbnb base rules. See Airbnb JavaScript Style Guide.

npm run lint

Unit

Mocha and Chai.

npm test

Usage

Environment variables

name	type	description	default	example
AWS_ACCESS_KEY_ID*	AWS	Specifies an AWS access key associated with an IAM user or role.	none	AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY*	AWS	Specifies the secret key associated with the access key. This is essentially the "password" for the access key.	none	wJalrXUtnFEMI/K7MDENG/ bPxRfiCYEXAMPLEKEY
DEBUG	Debug	Debug mode. See bugbug.	none	s3-lambo:*

*required

Import

const s3 = require('s3-lambo');

// s3 is an object of functions
const {
  getObjectContent,
  getObjectHash,
  listKeys,
  upload,
  uploadFile,
  uploadDirectory,
} = require('s3-lambo');

s3-lambo module exports an object of functions. You'll find the complete list of functions below.

• s3 <Object> with the following functions.

async getObjectContent(params)

Returns the content of an S3 object.

Note:

• based on the S3 object ContentType property, the function will return an object if application/json is found, a string if it starts with text/ or a buffer.

• params <Object> See AWS getObject.
• Returns: <Promise>
  • Resolve: <Object> | <String> | <Buffer>
  • Reject: <Error>
    • AWS_NO_SUCH_KEY
    • AWS_ERROR

Examples:

// to be run in an async context

// ContentType is application/json
await getObjectContent({
  Key: 'lambo.json',
  Bucket: 'my-bucket',
}); // { aventador: 'V12' }

// ContentType is text/plain
await getObjectContent({
  Key: 'lambo.txt',
  Bucket: 'my-bucket',
}); // "Aventador"

// no ContentType, seen as application/octet-stream by default
await getObjectContent({
  Key: 'lambo.js',
  Bucket: 'my-bucket',
}); // Buffer

async getObjectHash(params)

Returns the md5 hash of the S3 object content.

• params <Object> See AWS getObject.
• Returns: <Promise>
  • Resolve: <String>
  • Reject: <Error>
    • AWS_NO_SUCH_KEY
    • AWS_ERROR

Examples:

// to be run in an async context

await getObjectHash({
  Key: 'lambo.json',
  Bucket: 'my-bucket',
}); // "cc49759e9ae4eb66a270bb61b9fb6b32"

async listKeys(params[, opts])

List all keys of an S3 bucket.

• params <Object> See AWS listObjectsV2.
• opts <Object>
  • ignoreKeys <Array> Keys to be ignored. Default: []
  • ignoreRegExp <RegExp> Keys to be ignored. Default: null
  • startSlash <Boolean> Whether keys listed should start with a slash. Default: false
• Returns: <Promise>
  • Resolve: <Array> Default: []
  • Reject: <Error>
    • AWS_ERROR

Examples:

// to be run in an async context

await listKeys({
  Bucket: bucket,
});

// [
//   'index.css',
//   'index.html',
//   'index.js',
//   'package.json',
//   'private/lambo.png',
//   'public/lambo.json',
//   'sw.js',
// ]

await listKeys({
  Bucket: bucket,
},
{
  ignoreKeys: [
    'sw.js',
  ],
  // ignore keys starting with private/ or ending with .json
  ignoreRegExp: /(^private\/)|(\.json$)/,
  startSlash: true,
});

// [
//   '/index.css',
//   '/index.html',
//   '/index.js',
// ]

async upload(params[, options])

Upload content to an S3 bucket at the specified key.

Note:

• add S3 object's Content-Type metadata based on key's extension, set to application/octet-stream by default (see node-mime-types for more details).

• params <Object> See AWS upload.
• options <Object> See AWS upload. Default: {}
• Returns: <Promise>
  • Resolve: <Boolean>
  • Reject: <Error>
    • AWS_ERROR

Examples:

// to be run in an async context

await upload({
  Key: 'lambo.txt',
  Bucket: 'my-bucket',
  Body: 'Aventador',
  CacheControl: 86400,
}); // true, Content-Type metadata is automatically set to text/plain

async uploadFile({ path, params[, options] })

Upload a file to an S3 bucket at the specified key using streams.

Note:

• add S3 object's Content-Type metadata based on file's extension, set to application/octet-stream by default (see node-mime-types for more details).

• parameters <Object>
  • path <String> Relative or absolute path to a file.
  • params <Object> See AWS upload.
  • options <Object> See AWS upload. Default: {}
• Returns: <Promise>
  • Resolve: <Boolean>
  • Reject: <Error>
    • FS_ERROR
    • AWS_ERROR

Examples:

// to be run in an async context

await uploadFile({
  path: '../lambo.png',
  params: {
    Key: 'private/lambo.png',
    Bucket: 'my-bucket',
    Body: buffer,
    CacheControl: 86400,
  },
}); // true, Content-Type set to image/png

async uploadDirectory({ path, params[, options, rootKey, ignore] })

Upload a directory and its subdirectories to an S3 bucket recursively.

Note:

• the file's key once uploaded to a S3 bucket will match the path relative to its root directory;
• rootKey is the root AWS key to use, by default it is the bucket root, e.g. saying rootKey is public/images and you want to upload /Users/you/my-project/pics, files will be uploaded to s3://bucket/public/images, default to '';
• add S3 object's Content-Type metadata based on file's extension, set to application/octet-stream by default (see node-mime-types for more details);
• without clustering we found uploading a directory of 1254 files was nearly 2 times faster than the native AWS CLI sync method (it's Python underneath, Node.js should be faster, even faster with a Lambo V12);
• improve this by clustering the whole upload, some extra code/controls will be needed (based on files' length, number of files, available cores, etc.).

• parameters <Object>
  • path <String> Relative or absolute path to a file.
  • params <Object> See AWS upload.
  • options <Object> See AWS upload. Default: {}
  • rootKey <String> The root AWS key where to upload the directory's files. Default: ''
  • ignore <Array> A list of strings to ignore in the key to upload, could be absolute or relative path to the rootKey. Default: null
• Returns: <Promise>
  • Resolve: <Boolean>
  • Reject: <Error>
    • FS_ERROR
    • AWS_ERROR

Examples:

// to be run in an async context

// uploading ./dist directory:
// dist/index.html
// dist/error.html
// dist/css/index.css
// dist/js/index.js
// dist/js/sw.js

await uploadDirectory({
  path: './dist',
  params: {
    Bucket: 'my-bucket',
  },
}); // true

// results in the S3 bucket
// index.html
// error.html
// css/index.css
// js/index.js
// js/sw.js

// uploading ../lambo directory:
// lambo/lambo.png
// lambo/logo/lamborghini.png
// lambo/models/aventador.png
// lambo/models/huracan.png
// lambo/models/urus/urus.png
// lambo/models/urus/urus_pearl_capsule.png
// lambo/models/urus/urus_graphite_capsule.png

await uploadDirectory({
  path: '../lambo',
  params: {
    Bucket: 'my-bucket',
    CacheControl: 86400,
  },
  rootKey: 'public/images',
  ignore: ['urus/'],
}); // true

// results in the S3 bucket
// public/images/lambo.png
// public/images/logo/lamborghini.png
// public/images/models/aventador.png
// public/images/models/huracan.png

Errors

Object structure

Errors emitted by s3-lambo inherit the native Error prototype with an additional code property and toString method.

{
  name,
  code,
  message,
  stack,
  toString(),
}

Codes

name	code	description	module
AWSError
	AWS_NO_SUCH_KEY	Key in bucket does not exist.	src/index
	AWS_ERROR	aws-sdk module encountered an error.	src/index
FSError
	FS_ERROR	An error related to the file system was caught.	src/index
LibError
	UNKNOWN_ERROR	An unknown error was caugth.	src/index

Code of Conduct

This project has a Code of Conduct. By interacting with this repository, organization, or community you agree to abide by its terms.

Contributing

Please take also a moment to read our Contributing Guidelines if you haven't yet done so.

Support

Please see our Support page if you have any questions or for any help needed.

Security

For any security concerns or issues, please visit our Security Policy page.

License

MIT.