Skip to content

microlinkhq/keyv-s3

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 


Last version Coverage Status NPM Status

Amazon S3 storage adapter for Keyv.

Motivation

Microlink API has a built-in cache layer for speed up consecutive API calls, caching the response based in a configurable ttl. You can read more about how it works in our blog post.

Until now, we delegate into Redis for serving the cache layer because it's fast; However, it's also expensive when you have a certain size.

The service is serving more than 8 millions API calls per month, where every request is creating a cache entry; That means running out of space could be done relatively fast.

We wanted a cache layer that met the following premises:

  • Infinite space.
  • Failover tolerant.
  • Cheap at scale.

Knowing what we wanted, and keeping in mind Microlink API response payload is JSON, we found Amazon S3 bucket excellent storage.

Install

$ npm install @aws-sdk/client-s3 keyv-s3 --save

Usage

First, you need to create a S3 bucket to be used as you cache layer

You need to setup a properly bucket name to be possible access via HTTP.

For doing that, we setup a CNAME over the bucket name, in our case we use CloudFlare.

That's a necessary step because querying via S3 API is slower than query directly to the bucket.

Also, CloudFlare gives us an extra perfomance boost since it's on top of the Bucket domain associated, caching successive calls and save your S3 quota.

In case you are using CloudFlare as well, we have the following page rule associated:

Now you can connect with your S3 caching layer from code:

const KeyvS3 = require('keyv-s3')

const keyvS3 = new KeyvS3({
  region: 'us-east-1',
  namespace: 'c.microlink.io',
  accessKeyId: process.env.S3_ACCESS_KEY_ID,
  secretAccessKey: process.env.S3_SECRET_ACCESS_KEY
})

You can interact with keyvS3 using keyv instance methods.

Backblaze B2 support

For using Backblaze B2 as provider, you have to provide s3client with Blackbaze B2 endpoint.

Also, since Backblaze B2 buckets name doesn't allow a dot, you should to specify the hostname to be used for mapping the bucket:

const keyvS3 = new KeyvS3({
  region: 'us-east-1',
  namespace: 'c-microlink-io',
  hostname: 'c.microlink.io',
  s3client: new S3Client({
    region: process.env.B2_REGION,
    endpoint: `https://s3.${process.env.B2_REGION}.backblazeb2.com`,
    credentials: {
      accessKeyId: process.env.B2_ACCESS_KEY_ID,
      secretAccessKey: process.env.B2_SECRET_ACCESS_KEY
    }
  })
})

License

keyv-s3 © microlink.io, released under the MIT License.
Authored and maintained by Kiko Beats with help from contributors.

microlink.io · GitHub microlink.io · Twitter @microlinkhq

About

Amazon S3 storage adapter for Keyv.

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •