Skip to content
πŸ“ Caching tool for quicker builds in CI systems
JavaScript
Branch: master
Clone or download
Latest commit cff7ecc Feb 12, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib catch errors and log error path Feb 12, 2019
.gitignore add readme Jul 25, 2018
README.md update docs Aug 15, 2018
install-from-cache-example.js update docs Aug 15, 2018
install-from-cache-larger-example.js run locally Feb 12, 2019
package-lock.json 0.0.10 Feb 12, 2019
package.json 0.0.10 Feb 12, 2019

README.md

Cache me outside

Caching tool for quicker builds in CI systems

Usage

1. Configure the files you want to cache

/* code from ./install-from-cache-example.js */
const path = require('path')
const cacheMeOutside = require('cache-me-outside')

/* cache destination folder */
const cacheFolder = path.join('/opt/build/cache', 'storage')

/* Array of folders to cache */
const contentsToCache = [
  {
    contents: path.join(__dirname, 'node_modules'),
    handleCacheUpdate: 'npm install',
    shouldCacheUpdate: async (cacheManifest, utils) => {
      /* Your custom invalidation logic */

      /*
         - Dates cache last updated
         - Make remote api request
         - Diff dependencies files like package.json
         - Whatever you want
      */

      const updateCache = true
      return updateCache // Boolean
    },
  },
  // ... add more folders if you want
]

// Run cacheMeOutside
cacheMeOutside(cacheFolder, contentsToCache).then((cacheInfo) => {
  console.log('Success! You are ready to rock')
  cacheInfo.forEach((info) => {
    console.log(info.cacheDir)
  })
})

2. Add to your build step

Now that we have configured what we want to cache, we need to add this to our build step.

Inside package.json, or whatever build tool you are using, run the catch-me-outside script before your build step.

{
  "scripts": {
    "prebuild": "node ./install-from-cache-example.js",
    "build": "react-scripts build"
  }
}

Inside of CI system: npm run build will run through the diagram below.

  1. Check for the cached files
  2. If no cache, run the handleCacheUpdate commands or function (ie npm install)
  3. Then save the contents to the cache directory for the next run.

If any of shouldCacheUpdate return true, the cached files are invalidated and handleCacheUpdate is ran again.

If you omit shouldCacheUpdate, the hash of the folder contents will be used, so if any file changes within the contents you are caching, the handleCacheUpdate will run.

API

/* code from install-from-cache-larger-example.js */
const path = require('path')
const cacheMeOutside = require('./lib') // require('cache-me-outside')

/* Netlify cache folder */
let cacheFolder = path.join('/opt/build/cache', 'my-cache-folder')

/* Array of folders to cache */
const contentsToCache = [
  {
    /**
     * Directory of files to cache
     * @type {String}
     */
    contents: path.join(__dirname, 'node_modules'),
    /**
     * Command or Function to run on `shouldCacheUpdate = true`
     * @type {String|Function}
     */
    handleCacheUpdate: 'npm install && echo "this runs when cache is invalid"',
    /**
     * Sets whether or not cache should get updated
     * @param  {object}  cacheManifest contains useful info for custom invalidation
     * @param  {object}  utils         contains helpful functions for diffing
     * @return {Boolean}              Returns true or false
     */
    shouldCacheUpdate: async (cacheManifest, utils) => {
      // This example uses changes to package.json to invalid cached 'node_modules' folder
      const packageJson = path.join(__dirname, 'package.json')
      const packageJsonChanged = await utils.diff(packageJson)
      // You can check multiple files or run custom logic
      return packageJsonChanged
    },
  },
  {
    contents: path.join(__dirname, 'other/node_modules'),
    shouldCacheUpdate: function() {
      /* your custom cache invalidation logic here */
      return false
    },
    handleCacheUpdate: 'yarn install'
  },
  {
    contents: path.join(__dirname, 'serverless-test/.serverless'),
    handleCacheUpdate: () => {
      console.log('run my custom stuff here')
    }
    // if `shouldCacheUpdate` omitted will use contents folder hash
  },
]

/*
// local cache folder for testing
cacheFolder = path.resolve('./cache')
/**/

// Run cacheMeOutside
cacheMeOutside(cacheFolder, contentsToCache).then((cacheInfo) => {
  console.log('Success! You are ready to rock')
  cacheInfo.forEach((info) => {
    console.log(info.cacheDir)
  })
})

When the cache is saved it generates a cache.json manifest file. This is passed into shouldCacheUpdate if you want to use it to invalidate your cache.

{
  "createdOn": 1534296638475,
  "createdOnDate": "Wed, 15 Aug 2018 01:30:38 GMT",
  "modifiedOn": 1534300695541,
  "cacheDir": "/Users/davidwells/Netlify/cache-magic/cache/Netlify/cache-magic/serverless-test",
  "cacheDirContents": "/Users/davidwells/Netlify/cache-magic/cache/Netlify/cache-magic/serverless-test/.serverless",
  "contents": {
    "src": "/Users/davidwells/Netlify/cache-magic/serverless-test/.serverless",
    "hash": "0496d16c0a8b1d43ca2d3c77ca48a8e237fdb625",
    "files": {
      "stuff.txt": "11b80f260a5eea9e867a23ab7f96aff77080ff90"
    }
  }
}

How it works



                       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                       β”‚                                       β”‚
                       β”‚             npm run build             β”‚
                       β”‚                                       β”‚
                       β”‚     "node ./cache-me-script.js"       β”‚
                       β”‚                                       β”‚
                       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚
                                   Does cache exist?
                                           β”‚
                                           β”‚
                                           β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€Yes───────────┐
                                           β”‚                         β”‚
                                           β”‚                         β”‚
                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€Noβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                         β”‚
                 β”‚                                                   β–Ό
                 β”‚                                     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                 β”‚                                     β”‚                           β”‚
                 β”‚                                     β”‚  Check if Cache is valid  β”‚
                 β”‚                                     β”‚  via `shouldCacheUpdate`  β”‚
                 β”‚                                     β”‚                           β”‚
                 β”‚                                     β”‚                           β”‚
                 β–Ό                                     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                                     β”‚
   β”‚                           β”‚                               Is cache valid?
   β”‚  Run `handleCacheUpdate`  β”‚                                     β”‚
   β”‚    command or function    │◀────────────Not valid───────────────┴─────────────Valid─────┐
   β”‚                           β”‚                                                             β”‚
   β”‚                           β”‚                                                             β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                                                             β”‚
                 β”‚                                                                           β–Ό
                 β”‚                                                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                 β–Ό                                                    β”‚                                            β”‚
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                                      β”‚   Cache is good and no files that would    β”‚
  β”‚                            β”‚                                      β”‚      trigger an update have changed.       β”‚
  β”‚   Copy contents to cache   β”‚                                      β”‚                                            β”‚
  β”‚   directory for next run   β”‚                                      β”‚ Copy cache contents to source destination  β”‚
  β”‚                            β”‚                                      β”‚                                            β”‚
  β”‚                            β”‚                                      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
You can’t perform that action at this time.