diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml index 71e3ba8..f8a8553 100644 --- a/.github/FUNDING.yml +++ b/.github/FUNDING.yml @@ -1,3 +1 @@ -# These are supported funding model platforms - -github: "qix-" +github: 'qix-' diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..ed2c874 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,21 @@ +name: Test +on: [push, pull_request] +jobs: + test: + name: Test + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Check out repository + uses: actions/checkout@v2 + with: + submodules: recursive + fetch-depth: 0 + - name: Install dependencies + run: npm i + - name: Lint commit message + uses: wagoid/commitlint-github-action@v4 + - name: Lint source + run: npm run lint + - name: Test + run: npm run test diff --git a/.npmrc b/.npmrc index 6b5f38e..07cae4a 100644 --- a/.npmrc +++ b/.npmrc @@ -1,2 +1,3 @@ save-exact = true package-lock = false +update-notifier = false diff --git a/LICENSE b/LICENSE index 52e36d2..bc52915 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2020 Josh Junon +Copyright (c) 2020-2022 Josh Junon Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 82252d7..31763ab 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ functions. Yes, those really exist. ```console $ npm install --save scaly ``` + ## Usage Inspired by CPU caching, applications can build out data access layers for various @@ -18,11 +19,11 @@ and facilitate the propagation of data back up the chain in the event of misses. That's a lot of buzzwords - here's an example problem that Scaly solves: -- You have a database where an Employee ID number is associated with their First Name. -- In the most common case, that name isn't going to be updated frequently - if ever. -- Not only is it not updated frequently, but you have to look up the First Name often, hammering your database (MongoDB, for example). -- You set up a key/value store (e.g. Redis) to cache the entries - maybe with an hour TTL by default. -- You'd also like a per-service-instance memory cache with extremely low TTLs (in the order of seconds) for many subsequent requests, too. +- You have a database where an Employee ID number is associated with their First Name. +- In the most common case, that name isn't going to be updated frequently - if ever. +- Not only is it not updated frequently, but you have to look up the First Name often, hammering your database (MongoDB, for example). +- You set up a key/value store (e.g. Redis) to cache the entries - maybe with an hour TTL by default. +- You'd also like a per-service-instance memory cache with extremely low TTLs (in the order of seconds) for many subsequent requests, too. Your code might look something like this: @@ -74,27 +75,27 @@ export async function getFirstName(eid) { That's a lot of code. Here are some problems: -- This is **one** API call. There are a lot of branches, very unclear code, - and the repeating of e.g. the `setUsernameInMemcache()` duplicate call - can lead to bugs (especially if you inlined the `set/getUsernameInXXX()` - functions). -- The control flow is hard to follow. This is a simple getter from multiple - data stores - anything more complicated will result in even more code. -- It's not extensible. Adding a new data source requires surgical changes - to existing APIs that cannot be individually tested (easily, at least - without _extensive_ fragmentation). -- Multiply this code by 100x. That's a conservative number of datastore - operations a medium-sized application might have. -- Error handling is ambiguous - do you return an error value, or throw - an exception? How do I differentiate between user (request) error (e.g. - requesting an invalid ID) and an internal error (e.g. connection was - reset, invalid database credentials, etc.)? -- RedisLabs just went down. There's a bug in the fault-tolerant Redis - implementation and now any attempts at getting cached values fail. - Our upstream outage just turned into a downstream outage. We need to - re-deploy without any Redis implementation and fall-back to just - memcache and mongo. How do you do this when you have 100x callsites - that need to be modified? +- This is **one** API call. There are a lot of branches, very unclear code, + and the repeating of e.g. the `setUsernameInMemcache()` duplicate call + can lead to bugs (especially if you inlined the `set/getUsernameInXXX()` + functions). +- The control flow is hard to follow. This is a simple getter from multiple + data stores - anything more complicated will result in even more code. +- It's not extensible. Adding a new data source requires surgical changes + to existing APIs that cannot be individually tested (easily, at least + without _extensive_ fragmentation). +- Multiply this code by 100x. That's a conservative number of datastore + operations a medium-sized application might have. +- Error handling is ambiguous - do you return an error value, or throw + an exception? How do I differentiate between user (request) error (e.g. + requesting an invalid ID) and an internal error (e.g. connection was + reset, invalid database credentials, etc.)? +- RedisLabs just went down. There's a bug in the fault-tolerant Redis + implementation and now any attempts at getting cached values fail. + Our upstream outage just turned into a downstream outage. We need to + re-deploy without any Redis implementation and fall-back to just + memcache and mongo. How do you do this when you have 100x callsites + that need to be modified? Along with a host of other issues. @@ -157,27 +158,27 @@ try { So, what's happening here? -- Each layer is comprised of the same (sub)set of API methods (in the example case, every layer - has a `getFirstNameByEid(eid)` method). -- Each API method must be an async generator - i.e. `async *foo` (note the `*`). This allows both - the `await` and `yield` keywords - the former useful for application developers, and the latter - required for Scaly to work. -- The layer first checks if it can resolve the request. If it can, it `return`s the result. -- If it cannot resolve the request, but wants to be notified of the eventual result (e.g. for - inserting into the cache), it can choose to use the result of a `yield` expression, which - resolves to the result from a deeper layer. - - `yield` will not return if an error is `yield`ed or `throw`n by another layer. - - The API method does NOT need to `return` in this case (the return value is ignored anyway). - This is what makes the single-line implementation for Redis's layer above work. -- If it does _not_ care about the eventual result, it can simply `return;` or `return undefined;`. - Scaly will move on to the next layer without notifying this layer of a result later on. - In the event the very last (deepest) layer `return undefined`'s, an error is thrown - **all - API methods must resolve or error at some point**. -- If the layer method wishes to raise a **recoverable or user error**, it should `yield err;` (where - `err` is anything your application needs - a string, an `Error` object, or something else). - **The function will not resume after the yield expression**, effectively acting like a `throw` statement. -- If the layer method wishes to raise an **unrecoverable/exceptional error**, it should `throw`. - This should be reserved for unrecoverable (e.g. connection lost, bad DB credentials, etc.) errors. +- Each layer is comprised of the same (sub)set of API methods (in the example case, every layer + has a `getFirstNameByEid(eid)` method). +- Each API method must be an async generator - i.e. `async *foo` (note the `*`). This allows both + the `await` and `yield` keywords - the former useful for application developers, and the latter + required for Scaly to work. +- The layer first checks if it can resolve the request. If it can, it `return`s the result. +- If it cannot resolve the request, but wants to be notified of the eventual result (e.g. for + inserting into the cache), it can choose to use the result of a `yield` expression, which + resolves to the result from a deeper layer. + - `yield` will not return if an error is `yield`ed or `throw`n by another layer. + - The API method does NOT need to `return` in this case (the return value is ignored anyway). + This is what makes the single-line implementation for Redis's layer above work. +- If it does _not_ care about the eventual result, it can simply `return;` or `return undefined;`. + Scaly will move on to the next layer without notifying this layer of a result later on. + In the event the very last (deepest) layer `return undefined`'s, an error is thrown - **all + API methods must resolve or error at some point**. +- If the layer method wishes to raise a **recoverable or user error**, it should `yield err;` (where + `err` is anything your application needs - a string, an `Error` object, or something else). + **The function will not resume after the yield expression**, effectively acting like a `throw` statement. +- If the layer method wishes to raise an **unrecoverable/exceptional error**, it should `throw`. + This should be reserved for unrecoverable (e.g. connection lost, bad DB credentials, etc.) errors. `scaly(layer1, layer2, layerN)` returns a new object with all of the API methods between all layers. This means that if `layer1` has a `getFoo()` API method, and `layer2` has a `getBar()` method, the @@ -199,4 +200,4 @@ converted to a `yield err`). # License -Copyright © 2020, Josh Junon. Released under the [MIT License](LICENSE). +Copyright © 2020-2022, Josh Junon. Released under the [MIT License](LICENSE). diff --git a/index.d.ts b/index.d.ts index 2c75381..a9f3767 100644 --- a/index.d.ts +++ b/index.d.ts @@ -1,29 +1,39 @@ -type ScalyResult = AsyncGenerator; +type ScalyResult = AsyncGenerator< + Error_ | undefined, + ReturnValue | void, + ReturnValue +>; -type BoxedTupleTypes = - {[P in keyof T]: [T[P]]}[Exclude]; +type BoxedTupleTypes = { [P in keyof T]: [T[P]] }[Exclude< + keyof T, + keyof any[] +>]; -type UnionToIntersection = - (U extends any ? (k: U) => void : never) extends ((k: infer I) => void) ? I : never; +type UnionToIntersection = (U extends any ? (k: U) => void : never) extends ( + k: infer I +) => void + ? I + : never; -type UnboxIntersection = T extends {0: infer U} ? U : never; +type UnboxIntersection = T extends { 0: infer U } ? U : never; declare function scaly< T, S extends any[], - U = T & UnboxIntersection>>, + U = T & UnboxIntersection>> >( target: T, ...sources: S ): { - [P in keyof U]: U[P] extends (...args: any[]) => ScalyResult - ? (...args: Parameters) => Promise<[true, ReturnValue] | [false, Error_]> - : never + [P in keyof U]: U[P] extends ( + ...args: any[] + ) => ScalyResult + ? ( + ...args: Parameters + ) => Promise<[true, ReturnValue] | [false, Error_]> + : never; }; -export { - scaly, - ScalyResult, -}; +export { scaly, ScalyResult }; export default scaly; diff --git a/index.js b/index.js index b56c442..ee7ef09 100644 --- a/index.js +++ b/index.js @@ -4,9 +4,9 @@ module.exports = (...layers) => { } // eslint-disable-next-line unicorn/no-array-reduce - const allOperations = new Set(layers.reduce( - (acc, layer) => acc.concat(Object.keys(layer)) - , [])); + const allOperations = new Set( + layers.reduce((acc, layer) => acc.concat(Object.keys(layer)), []) + ); const DB = {}; @@ -27,16 +27,16 @@ module.exports = (...layers) => { const gen = layer[op](...args); // Run until first return/yield - const {value: result, done} = await gen.next(); + const { value: result, done } = await gen.next(); // Did we return? if (done) { if (result !== undefined) { // Layer returned a value; propagate it up // to all other layers who asked for it. - await Promise.all(pendingGenerators.map( - gen => gen.next(result), - )); + await Promise.all( + pendingGenerators.map(gen => gen.next(result)) + ); return [true, result]; } @@ -71,7 +71,9 @@ module.exports = (...layers) => { // a user cannot fix this case by changing the // inputs, for example). throw new Error( - `operation was not handled by any configured layers: ${op} (attempted layers: ${attemptedLayers.join(', ')})`, + `operation was not handled by any configured layers: ${op} (attempted layers: ${attemptedLayers.join( + ', ' + )})` ); }; } diff --git a/index.test-d.ts b/index.test-d.ts index 62b27bd..b4eede2 100644 --- a/index.test-d.ts +++ b/index.test-d.ts @@ -1,45 +1,45 @@ -import {expectType} from 'tsd'; -import scaly, {ScalyResult} from '.'; +import { expectType } from 'tsd'; +import scaly, { ScalyResult } from '.'; const db = { maxUsers: 10, cursor: 0, - users: new Map(), + users: new Map(), - async * registerUser(username: string): ScalyResult { + async *registerUser(username: string): ScalyResult { if (this.users.size >= this.maxUsers) { yield 'tooManyUsers'; } const id = this.cursor++; - this.users.set(id, {username, messages: []}); + this.users.set(id, { username, messages: [] }); return id; }, - async * getUsername(id: number): ScalyResult { + async *getUsername(id: number): ScalyResult { const user = this.users.get(id); return user ? user.username : yield 'noSuchUser'; }, - async * getMessages(id: number): ScalyResult { + async *getMessages(id: number): ScalyResult { const user = this.users.get(id); return user ? user.messages.slice() : yield 'noSuchUser'; }, - async * sendMessage(id: number, message: string): ScalyResult { + async *sendMessage(id: number, message: string): ScalyResult { const user = this.users.get(id); return user ? (user.messages.push(message), null) : yield 'noSuchUser'; }, - async * checkDBStatus(): ScalyResult { + async *checkDBStatus(): ScalyResult { return this.users.size < this.maxUsers ? null : yield 'tooManyUsers'; - }, + } }; const cache = { usernames: new Map(), - async * getUsername(id: number): ScalyResult { + async *getUsername(id: number): ScalyResult { const username = this.usernames.get(id); if (username) { return username; @@ -48,9 +48,9 @@ const cache = { this.usernames.set(id, yield); }, - async * checkCacheStatus(): ScalyResult { + async *checkCacheStatus(): ScalyResult { return null; - }, + } }; const api = scaly(db, cache); diff --git a/package.json b/package.json index c8942b2..b7372ef 100644 --- a/package.json +++ b/package.json @@ -1,52 +1,57 @@ { - "name": "scaly", - "version": "1.0.0", - "description": "Minimalistic, composable cache and database layering inspired by CPU caches", - "main": "index.js", - "types": "index.d.ts", - "repository": "qix-/scaly", - "files": [ - "index.js", - "index.d.ts", - "LICENSE" - ], - "scripts": { - "test": "best -v -I test.js && tsd && xo" - }, - "keywords": [ - "cache", - "layers", - "database", - "generators", - "fallback" - ], - "xo": { - "ignore": [ - "./index.test-d.ts" - ], - "rules": { - "require-yield": [ - 0 - ], - "unicorn/no-reduce": [ - 0 - ], - "no-await-in-loop": [ - 0 - ], - "operator-linebreak": [ - 0 - ], - "unicorn/prefer-module": [ - 0 - ] - } - }, - "author": "Josh Junon (https://github.com/qix-)", - "license": "MIT", - "devDependencies": { - "@zeit/best": "0.7.3", - "tsd": "0.17.0", - "xo": "0.44.0" - } + "name": "scaly", + "version": "1.0.0", + "description": "Minimalistic, composable cache and database layering inspired by CPU caches", + "main": "index.js", + "types": "index.d.ts", + "repository": "qix-/scaly", + "files": [ + "index.js", + "index.d.ts", + "LICENSE" + ], + "scripts": { + "format": "prettier --write --ignore-path .gitignore .", + "lint": "prettier --check --ignore-path .gitignore .", + "format:staged": "pretty-quick --staged", + "lint:commit": "commitlint -x @commitlint/config-conventional --edit", + "test": "best -v -I test.js && tsd" + }, + "keywords": [ + "cache", + "layers", + "database", + "generators", + "fallback" + ], + "author": "Josh Junon (https://github.com/qix-)", + "license": "MIT", + "devDependencies": { + "@zeit/best": "0.7.3", + "tsd": "0.17.0", + "@commitlint/cli": "16.1.0", + "@commitlint/config-conventional": "16.0.0", + "@vercel/git-hooks": "1.0.0", + "prettier": "2.5.1", + "pretty-quick": "3.1.3" + }, + "publishConfig": { + "access": "public", + "tag": "latest" + }, + "git": { + "pre-commit": "format:staged", + "commit-msg": "lint:commit" + }, + "prettier": { + "useTabs": true, + "semi": true, + "singleQuote": true, + "jsxSingleQuote": false, + "trailingComma": "none", + "arrowParens": "avoid", + "requirePragma": false, + "insertPragma": false, + "endOfLine": "lf" + } } diff --git a/test.js b/test.js index 71b925b..5dec45e 100644 --- a/test.js +++ b/test.js @@ -26,7 +26,7 @@ function makePersistentDB() { userCount: 0, sessionCount: 0, users: {}, - sessions: {}, + sessions: {} }; return { @@ -34,17 +34,17 @@ function makePersistentDB() { return 'pdb'; }, - async * createSession(uid) { + async *createSession(uid) { const token = db.sessionCount++; if (!(uid in db.users)) { yield [['pdb'], 'no such user ID']; } - db.sessions[token] = {uid}; + db.sessions[token] = { uid }; return [['pdb'], token]; }, - async * destroySession(token) { + async *destroySession(token) { if (!(token in db.sessions)) { yield [['pdb'], 'invalid token']; } @@ -53,13 +53,13 @@ function makePersistentDB() { return [['pdb'], true]; }, - async * getUID(token) { + async *getUID(token) { return db.sessions[token] ? [['pdb'], db.sessions[token].uid] : yield [['pdb'], 'invalid token']; }, - async * getUsername(token) { + async *getUsername(token) { const session = db.sessions[token]; if (!session) { yield [['pdb'], 'invalid token']; @@ -73,24 +73,24 @@ function makePersistentDB() { return [['pdb'], user.username]; }, - async * addUser(username) { + async *addUser(username) { if (!username || typeof username !== 'string') { yield [['pdb'], 'invalid username (expected non-blank string)']; } const uid = db.userCount++; - db.users[uid] = {uid, username}; + db.users[uid] = { uid, username }; return [['pdb'], uid]; }, - async * deleteUser(uid) { + async *deleteUser(uid) { if (!(uid in db.users)) { yield [['pdb'], 'no such user ID']; } delete db.users[uid]; return [['pdb'], true]; - }, + } }; } @@ -110,12 +110,12 @@ function makeMemoryCache(ttl) { const cache = new Map(); - const store = (k, v) => cache.set(k, {v, ttl}); + const store = (k, v) => cache.set(k, { v, ttl }); const load = k => { const record = cache.get(k); if (record) { - if ((--record.ttl) === 0) { + if (--record.ttl === 0) { cache.delete(k); } @@ -128,14 +128,14 @@ function makeMemoryCache(ttl) { return 'mc'; }, - async * createSession(uid) { + async *createSession(uid) { const [trace, token] = yield; const key = `token:uid:${token}`; store(key, uid); trace.push('mc'); }, - async * destroySession(token) { + async *destroySession(token) { const key = `token:uid:${token}`; cache.delete(key); // No return; hand off control to the @@ -143,7 +143,7 @@ function makeMemoryCache(ttl) { // here with the result. }, - async * getUID(token) { + async *getUID(token) { const key = `token:uid:${token}`; const value = load(key); if (value !== undefined) { @@ -153,82 +153,55 @@ function makeMemoryCache(ttl) { const [trace, uid] = yield; store(key, uid); trace.push('mc'); - }, + } }; } const makeDB = memoryCacheTTL => // Order matters! - scaly( - makeMemoryCache(memoryCacheTTL || 1), - makePersistentDB(), - ); + scaly(makeMemoryCache(memoryCacheTTL || 1), makePersistentDB()); exports.createUser = async () => { const db = makeDB(); - equal( - await db.addUser('qix'), - [true, [['pdb'], 0]], - ); + equal(await db.addUser('qix'), [true, [['pdb'], 0]]); }; exports.errorCreateUserEmpty = async () => { const db = makeDB(); - equal( - await db.addUser(''), - [false, [['pdb'], 'invalid username (expected non-blank string)']], - ); + equal(await db.addUser(''), [ + false, + [['pdb'], 'invalid username (expected non-blank string)'] + ]); }; exports.createUserMultiple = async () => { const db = makeDB(); - equal( - await db.addUser('qix'), - [true, [['pdb'], 0]], - ); - equal( - await db.addUser('qux'), - [true, [['pdb'], 1]], - ); + equal(await db.addUser('qix'), [true, [['pdb'], 0]]); + equal(await db.addUser('qux'), [true, [['pdb'], 1]]); }; exports.deleteUser = async () => { const db = makeDB(); - equal( - await db.addUser('qix'), - [true, [['pdb'], 0]], - ); - equal( - await db.deleteUser(0), - [true, [['pdb'], true]], - ); + equal(await db.addUser('qix'), [true, [['pdb'], 0]]); + equal(await db.deleteUser(0), [true, [['pdb'], true]]); }; exports.errorDeleteUnknownUser = async () => { const db = makeDB(); - equal( - await db.deleteUser(1337), - [false, [['pdb'], 'no such user ID']], - ); + equal(await db.deleteUser(1337), [false, [['pdb'], 'no such user ID']]); }; exports.errorCreateSessionBadUser = async () => { const db = makeDB(); - equal( - await db.createSession(1337), - [false, [['pdb'], 'no such user ID']], - ); + equal(await db.createSession(1337), [false, [['pdb'], 'no such user ID']]); }; exports.createSession = async () => { const db = makeDB(); - equal( - await db.addUser('qix'), - [true, [['pdb'], 0]], - ); + equal(await db.addUser('qix'), [true, [['pdb'], 0]]); equal( await db.createSession(0), - [true, [['pdb', 'mc'], 0]], // First PDB creates it, then MC caches it + [true, [['pdb', 'mc'], 0]] // First PDB creates it, then MC caches it ); equal( await db.destroySession(0), @@ -237,82 +210,55 @@ exports.createSession = async () => { // This is probably not how it should be designed in the real-world, but // this is supposed to test the `return undefined` case, which has special // semantics in Scaly (not an error, but not a 'hit'/result). - [true, [['pdb'], true]], - ); - equal( - await db.deleteUser(0), - [true, [['pdb'], true]], + [true, [['pdb'], true]] ); + equal(await db.deleteUser(0), [true, [['pdb'], true]]); }; exports.errorDestroyInvalidSession = async () => { const db = makeDB(); - equal( - await db.destroySession(1337), - [false, [['pdb'], 'invalid token']], - ); + equal(await db.destroySession(1337), [false, [['pdb'], 'invalid token']]); }; exports.errorQueryUIDInvalidToken = async () => { const db = makeDB(); - equal( - await db.getUID(1337), - [false, [['pdb'], 'invalid token']], - ); + equal(await db.getUID(1337), [false, [['pdb'], 'invalid token']]); }; exports.errorQueryUsernameInvalidToken = async () => { const db = makeDB(); - equal( - await db.getUsername(1337), - [false, [['pdb'], 'invalid token']], - ); + equal(await db.getUsername(1337), [false, [['pdb'], 'invalid token']]); }; exports.queryUID = async () => { const db = makeDB(2); // Serve 2 fetches from memory cache before invalidating (simulates TTL in the real-world) - equal( - await db.addUser('qix'), - [true, [['pdb'], 0]], - ); - equal( - await db.createSession(0), - [true, [['pdb', 'mc'], 0]], - ); + equal(await db.addUser('qix'), [true, [['pdb'], 0]]); + equal(await db.createSession(0), [true, [['pdb', 'mc'], 0]]); equal( await db.getUID(0), - [true, [['mc'], 0]], // Cache hit (1 left) + [true, [['mc'], 0]] // Cache hit (1 left) ); equal( await db.getUID(0), - [true, [['mc'], 0]], // Cache hit (0 left, next hits persistence) + [true, [['mc'], 0]] // Cache hit (0 left, next hits persistence) ); equal( await db.getUID(0), - [true, [['pdb', 'mc'], 0]], // Hit persistence layer first, then cache + [true, [['pdb', 'mc'], 0]] // Hit persistence layer first, then cache ); equal( await db.getUID(0), - [true, [['mc'], 0]], // Value was re-warmed again + [true, [['mc'], 0]] // Value was re-warmed again ); }; exports.queryUsername = async () => { const db = makeDB(2); // Serve 2 fetches from memory cache before invalidating (simulates TTL in the real-world) - equal( - await db.addUser('qix'), - [true, [['pdb'], 0]], - ); - equal( - await db.createSession(0), - [true, [['pdb', 'mc'], 0]], - ); + equal(await db.addUser('qix'), [true, [['pdb'], 0]]); + equal(await db.createSession(0), [true, [['pdb', 'mc'], 0]]); equal( await db.getUID(0), - [true, [['mc'], 0]], // Cache hit - ); - equal( - await db.getUsername(0), - [true, [['pdb'], 'qix']], + [true, [['mc'], 0]] // Cache hit ); + equal(await db.getUsername(0), [true, [['pdb'], 'qix']]); };