Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: getHashedStaticPath, append hashes to URLs for cache busting #431

Closed
wants to merge 110 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
c55e28a
hash initial
gurgunday Feb 1, 2024
491859e
refactor
gurgunday Feb 1, 2024
2ab65a0
use glob
gurgunday Feb 1, 2024
65ed7c6
hashes
gurgunday Feb 1, 2024
d5a67e2
merge main
gurgunday Feb 1, 2024
7bdd289
change msg
gurgunday Feb 1, 2024
1dbf421
remove dynamic unhash
gurgunday Feb 1, 2024
c910b78
fix
gurgunday Feb 1, 2024
26ec2ba
fix
gurgunday Feb 1, 2024
f09d85a
fix
gurgunday Feb 1, 2024
4a3a40e
add tests
gurgunday Feb 1, 2024
5e21320
add tests
gurgunday Feb 1, 2024
9fb0a2a
remove comment
gurgunday Feb 1, 2024
ab1c1ff
fix windows
gurgunday Feb 1, 2024
72f8965
escape more consistently
gurgunday Feb 1, 2024
731a3f3
fix error
gurgunday Feb 1, 2024
f70bbe3
format
gurgunday Feb 1, 2024
a00c931
add type
gurgunday Feb 1, 2024
ce1a788
add README
gurgunday Feb 1, 2024
1ea8ae8
typo
gurgunday Feb 1, 2024
67c3add
use globSync
gurgunday Feb 1, 2024
07250db
tests
gurgunday Feb 1, 2024
403e7f8
tests
gurgunday Feb 1, 2024
3276917
windows
gurgunday Feb 1, 2024
be525c2
posix
gurgunday Feb 1, 2024
64b1073
don't call path.join
gurgunday Feb 1, 2024
f75aed3
use foo
gurgunday Feb 2, 2024
587c5c4
simplify
gurgunday Feb 2, 2024
20b5a26
example
gurgunday Feb 2, 2024
ca3606a
return pathname if hash gen fails
gurgunday Feb 2, 2024
0194e69
use glob
gurgunday Feb 2, 2024
9dff32b
refactor
gurgunday Feb 2, 2024
3bd86ff
really ignore files
gurgunday Feb 2, 2024
5075b62
type
gurgunday Feb 2, 2024
76968b4
make sure getHashedAsset returns a string
gurgunday Feb 2, 2024
e607765
add example
gurgunday Feb 2, 2024
1feca50
Add fastq dependency for task queueing
gurgunday Feb 2, 2024
2f84451
typo
gurgunday Feb 2, 2024
ff342d6
replace p-limit with fastq
gurgunday Feb 2, 2024
88989b7
add back jsdoc
gurgunday Feb 2, 2024
c231e52
remove copilot comments
gurgunday Feb 2, 2024
8594fb7
top lvl await
gurgunday Feb 2, 2024
5fb4046
use availableParallelism
gurgunday Feb 2, 2024
16d436b
typo
gurgunday Feb 2, 2024
fce66eb
pollyfill
gurgunday Feb 2, 2024
6ad51f0
ignore
gurgunday Feb 2, 2024
03b9f78
use async iterator
gurgunday Feb 2, 2024
bc9c061
use async iterator
gurgunday Feb 2, 2024
0fb66bb
start with undefined
gurgunday Feb 2, 2024
c5cf0ba
use Glob
gurgunday Feb 2, 2024
f92aba2
hash
gurgunday Feb 2, 2024
d046eb0
remove istanbul
gurgunday Feb 2, 2024
1b65f15
add another test
gurgunday Feb 2, 2024
c3a21b6
update script
gurgunday Feb 2, 2024
efa2900
extract tmp
gurgunday Feb 2, 2024
e3f7bc3
Refactor generateHashes function signature
gurgunday Feb 2, 2024
e28166e
make sure node_modules is not there and add another test
gurgunday Feb 2, 2024
f257e5c
use os.cpus
gurgunday Feb 2, 2024
9efd8d5
update signature
gurgunday Feb 3, 2024
4e4b2fb
bump
gurgunday Feb 3, 2024
8d8cd42
update readme
gurgunday Feb 3, 2024
a06f496
typo
gurgunday Feb 3, 2024
3a5b9e9
improve readme
gurgunday Feb 3, 2024
923ab52
update readme
gurgunday Feb 3, 2024
0e4f5c9
typo
gurgunday Feb 3, 2024
5b8f8ff
update readme
gurgunday Feb 3, 2024
a794124
update test
gurgunday Feb 3, 2024
9f12cac
move the script to the README
gurgunday Feb 3, 2024
720629a
typo
gurgunday Feb 3, 2024
a11247b
more docs
gurgunday Feb 3, 2024
a6c7bef
use fastify.route
gurgunday Feb 3, 2024
a9d3534
lint
gurgunday Feb 3, 2024
a58b39f
remove dup test
gurgunday Feb 3, 2024
49c761f
use symbol
gurgunday Feb 3, 2024
fea7e75
remove unused error init
gurgunday Feb 3, 2024
083fb93
simplify
gurgunday Feb 3, 2024
9fd9adf
Fix teardown function in static.test.js
gurgunday Feb 3, 2024
41f10f0
disable wildcard by default
gurgunday Feb 4, 2024
33f4930
update README
gurgunday Feb 4, 2024
0ebbfbd
don't generate empty object for no reason
gurgunday Feb 4, 2024
a5785d1
reorder imports
gurgunday Feb 4, 2024
28b6347
reorder imports
gurgunday Feb 4, 2024
965b668
rename var
gurgunday Feb 4, 2024
0085e5b
memory: only store hashes in the map
gurgunday Feb 4, 2024
0e4fbea
make path configurable
gurgunday Feb 4, 2024
ae10ffe
simplify and revert name change
gurgunday Feb 4, 2024
c9393f5
remove unrelated changes
gurgunday Feb 4, 2024
78ed80a
remove unrelated changes
gurgunday Feb 4, 2024
b03ff10
use hashPath instead of location
gurgunday Feb 4, 2024
07b0244
refactor getHashedAsset
gurgunday Feb 4, 2024
7969fbd
add pregenerated example
gurgunday Feb 4, 2024
a5c83ef
use node: imports
gurgunday Feb 5, 2024
942bccc
use node: imports
gurgunday Feb 5, 2024
b504798
Merge branch 'master' into hash
gurgunday Feb 5, 2024
9e3f9b3
use executable scripts
gurgunday Feb 5, 2024
3ab6db6
Merge branch 'hash' of https://github.com/gurgunday/fastify-static in…
gurgunday Feb 5, 2024
3c37ab4
update readme
gurgunday Feb 5, 2024
b4475b7
dd hash-gen
gurgunday Feb 5, 2024
f4274c7
change params
gurgunday Feb 5, 2024
e489afd
typo
gurgunday Feb 5, 2024
ded393d
move to bin
gurgunday Feb 5, 2024
64fd79c
apply suggestions 1
gurgunday Feb 5, 2024
fd94f70
apply suggestions 2
gurgunday Feb 5, 2024
14da551
use named param
gurgunday Feb 5, 2024
8428987
append to the end
gurgunday Feb 5, 2024
5b47060
call it skipPatterns
gurgunday Feb 5, 2024
19f1cf6
Merge branch 'master' of https://github.com/fastify/fastify-static in…
gurgunday Feb 6, 2024
418b9c8
Merge branch 'master' of https://github.com/fastify/fastify-static in…
gurgunday Feb 6, 2024
ccea963
only throw when not file not found error
gurgunday Feb 6, 2024
83a1a77
use ?hash=
gurgunday Feb 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
60 changes: 60 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,66 @@ for getting the file list.
This option cannot be set to `false` with `redirect` set to `true` on a server
with `ignoreTrailingSlash` set to `true`.

#### `hash`

Default: `undefined`

Enabling `hash` will turn off the `wildcard` option if it wasn't explicitly set to `true`, and both cannot be set to `true`. When enabled, `hash` lets the user access assets dynamically using the decorated `getHashedStaticPath` function. This in turn makes possible the usage of a very high `maxAge` so that the content can be cached as long as possible. If any modifications are made to a file, its hash will simply be recalculated during the next startup and the cache will bust for that asset.

##### Example:

```js
const fastify = require('fastify')()
fastify.register(require('fastify-static'), {
root: path.join(__dirname, 'public'),
hash: true,
immutable: true,
maxAge: 31536000 // You can set a very long maxAge
})

fastify.listen(3000, (err) => {
if (err) throw err
console.log('Server listening at http://localhost:3000')
})
```

Then in your templates:

```handlebars
<script src="{{ fastify.getHashedStaticPath('js/main.js') }}"></script>
<link href="{{ fastify.getHashedStaticPath('css/main.css') }}" rel="stylesheet">
```

**Note:** Hashes can be generated in advance to speed up cold start times, ideally during the build phase:

```sh
npx fastify-static-prehash <root-paths> <write-location> <include-dot-files> <ignore-patterns>
```

##### Example:

```sh
npx fastify-static-prehash example/public/,example/public2/ example/hashes.json true '**/*.json'
```

And then, during plugin initialization:

```js
const fastify = require('fastify')()
fastify.register(require('fastify-static'), {
root: path.join(__dirname, 'public'),
hash: { path: path.join(__dirname, 'example/hashes.json') },
serveDotFiles: true,
immutable: true,
maxAge: 31536000
})

fastify.listen(3000, (err) => {
if (err) throw err
console.log('Server listening at http://localhost:3000')
})
```

#### `allowedPath`

Default: `(pathName, root, request) => true`
Expand Down
35 changes: 35 additions & 0 deletions bin/fastify-static-prehash.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/usr/bin/env node

'use strict'

const { generateHashes } = require('../lib/hash')
const path = require('path')

async function run () {
let [rootPaths, writeLocation, includeDotFiles, ignorePatterns] = process.argv.slice(2)

if (rootPaths === undefined || writeLocation === undefined) {
console.error('Usage: hash <root-paths> <write-location> <include-dot-files> <ignore-patterns>')
process.exit(1)
}

rootPaths = rootPaths.split(',').map(p => path.resolve(p.trim()))
includeDotFiles = includeDotFiles === 'true'
ignorePatterns = ignorePatterns ? ignorePatterns.split(',') : []

try {
await generateHashes({
rootPaths,
includeDotFiles,
skipPatterns: ignorePatterns,
writeToFile: true,
outputPath: writeLocation
}
)
console.log('Hashes generated successfully.')
} catch (error) {
console.error('Error generating hashes:', error)
}
}

run()
8 changes: 8 additions & 0 deletions example/hashes.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"index.js": "1f49271e60e53b00",
"index.html": "8e27d2f424d541b4",
"index.css": "712708f47c759d37",
"images/sample.jpg": "4c67e0488b3afa81",
"test.html": "6944c8b16078a588",
"test.css": "b305efc33fe211f7"
}
33 changes: 33 additions & 0 deletions example/server-hash-pregenerated.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import fastify from 'fastify'
import fastifyStatic from '../index.js'

const server = fastify()

await server.register(fastifyStatic, {
root: new URL('./public', import.meta.url).pathname,
prefix: '/assets/',
hash: { path: new URL('hashes.json', import.meta.url).pathname },
immutable: true,
maxAge: 31536000 * 1000 // 1 year
})

server.register(import('fastify-html'))

// Define a route for the root URL
server.get('/', async (request, reply) => {
return reply.html`
<html>
<head>
<link rel="stylesheet" href="${server.getHashedStaticPath('index.css')}">
</head>
<body>
<h1>Hello, world!</h1>
<img src="${server.getHashedStaticPath('images/sample.jpg')}" alt="An image">
</body>
</html>
`
})

// Start the server
await server.listen({ port: 3000 })
console.log('Server is running on port 3000')
33 changes: 33 additions & 0 deletions example/server-hash.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import fastify from 'fastify'
import fastifyStatic from '../index.js'

const server = fastify()

await server.register(fastifyStatic, {
root: new URL('./public', import.meta.url).pathname,
prefix: '/assets/',
hash: true,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to be annoying about names, this is take-it-or-leave-it feedback, but just reading this code without reading the readme, I feel like the option hash: true could mean a variety of different things. I'd probably guess it means "
"add an e-tag header", not that it allows fancy path building for CDN busting hashes.

immutable: true,
maxAge: 31536000 * 1000 // 1 year
})

server.register(import('fastify-html'))

// Define a route for the root URL
server.get('/', async (request, reply) => {
return reply.html`
<html>
<head>
<link rel="stylesheet" href="${server.getHashedStaticPath('index.css')}">
</head>
<body>
<h1>Hello, world!</h1>
<img src="${server.getHashedStaticPath('images/sample.jpg')}" alt="An image">
</body>
</html>
`
})

// Start the server
await server.listen({ port: 3000 })
console.log('Server is running on port 3000')
80 changes: 59 additions & 21 deletions index.js
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
'use strict'

const { readFile } = require('node:fs/promises')
const { PassThrough } = require('node:stream')
const path = require('node:path')
const { fileURLToPath } = require('node:url')
const { statSync } = require('node:fs')
const { glob } = require('glob')
const { Glob } = require('glob')
const fp = require('fastify-plugin')
const send = require('@fastify/send')
const encodingNegotiator = require('@fastify/accept-negotiator')
const contentDisposition = require('content-disposition')

const dirList = require('./lib/dirList')
const { generateHashes } = require('./lib/hash')

const endForwardSlashRegex = /\/$/u
const doubleForwardSlashRegex = /\/\//gu
const asteriskRegex = /\*/gu
const endForwardSlashRegex = /\/$/u

const kFileHashes = Symbol('fileHashes')

const supportedEncodings = ['br', 'gzip', 'deflate']
send.mime.default_type = 'application/octet-stream'
Expand Down Expand Up @@ -59,6 +62,27 @@ async function fastifyStatic (fastify, opts) {
: prefix + '/'
}

if (opts.hash) {
if (opts.wildcard) {
throw new Error('"wildcard" has to be disabled to use "hash"')
}

opts.wildcard = false

fastify.decorate('getHashedStaticPath', getHashedStaticPath)
if (opts.hash.path) {
const hashesContent = await readFile(opts.hash.path, 'utf8')
fastify.decorate(kFileHashes, new Map(Object.entries(JSON.parse(hashesContent))))
} else {
fastify.decorate(kFileHashes, await generateHashes({
rootPaths: opts.root,
includeDotFiles: opts.serveDotFiles,
skipPatterns: opts.hash.skipPatterns
}
))
}
}

// Set the schema hide property if defined in opts or true by default
const routeOpts = {
constraints: opts.constraints,
Expand Down Expand Up @@ -112,11 +136,14 @@ async function fastifyStatic (fastify, opts) {
throw new Error('"wildcard" option must be a boolean')
}
if (opts.wildcard === undefined || opts.wildcard === true) {
fastify.head(prefix + '*', routeOpts, function (req, reply) {
pumpSendToReply(req, reply, '/' + req.params['*'], sendOptions.root)
})
fastify.get(prefix + '*', routeOpts, function (req, reply) {
pumpSendToReply(req, reply, '/' + req.params['*'], sendOptions.root)
fastify.route({
...routeOpts,
exposeHeadRoute: true,
method: 'GET',
url: `${prefix}*`,
handler (req, reply) {
pumpSendToReply(req, reply, `/${req.params['*']}`, sendOptions.root)
}
})
if (opts.redirect === true && prefix !== opts.prefix) {
fastify.get(opts.prefix, routeOpts, function (req, reply) {
Expand All @@ -127,18 +154,21 @@ async function fastifyStatic (fastify, opts) {
const indexes = opts.index === undefined ? ['index.html'] : [].concat(opts.index)
const indexDirs = new Map()
const routes = new Set()
const globPattern = '**/**'

const roots = Array.isArray(sendOptions.root) ? sendOptions.root : [sendOptions.root]
for (let i = 0; i < roots.length; ++i) {
const rootPath = roots[i]
const posixRootPath = rootPath.split(path.win32.sep).join(path.posix.sep)
const files = await glob(`${posixRootPath}/${globPattern}`, { follow: true, nodir: true, dot: opts.serveDotFiles })
for (let rootPath of roots) {
rootPath = rootPath.split(path.win32.sep).join(path.posix.sep)
if (!rootPath.endsWith('/')) {
rootPath += '/'
}

const filesIterable = new Glob('**/**', {
cwd: rootPath, absolute: false, follow: true, nodir: true, dot: opts.serveDotFiles
})

for (let i = 0; i < files.length; ++i) {
const file = files[i].split(path.win32.sep).join(path.posix.sep)
.replace(`${posixRootPath}/`, '')
const route = (prefix + file).replace(doubleForwardSlashRegex, '/')
for (let file of filesIterable) {
file = file.split(path.win32.sep).join(path.posix.sep)
const route = prefix + file

if (routes.has(route)) {
continue
Expand Down Expand Up @@ -168,14 +198,13 @@ async function fastifyStatic (fastify, opts) {
}

const allowedPath = opts.allowedPath

function pumpSendToReply (
request,
reply,
pathname,
rootPath,
rootPathOffset = 0,
pumpOptions = {},
pumpOptions,
checkedEncodings
) {
const options = Object.assign({}, sendOptions, pumpOptions)
Expand Down Expand Up @@ -237,7 +266,7 @@ async function fastifyStatic (fastify, opts) {

wrap.getHeader = reply.getHeader.bind(reply)
wrap.setHeader = reply.header.bind(reply)
wrap.removeHeader = () => {}
wrap.removeHeader = noop
wrap.finished = false

Object.defineProperty(wrap, 'filename', {
Expand Down Expand Up @@ -381,10 +410,12 @@ async function fastifyStatic (fastify, opts) {

function setUpHeadAndGet (routeOpts, route, file, rootPath) {
const toSetUp = Object.assign({}, routeOpts, {
method: ['HEAD', 'GET'],
exposeHeadRoute: true,
method: 'GET',
url: route,
handler: serveFileHandler
})

toSetUp.config = toSetUp.config || {}
toSetUp.config.file = file
toSetUp.config.rootPath = rootPath
Expand All @@ -397,6 +428,11 @@ async function fastifyStatic (fastify, opts) {
const routeConfig = req.routeOptions?.config || req.routeConfig
pumpSendToReply(req, reply, routeConfig.file, routeConfig.rootPath)
}

function getHashedStaticPath (unhashedRelativePath) {
const hash = fastify[kFileHashes].get(unhashedRelativePath)
return `${prefix}${unhashedRelativePath}${hash ? `?hash=${hash}` : ''}`
}
}

function normalizeRoot (root) {
Expand Down Expand Up @@ -549,6 +585,8 @@ function getRedirectUrl (url) {
}
}

function noop () {}

module.exports = fp(fastifyStatic, {
fastify: '4.x',
name: '@fastify/static'
Expand Down