feat: move static pages out of function bundle #728
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
type: feature
lindsaylevine
approved these changes
Oct 21, 2021
|
|
||
| const pagesManifest = await readJson(pagesManifestPath) | ||
| // Arbitrary limit of 10 concurrent file moves | ||
| const limit = pLimit(10) |
There was a problem hiding this comment.
We want to avoid having too many files open errors if there are loads of them, but equally we want to avoid blocking on a single large file. A better number might be 1, or 2, or the number of cpus, or soemhting else. In lieu of actually testing (yet), I picked an arbitrary number.
|
|
||
| return readfileOrig(file, options) | ||
| } | ||
| } |
There was a problem hiding this comment.
the comments are so great!! 🙏 this is genius
There was a problem hiding this comment.
I'm not sure if it's great or terrible, tbh
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because we're using Next's routing, we're serving all pages via the function. This includes static pages. This has a couple of downsides. First, some users have large numbers of pre-rendered pages, which can lead to large zip sizes. Second, it means we're not making the best use of the CDN.
This PR moves static page files out of the
server/pagesdirectory so they're no longer bundled with the function, and instead puts them directly into the publish directory so they're served by the CDN instead, and with redirect shadowing these are served instesd of calling the function. This is mostly fine, but unfortunately there are still some situations where Next will try to load the files. This is mostly where there are rewrites that point at these static pages. In this situations, if there is no file then the user will incorrectly see a 404 rather than the page from the CDN. This PR adds an extra workaround for this. There is only one place in the Next codebase where these files are loaded, which is in IncrementalCache, wherefs.promises.readFIleis used. By monkey-patchingpromiseswith a wrapper forreadFile, we can check if Next is tryign to read one of these files. We know which files they are, as we've written out a manifest of all the files we moved. If it is one of these files then we load the file from the CDN, cache it in the lambda, then return the cached file instead. Next sees it as having loaded the file directly, and is none the wiser.I hate monkey-patching, but this should be a relatively uncommon code path. I've put the whole thing behind a flag
EXPERIMENTAL_MOVE_STATIC_PAGESfor now, and can let users who have lots of prerendered pages try it. With some instrumentation, tests and maybe improved implementation this might be worht makign the default.