Doc replacements are a one-time thing

[Rich-Harris]

IIUC our sync-docs stuff is broken — a directive like > TYPES: svelte will be replaced with the content, which is then written back to disk. The next time the script runs, there is no > TYPES: svelte, so nothing ever subsequently updates.

If we really must have authored and generated content in the same file — and I maintain that we'd probably be happier in the long term if we didn't! — then at the very least we'll need to preserve these directives. Or rather, we'll need start and end boundaries, so something like this:

> START TYPES: svelte
> (inserted content here)
> END TYPES: svelte

[Rich-Harris]

Just realised that the idea we proposed for making it possible to mix and match authored and generated content — checksums for generated content — doesn't really work unless a) the entire file is generated or b) the checksums apply to individual replacements.

Really think we'd be so much better off with runtime includes.

[dummdidumm]

Your understanding is incorrect. The next time the sync script runs, it will use the source again, which will still contain the > TYPES directive as before.

Runtime includes have a dozen of drawbacks, and I listed them out and I thought we collectively agreed. Yes there may be a tiny few drawbacks, but they're nowhere near as bad as going back to runtime includes. I'm firmly against going down that route.

Edit: I glanced at the PR, and the runtime includes are not as bad as I thought they are, since the result is at least written to disk. Still I'd like to properly discuss this on monday.

[Rich-Harris]

Yeah I wasn't going to merge it without discussion. My confusion stemmed from the fact that some of the contents of content are authored in this repo while other stuff comes from other repos (I got muddled up and thought it was only the reference stuff that was coming from outside, I think I was thrown by the 50-reference vs 98-reference thing), and the fact that we're rewriting files inside content/docs (I missed the cpSync above it)

[Rich-Harris]

(Oh, and the fact that I couldn't get the damn file to update seemingly no matter what I did, which is sort of the root of the issue — the way we do things doesn't really leave sufficient breadcrumbs to reverse engineer the process, at least not for a simple soul such as myself)

[dummdidumm]

If this was related to code snippets not changing, you may have run into a code snippet caching problem which is separate.

[Rich-Harris]

Yeah. I wonder if we should do the shiki-ification during syncing, so that we don't need the cache. It would make syncing a lot slower of course, but it would make deployments significantly faster

[Rich-Harris]

Alternatively we could do both — generate the snippets during sync AND keep the cache, but make it explicit when you run the script so that if things don't budge it's obvious why

[dummdidumm]

One drawback of doing too much ahead of time and commuting it to disk is that it would get harder to differentiate between the actual content and all the dom stuff that is generated around it. It would especially annoying for translators.
AFAIK deployments today are also fast, because Vercel caches node modules between builds and since the snippet cache is in there it's fast.

[Rich-Harris]

@trueadm if you're looking for a decent first issue with which to get your hands dirty in this codebase, this could be a good one as it touches a few different moving parts.

Right now, when we run USE_GIT=true pnpm sync-docs inside apps/svelte.dev (you only need USE_GIT the first time, to fetch the repos and put them inside the project folder), we copy over the documentation in the svelte and kit repos and replace any > TYPES: blah placeholders. Any code blocks, including those that replace > TYPES: blah etc, are left as-is.

Later, when we render the page, we use Shiki Twoslash to highlight the code and also add these tooltips:

This is an expensive process, and as such the result is cached inside node_modules/.snippets. But this is flawed — the cache key is based on the source/language/name of the type being transformed, not the code that transforms it, so breakage is not infrequent. And because Vercel caches node_modules, stale snippets will sometimes get deployed (conversely, if you deploy without a cache, it takes a long time to generate everything).

What I therefore propose:

1. We upgrade to modern Twoslash, as it is actively maintained unlike the very old version of shiki-twoslash we're currently using
2. We incorporate the snippet generation into the sync-docs phase — i.e. instead of writing to node_modules/.snippets, we write them to a snippets directory which is checked in, and later used to render pages
3. Stretch goal: we continue using a cache so that sync-docs is fast, but we incorporate the mtime of the most recently modified module that contributes to the transformation

A semi-related issue is #180, since it would be nice to unify the markdown process between the docs and the tutorial, and use the +++ mechanism used to highlight stuff in the tutorial...

...instead of the clunky old diff blocks we currently use. But that might be awkward since once transformation involves Twoslash and the other doesn't.