Why are AST transforms required for the sdk-js in sveltekit?

[samuelstroschein]

@benjaminpreiss @ivanhofer Why are AST transforms required in the sdk-js?

I found #615 that introduced AST transforms. However, I can't find the reasoning.

[benjaminpreiss]

Hey, thanks for the Q.

We need to transform some source code in the vite plugin. One might think this could just be done with regexes, but it's much more reliable to parse the JS code as an AST to target specific elements when transforming.

Does that answer your question?

[samuelstroschein]

We need to transform some source code in the vite plugin.

Why does source code need to be transformed?

One might think this could just be done with regexes

No no. Been there done that. Everyone who rarely works with ASTs runs into this issue, myself included :D

[benjaminpreiss]

We enable the developer to use the inlang sdk by offering import {i, ...} from '@inlang/sdk-js'.
This is just a front for type safety, however. In order to make the code from the developers work, we have to replace the code we receive in the plugin into something else.

E.g.

import {i, ...} from '@inlang/sdk-js'

console.log(i('language'))

Turns into

import {getRuntimeFromContext} from '@inlang/something'
import {get} from 'svelte/store'

let {i} = getRuntimeFromContext()
console.log(get(i('language')))

This is just a very simple example for the transforms we have to make - does this help?

[samuelstroschein]

It does help but I am wondering. Couldn't you achieve the desired effect of swapping runtime-specific code with dynamic imports without running into transform-related issues like #687, #792 and high effort of building the transforms?

let serverRuntime;

export function i() {
  if (runningOnServer) {
    if (serverRuntime === undefined){ 
      serverRuntime = import("@inlang/sdk-js/server") 
    }
    return serverRuntime
  } else {
  // return client etc.
  }

}

[ivanhofer]

Then you would need to use it like this:

<h1>{await i('welcome')}</h1>

And it would probably introduce more problems than it solves, because people will always use the function in a wrong way ^^
This alone should be reason enough to not do it.

But the real reason is:

the whole easy to use solution relies on the AST transformation we make. There are 10+ different ways how i('welcome') can be used (different files, different contexts). A user would need to set this up himself and needs to know where he needs to use which import. At that point we wouln't need yet another i18n library and everyone should just use typesafe-i18n.

[samuelstroschein]

Just crossed my mind: Vite tree-shakes unused imports/we can make vite tree shake and therefore don't need dynamic imports/async functions.

It seems to me that ease of use of the i function can be achieved during "runtime" with conditionals. Let users use i and figure out under the hood what to run/import. The outcome seems to be identical to the AST transforms but more bulletproof and maintainable. AST transform bugs keep coming in (#849, #843), and will continue to come in. Adjusting code through transforms must work 100% correctly with every update of SvelteKit. Ensuring 100% working AST transforms for every tiny update seems like a losing battle which will lead to constant bugs and therefore frustrate users.

Here is pseudocode of a minimal footprint runtime solution:

import { serverIFunction } from "./server.js"
import { layoutIFunction } from "./server.js"


export function i() {
  // if runningOnServer is tree shaken 
  if (runningOnServer) {
    // serverIFunction will not be bundled on the client 
    return serverRuntime
  } else if (runningOnClient) {
    return clientIFunction
  } else {
    return layoutIFunction
    }
}

Conditional logic during runtime (with minimal footprint due to tree shaking) seems easier than trying to transform people's code through AST transforms, leading to a better sdk-js. Besides, we will be able to ship faster. Tree-shaking can be optimized over time. The number 1 goal should be shipping a stable sdk-js fast, regardless of how optimized the solution is. AST transforms must work 100%, there is no incremental optimization possibility for AST transforms.

Summary

• Can AST transforms be "bug-free" given that code can have any shape (endless edge cases) and every update of svelte-kit potentially crashes AST transforms?
• Are AST transforms required/worth it to achieve ease of use?

@ivanhofer Are you certain that AST transforms are required?

[ivanhofer]

All of the bugs that were opened before, occurred due to the fact that we haven't had time to implement everything. We have realeased a version for the Svelte Summit that works with code written in the style of the SvelteKit documentation. I already knew that those bug will occur and we are solving them with the refactoring.

@ivanhofer Are you certain that AST transforms are required?

If you know how we can map a request coming from hooks.server.js to a request passing load in +layout.server.js then we don't need AST transforms. Doing stuff on the server and it's shared context is more complex then just running a single js file on the client. Why do you think that nobody has figured out i18n if it was "just a dumb" if-else statement?

[samuelstroschein]

Why do you think that nobody has figured out i18n if it was "just a dumb" if-else statement?

Because nobody is using a compiler-based approach via a build step (vite).

For context: I interpret, and I might not get it, the AST transforms as "minimal runtime footprint optimization". If that is the case, using the build step (that no other i18n lib uses), could lead to the same outcome with less effort.

If you know how we can map a request coming from hooks.server.js to a request passing load in +layout.server.js then we don't need AST transforms.

Can you elaborate what end-user functionality is enabled by mapping requests from hooks.server.js to other files?

I don't seem to understand what AST transforms enable [which can't be achieved otherwise]. My interest is two-fold. First, understand the architectural decisions of the SDK, and second, maybe help improve the architecture (or not, once I grasp what AST transforms enable!).

[ivanhofer]

For context: I interpret, and I might not get it, the AST transforms as "minimal runtime footprint optimization".

This will be one of the benefits in the future. We are not at this point right now. For now the vite approach is here so develpers don't have to touch 5+ files and add 70+ LOC to configure an i18n library.

Can you elaborate what end-user functionality is enabled by mapping requests from hooks.server.js to other files?

Take a look at all the files in https://github.com/inlang/inlang/tree/main/source-code/sdk-js/src/adapter-sveltekit/examples/lang-in-slug/src. This would be the most developer-friendly way of achieving the i18n configuration without a compiler.

[benjaminpreiss]

Dear all,

I'd like to give my thoughts to this discussion, by going through all the necessary steps outlined in the Refactoring master plan and sharing my evaluation regarding feasibility.

In general, we transform:

• regular JS files / regular svelte files
• special JS files (e.g. layout.server.js, where we need to wrap the load function)
• special svelte files (like +layout.svelte)

Regular JS files / regular svelte files

In these files, we run transformJs(), which means:

• replaceSDKImports(): Replace the import from the SDK with some lines and replace all occurrences of the imports with placeholders (e.g. i → $$_INLANG_I_$$) (Virtual imports ✅, bulletproof AST transform ❌)
• replacePlaceholders(): Replace the introduced placeholders (like $$_INLANG_I_$$ → i or $i or get(i)) (Virtual imports ❌, bulletproof AST transform ✅)

replacePlaceholders can't be done without AST transforms, as we need to prefix some occurrences in the MIDST of the source code with a $ sign to get reactivity (example)

replaceSDKImports is not bulletproof, as identifying the occurrence of an inlang SDK export for replacement with a placeholder isn't always easy:

import {i} from "@inlang/sdk-js"

console.log(i('hello'))

[1, 2, 3, 4].forEach((v, i) => {
     console.log(i)
})

Here, we would need to write an AST transform that understands that the second console.log actually doesn't reference an inlang SDK module and therefore does not replace i here.

For regular svelte files, the functionality is similar: transformSvelte()

Special JS files

Here, we need to wrap some function declarations or create them from scratch and rewrite some parameters (which is the tricky part). export const load: LayoutLoad = ({ locals }) => {} becomes export const load = initRootLayoutServerLoadWrapper().wrap(({ locals }) => {}

• wrapAndAddParameters(): Wrap certain function definitions and add parameters (Virtual imports ❌, bulletproof AST transform ❌)

The complexity of changing the parameters of a certain function lies in identifying where to rewrite these parameters:

function fn2(someparameters) {
    console.log(i('tricky'))
}
const fn1 = fn2
export const load = fn1

should become

function fn2(someparameters, {i}) {
    console.log(i('tricky'))
}
const fn1 = fn2
export const load = inlangWrapper(fn1)

Here, we need to rewrite the function parameters of fn3, as our wrapper provides i as the second parameter and because it is actually the original definition of load. But things can quickly become untraceable if the declaration chain is not a direct assignment:

import {fn3} from 'somewhere'

function fn2(someparameters) {
    console.log(i('tricky'))
}
const fn1 = fn3(fn2)
export const load = fn1

Here we can not trace back what fn3 actually does and whether it will pass on {i} from the inlang wrapper to fn2. If we can find another way to implement what we need to do here, it is STRONGLY ADVISABLE.

That said, I am confident that we can cover most of the cases quickly / have them covered.

Special svelte files

Here we just need to replace the SDK imports with some lines (all at the top of the script tag), wrap some svelte markup

• Add some imports and lines at the top of the file (Virtual imports ✅, bulletproof AST transform ✅)
• Wrap and insert in markup (Virtual imports ✅, bulletproof AST transform ✅)

... and then run the regular svelte file transforms.

Summary

• Maybe there can be another way of implementing the wrap functionality in e.g. +layout.server.js. We already know that an alternative to adding {i} to the load function definition can be retrieving i from the locals by calling getRuntimeFromLocals(event.locals).i(...). But then we still need to provide event in the function parameters:

"import { initHandleWrapper } from "@inlang/sdk-js/adapter-sveltekit/server";
import { sequence } from "@sveltejs/kit/hooks";
import { i } from "@inlang/sdk-js";

const seq1 = async ({ event, resolve }) => {
console.log(getRuntimeFromLocals(event.locals).i("welcome"));
    return resolve(event);
};

export const handle = initHandleWrapper({
    inlangConfigModule: import("../inlang.config.js"),
    getLanguage: () => undefined
}).wrap(sequence(seq1));"

How can we improve this so that we don't need to change function parameters?

• Replacing i with a placeholder $$_INLANG_I_$$ has its risks, as we cannot always be sure that there isn't another definition of i. Can we instead directly export a piece of code from the SDK that provides us with reactivity for i without needing to wrap it where it is used?. E.g. including some logic in the function definition of i that calls get() on the underlying store whenever i() is called?

Final words

Regardless of how we decide here, I am very confident that we can get the AST transforms to be very stable - we are actually pretty close to covering most cases.

[samuelstroschein]

Thank you for the elaborate reply. I still don't understand what end-user benefits AST transforms have. I asked my question in another thread #837 (reply in thread). If you know the answer, feel free to reply there.

[benjaminpreiss]

The answer is simple - Certain things are not possible without AST transforms (targeted replacements of code pieces). So the more important question is: Do we need these targeted replacements? And it turns out that we currently need these (@ivanhofer, correct me if I am wrong).

For you understanding, what are targeted replacements - look at this svelte file:

<script>
import {i} from '@inlang/sdk-js'
console.log("blue")
console.log(i('welcome'))
</script>
<div>{i('welcome2')}</div>

If we need to prefix i with a $, then we need AST transforms to identify the exact locations on where we need to insert the $ sign.