From 9c443f5e155fe6a9d11d0a432b4585c27a7c3274 Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Fri, 19 Sep 2025 19:14:20 -0500 Subject: [PATCH 1/7] New compiler docs --- docs/website/assets/sum-1-to-100.svg | 1 + docs/website/how-it-works.md | 57 ++++++++----- docs/website/new-compiler.md | 123 +++++++++++++++++++++++++++ sidebars.js | 1 + 4 files changed, 162 insertions(+), 20 deletions(-) create mode 100644 docs/website/assets/sum-1-to-100.svg create mode 100644 docs/website/new-compiler.md diff --git a/docs/website/assets/sum-1-to-100.svg b/docs/website/assets/sum-1-to-100.svg new file mode 100644 index 00000000..d751744d --- /dev/null +++ b/docs/website/assets/sum-1-to-100.svg @@ -0,0 +1 @@ +definesum1to100setsumto0setito1repeat100changesumbyichangeiby1 \ No newline at end of file diff --git a/docs/website/how-it-works.md b/docs/website/how-it-works.md index 164f18f6..1d80fefa 100644 --- a/docs/website/how-it-works.md +++ b/docs/website/how-it-works.md @@ -7,29 +7,46 @@ hide_table_of_contents: true TurboWarp uses a *compiler* while Scratch uses an *interpreter*. This allows TurboWarp to run somewhere between 10-100x faster depending on the project, but it makes live script editing [impracticable](#live-script-editing). -export const Test = ({name, id, scratch, tw}) => ( - - {name} - {scratch} - {tw} - -); - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + +
TestScratchTurboWarp
TestScratchTurboWarp
+ +
Sort 200000 random items
+
Lower is better
+ 		8.871 seconds0.0451 seconds
+ +
res=1 samp=10 dof=.08
+
Lower is better
+ 		814 seconds15 seconds
+ +
Hashes per second
+
Higher is better
+ 		146 per second3010 per second
-(Tested in Chromium 103 on Linux) +(Tested using Chromium 140, Arch Linux, i7 4790k) Consider the following script: diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md new file mode 100644 index 00000000..4b37ff3d --- /dev/null +++ b/docs/website/new-compiler.md @@ -0,0 +1,123 @@ +--- +slug: /new-compiler +hide_table_of_contents: true +--- + +# New Compiler + +The compiler is the part of TurboWarp that converts projects to JavaScript. On September 19, 2025, we released a new compiler that is better at analyzing the scripts inside projects to generate even faster JavaScript. + +This is the largest single change we've ever made. We've tried to test it thoroughly, but **it's quite likely there are still a few more bugs**. You should report broken projects at [the usual place](https://scratch.mit.edu/users/GarboMuffin/#comments). If your project is broken, you can use https://experiments.turbowarp.org/old-compiler/ instead until we fix the bug in the new compiler. + +:::warning +The new compiler breaks a small handful of custom extensions. See the [extensions](#extensions) section below for details and workarounds. +::: + +The new compiler is currently only available on the website. The packager, desktop app, and other experiment branches will be updated later. + +## Performance comparison {#performance} + +The performance gain varies a lot depending on the project. Some projects run twice as fast, while others are unchanged, but there should be no projects that run slower. + + + + + + + + + + + + + + + + + + + + + + + + + + +
TestOld CompilerNew Compiler
+ +
Time until shell
+
Lower is better
+ 		21 seconds12 seconds
+ +
Hashes per second
+
Higher is better
+ 		2711 per second3010 per second
+ +
Sort 200000 random items
+
Lower is better
+ 		0.0515 seconds0.0451 seconds
+ +(Chromium 140, Arch Linux, i7 4790k, warp timer disabled) + +## Limitations {#limitations} + +### Warp timer {#warp-timer} + +The [warp timer](warp-timer) forces blocks marked as "run without screen refresh" to briefly pause after running for 500 ms to prevent an infinite loop from causing you to lose unsaved work. This is why the warp timer is automatically enabled when you open the TurboWarp editor. + +Unfortunately, the warp timer breaks many of the assumptions that the new compiler relies on to optimize projects, so do not expect major performance improvements while you are in the editor or otherwise have the warp timer enabled. They won't run slower than they did in the old compiler, though. + +### Extension compatibility {#extensions} + +All extensions included in the TurboWarp extension list will work the same, and a vast majority of custom extensions will also continue to work the same. + +A small handful of custom extensions use an API called `i_will_not_ask_for_help_when_these_break` to integrate more directly with the compiler. We gave this API that crazy name because we knew it was going to break at some point, and we didn't want a small handful of extensions to restrict us from being able to change the compiler's internals when needed. If your project requires these extensions, you can use https://experiments.turbowarp.org/old-compiler/ instead until extensions become compatible. + +## Brief technical overview {#technical-overview} + +The broad idea is that the new compiler tracks what type a variable has at all points, including through conditionals and loops. This allows it to remove unnecessary type conversions and generate more specialized code. + +Consider this script that sums integers from 1 to 100. Suppose that the block is marked as "run without screen refresh" and warp timer is disabled. + +![](./assets/sum-1-to-100.svg) + +The old compiler generates: + +```js +b0.value = 0; +b1.value = 1; +for (var a0 = 100; a0 >= 0.5; a0--) { + b0.value = ((+b0.value || 0) + (+b1.value || 0)); + b1.value = ((+b1.value || 0) + 1); +} +``` + +`(+b0.value || 0)` is how the compiler converts values to numbers. In this case it is redundant as the variables are already always numbers, but the only compiler did not realize this. + +The new compiler can reason about the loops and does realize that the variables are always numbers, so it generates: + +```js +b0.value = 0; +b1.value = 1; +for (var a0 = 100; a0 >= 0.5; a0--) { + b0.value = (b0.value + b1.value); + b1.value = (b1.value + 1); +} +``` + +Much simpler and would run a bit faster. + +## Credits {#credits} + +[Tacodiva](https://scratch.mit.edu/users/Tacodiva7729/) did largely all the hard parts and persevered through an impossibly long review process. + +Early testers helped us find and fix many bugs before release: + + * Krypto + * Vadik1 + * SpinningCube + * Dominic + * scratchfakemon + * lspectroniztar + * termy diff --git a/sidebars.js b/sidebars.js index 0aa21d2e..169e9235 100644 --- a/sidebars.js +++ b/sidebars.js @@ -20,6 +20,7 @@ module.exports = { 'website/settings/disable-compiler' ] }, + 'website/new-compiler', 'website/embedding', 'website/how-it-works', 'website/javascript', From 2079f8bf332d4b3062c7f05ae195b74f0e422fd2 Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Fri, 19 Sep 2025 19:18:33 -0500 Subject: [PATCH 2/7] add the links i could find --- docs/website/new-compiler.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md index 4b37ff3d..88802b96 100644 --- a/docs/website/new-compiler.md +++ b/docs/website/new-compiler.md @@ -110,14 +110,14 @@ Much simpler and would run a bit faster. ## Credits {#credits} -[Tacodiva](https://scratch.mit.edu/users/Tacodiva7729/) did largely all the hard parts and persevered through an impossibly long review process. +[Tacodiva](https://scratch.mit.edu/users/Tacodiva7729/) did almost all the work and persevered through an impossibly long review process. Early testers helped us find and fix many bugs before release: - * Krypto - * Vadik1 - * SpinningCube - * Dominic + * [Krypto](https://scratch.mit.edu/users/KryptoScratcher/) + * [Vadik1](https://scratch.mit.edu/users/Vadik1/) + * [SpinningCube](https://scratch.mit.edu/users/SpinningCube/) + * [GamingWithDominic](https://scratch.mit.edu/users/GamingWithDominic/) * scratchfakemon - * lspectroniztar + * [LSPECTRONIZTAR](https://scratch.mit.edu/users/LSPECTRONIZTAR/) * termy From fbf32347e38c0e18b187091ca82e1e8d4b00c79c Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Fri, 19 Sep 2025 19:24:34 -0500 Subject: [PATCH 3/7] clarify "tested using" --- docs/website/new-compiler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md index 88802b96..fefac5c7 100644 --- a/docs/website/new-compiler.md +++ b/docs/website/new-compiler.md @@ -58,7 +58,7 @@ The performance gain varies a lot depending on the project. Some projects run tw -(Chromium 140, Arch Linux, i7 4790k, warp timer disabled) +(Tested using Chromium 140, Arch Linux, i7 4790k, warp timer disabled) ## Limitations {#limitations} From 3a402eeb440c446978e87e599c1038dfd8c3ae20 Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Fri, 19 Sep 2025 19:42:59 -0500 Subject: [PATCH 4/7] revise --- docs/website/new-compiler.md | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md index fefac5c7..b7cc3018 100644 --- a/docs/website/new-compiler.md +++ b/docs/website/new-compiler.md @@ -76,37 +76,41 @@ A small handful of custom extensions use an API called `i_will_not_ask_for_help_ ## Brief technical overview {#technical-overview} -The broad idea is that the new compiler tracks what type a variable has at all points, including through conditionals and loops. This allows it to remove unnecessary type conversions and generate more specialized code. +The broad idea is that the compiler analyzes scripts to determine what kinds of values each variable may have at each point in the script. This allows it to remove unnecessary type conversions and generate more specialized code. The old compiler tried to do this too, but was unable to reason about loops or conditionals. Consider this script that sums integers from 1 to 100. Suppose that the block is marked as "run without screen refresh" and warp timer is disabled. -![](./assets/sum-1-to-100.svg) +![Script that sums 1 to 100](./assets/sum-1-to-100.svg) The old compiler generates: ```js -b0.value = 0; -b1.value = 1; +sum.value = 0; +i.value = 1; for (var a0 = 100; a0 >= 0.5; a0--) { - b0.value = ((+b0.value || 0) + (+b1.value || 0)); - b1.value = ((+b1.value || 0) + 1); + sum.value = ((+sum.value || 0) + (+i.value || 0)); + i.value = ((+i.value || 0) + 1); } ``` -`(+b0.value || 0)` is how the compiler converts values to numbers. In this case it is redundant as the variables are already always numbers, but the only compiler did not realize this. +`(+something.value || 0)` is how the compiler converts a variable to a number. In this case it is redundant as the variables are already always numbers, but the old compiler did not realize this. -The new compiler can reason about the loops and does realize that the variables are always numbers, so it generates: +The new compiler instead generates: ```js -b0.value = 0; -b1.value = 1; +sum.value = 0; +i.value = 1; for (var a0 = 100; a0 >= 0.5; a0--) { - b0.value = (b0.value + b1.value); - b1.value = (b1.value + 1); + sum.value = (sum.value + i.value); + i.value = (i.value + 1); } ``` -Much simpler and would run a bit faster. +No more unnecessary type conversions. Small changes like this can add up to be significant. + +If warp timer is enabled or the script is not marked as "run without screen refresh", the new compiler generates the same code as the old compiler. This is because the repeat block might pause before it finishes all iterations, so other scripts have the opportunity to change variables in unknown ways. For example, if another script changed the `i` variable to a string, `i.value + 1` would act like the `join` block instead of the `+` block. + +(We manually cleaned up the JavaScript for this page. The actual code has no formatting and does not have meaningful variable names.) ## Credits {#credits} @@ -118,6 +122,6 @@ Early testers helped us find and fix many bugs before release: * [Vadik1](https://scratch.mit.edu/users/Vadik1/) * [SpinningCube](https://scratch.mit.edu/users/SpinningCube/) * [GamingWithDominic](https://scratch.mit.edu/users/GamingWithDominic/) - * scratchfakemon + * [Scratch_Fakemon](https://scratch.mit.edu/users/Scratch_Fakemon/) * [LSPECTRONIZTAR](https://scratch.mit.edu/users/LSPECTRONIZTAR/) * termy From eb6704fc96c2882145f14bbec3e9f4b687aaa6c4 Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Fri, 19 Sep 2025 20:01:18 -0500 Subject: [PATCH 5/7] add link for the last person --- docs/website/new-compiler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md index b7cc3018..8ae04fe3 100644 --- a/docs/website/new-compiler.md +++ b/docs/website/new-compiler.md @@ -124,4 +124,4 @@ Early testers helped us find and fix many bugs before release: * [GamingWithDominic](https://scratch.mit.edu/users/GamingWithDominic/) * [Scratch_Fakemon](https://scratch.mit.edu/users/Scratch_Fakemon/) * [LSPECTRONIZTAR](https://scratch.mit.edu/users/LSPECTRONIZTAR/) - * termy + * [Terminal](https://scratch.mit.edu/users/windowscj/) From 2667d0e98c0788e12f74777a897bf2c9d99fda41 Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Fri, 19 Sep 2025 21:02:34 -0500 Subject: [PATCH 6/7] delay --- docs/website/new-compiler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md index 8ae04fe3..2360a53a 100644 --- a/docs/website/new-compiler.md +++ b/docs/website/new-compiler.md @@ -5,7 +5,7 @@ hide_table_of_contents: true # New Compiler -The compiler is the part of TurboWarp that converts projects to JavaScript. On September 19, 2025, we released a new compiler that is better at analyzing the scripts inside projects to generate even faster JavaScript. +The compiler is the part of TurboWarp that converts projects to JavaScript. On September 20, 2025, we released a new compiler that is better at analyzing the scripts inside projects to generate even faster JavaScript. This is the largest single change we've ever made. We've tried to test it thoroughly, but **it's quite likely there are still a few more bugs**. You should report broken projects at [the usual place](https://scratch.mit.edu/users/GarboMuffin/#comments). If your project is broken, you can use https://experiments.turbowarp.org/old-compiler/ instead until we fix the bug in the new compiler. From e8ea6cbf941d746646232def4a1baae6b4668026 Mon Sep 17 00:00:00 2001 From: Thomas Weber Date: Sat, 20 Sep 2025 00:21:44 -0500 Subject: [PATCH 7/7] Update new-compiler.md --- docs/website/new-compiler.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/website/new-compiler.md b/docs/website/new-compiler.md index 2360a53a..b841cbb2 100644 --- a/docs/website/new-compiler.md +++ b/docs/website/new-compiler.md @@ -108,7 +108,7 @@ for (var a0 = 100; a0 >= 0.5; a0--) { No more unnecessary type conversions. Small changes like this can add up to be significant. -If warp timer is enabled or the script is not marked as "run without screen refresh", the new compiler generates the same code as the old compiler. This is because the repeat block might pause before it finishes all iterations, so other scripts have the opportunity to change variables in unknown ways. For example, if another script changed the `i` variable to a string, `i.value + 1` would act like the `join` block instead of the `+` block. +If warp timer is enabled or the script is not marked as "run without screen refresh", the new compiler generates the same code as the old compiler. This is because the repeat block might pause before it finishes all iterations, so other scripts have the opportunity to change variables in unknown ways. For example, if another script changed the "i" variable to a string, `i.value + 1` would act like the "join" block instead of addition. The type conversion would be necessary to ensure the script always works correctly in this case. (We manually cleaned up the JavaScript for this page. The actual code has no formatting and does not have meaningful variable names.)