-
-
Notifications
You must be signed in to change notification settings - Fork 149
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Update deps. * Start on blog post. * Finish blog post. * Add image.
- Loading branch information
Showing
11 changed files
with
1,805 additions
and
1,603 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
593 changes: 297 additions & 296 deletions
593
.yarn/releases/yarn-4.1.1.cjs → .yarn/releases/yarn-4.2.2.cjs
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| --- | ||
| slug: moon-v1.25 | ||
| title: moon v1.25 - New task runner and console reporter | ||
| authors: [milesj] | ||
| tags: [task, runner, console, reporter, operation] | ||
| image: ./img/moon/v1.25.png | ||
| --- | ||
|
|
||
| In this release, we focused primarily on rewriting our task runner, and improving our console. | ||
|
|
||
| <!--truncate--> | ||
|
|
||
| ## New task runner implementation | ||
|
|
||
| It's been over a month since our last release, but we've been really busy rewriting our task runner | ||
| from the ground up! In other build systems, a task runner is typically the orchestator that runs | ||
| multiple tasks and manages their state. In moon this is known as the action pipeline (or just | ||
| pipeline), and a task runner is simply the execution of a single task. However, executing a single | ||
| task is quite involved, as we need to generate a unique hash, check the cache, hydrate outputs if a | ||
| cache hit, actually execute the task as a child process, and much more! | ||
|
|
||
| Task running is some of the oldest code in moon, as it was part of the initial MVP. Because of this, | ||
| it hasn't changed much, but moon has grown quite large and it was time to revisit it with better | ||
| design patterns and practices. Furthermore, since the task runner is so critical to moon itself, we | ||
| wanted to ensure it worked correctly, and spent more time than usual implementing, testing it, and | ||
| verifying edge cases. | ||
|
|
||
| With this new task runner, we... | ||
|
|
||
| - Improved handling and reliability of output archiving (cache miss) and hydration (cache hit). | ||
| - Streamlined the task execution (child process) flow. | ||
| - Increased performance by optimizing or removing certain code paths. | ||
|
|
||
| > If you're interested in how the task runner was implemented, feel free to take a look at the | ||
| > [Rust crate](https://github.com/moonrepo/moon/tree/master/nextgen/task-runner), and the | ||
| > [pull request](https://github.com/moonrepo/moon/pull/1463) itself. | ||
| ### Fine-grained operations | ||
|
|
||
| A major goal of moon is bubbling up information to the user that is applicable to the current | ||
| workflow, but what about when that workflow must be debugged or optimized? At that point, it was | ||
| almost impossible without digging into the source code. | ||
|
|
||
| To make a step in this direction, as part of the new task runner we now track timing information for | ||
| individual parts of the run execution, and we're calling these parts operations. An operation is | ||
| anything from generating a hash, creating a tarball archive, unpacking the archive (cache | ||
| hydration), task execution (the child process), and more. | ||
|
|
||
| This timing information is useful in figuring out why a certain task is slower than expected, and | ||
| which operation is actually causing the slowness. It also helps to uncover which operations were | ||
| actually ran for an action, which were skipped, so on and so forth. At this point in time, the | ||
| operations information is only included in the run report, located at `.moon/cache/runReport.json`. | ||
| In the future, we plan to display this information in a nice UI. | ||
|
|
||
| For an example of this in action, here's a list of all operations that were executed when running | ||
| the `build` task for our website. | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "duration": { | ||
| "secs": 0, | ||
| "nanos": 609156875 | ||
| }, | ||
| "finishedAt": "2024-05-27T00:14:54.286628", | ||
| "meta": { | ||
| "type": "hash-generation", | ||
| "hash": "10606e37c5e6ab4008007b30275f1682bae32dca71650ce173eb386d5b6c3309" | ||
| }, | ||
| "startedAt": "2024-05-27T00:14:53.677526", | ||
| "status": "passed" | ||
| }, | ||
| { | ||
| "duration": { | ||
| "secs": 0, | ||
| "nanos": 32834 | ||
| }, | ||
| "finishedAt": "2024-05-27T00:14:54.286667", | ||
| "meta": { | ||
| "type": "output-hydration" | ||
| }, | ||
| "startedAt": "2024-05-27T00:14:54.286634", | ||
| "status": "skipped" | ||
| }, | ||
| { | ||
| "duration": { | ||
| "secs": 15, | ||
| "nanos": 789003125 | ||
| }, | ||
| "finishedAt": "2024-05-27T00:15:10.075113", | ||
| "meta": { | ||
| "type": "task-execution", | ||
| "command": "docusaurus build", | ||
| "exitCode": 0 | ||
| }, | ||
| "startedAt": "2024-05-27T00:14:54.286950", | ||
| "status": "passed" | ||
| }, | ||
| { | ||
| "duration": { | ||
| "secs": 17, | ||
| "nanos": 214634292 | ||
| }, | ||
| "finishedAt": "2024-05-27T00:15:27.289995", | ||
| "meta": { | ||
| "type": "archive-creation" | ||
| }, | ||
| "startedAt": "2024-05-27T00:15:10.075686", | ||
| "status": "passed" | ||
| } | ||
| ] | ||
| ``` | ||
|
|
||
| Because of these new operations, we can clearly see above that the archive creation process is | ||
| taking 17 seconds, which is 2 seconds longer than the build itself! Without this information, we | ||
| would have never known that the archive was taking this long, but now we do, and we can optimize it | ||
| in a future release! | ||
|
|
||
| ### Run summaries | ||
|
|
||
| Because of the new task runner and the new console ([below](#new-console-reporting-layer)), we have | ||
| the ability to bubble up more information than before. Based on requests from the community, we've | ||
| taken the output from [`moon ci`](/docs/commands/ci) and applied it to both | ||
| [`moon check`](/docs/commands/check) and [`moon run`](/docs/commands/run) behind the `--summary` | ||
| flag. | ||
|
|
||
| When this flag is passed, we will now summarize all actions that have ran in the pipeline (not just | ||
| task related ones), and include failed tasks for review. For example, here's the output of | ||
| `moon check website --summary` on moon's repository. | ||
|
|
||
| ``` | ||
| $ moon check website --summary | ||
| ▪▪▪▪ types:build (cached, 1ms, 21bd9add) | ||
| ▪▪▪▪ runtime:build (cached, e8363e65) | ||
| ▪▪▪▪ website:typecheck (cached, 0ab91eaa) | ||
| ▪▪▪▪ website:format (cached, 07ae2388) | ||
| ▪▪▪▪ website:test (cached, 11d33e2e) | ||
| ▪▪▪▪ website:lint (cached, 2197fbb1) | ||
| ▪▪▪▪ website:build (10606e37) | ||
| ▪▪▪▪ website:build (15s 789ms, 10606e37) | ||
| SUMMARY | ||
| pass SyncWorkspace | ||
| skip SetupNodeTool(20.13.1) (skipped, 250ms) | ||
| skip InstallNodeDeps(20.13.1) (skipped, 13ms, f341872f) | ||
| pass SyncNodeProject(types) (1ms) | ||
| pass SyncNodeProject(runtime) (1ms) | ||
| pass RunTask(types:build) (cached, 140ms, 21bd9add) | ||
| pass SyncNodeProject(website) (1ms) | ||
| pass RunTask(runtime:build) (cached, 32ms, e8363e65) | ||
| pass RunTask(website:build) (33s 614ms, 10606e37) | ||
| pass RunTask(website:format) (cached, 59ms, 07ae2388) | ||
| pass RunTask(website:lint) (cached, 101ms, 2197fbb1) | ||
| pass RunTask(website:test) (cached, 64ms, 11d33e2e) | ||
| pass RunTask(website:typecheck) (cached, 59ms, 0ab91eaa) | ||
| STATS | ||
| Actions: 11 completed (6 cached), 2 skipped | ||
| Time: 34s 52ms | ||
| ``` | ||
|
|
||
| ## New console reporting layer | ||
|
|
||
| For the most part, when something in moon needed to be printed to the console, we would simply print | ||
| it directly at that point in time, anywhere within the codebase. While this worked, it made it | ||
| difficult to orchestrate output from different parts of the codebase, and in the context of Rust, | ||
| each call to stdout/stderr [locks the I/O stream](https://nnethercote.github.io/perf-book/io.html), | ||
| resulting in performance loss. | ||
|
|
||
| To work around this, in [v1.21](./moon-v1.21) we implemented a new | ||
| [console layer](https://github.com/moonrepo/moon/tree/master/nextgen/console) that buffers all stdio | ||
| writes, and prints them on 100ms intervals. This avoids locking on every call, and instead batches | ||
| them. To expand on this further, in this release we've implemented a new | ||
| [reporter layer](https://github.com/moonrepo/moon/tree/master/nextgen/console-reporter), with a | ||
| well-defined interface that is used to print checkpoints (the 4 squares), and status updates from | ||
| the action pipeline (and the new task runner). | ||
|
|
||
| This reporter layer enables new console UI implementations in the future based on your preferences. | ||
| For example, an [interactive UI](https://ratatui.rs/) composed of tables, tabs, and more, | ||
| representing the current state of the pipeline. | ||
|
|
||
| ## Other changes | ||
|
|
||
| View the [official release](https://github.com/moonrepo/moon/releases/tag/v1.25.0) for a full list | ||
| of changes. | ||
|
|
||
| - Greatly reduced the amount of concurrent locks being held during task execution. May see slight | ||
| performance improvements. | ||
| - Updated external configuration files (via https extends) to be cached for 24 hours. | ||
| - Updated macOS binaries to be built on macos-12 instead of macos-11. | ||
| - Updated proto to v0.35.4 (from v0.34.4). |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.