Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sdk): simulator state #5164

Merged
merged 40 commits into from
Dec 14, 2023
Merged

feat(sdk): simulator state #5164

merged 40 commits into from
Dec 14, 2023

Conversation

Chriscbr
Copy link
Contributor

@Chriscbr Chriscbr commented Dec 7, 2023

This PR adds the capability for the Wing simulator to store resource state across compilations. This means if you performed a lot of work to fill a cloud.Bucket with objects or values in cloud.Counter, it will be remembered when you recompile and reload the application as long as the resource is still in your app. This will be extended to other resources like Queue and Workload in the future.

Here's a screen recording - notice how even as the app is updated with new resources or different inflight code, the state of resources doesn't go away.

simulator-state-720.mov

Notes:

  • The cloud.TestRunner resource is now only added to a Wing app's construct tree when it is compiled in test mode (with wing test).
  • The Wing Console now runs two copies of the app, one compiled in the default mode, and one compiled with tests. The resource graph only shows the default app, but the tests panel shows a list of all tests from the app compiled with the simulator, and clicking any of the tests will run them with fresh state every time.

Future work:

Checklist

  • Title matches Winglang's style guide
  • Description explains motivation and solution
  • Tests added (always)
  • Docs updated (only required for features)
  • Added pr/e2e-full label if this feature requires end-to-end testing

By submitting this pull request, I confirm that my contribution is made under the terms of the Wing Cloud Contribution License.

@monadabot
Copy link
Contributor

Thanks for opening this pull request! 🎉
Please consult the contributing guidelines for details on how to contribute to this project.
If you need any assistence, don't hesitate to ping the relevant owner over Slack.

Topic Owner
Wing SDK and utility APIs @chriscbr
Wing Console @ainvoner, @skyrpex, @polamoros
JSON, structs, primitives and collections @hasanaburayyan
Platforms and plugins @hasanaburayyan
Frontend resources (website, react, etc) @tsuf239
Language design @eladb
VSCode extension and language server @markmcculloh
Compiler architecture, inflights, lifting @yoav-steinberg
Wing Testing Framework @tsuf239
Wing CLI @markmcculloh
Build system, dev environment, releases @markmcculloh
Library Ecosystem @chriscbr
Documentation @hasanaburayyan
SDK test suite @tsuf239
Examples @skorfmann
Wing Playground @eladcon

@monadabot
Copy link
Contributor

monadabot commented Dec 8, 2023

Console preview environment is available at https://wing-console-pr-5164.fly.dev 🚀

Last Updated (UTC) 2023-12-14 16:12

@monadabot
Copy link
Contributor

monadabot commented Dec 8, 2023

Benchmarks

Comparison to Baseline ⬜⬜⬜⬜⬜🟩⬜⬜⬜⬜⬜⬜⬜
Benchmark Before After Change
version 76ms±3.43 72ms±0.38 -4ms (-5.7%)⬜
jsii_big.test.w -t sim 3358ms±16.85 3369ms±17.12 +11ms (+0.33%)⬜
jsii_big.test.w -t tf-aws 3458ms±15.67 3458ms±9.31 ...
functions_1.test.w -t sim 552ms±4.1 551ms±4.08 -1ms (-0.16%)⬜
functions_1.test.w -t tf-aws 1530ms±27.69 1526ms±21.54 -4ms (-0.27%)⬜
hello_world.test.w -t sim 557ms±4.37 547ms±3.92 -9ms (-1.68%)🟩
hello_world.test.w -t tf-aws 4660ms±32.52 4727ms±45.1 +67ms (+1.43%)⬜
functions_10.test.w -t sim 610ms±3.4 612ms±9.34 +2ms (+0.3%)⬜
functions_10.test.w -t tf-aws 3289ms±24.31 3307ms±35.2 +18ms (+0.54%)⬜
jsii_small.test.w -t sim 524ms±7.23 521ms±3.88 -3ms (-0.63%)⬜
jsii_small.test.w -t tf-aws 650ms±5.22 645ms±7.68 -5ms (-0.71%)⬜
empty.test.w -t sim 516ms±2.94 521ms±3.58 +5ms (+0.97%)⬜
empty.test.w -t tf-aws 642ms±5.46 641ms±4.98 0ms (-0.07%)⬜

⬜ Within 1.5 standard deviations
🟩 Faster, Above 1.5 standard deviations
🟥 Slower, Above 1.5 standard deviations

Benchmarks may vary outside of normal expectations, especially when running in GitHub Actions CI.

Results
name mean min max moe sd
version 72ms 71ms 73ms 0ms 1ms
jsii_big.test.w -t sim 3369ms 3339ms 3412ms 17ms 24ms
jsii_big.test.w -t tf-aws 3458ms 3446ms 3483ms 9ms 13ms
functions_1.test.w -t sim 551ms 544ms 563ms 4ms 6ms
functions_1.test.w -t tf-aws 1526ms 1474ms 1559ms 22ms 30ms
hello_world.test.w -t sim 547ms 539ms 556ms 4ms 5ms
hello_world.test.w -t tf-aws 4727ms 4644ms 4854ms 45ms 63ms
functions_10.test.w -t sim 612ms 588ms 634ms 9ms 13ms
functions_10.test.w -t tf-aws 3307ms 3241ms 3391ms 35ms 49ms
jsii_small.test.w -t sim 521ms 515ms 528ms 4ms 5ms
jsii_small.test.w -t tf-aws 645ms 635ms 670ms 8ms 11ms
empty.test.w -t sim 521ms 512ms 529ms 4ms 5ms
empty.test.w -t tf-aws 641ms 629ms 651ms 5ms 7ms
Last Updated (UTC) 2023-12-14 16:47

monadabot and others added 3 commits December 8, 2023 23:01
Signed-off-by: monada-bot[bot] <monabot@monada.co>
Signed-off-by: monada-bot[bot] <monabot@monada.co>
@monadabot monadabot added the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Dec 8, 2023
@Chriscbr Chriscbr removed the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Dec 8, 2023
Copy link
Contributor

@eladb eladb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. See a few notes.

I am okay to get this over the finish line and then think what's the best way to support things like keeping containers alive (the additional requirement we discussed yesterday).

libs/wingsdk/src/core/app.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/core/synth-roots.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/simulator/simulator.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/simulator/simulator.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/simulator/simulator.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/std/test.ts Show resolved Hide resolved
libs/wingsdk/src/target-sim/bucket.inflight.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/target-sim/bucket.inflight.ts Outdated Show resolved Hide resolved
libs/wingsdk/src/target-sim/counter.inflight.ts Outdated Show resolved Hide resolved
@Chriscbr Chriscbr removed the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Dec 13, 2023
@monadabot monadabot added the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Dec 13, 2023
@Chriscbr Chriscbr removed the ⚠️ pr/review-mutation PR has been mutated and will not auto-merge. Clear this label if the changes look good! label Dec 13, 2023
@Chriscbr Chriscbr removed the 🚧 pr/do-not-merge PRs with this label will not be automatically merged by mergify. label Dec 14, 2023
Copy link
Contributor

@polamoros polamoros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes done in the console app look good to me 👍

Copy link
Contributor

mergify bot commented Dec 14, 2023

Thanks for contributing, @Chriscbr! This PR will now be added to the merge queue, or immediately merged if rybickic/simulator-state is up-to-date with main and the queue is empty.

mergify bot added a commit that referenced this pull request Dec 14, 2023
@mergify mergify bot merged commit 698e645 into main Dec 14, 2023
14 of 15 checks passed
@mergify mergify bot deleted the rybickic/simulator-state branch December 14, 2023 19:59
@monadabot
Copy link
Contributor

Congrats! 🚀 This was released in Wing 0.51.33.

mergify bot pushed a commit that referenced this pull request Dec 16, 2023
Fixes #5228

We discovered an issue where invalid errors were thrown in unit tests where a message is pushed to a queue and the queue had a consumer function. The problem occurred because a queue's simulation periodically processes messages, but this continued even after the queue consumer function is deleted during the simulator's cleanup. (I've shared some verbose test logs from the minimal repro in #5228 here: https://gist.github.com/Chriscbr/23fb248fb65d9e349da6e3347c90c866).

To fix this, I updated the EventMapping resource so it also removes the its function from the queue's subscribers when the EventMapping is cleaned up. By doing this, we guarantee the function is removed from the queue's subscribers before the function is cleaned up, so when the queue tries processing messages, it will send the messages to another consumer or simply retain them in memory.

Misc:
- I also cleaned up some dead code that was previously used by the CLI for picking out distinct tests to run (no longer needed after #5164)

## Checklist

- [x] Title matches [Winglang's style guide](https://www.winglang.io/contributing/start-here/pull_requests#how-are-pull-request-titles-formatted)
- [x] Description explains motivation and solution
- [x] Tests added (always)
- [ ] Docs updated (only required for features)
- [ ] Added `pr/e2e-full` label if this feature requires end-to-end testing

*By submitting this pull request, I confirm that my contribution is made under the terms of the [Wing Cloud Contribution License](https://github.com/winglang/wing/blob/main/CONTRIBUTION_LICENSE.md)*.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants