Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Miniflare 3] Implement simulators as Durable Objects #656

Merged
merged 25 commits into from Sep 5, 2023
Merged

[Miniflare 3] Implement simulators as Durable Objects #656

merged 25 commits into from Sep 5, 2023

Commits on Sep 5, 2023

  1. Move some shared helpers into workers directory 

    These will be needed when we port simulators to workers.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    d363e7e View commit details
    Browse the repository at this point in the history

  2. Implement fake timers for Workers 

    These were previously implemented in Node, but we'll need them in
Durable Objects to aid with testing.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    b1f5bd6 View commit details
    Browse the repository at this point in the history

  3. Implement SQL helpers for Workers 

    These provide a similar API to what we had with `better-sqlite3`.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    fe6b263 View commit details
    Browse the repository at this point in the history

  4. Implement decorator router for Workers 

    This will allow us to use HTTP method decorators like `@GET` in our
Durable Object simulators.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    ae5c947 View commit details
    Browse the repository at this point in the history

  5. Implement blob store for Workers 

    This implements the same blob store API we previously had in Node,
but backed by a `workerd` disk directory service instead.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    ed23f7e View commit details
    Browse the repository at this point in the history

  6. Implement simulator Durable Object base class 

    Implements a `MiniflareDurableObject` class that provides router,
storage and fake timer APIs similar to Miniflare's existing ones.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    5d08f29 View commit details
    Browse the repository at this point in the history

  7. Implement expiring key-value-metadata store for Workers 

    This will be used as the base for KV and Cache simulators.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    48b0d3a View commit details
    Browse the repository at this point in the history

  8. Implement miniflare:shared and miniflare:zod extensions 

    This implements 2 `workerd` "extensions": modules that can be
imported by any other worker. `miniflare:shared` contains a bunch of
helpers, including the Durable Object base class. `miniflare:zod`
includes `zod` and a few useful schemas. `zod` is a relatively large
library, so we'd only like to include it in the config once. Not all
simulators will depend on it too, so it's separated from
`miniflare:shared`.

Miniflare's build script has been updated to write embedded Workers
to files, that are lazily read as required. `esbuild`'s support for
`tsconfig` paths is used to load `miniflare:zod` in Node.js code too.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    95d1d6f View commit details
    Browse the repository at this point in the history

  9. Implement simple Worker testing 

    This implements an AVA macro for a test that should run within
`workerd`. This allows us to test functionality ported to Workers.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    c02e0f5 View commit details
    Browse the repository at this point in the history

  10. Add tests for Workers decorator router

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    b9da64f View commit details
    Browse the repository at this point in the history

  11. Add tests for Workers blob store

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    4670191 View commit details
    Browse the repository at this point in the history

  12. Implement generic logging endpoint in loopback server 

    This replaces the loopback server's support for logging requests with
a more generic endpoint, that supports logging any arbitrary message.
This will be needed by Durable Object simulators that want to write
to the console, without `workerd`'s verbose prefixing. In particular,
Queues logs dispatches, and Cache logs first usage when the
`cacheWarnUsage` option is enabled.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    6b40ad3 View commit details
    Browse the repository at this point in the history

  13. Rewrite KV tests using proxy 

    Previously, we were only testing the `KVGateway`. In preparation for
porting the KV implementation to a worker, rewrite the tests to test
the full KV flow, including the router.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    e1b4a74 View commit details
    Browse the repository at this point in the history

  14. Add tests for Workers Sites

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    4f47465 View commit details
    Browse the repository at this point in the history

  15. Implement KV simulator as Durable Object

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    64b8232 View commit details
    Browse the repository at this point in the history

  16. Implement Workers Sites simulator as a Worker

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    7405f88 View commit details
    Browse the repository at this point in the history

  17. Implement Cache simulator as Durable Object

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    8b195af View commit details
    Browse the repository at this point in the history

  18. Implement R2 simulator as Durable Object 

    Fixes cloudflare/workers-sdk#3626
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    efbecb0 View commit details
    Browse the repository at this point in the history

  19. Implement D1 simulator as Durable Object

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    3a95517 View commit details
    Browse the repository at this point in the history

  20. Implement Queues simulator as Durable Object

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    6b1288c View commit details
    Browse the repository at this point in the history

  21. Remove unused dependencies 

    ...and move dependencies only used in Workers to `devDependencies`.
They're bundled in with the Worker code, so don't need to be
installed again. Importantly, this change removes `better-sqlite3`,
meaning Miniflare no longer has any native dependencies other than
`workerd`.

Fixes #599

Fixes cloudflare/workers-sdk#3423
Fixes cloudflare/workers-sdk#3449
Fixes cloudflare/workers-sdk#3531
Fixes cloudflare/workers-sdk#3534
Fixes cloudflare/workers-sdk#3708
Fixes cloudflare/workers-sdk#3746

Closes cloudflare/cloudflare-docs#10241
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    518ebb0 View commit details
    Browse the repository at this point in the history

  22. Use single connection Client dispatcher 

    When combined with running tests in serial, this should ensure we
only have at most two TCP connections open to `workerd` at a time.
    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    7ff9dd6 View commit details
    Browse the repository at this point in the history

  23. Use process isolation for tests

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    309673d View commit details
    Browse the repository at this point in the history

  24. Add tests for KV, R2 and D1 migrations

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    20583ab View commit details
    Browse the repository at this point in the history

  25. Enable @typescript-eslint/no-non-null-assertion rule

    @mrbbot
    mrbbot committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    4b4dc30 View commit details
    Browse the repository at this point in the history