Turborepo Implementation & Monorepo standardization

[JacobMGEvans]

👋🏻 This has been a long time in the making #1990, we are finally here! The Repo has undergone a lot of initial growth and evolution as we worked to make the migration happen to TypeScript and further provide better development experiences for our users, this is now us doing the same for ourselves which will enable us to evolve AGAIN!

Why TurboRepo & Monorepo?

The goal of providing an architecture & system that decouples the Wrangler core so that it can be used in external modules, we would like to move the workers-sdk towards a Monorepo structure and manage it with a modern tooling built for Monorepos like Turborepo.

This would help us standardize and enforce Monorepo conventions. Additionally packages will share a core config, i.e. TSConfig & ESLint config, while also extending that to their own isolated config and environments, this will prevent global configs becoming increasing complex and potentially conflicts between local configs that don't necessarily need the global config.

This move will help to reduce build times and test failures by enabling caching of builds and by intelligently running tests and other checks only on parts of the codebase that have changed or are dependent on each other.

Future improvements & things this PR facilitates:

• Turborepo codegen for spinning up and wiring up new packages or other templateable things Docs
• Remote caching of builds, tests, checks, etc... which will be built custom using TurboRepo Remote Cache API Docs
• Further isolation and decentralizing of configurations and the packages themselves, i.e. Wrangler getting broken into wrangler/cli, wrangler/core, etc...
• We will need to follow up this work with specific configuration on how we want to handle Templates, some templates have VERY different local setups & config than the rest of workers-sdk due to the nature of them being created outside of the workers-sdk repo and migrated in later
• TypeScript Composition setup for Projects in monorepo

Conclusion

Overall, this is a move to improve the developer experience of a repo that many teams contribute to on a daily basis.

Comparisons

CI/CD Checks

TurboRepo Checks time (Linting, Types & Formatting combined in parallel)

Before TurboRepo

Local Terminal Runs

TurboRepo Checks - uncached (Linting, Types & Formatting combined highly parallel)

TurboRepo Checks - Cached (Linting, Types & Formatting combined)

Before TurboRepo - Linting, Types & Formatting concurrent w/ no caching comparison

Special Thanks

@anthonyshew - https://twitter.com/anthonysheww

[changeset-bot]

⚠️ No Changeset found

Latest commit: df8b510

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/5245269842/npm-package-wrangler-3261

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/3261/npm-package-wrangler-3261

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/5245269842/npm-package-wrangler-3261 dev path/to/script.js

Additional artifacts:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/5245269842/npm-package-cloudflare-pages-shared-3261

Note that these links will no longer work once the GitHub Actions artifact expires.

[codecov]

Codecov Report

Merging #3261 (df8b510) into main (2ce2423) will decrease coverage by 0.07%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##             main    #3261      +/-   ##
==========================================
- Coverage   75.42%   75.36%   -0.07%     
==========================================
  Files         182      183       +1     
  Lines       11070    11022      -48     
  Branches     2903     2893      -10     
==========================================
- Hits         8350     8307      -43     
+ Misses       2720     2715       -5     

Impacted Files	Coverage Δ
packages/wrangler/src/config/index.ts	90.38% <ø> (ø)
...kages/wrangler/src/constellation/createProject.tsx	83.33% <ø> (ø)
...s/wrangler/src/constellation/deleteProjectModel.ts	91.30% <ø> (ø)
packages/wrangler/src/constellation/listModel.tsx	93.75% <ø> (ø)
...ackages/wrangler/src/constellation/uploadModel.tsx	84.61% <ø> (ø)
packages/wrangler/src/constellation/utils.ts	93.18% <ø> (ø)
packages/wrangler/src/d1/backups.tsx	29.11% <ø> (ø)
packages/wrangler/src/d1/create.tsx	31.25% <ø> (ø)
packages/wrangler/src/d1/delete.ts	34.78% <ø> (ø)
packages/wrangler/src/d1/execute.tsx	55.10% <ø> (ø)
... and 23 more

... and 7 files with indirect coverage changes

[petebacondarwin]

Looking good. I asked a couple of clarifying questions before approving.

[petebacondarwin]

Suggested change

	"test:watch": "npx vitest"
	"test:watch": "vitest"

??

[JacobMGEvans]

vitest run will never watch and vitest will watch in local dev mode. TurboRepo will hang sometimes with vitest. Additionally npx is unnecessary, everything should be a dependency in the packages as a Monorepo standard convention.

[petebacondarwin]

Yeah this comment was about the fact that there is inconsistent use (or non-use) of npx in these scripts. If a tool is installed locally (which it should be!) then npx should not be needed - for wrangler, vitest, etc.

[JacobMGEvans]

I agree there should be a consistent enforcement.

[petebacondarwin]

Running locally, npm run test fails the local-mode-tests - they timed-out and then turbo appears to hang.

I did a git clean and npm ci just to be sure, and then npm run test failed on edge-preview-authenticated-proxy:test, which said it didn't have wrangler build available.

So I added "dependsOn": ["^topological"], to the test section in turbo.json, and it then failed with the same error on local-mode-tests:test.

Then I tried just running it again and now it failed on node-app-pages:test with a timeout.

I think we need to ensure that all these tests can run successfully from a clean clone and install of the project before we land this. It feels like there is still some configuration that is missing?

(It may be that the timeouts are due to the workerd permission pop-ups blocking the tests; and if you don't click accept quickly enough then the tests timeout?)

[GregBrimble]

I have reviewed the code relevant to Pages, as well as the global-level stuff and smatterings not relevant to Pages. Pretty much all good to me. One tiny tiny nit and a suggestion about package naming (which isn't strictly relevant to this PR, but you do add some new packages which fall under have the same 'problem').

LGTM, thanks @JacobMGEvans! ✅

[JacobMGEvans]

Current status locally.

[anthonyshew]

🥳 🥳 🥳