Dev Experience + Bundler Overhaul + BBUI in monorepo

[aptkingston]

Description

This PR adds BBUI to the monorepo, and totally overhauls the whole experience of developing Budibase.

This is pretty much impossible to review - but the best way would be to check it out and try it.

TLDR

• The builder, BBUI and standard components use Vite instead of Rollup
• Envoy proxies to the builder dev server (no CORS issues, cookies still work)
• No dev bundlers in BBUI or standard components for less load
• Lerna correctly symlinks everything together
• No manual symlinking or usages of /tmp directory
• Server static routes cleaned up
• Packages reference the source code of other packages (svelte keyword in package.json) and bundle it
• Builds are much smaller (~50%) and more optimized
• You get instant updates without the page even refreshing 🎉
• The new dev URL is now host:10000/builder rather than host:4001/_builder

Dev setup

Currently we use a script symlinkDev to link our local packages together, and some of them had relative paths going outside of the package to reach another package (eg requesting ../client from the builder package). Instead, lerna now links all dependencies together, which means that the packages in each node_modules folders are actually symlinks to your version on disk. That means that you only need to worry about a single file path in dev or prod. As we don't publish the builder package, the builder still outputs production builds to the server package, but this is the only exception.

lerna link is the command that does this, and it's been added to both the bootstrap command and also to top level dev command, to ensure everything is always linked.

Bundlers

Currently we have rollup running inside the builder, BBUI, standard components, string templates and client. Each of these packages uses the built output of the others, and then recompiles when changes are detected. This means that making a change to standard-components causes standard-components to recompile, followed by client, followed by builder, and finally the sever will reload. This takes anywhere from 20-60 seconds depending on the speed of your machine.

Now, each package uses the source of the other packages and bundles it directly, cutting out the "middle men" bundlers. If you make a change to standard-components then the client directly picks that up and recompiles without any intermediate step of bundling standard-components itself, reducing the overload load on the dev machine and the amount of time spent bundling.

The builder, BBUI and standard components all use Vite now instead of rollup, although neither BBUI or standard component run any sort of dev server now - because the builder directly uses the source code. The builder does run a Vite dev server on port 3000, and envoy has been updated to proxy through to it.

budibase-client.js and backwards compatibility

The management and bundling of the client library is a tricky subject due to our requirements, but it's been simplified now. We require a separate client library to be built, so that apps can be versioned and maintained in the long term - otherwise we could simply bundle up the client inside the builder and be done with it.

Currently, the client library is bundled and symlinked to the /tmp/.budibase directory. The server then serves this latest version in dev, or serves the stored version for an app in prod. The server also has endpoints to serve the client library bundle to preview apps, and to serve the builder with component definitions of the client library. Currently these endpoints are not in sync, which means the component list that the builder gets is not always correct for the version of the client library that the app is using.

With these changes, the client library is bundled as normal and no symlinking is required. Due to lerna symlinking all packages together automatically, the server can now reference the latest version of the client lib in its node_modules directory. The server also ensures that all endpoints referencing client library are in sync - that's to say that the version of the client library being served and the component definitions returned are always correct for the version of the client library. There is also a new client library bundle URL sent down in the app package. The builder injects this URL into the iframe preview - meaning that now you can actually develop an old app with the correct version of the client library, whereas previously the builder iframe preview and the builders component list always used the latest version of the client library.

Here's how the client library versions are now handled:

• When running in dev, you will always get the latest version of the client library. This means you can develop the client library and see your changes. You'll still need to reload the page manually to see your client library changes, but the time is much shorter (~5s).
• When running in prod, you will always get the version of the client library that is stored for the app. The builders component list, iframe preview, real preview and deployed apps will all use this same version.

All we need to implement now is a way to manually choose to update your client library version, and then we would be well on our way to properly supporting backwards compatibility - at least in terms of the client library.

Builder assets

Any builder assets can be properly imported into svelte files as needed, and they will be bundled with the builder. This also means that only the assets that are actually used are now bundled with the builder. This makes it much easier to also manage import paths to assets. For instance, here's how using an image now looks:

<script>
  import Logo from "/assets/logo.svg"
</script>

<img src={Logo} />

Previously all assets were imported in the main.js file, and things like images relied on using hardcoded URL's which were expected to exist at runtime.

[shogunpurple]

Awesome work 🔥 - this makes life so much easier for everyone and makes the platform much easier to develop on.

However I am seeing an issue when running on my branch:

• Ran a yarn clean
• ran yarn && yarn bootstrap
• yarn dev

When I hit localhost:10000/builder I'm getting:

upstream connect error or disconnect/reset before headers. reset reason: connection failure

Seems like it might just be something misconfigured in envoy and it can't reach the load balancer targets (getting 503). @kevmodrome is having the same issue. (Mac related maybe?)

[shogunpurple]

Ah - it appears to be the IP addresses added to the envoy config for dev.

172.17.0.1 works on linux because that static IP gets assigned when docker runs directly but since OSX uses docker machine we need:

host.docker.internal

This won't work on linux though 😖

[mike12345567]

Ah - it appears to be the IP addresses added to the envoy config for dev.

172.17.0.1 works on linux because that static IP gets assigned when docker runs directly but since OSX uses docker machine we need:

host.docker.internal

This won't work on linux though 😖

Damn this is something I was worried about, finding some way for envoy to redirect requests outside of docker is quite difficult, its pretty abnormal behaviour for a docker environment.

Wondering if we could do something like detect where the request came from with envoy and bounce the request back to that address but a different port... actually that probably won't work with virtual machines and running the browser etc outside the VM.

I'm thinking what we might have to do is in the manage.js script detect the OS type and then inject the correct host IP based on that, but that would require re-writing the envoy dev YAML every time, which would be pretty nasty and a fairly complex process to keep maintained (versus just having the file).

[kevmodrome]

Found this: docker/for-linux#264

[mike12345567]

Found this: docker/for-linux#264

extra_hosts:
      - "host.docker.internal:host-gateway"

Adding this to Envoy config in dev might be worth a try, i'll give it go locally, looks like its mentioned as a rough workaround?

[codecov-io]

Codecov Report

Merging #1361 (9c0a879) into next (6851f59) will decrease coverage by 0.18%.
The diff coverage is 54.28%.

❗ Current head 9c0a879 differs from pull request most recent head 6d5b136. Consider uploading reports for the commit 6d5b136 to get more accurate results

@@            Coverage Diff             @@
##             next    #1361      +/-   ##
==========================================
- Coverage   84.83%   84.64%   -0.19%     
==========================================
  Files         116      116              
  Lines        3191     3211      +20     
  Branches      415      417       +2     
==========================================
+ Hits         2707     2718      +11     
- Misses        230      239       +9     
  Partials      254      254              

Impacted Files	Coverage Δ
packages/server/src/api/controllers/row.js	80.95% <ø> (ø)
...ackages/server/src/api/controllers/search/index.js	0.00% <0.00%> (ø)
packages/server/src/api/routes/component.js	100.00% <ø> (ø)
packages/server/src/api/routes/search.js	0.00% <0.00%> (ø)
packages/server/src/middleware/authenticated.js	82.92% <0.00%> (ø)
packages/server/src/middleware/authorized.js	74.19% <ø> (+0.66%)	⬆️
packages/server/src/api/controllers/auth.js	83.72% <50.00%> (-3.46%)	⬇️
packages/server/src/utilities/index.js	74.13% <60.00%> (-3.87%)	⬇️
packages/server/src/api/controllers/application.js	94.11% <100.00%> (+0.05%)	⬆️
packages/server/src/api/index.js	93.54% <100.00%> (ø)
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6851f59...6d5b136. Read the comment docs.

[kevmodrome]

VERY hyped about this! Looks great @aptkingston 🥳

[aptkingston]

Cheers for the contributions to get this working cross-platform!