Experimental compiler refactor #133

matthewmueller · 2022-06-06T06:27:15Z

I'd like to take one more crack at the compiler refactor. It's important to get this process right because all the generators will fit within this overall process. Writing generators is the easy part.

The goal is to:

Transition from bud starting a long-running generated cli process that starts another long-running app process to bud optionally running a short-lived generated cli process, then running a long-running app process.

Additionally:

Gather all the generators and runtimes into a simple framework/ directory

This will hopefully make the code base much easier to follow, without compromising on flexibility.

CLAassistant · 2022-06-23T04:00:11Z

All committers have signed the CLA.

matthewmueller · 2022-06-23T06:06:34Z

Woohoo, it's aliveeeeee. Just need to do a bit more cleanup and it's good to go!

…ived process

…d overlay being needed in app

…n bud and the app

…orms

…r. fix imhash tests after refactor.

…ting stale code

matthewmueller · 2022-06-25T21:52:41Z

.github/workflows/test.yml

-      # main branch. See: https://github.com/golang/go/issues/53226
-      - name: Install bud binary into $PATH
-        run: GOPRIVATE=* go install github.com/livebud/bud@main
-


No longer necessary! There's no more spawning of a v8server when using bud run. Instead we've combined this with an internal local "bud server", that handles server-side rendering, hot reloads, and bundling client-side files.

This bud server is only run with bud run, when you bud build v8 is bundled, there's no hot reloads and all client-side files are pre-build and embedded.

matthewmueller · 2022-06-25T21:53:41Z

Makefile

@@ -50,7 +50,7 @@ example.hn.watch:
 # Go
 ##

-GO_SOURCE := ./internal/... ./package/... ./runtime/...
+GO_SOURCE := ./internal/... ./package/... ./framework/...


I wanted to combine internal/generator and runtime/generator into one package so that would put all the generators in one place. Framework felt like a better word for this because not all the packages in framework/ used at runtime, though I'm still tempted to change it to generator/

matthewmueller · 2022-06-25T21:55:58Z

framework/app/app.go

@@ -0,0 +1,45 @@
+package app
+


app is the new entrypoint into the program. In the previous bud, there was a generated main and program. This was over-engineering at this point in bud's journey, so I opted to simplify things for the time being.

matthewmueller · 2022-06-25T21:57:03Z

framework/app/app.gotext

+		return err
+	}
+	// Inform bud that we're ready
+	budClient.Publish("app:ready", nil)


The bud client talks to the server during bud run and informs the server that the app is ready. This allows tests to be less flaky and fail faster.

matthewmueller · 2022-06-25T21:57:49Z

framework/controller/controllerrt/request/unmarshal_test.go

@@ -5,8 +5,8 @@ import (
 	"net/http/httptest"
 	"testing"

+	. "github.com/livebud/bud/framework/controller/controllerrt/request"


Not totally happy with controllerrt, but it's short for "controller runtime"

matthewmueller · 2022-06-25T22:09:19Z

internal/cli/bud/bud.go

+}
+
+// Module finds the go.mod file for the application
+func Module(dir string) (*gomod.Module, error) {


internal/cli/bud contains some shared configuration, the root command and some helper functions that are used by the rest of the CLI

matthewmueller · 2022-06-25T22:12:05Z

internal/cli/toolfscat/toolfscat.go

+	"github.com/livebud/bud/internal/cli/bud"
+)
+
+func New(bud *bud.Command, in *bud.Input) *Command {


Added some tools to inspect the generated files

matthewmueller · 2022-06-25T22:12:23Z

internal/cli/toolfstxtar/toolfstxtar.go

+	}
+	// Print the archive to stdout
+	fmt.Fprintln(os.Stdout, string(txtar.Format(ar)))
+	return nil


This one is going to be handy for reproing issues

matthewmueller · 2022-06-25T22:13:36Z

internal/exe/exe.go

+	"os/exec"
+
+	"github.com/livebud/bud/internal/once"
+)


This package simplifies the previous exe package to consolidate how subprocesses are managed in bud. Instead of wrapping exec.Command, it can turn an exec.Cmd into a Process.

Additionally, it can allow you to configure commands earlier with shared env / stdio / files etc. that can be spawned at a later time.

matthewmueller · 2022-06-25T22:15:35Z

internal/format/format.go

+	return fmt.Sprintf(format(message), args...)
+}
+
+func Errorf(message string, args ...interface{}) error {


Added a way to add multi-line errors more easily.

matthewmueller · 2022-06-25T22:25:50Z

This PR concludes a large internal refactor I've been working on since launch. The goals of this PR:

Make it easier to understand. The double generated binary building was something that often confused me.
Make it easier to contribute. I'm so impressed with the contributions so far, with this refactor it should be even easier.
Make it faster during development. By far the slowest step in the build process is running go build. If we didn't need to do the double-compile, it should be 2x faster.

In general, now that I'm starting to get a better understanding of what people are expecting from the framework, I can start moving the codebase more towards that.

The results of this PR:

There's no more generated cli, we're just generating an app! We will find solutions when the time comes for all the planned features that led to the generated cli design.
All generators and helper runtimes live in framework/, supporting packages that are needed by the generated code live in packages/, everything else goes in internal/.
Tests are faster and less flaky because no more generated CLI and a pubsub system that will tell the test when the app is ready
Tests are more end-to-end because they only support two commands: cli.Run and cli.Start.
No more v8client/v8server, everything has been consolidated into the bud client/server that's run during bud run.
Debug logging! Added --log=debug to bud and the test suite. Slowly passing the logger through the subsystems.

Since so much of the codebase has changed, I'm pretty sure there's going to be a few minor regressions, but we can address them as they pop up. The core workflows have remained unchanged.

This PR pretty much wraps up v0.2. Have a look at the discussion to learn more about next steps.

matthewmueller marked this pull request as draft June 6, 2022 06:27

matthewmueller changed the title ~~Experimental refactor~~ Experimental compiler refactor Jun 6, 2022

matthewmueller marked this pull request as ready for review June 23, 2022 04:03

matthewmueller force-pushed the framework-dir branch from fa3b405 to 0fe9387 Compare June 23, 2022 05:39

matthewmueller mentioned this pull request Jun 25, 2022

Improve visual for hot reload #131

Merged

matthewmueller force-pushed the framework-dir branch from 2162ec9 to 5fb08b1 Compare June 25, 2022 19:34

matthewmueller added 22 commits June 25, 2022 16:46

wip experiment to clarify streams, apis and make generators a short-l…

3f17926

…ived process

wip: move generators to framework/

6f1228b

add more snapshot tests and simplify app generator

63ca7c3

framework/app: update comment

bfa0057

internal/golden: swap out autogold with custom txtar implementation

51ba9b4

'bud run' except for hot reload.

c9bb8eb

added initial budserver and budproxy to replace hot server, v8pipe an…

49fe21d

…d overlay being needed in app

design and integrate dev server and client to handle all comms betwee…

17eb829

…n bud and the app

got hot reload working again

052c67f

don't log error when context is canceled (from a sigint)

eadc9ba

finish migrating commands to new CLI

2c58e0c

testing 'bud run' appears to be working now

0fe7501

controller tests have been refactored

7d0effc

add more debug logging. fix fscache. add 'bud tool fs'

2bdcd81

get most framework/ tests passing.

ea60e27

replaced hot. started planning to simplify the io.

07ffda0

completely broken, but starting to come together.

af7df88

wip pubsub eventing

a67a2ab

get the cli commands and hot reload working again

c891eed

do tests actually pass? o_O

b961507

merge in updates from main

0fa3d8d

loosen mime type checking to account for inconsistencies across platf…

a6fbe32

…orms

matthewmueller added 7 commits June 25, 2022 16:46

move the runtimes into the framework

38e7afc

wip migrating hot reload improvements into compiler refactor

6817a12

integrate @012e's prompter into the compiler refactor

e91d392

rename testlog.Log() to testlog.New(). rename to bud client and serve…

7f327e5

…r. fix imhash tests after refactor.

fix makefile source

c123ddb

remove installing bud into PATH since we no longer use v8client

4c4b459

attempt to purge cache, for some reason github actions seem to be tes…

84c6a24

…ting stale code

matthewmueller force-pushed the framework-dir branch from 2a11558 to 84c6a24 Compare June 25, 2022 21:46

matthewmueller commented Jun 25, 2022

View reviewed changes

matthewmueller merged commit 97d9b13 into main Jun 25, 2022

matthewmueller deleted the framework-dir branch June 25, 2022 22:33

This was referenced Jun 26, 2022

framework/view: fix 'bud build' not referencing the correct client-side filepath for hydration. #154

Merged

framework/controller: test that we can pass custom env vars through bud run. closes #151. #159

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental compiler refactor #133

Experimental compiler refactor #133

matthewmueller commented Jun 6, 2022 •

edited

Loading

CLAassistant commented Jun 23, 2022 •

edited

Loading

matthewmueller commented Jun 23, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller Jun 25, 2022

matthewmueller commented Jun 25, 2022 •

edited

Loading

Experimental compiler refactor #133

Experimental compiler refactor #133

Conversation

matthewmueller commented Jun 6, 2022 • edited Loading

CLAassistant commented Jun 23, 2022 • edited Loading

matthewmueller commented Jun 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewmueller commented Jun 25, 2022 • edited Loading

matthewmueller commented Jun 6, 2022 •

edited

Loading

CLAassistant commented Jun 23, 2022 •

edited

Loading

matthewmueller commented Jun 25, 2022 •

edited

Loading