Repository navigation
Replies: 1 comment
|
The design is settled, and the body above now records every decision. It's filed as five stories, each blocked by the one before:
Settled decisions
humus adopts this through a proposal of its own and z5labs/humus#629, as the Migration section says. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Add a nested module,
github.com/z5labs/bedrock/host, that holds the type a framework built on bedrock uses to hand an App the network and the filesystem. The module has three parts:host.Host[F], withListen,Dialand a filesystemFthat each framework declares;host.OSFS, the real machine's filesystem, a struct with a method for every file function of packageos, andhost.OS(), the real machine's Host;hosttest, an in-memory network and a filesystem confined to the test's temp directory, which runs inside atesting/synctestbubble.bedrock defines the type and never passes one.
bedrock.ProcessandApp.Rundon't change, andbedrock.Maindoesn't build a Host. A framework passes the Host in, as humus does today throughSetup.Host.The code already exists. It is humus's
Hostfrom z5labs/humus#630 and humustest's in-memoryHostfrom z5labs/humus#633, moved here. What changes is the filesystem, designed with a second framework in mind (a command-line framework built on bedrock). bedrock fixes what each filesystem method means and ships the implementations, and each framework chooses the methods its users see. A few names change too.Background
bedrock's package doc leaves the host to the framework, in Beyond the boundary:
humus filled that gap:
humus.HostholdsListen,Dialand anFSthat hands out*os.File. The design was settled in the lifecycle review and its revision. It imports onlycontext,io/fs,netandos.humustest.Hostis a network ofnet.Pipeconnections that starts no goroutines, and a real filesystem int.TempDir()confined withos.Root(Proposal: humustest — whole-service testing on top of bedrocktest humus#633). Itshost.go,network.goandfs.goimport only the standard library, plus humus for the two types.Neither file has anything to do with services. They are the second boundary, the one bedrock leaves open, and every framework on bedrock needs one.
A second framework is coming. A command-line framework on bedrock is planned as a project of its own. humus's conventions are a service's (logs on stdout, a grace period sized for Kubernetes, exit codes 0, 1 and 78), and a CLI needs different answers to the same questions. CLIs also touch files and the network more than services do, and their tests need the same hermetic host. Today the CLI framework could only do one of two things, and both are bad:
humus.Host, and importing it links in the OpenTelemetry SDK, every OTLP exporter and grpc-go:go list -depslists 126 gRPC and OTLP packages. It would also tie the CLI framework's design and releases to humus's.OSHostare about 150 lines. The test host is about 760 more (host.go,network.goandfs.go), with about 1,400 lines of tests. Two copies would drift apart, and what drifts is the contract: what a relative name resolves against, which error a missing file returns, when a test's network refuses a dial. A team's shared code, such as a helper that loads its TLS bundle or a client constructor that dials, would then behave one way under humus and another under the CLI framework, and only one of the two test hosts would match the machine it stands in for.bedrock already carries a piece of the same thing.
example/httpservertakesListenas a field, and its test builds a 60-line in-memorypipeListener.The precedent is fit, circuit and retry (#578, #595, #606). Each is something frameworks share, kept in a nested module so the root module stays exactly the process model. In #595's words, bedrock ships the model and the framework ships the wiring. For the Host, bedrock ships the contract and its implementations, and each framework chooses how much of them its users see. bedrock standardizes; humus and the CLI framework are the opinionated layers that decide what to enforce.
Proposed API
A framework declares the filesystem its users see, and an alias hides the type parameter from them. Here is a CLI framework that gives a command five methods:
The framework's test helper builds a Host the same way, from hosttest's
Listen,DialandFS. Both assignments toFSare checked when they compile.An App built on bedrock alone does the same. It declares only the methods it calls and takes the Host as a field, the way
example/httpservertakesListen. Here is a small CLI that downloads a URL into a file, and calls one method:Its test runs the whole App, the server it calls and the file it writes, with no real socket and no file outside the test's directory:
Every block above compiles against a prototype of this API. Written in humus's types, the example and its test also pass with
-racetoday againsthumus.Hostandhumustest.Host.Design decisions
A nested module, not the root module. It follows fit, circuit and retry, so the root module stays exactly the process model. Its library code imports only the standard library. Neither module requires the other, not even for tests, as with fit, circuit and retry, so a framework that wants only the Host gets nothing else. The alternative was
bedrock.Hostnext tobedrock.Process, and it was rejected:os.go, which is today the only file that touches the real process.bedrock defines the Host and never passes one.
Processstays what exec hands a process, andApp.RunandMaindon't change. Each framework decides how an App receives its Host: humus throughSetup.Host, a CLI framework through its own types, and an App on bedrock alone through a field.bedrock fixes the contract, and each framework chooses the surface.
Hostis generic over the filesystem it carries. bedrock defines what every filesystem method means and ships the two implementations,OSFSand hosttest'sFS. A framework declares the interface its users see and hides the type parameter behind an alias,type Host = host.Host[FS]. humus and a CLI framework are opinionated, and each exposes as much of the filesystem as its users need. bedrock isn't, so it doesn't choose for them.OSFSand hosttest'sFSassign to any framework interface they satisfy. A framework that asks for a methodOSFSlacks gets a compile error at the assignment:host.OSFS does not implement FS (missing method Chdir).host.Host[humus.FS]andhost.Host[cli.FS]are different types. Shared code that needs only the network is generic, such asfunc NewClient[F host.FS](h host.Host[F]) *http.Client. Shared code that needs the filesystem takes the smallest interface it calls, such asinterface{ Open(string) (*os.File, error) }, which every framework's FS satisfies.FSinterface in a non-genericHost, and it was rejected. A framework that wanted a different surface would have to declare a Host of its own, or reach its own methods through a type assertion, which is checked only at run time.Accept interfaces, return structs.
OSFSand hosttest'sFSare structs.OSFShas no fields, sovar fsys host.OSFSis the real filesystem and it needs no constructor. bedrock exports no interface of filesystem methods. Each consumer, a framework or an App, declares the methods it calls, asfetchdeclaresOpenFile.OSand hosttest.OS()returnsHost[OSFS], which a framework or an App copies into a Host of its own filesystem, and hosttest'sHosthas noHostmethod.mainand the test each build the App's Host from parts, as in the example above.OSFSinterface with every method, and it was rejected.OSand hosttest would hand out an interface, and a framework that embedded the interface whole would gain every method bedrock adds, whether or not it chose to. A framework that wants every method lists them.FSis sealed. Its one method is unexported, soFSis implemented only in this module. A type elsewhere satisfies it by embedding a value from this module, which is how a test wraps a filesystem to change one method. This is the pattern oftesting.TB, whose private method exists "to prevent users implementing the interface and so future additions to it will not violate Go 1 compatibility". The seal gives two things:OSFSor hosttest'sFS, possibly wrapped, and the conformance table holds the two to the same results. So the compiler backs the decision below that there is no fake filesystem to keep in line with the real one. A filesystem written from scratch is still possible, by embeddinghost.FSitself, but it's a deliberate step, and whoever takes it keeps up with every method by hand.OSFSor hosttest'sFS, possibly wrapped, and both have every method. So when a framework adds a method to its interface, its users' wrappers gain it by promotion. Adding a method toOSFSitself breaks no one, because it's a struct. SoOSFSfollows packageosfrom one Go release to the next, and its method set doesn't have to be settled up front.The alternative was an
anyconstraint, under which any type would do, including a hand-written fake that drifts from the real filesystem.OSFSis every file function of packageos. It has one method for each function inosthat names a file, except those that read or change process-wide state:Chdir,TempDir,UserHomeDir,UserConfigDirandUserCacheDir.Getwdis the one exception, because relative names need it (below).Create,OpenRoot,OpenInRoot,ReadFile,WriteFile,Truncate,CopyFS,Link,Symlink,Readlink,Lstat,Chmod,Chown,Lchown,ChtimesandGetwd.os.Rootcall.os.RootlacksCreateTemp,MkdirTemp,ReadDir,Truncate,CopyFS,OpenInRoot,DirFSandGetwd, so hosttest builds those from the others.os.Rootso that a../entry can't escape. With every method inOSFS, each framework picks what it needs, and bedrock never picks for it.A framework declares its FS in one of two ways, each checked when it compiles:
host.FS, like the CLI framework above. It may list every method ofOSFS. Its surface grows only when the framework adds a method.OSFSwith a method it lacks.OSFSalone no longer satisfies the interface, so the framework wraps it, asstruct{ host.OSFS }plus the method.A framework's FS can't embed the
io/fsinterfaces. Each of them embedsfs.FS, whoseOpenreturnsfs.File, so an interface that listsOpenand embedsfs.ReadDirFSfails withduplicate method Open.DirFSis the way intoio/fs.bedrock never converts a value to
F. Inside a generic function, a value can becomeFonly through a type assertion, which is checked at run time. SoOSreturnsHost[OSFS], hosttest'sFSmethod returns*FS, and a framework or an App builds its own Host from their fields,Host{Listen: h.Listen, Dial: h.Dial, FS: h.FS}, a line the compiler checks. There is nofunc OS[F FS]() Host[F].The rest is as humus settled it in Proposal: the humus App lifecycle humus#630.
Dialthat fails, and keeps the rest.*os.File, not an in-memory fake. Libraries that need a real file or a path keep working in tests, such as SQLite drivers, archive libraries and programs started withos/exec, and there is no fake filesystem to keep in line with the real one. It's a one-way decision: an*os.Fileis always an OS handle, so moving to an in-memory filesystem later would change every signature.OSFSuses plainoscalls. It can't beos.OpenRoot("/"), becauseos.Rootrefuses to follow absolute symbolic links even when it is rooted at/.Getwd, because relative names need a working directory. WithOSFS, relative names resolve against the process's working directory, and with hosttest against the test's directory.filepath.Absreads the real working directory throughos.Getwd, so in a test it produces a path outside the test's directory, which hosttest refuses. Code that needs an absolute path joins the filesystem'sGetwdinstead. There is noChdir, because the working directory is process-wide state. Strictly, the working directory is something exec hands a process, but it means nothing without a filesystem, so it belongs with the filesystem rather than inProcess.hosttest is humustest's
Host, moved. Its rules follow a real network, as Proposal: humustest — whole-service testing on top of bedrocktest humus#633 settled them:0.0.0.0or::) conflicts with any other listener on its port;*net.OpError.Dialrefuses rather than waits. A test that dials before the App listens has dialled too early, and waiting would hide that. The test waits withsynctest.WaitorListeninginstead.hosttest's
FSgets the unexported method by embeddinghost.FSthrough an unexported alias, so the field stays out of its docs. It must not embedOSFSfor the method instead. Every method it forgot to write would then be the real filesystem's, and would compile and escape the test's directory. The same holds for any filesystem that confines another. No exported interface lists every method, so the conformance table checks that hosttest'sFShas them all (see Testing).hosttest's
Symlinkrefuses an absolute target with a*os.LinkErrorwhoseErriserrors.ErrUnsupported.os.Rootwon't follow an absolute link even when it points inside the root, so a link that works underOSFSwould fail under hosttest at its first use, far from the call that made it. Refusing it fails at the cause.os.CopyFScopies a symbolic link withSymlink, so hosttest'sCopyFSrefuses an fsys holding an absolute link the same way. These two are the FS conformance table's one exception.No lint rule for the host boundary. forbidigo doesn't ban
net.Listen,net.Dialoros's file API outsideOSandOSFS. Because the filesystem hands out*os.File, user code works withostypes directly, and a ban would get in its way. humus decided the same in Proposal: the humus App lifecycle humus#630 (open question 9), and bedrock's list covers only the process boundary. A framework that wants the host boundary enforced for its users adds the rule itself.No HTTP client helper. A client over the Host's network is one line,
&http.Transport{DialContext: h.Dial}, and its settings, such as keep-alives and TLS, are the framework's choice. humustest'sHTTPClientstays in humustest.New ambient leaks are documented. The host package doc lists what reads the real machine behind the Host's back:
os.Getwdandfilepath.Abs;http.DefaultTransportandnet.DefaultResolver, which dial the real network;os/exec, which sees the real network and the whole filesystem.bedrock's own Ambient leaks list gains
os.TempDir,os.UserHomeDir,os.UserConfigDirandos.UserCacheDir, which read the real environment. A framework computes them fromp.Environmentinstead.Testing
hostandhosttestis 100.0%, as for every module, and the coverage gate covers the new module.testing/synctestbubbles, and none of them opens a real socket.OS()'s network on loopback, and hosttest's tests run one over hosttest's network inside a synctest bubble. Their certificates come from unexported test helpers that use only the standard library. bedrock exports no certificate helpers.conformancebuild tag, onhost.OS()over real loopback sockets, which needs127.0.0.2on the loopback interface, as Linux has. A second table does the same forOSFS. Each method runs onOSFSin at.TempDir()and on hosttest'sFS, and must give the same result, including a*fs.PathErrorthat names the path the caller passed. The table calls the methods through an unexported interface that lists all ofOSFS's, so it doesn't compile if hosttest'sFSlacks one. The coverage gate fails a method added toOSFSuntil the table calls it. As with bedrocktest's fake signals, the fake can't drift from what it stands in for.host.Host[FS], because the host module doesn't require the bedrock module. The CLI framework's FS and its alias are a whole-file example in host's.Migration from today
CLAUDE.mdcommands and package layout.tag-modules.shalready finds every nestedgo.mod, so releases need nothing.osfunctions above.example/httpserverkeeps itsListenfield. It shows the process model with nothing else, and the root module doesn't depend on nested modules.humus.FSlists every method ofOSFSor a subset, withtype Host = host.Host[FS]; humus hasn't released since v0.15.0, so the old types go without a deprecation period;humustest.Hostis built on hosttest or replaced by it;Telemetry,StartandTestService.Settled questions
OSFSmethod set. The rule stands: everyosfunction that names a file, minus the five that read or change process-wide state, plusGetwd.OpenInRoot,CopyFSandTruncatestay in.host.OS()andhosttest.New(t), nothost.OSHost()orhosttest.NewHost(t).CA. host's own tests cover TLS over both networks (see Testing). A bedrock package of TLS, CA and certificate helpers may deserve a proposal of its own.NewOSFS.OSFShas no fields, sovar fsys host.OSFSis the real filesystem, andOS()stays.Symlinkrefuses them (see Design decisions).Out of scope
Process,App.RunorMain.io.Writerseam in its own code.testing/synctestandtesting/cryptotest.ListenPacket, Unix sockets and name resolution. hosttest's network is TCP, with names compared as written.p.Environment.Chdir, because the working directory is process-wide state.All reactions