Question: is there a method to mount some files/directories from current working directory into the container?

[Jamesits]

I'm testing the ability of bass to do aggressive caching on a large project that need to be compiled multiple times, and within one step of the compilation, minor predictable changes need to be applied (some constants in a certain file needs to be changed every time). I'm thinking of generating the required files locally, then mounting the files into the container with with-mount <trunk> ./local-dir ./src) but it seems to result in a empty directory being mounted. Is there a method to achieve this, or by design bass require a fixed source from a fixed input?

BTW, this is my first time writing that much Lisp, feel free to point out if I've done something wrong.

[vito]

EDIT: This response was only accurate for about a week. See #6 (comment).

Hey!

(Bookkeeping: converted to a Discussion)

You guessed it:

or by design bass require a fixed source from a fixed input?

Bass's focus is building reproducible artifacts, so it maintains a pretty strict boundary between the script and the host machine. There's no way to write to the host machine, there's no way to read arbitrary files (besides (load)) or environment variables, and there's no way to propagate them to thunks. The goal is for Bass scripts to always produce the same results regardless of where they run. Right now Bass only has the Docker runtime, so it's tempting to try to feed it local data, but someday soon I'd like to implement a Kubernetes and/or Nomad runtime for running thunks across a cluster.

Jury's out as to whether this can work in the long run - it's all experimental, but this is a pretty core aspect of the experiment.

To explain part of your question, (with-mount <thunk> ./foo/ ./bar/) (two local paths) creates a bind-mount within the container. It's typically used to just make sure a file or directory exists for writing data prior to running a command which writes to it. Here's a test demonstrating it:

bass/runtimes/testdata/mount-local.bass

Lines 1 to 5 in 585f9ba

	(def create
	(-> ($ sh -c "echo 1 >> ./file && echo 2 >> ./dir/subfile")
	(in-image "alpine")
	(with-mount ./dir/ ./dir/)
	(with-mount ./file ./file)))

Back to the original ask: I'm not sure what kind of build flow you're going for, but it might be worth pointing out that Bass is meant to be the engine that runs your various build and delivery tools which themselves act on local files (mounted in by Bass). It doesn't replace tools like Bazel, make, and so on - it just forces you to make the entire toolchain reproducible. The caching is also pretty coarse; it can't be used to make existing build tools faster, it can only be used to prevent redundant builds of the exact same thunk.

Hope this helps!

[Jamesits]

I'm running a research project with PLCT which aims to track how software (compiler variants, compiler configuration, OS, library) changes impact the real-world performance of existing RISC-V processors in a reproducible way. The process involves recompiling a lot different things with a Cartesian product of pre-defined changes then run the artifacts on the ~600 boards of different vendors and a lot software emulators we currently have.

And the worst thing is: One can't just modify one const in any piece of code to be a variable so I can supply its value at runtime. For embedded systems, making any one thing dynamic will affect everything else, from compiler's loop optimization/unrolling to CPU's cache hit rate and branch prediction result. Having a loop running X times is a lot different than from having it running 100 times or 200 times.

This is a really weird use case for any existent software. I've surveyed a lot CI/CD related tools, but:

• Few CI tools can handle a matrix of >1000 parallel tasks in a single job and have ~100 jobs queued at the same time without having performance problems either on the scheduler or on the UI.
• Most of them are designed to fetch the sources from a server (some git server for example), but because the permutations are generated by code in batch, creating one commit for every permutation we have is not very realistic.
• I really want aggressive build caching, because it saves time (a lot!). But existent CI runners do caching at task level, so if I run the whole build process in one task, the cache is useless. If I split different commands into different tasks, then there will be more problems on the scheduler performance.

These thoughts gradually led me here. I think Bass has the correct abstraction on doing reproducible cacheable builds, but I still need to have a limited configuration interface so I can change the result one variable at a time, and have an interface like Concourse's fly execute without needing to create a commit beforehand. Maybe I'll have the time to write my own little tool mimicking Bass's approach but with the capabilities required by my project.

Really thank you for the explanation!

[vito]

No problem! Thanks for the details, it helps put things in perspective.

That sounds like a pretty interesting use case. I can sort of see how I might approach that with Bass, but it's a little blurry and some parts just aren't there yet.

It's not implemented yet, but Bass's multi-platform design should support running across an arbitrary number of machines - assuming they're able to run Bass, which might be an adventure of its own with that many variants. The basic plan is to implement a runtime that just runs bass commands over SSH. Pretty barebones, but keeps things stateless and flexible (each machine can configure its own runtime pool, possibly using its local network). I did a spike on this a while back and had Windows <=> Linux interop working, but I haven't pulled that work in properly yet. Interested in hearing other ideas there.

Some thoughts point-by-point, though you've probably already been through this since you're here:

• Since Bass is just an interpreter that runs on the local machine, there's way less overhead compared to a stateful CI service, but I haven't done any large-scale tests. I'm sure there's some low-hanging fruit and "interesting" performance characteristics I haven't had to think about yet. The progress UI is also pretty jank when a ton of things run - gonna be working on that today.
• Generating code and triggering other thunks should work pretty well - Bass doesn't really care where artifacts and values are coming from, it's just running commands in containers and passing data around. You should be able to run a thunk that emits a JSON stream, iterate over the values, and pass them to other thunks. Explicit parallelism isn't supported yet though - not sure how I want that to look at the language level.
• Bass's caching might be enough since there's less of a penalty for running separate thunks. You might even be able to do pretty fine-grained caching by having a thunk that just warms the cache, and then pass that path to other thunks that use it.

As for a configuration interface, one simple thing you can do today is just (def) local values and (load) them into your script like so:

bass/demos/actions-checkout.bass

Lines 1 to 7 in 585f9ba

	(import (load (*dir*/secrets))
	*access-token*)
	(import (load (.actions {:GITHUB_REPOSITORY "vito/bass"
	:GITHUB_TOKEN *access-token*}))
	load-action
	run-action)

It's a little clumsy if you're trying to run with a bunch of different values interactively, since you have to keep updating the file, though I guess you could just do a loop in code too. To make this better I think it'd be reasonable to support a fly execute type interaction for passing simple values in, since those still fit reproducible builds.

(Sidebar: there needs to be better support for passing secrets in safely - I don't want them to end up embedded in the JSON artifact thunks that people would want to publish.)

If you keep tinkering with Bass please feel free to open issues for any confusing error messages you find. I'd like for them to have Elm-tier helpfulness, but haven't had time to go over them all yet.

Also feel free to yoink parts of Bass for your own tool - it could probably be split into separate components (runtime vs language) but I'm just keeping things coupled for now to support rapid iteration.

[vito]

Just an update, I'm considering adding support for passing host paths to thunks. I have a working proof of concept that I might just push once it's properly tested, etc. (edit: pushed to wip-host-paths)

Following 9b78421, images can be built and used in code without ever having to be pushed to a registry. Being freed from managing images in a registry really makes it feel a lot nicer for local development, and it feels like a natural next step to extend that to source code, too.

Here's how it works: to pass a local directory to thunks, create a Bass script in it and define a binding for *dir*. Modules that need it can then (load) the script relative to their own *dir*.

This way you don't have to pass context dir paths to every bass command you run in the workspace (everything is in code), and it's still hard for a Bass script to access arbitrary files, since they'll have to exactly name a script somewhere relative to its own *dir* on the host machine.

One trade-off is that for now Bass only supports whatever platforms Buildkit supports. But overall it feels a lot more useful.

[Jamesits]

This is very interesting! I'm going to try the new branch today.

[vito]

Cool! Fyi you'll need buildkitd running now - there are scripts under hack/ that should help. Planning to make it more seamless in the future.

[vito]

Merged this, along with cache mounts, in #16

[vito]

Marked this as the answer since it's more accurate than the previous reply now.