Library dependency system

[bgilbert]

Allow me to give a bigger picture for this.

Butane is (becoming) another provisioning/configuration management tools, à la Ansible, Chef or puppet.

Contrary to these tools, it lacks essential features, including the "merge" of different configuration files, because code de-duplication:

• In Ansible, we uses roles (backend: ansible-galaxy) to use in playbooks.
• In Chef, we uses recipes (backend: chef-server/automate) to use in cookbooks.
• In Terraform, we uses modules (backend: Terraform registry) to use in other modules.
• In Python, we uses pip (backend: pypi) to use in python scripts.
• PHP: composer with packagist, Java: gradle with maven, etc, etc.

For now, Butane lacks this kind of “library dependency system”:

1. This issue but also…
2. …the backend to enable it - like for example, a very simple implementation: get dependency libraries from git.
3. Variables. Being able to merge butane files will be quickly limited without variables (Add a substitution pass #111), not only for secrets, but also dynamic values: server configuration moved from static to dynamic. And now, from dynamic to abstracted (IPs changes all the time, naming conventions evolve, server are treated as cattle, security becomes real-time…). How can we managed abstracted machines with a static provisioning tool?
4. …Without workaround. With things like manual variables injection or makefiles - it becomes very hard to explode a butane YAML to its relevent subparts: unit files, configurations files, scripts… When everything is in the butane YAML file, how do you lint and unit test bash scripts? How to you check configuration file syntax?
5. Documented good practices!

I think what is happening with butane/ignition and coreOS is very exciting! Although, I believe Butane is now the heart of coreOS adoption for DevOps and Sysadmins, but compared to other provisioning tools, it’s still a bit dry.

Can you consider not only this feature, but plan for the need to have a proper library management system for Butane?

Originally posted by @gui-don in #118 (comment)

[bgilbert]

Contrary to these tools, it lacks essential features, including the "merge" of different configuration files, because code de-duplication:

• In Ansible, we uses roles (backend: ansible-galaxy) to use in playbooks.
• In Chef, we uses recipes (backend: chef-server/automate) to use in cookbooks.
• In Terraform, we uses modules (backend: Terraform registry) to use in other modules.
• In Python, we uses pip (backend: pypi) to use in python scripts.
• PHP: composer with packagist, Java: gradle with maven, etc, etc.

Butane is currently designed around the assumption that all Butane configs are under the administrator's control. In other words, while a user might copy some config fragments from the Internet, they'll all ultimately be maintained in something like a single Git repo controlled by the user.

That model may or may not be the right one, but I think there are some arguments in favor. Butane is intended as one tool in the toolbox, not as a grand unified configuration management engine. In some cases it'll be integrated into other components in the stack, e.g. Matchbox. And I don't think we want to build Yet Another Tool-Specific Registry Format.

As a practical matter, I'm not sure that Butane configs will be complex enough to merit a lot of generic reuse. Unlike, say, many Ansible systems, user application code should all be running in containers. So, I'd expect the Butane config to be used mostly for disk configuration, some systemd services, and maybe a few host config files - and not much more.

Do you have an example where this sort of mechanism would be especially useful?

3. Variables. Being able to merge butane files will be quickly limited without variables (Add a substitution pass #111), not only for secrets, but also dynamic values: server configuration moved from static to dynamic. And now, from dynamic to abstracted (IPs changes all the time, naming conventions evolve, server are treated as cattle, security becomes real-time…). How can we managed abstracted machines with a static provisioning tool?

We don't document this very well right now, but Afterburn metadata is the intended mechanism for node-specific network configuration.

4. …Without workaround. With things like manual variables injection or makefiles - it becomes very hard to explode a butane YAML to its relevent subparts: unit files, configurations files, scripts… When everything is in the butane YAML file, how do you lint and unit test bash scripts? How to you check configuration file syntax?

For files that also need to be processed by external tools, you can use --files-dir and the local or trees directive. I can see how that could be unsatisfactory, though, especially for systemd units where local inclusions are not possible. In principle we could add a simulation mode that generates a directory tree.

5. Documented good practices!

I'm in favor! Any particular gaps you'd like to call out?

[ghost]

Hi @bgilbert! Thanks for your answer.

Butane is currently designed around the assumption that all Butane configs are under the administrator's control. In other words, while a user might copy some config fragments from the Internet, they'll all ultimately be maintained in something like a single Git repo controlled by the user.

My main concern is code duplication. To solve it, being able to reuse snippets of code (on the internet, maybe, but mainly my own snippets of code). For now, we can use the merge feature with ignition files, but because Ignition is encoded code, we cannot pass variables dynamically. More on that later.

That model may or may not be the right one, but I think there are some arguments in favor. Butane is intended as one tool in the toolbox, not as a grand unified configuration management engine. In some cases it'll be integrated into other components in the stack, e.g. Matchbox. And I don't think we want to build Yet Another Tool-Specific Registry Format.

I get that. Maybe butane will not be the answer for my need.
However, for now there is no other options to provision CoreOS machines: we cannot use Ansible, Chef or Saltstack with CoreOS. Such tools would need to run in a docker container, which is awkward for a provisioner. Plus, CoreOS is a immutable system, while the provisioning tools above works mainly for mutable systems.

As a practical matter, I'm not sure that Butane configs will be complex enough to merit a lot of generic reuse. Unlike, say, many Ansible systems, user application code should all be running in containers. So, I'd expect the Butane config to be used mostly for disk configuration, some systemd services, and maybe a few host config files - and not much more.

That is a good argument. However, consider how many services needs to be setup in a state-of-the-art machine today:

• basic configuration (disks, networks, users: the basics)
• the service itself (in a docker container, still, needs config - or config file - to be passed + 1 system unit minimum)
• health checks
• monitoring
• centralized logging
• security watchdogs
• some cloud specific tools: like AWS agent

I believe this is quite a lot!
Now, let my give you my situation (that I believe will be shared with many people).

Some important predicates:

• I want CoreOS. RO system and OS tree automated updates are just to good not to use.
• I'm gonna manage 30+ machines, all in the cloud.
• I want and I have automated pipelines, so I need the provisioning process to occur without human intervention but code.

Let’s say I have 30+ different-purpose web workers, here is what I would need to configure on each of them:

• Users = butane basics (might be the same, might not)
• service = unit+ configuration (different all the time)
• health checks = unit + configuration + script (might be sometimes the same on some machines)
• monitoring = unit + configuration system + configuration service (same unit unit/configuration system, different configuration/service)
• centralized logging = unit + configuration system + configuration service (same unit unit/configuration system, different configuration/service)
• security watchdogs = unit + configuration (same everywhere with minor configuration changes)
• cloud specific tools = unit (same everywhere)

That’s not all. As you know:

• companies need some kind of separation by environment: prod, staging, dev, etc.
• companies have teams/projects, requiring different tools, but also, sometimes the same tools, configured slightly differently.

Because of this "spaghetti bowl" compartmentalization, handling that many machines in the same git repository is simply not working in the long run (illustrated by the move towards micro services). That’s why I've been working for years with a highly granular code, each single machine having its own repository. Automation solving the problem of maintenance of that many repositories.

With any other tool, libraries offers what I need: reusable, generic (or business-generic: generic but for my company) pieces of code, that I glue together with context variables in a repository representing a machine. The benefits are huge: every single component can be linted and tested independently and versioned making the overall system very robust:

• It’s easy to deploy a test machine of a service and do QA on it.
• Bugs are less frequent because each small component is tested automatically and independently from each others
• It’s easy to rollback, because each component of the system is versioned.

With Butane… It is hard to deal with without a dependency system. I believe it’s gonna be an adoption problem for CoreOS, especially in companies. While a lot of people do static configurations and duplicate their code, it’s a huge brake for companies growth as the same things are repeated again and again.

We don't document this very well right now, but Afterburn metadata is the intended mechanism for node-specific network configuration.

It’s not only network configuration that needs to be passed. Secrets are one other important thing but also some flat values that are defined elsewhere. For example, I might define application ports, domain names, configuration values that I want to toggle, firewall rules, etc in infrastructure code (CloudFormation, Terraform…). These values are sometimes known at runtime, so I have no choice but pass them to Butane at runtime. Right now, I use Terraform templating system to inject these values but, it forces me to have a single butane file, otherwise needing to know what variables is in what file, which is a nightmare to maintain as it breaks separation of concerns.

If Butane itself would accept variables as inputs and would be able to replace them before compile, it would change everything: no need to use any external template system and I would be able to split the butane code to simplify linting and unit testing.

Documented good practices!

First and foremost good practice is code duplication avoidance because consequences are dire! I was mentioning that if a dependency system is implemented in Butane one day. Technicalities about such system would surely be documented, but good practices are often overlooked. Ansible is a very bad example in the matter: it’s really hard to know how to do things correctly just be reading the documentation.

---

That was a very long answer, I hope it will spark more discussions. I'll be happy to clarify some points if required. Also, if you see the purpose of all that but believe it should not belong to Butane itself, what can I leverage to make this happen in this community?

[bgilbert]

Thanks for the detailed response; that's very helpful.

Your usage model is largely aligned with how we think about FCOS (immutable cloud nodes with automated provisioning and declarative configuration), but interestingly, you're using individually configured "pet nodes". We think of large-scale FCOS deployment in terms of undifferentiated "cattle nodes", each of which can have arbitrary workloads scheduled onto them via some orchestrator (which is often but not necessarily Kubernetes). On the Butane side, it means the config matrix is typically environment x node-type, independent of workload. In your case, you're using Butane as a central component of your orchestration, so your configs also need to know about workloads. That puts more pressure on the Butane design.

You can still use Butane in your model, but you'll have to add some infrastructure around it. If I were building the system you're describing with today's Butane, my first cut would use Butane + Git submodules + some templating system (maybe just envsubst) + Make. (I don't know the Terraform templating system, but I assume there's no inherent reason templating couldn't be applied to multiple source files before transpilation.) This currently isn't very ergonomic (because it requires external templating and transpilation of each file separately) but everything you're describing should be possible today. That's what I meant when I said that Butane is one tool in the toolbox: other tools or custom glue can integrate Butane into use cases not contemplated by the design.

We should still have a useful set of features, though. I think you're asking for three things: building a config from smaller ones (#118) that are maintained in separate repositories (this bug) and templated with context variables (#111). I think both #111 and #118 are worth pursuing in some form, for improved ergonomics. I'd like more information on the separate repo part, because I don't think Butane should handle this; git submodule does a better job than we could. Is there some reason existing tools couldn't handle that for you?

In any event, we should certainly document a suggested workflow.

[ghost]

Hello @bgilbert (sorry for the late answer)

but interestingly, you're using individually configured "pet nodes". We think of large-scale FCOS deployment in terms of undifferentiated "cattle nodes"

That’s correct.
My intention is to go there, with either Nomad or Kubernetes. When I'm pondering the container orchestration end goal with what I have (old infrastructure), CoreOS gives this interesting middle ground, like a step between "old" mutable servers and an unified platform for containers.

If I do go full modern infrastructure, it’s hard to see if I will not hit a wall, not because I do not trust these tools to work, but I'm afraid the software I migrate might not be mature enough.
Butane/CoreOS still gives me the ability to check my goals: resilience, HA, 100% automation, fault tolerance and security. So, if workloads runs on CoreOS for months, migrating to Nomad or K8S becomes background work - nobody waits for anymore feature (I really see Nomad or K8S as tools to save money: "more workloads on less compute", and also, no more hard bindings to expensive public cloud services ;) )

In essence, I think that immutable-like OS like CoreOS and others are too good not to use, even in a kind of "pet" mode as you put it.

I'd like more information on the separate repo part, because I don't think Butane should handle this; git submodule does a better job than we could. Is there some reason existing tools couldn't handle that for you?

As far as I know git submodules will not handle pinning properly.

The main (huge) advantage of separate repos is to be able to pins, hence, facilitate lifecycle of the code: try an update by pinning new version? Introduced some bugs? Rollback the pin. I'm not just talking about one repo depending on another one: but potentially 20 repos depending on 20, depending on 20… Any change in a transitive dependency can break several parts, let’s say, when someone uses a "feature" that was not intended originally (the linux kernel have this problem a LOT). Again: by pinning, it’s possible to let repositories that are ready to update while giving the time for everyone to fix their things before updating: the agility to test, update and rollback easily.
With a single repository, a single failing update is a blocker. The more dependencies I have - that is to say, the more i do my job well and de-duplicate the code - the more likely it will not be able to update anything. In real life, it means every change of code is a very risky, long and painful task… For instance, when dep code is changed: how to deal with code that uses dep in an unexpected way? How to deal with legacy code that uses dep? How to deal with another team that want to include dep in their code? Pinning/Tagging is a convenient solutions to this problem.

I'm pretty sure this is debatable as it’s mainly from my personal experience, but so far I verified that in many context.

For me, multi repositories have two advantages: be able to handle tags and pinning (as I demonstrate here) and to fine-tune users permissions in multi-team complex scenarios.

---

After some months, here is how we solve the issues I mentioned:

We create "snippets" of Butane code, one per repository, that we tag. A pipepline deploy each Butane repository as a pinned artifact.
In a Terraform pipeline, each workload code fetches all the artifacts needed to achieve a full-OS configuration depending on the need.
The Butane "snippets" (really: Terraform templates) are compiled together - with dynamic variables injection - using a Terraform provider and Terraform templates. Each service then have its own, final ignition file that is used to boot-up VMs.

While it achieves what Iwe want to do, the big drawback that it’s very dependent on Terraform, the Terraform provider and a rather complex pipeline (manual artifact fetching).

[bgilbert]

Thanks for the additional info. What specifically about the git submodule system won't work for you? Submodules are pinned to a specific commit of the submodule repo, and the pin is moved by making a new commit that updates the pin object.

[ghost]

We find it hard to maintain inter-dependencies, updates, having to use many command lines to make sure the submodule is properly updated. I’ve been working on medium-size projects with 50 plus dependencies. Just imagine having to set them up manually instead of just maintaining a requirements.txt, or composer.json or gradle files…
Moreover, I don't really think it’s possible to pin properly a submodule. Sure, you can submodue add and checkout tag, but that won't be save in the parent repository, so basicllay, it means I have to create my own system to properly pin the submodules. In .gitsubmodules it’s not possible to pin a tag. Unless you know a way that I'm not aware of…
Last but not least, pinning is often handled by less technical people. Where I come from, code is written by highly skilled individuals but maintenance pinning for updates might be done by juniors or even people unfamiliar with code. Changing a version in a file is easy. Changing a submodule branch/tag requires heavy documentation.

I see git submodule as a neat solution when you have to split code into multiple repositories of a single application, not to simulate a dependency system. As a consequence it lacks all the interesting features of the such systems:

• transitive dependency discovery (recursive)
• efficiency (dependency only downloaded once and dev VS prod dependencies)
• easy pinning (pinning is visible the code; pinned ONCE, no duplication + sometimes floating pins (e.g; > 3.1 for all 3.1.X versions)
• easy, single-step, CI.CD friendly command lines (composer install; pip install -d; yarn; gradle, etc)
• authentication to private code repositories (pip, packagist, npm, maven…)
• …

Just to be clear: I'm not imagining all the the above for Butane. I believe that just the ability to call "snippets" of code with some kind of inheritance+ transitive dependency discovery would already be amazing. A dummy code example:

variant: fcos
version: 1.6.0
snippets:
   -
    name: healthcheck-system
    engine: git
    url: https;//gitrepo/my/git.butane-dep.git
    ref: 3.2.3
   -
    name: monitoring
    engine: git
    url: https;//gitrepo/my/git.monitoring-dep.git
    ref: 2

(engine is an implementation of a code repository. git would be largely sufficient for a long time)

I know it’s a lot more that it appears to be on this code example but:

• it would greatly facilitate de-deduplication of code;
• encourage code reuse (e.g; imagine having public butane snippets to handle this logging tool or that monitoring tool…)
• offer an official way of splitting butane code (submodule still looks a lot "hacky" compared to that)
• allow CI/CD testing because any butane code would mention all its dependencies. So, every butane code is guaranteed to be deployable/testable without any manual copy pasting or submodule setup to work.
• remove the need of home-made external tools to do the glue between snippets of butane code.

[bgilbert]

Moreover, I don't really think it’s possible to pin properly a submodule. Sure, you can submodue add and checkout tag, but that won't be save in the parent repository

Yes, it will. The parent repository stores the intended commit hash of each submodule.

Last but not least, pinning is often handled by less technical people. Where I come from, code is written by highly skilled individuals but maintenance pinning for updates might be done by juniors or even people unfamiliar with code. Changing a version in a file is easy. Changing a submodule branch/tag requires heavy documentation.

One possibility is to write some wrapper code that provides a friendlier interface. You're right, though, that certain features like floating pins can't be handled by submodules.

Just to be clear: I'm not imagining all the the above for Butane. I believe that just the ability to call "snippets" of code with some kind of inheritance+ transitive dependency discovery would already be amazing.

#118 covers inheritance between Butane configs, and I do think we should pursue that. I believe the separate repository handling can be handled effiectively by another tool, with a separate config file, that clones the correct repositories and checks out the appropriate commits. (The tool could reduce disk space requirements by keeping a central archive of repos and creating shared clones as needed.) For example:

main.bu:

variant: fcos
version: 1.6.0
ignition:
  config:
    merge:
      - local_butane: healthcheck-system/main.bu
      - local_butane: monitoring/main.bu

modules.yaml:

modules:
   -
    name: healthcheck-system
    engine: git
    url: https;//gitrepo/my/git.butane-dep.git
    ref: 3.2.3
   -
    name: monitoring
    engine: git
    url: https;//gitrepo/my/git.monitoring-dep.git
    ref: 2

Conveniently, the tool that reads modules.yaml doesn't actually need to understand Butane at all.

(Though, I've been assuming that child Butane configs would inherit the --files-dir of their parent. That wouldn't work for this use case, and may need some more thought.)

In general, the workflow you're describing is substantially different from the primary way we expect Butane to be used (xref #301 (comment)), and the functionality you're looking for would take a substantial amount of work to implement. This might be worth doing, but is not going to be on the Butane roadmap for a long time, if ever. To avoid getting anyone's hopes up who may run across this ticket, I'll go ahead and close it.

I really do appreciate the discussion. I'd strongly encourage you to either use an existing general-purpose dependency management system, if there's a suitable one, or to consider writing and releasing a tool that meets your requirements. That tool could even offer native integration with Butane if that would be useful.

[ghost]

Understood. 👍
Our workflow and usage will also improve during the following months. I may come back to discuss other features.

Many thanks for listening!