Composite Plan/Package/Service

[reset]

User Story

As a plan author, I need a way to represent a service which is made up of one or more processes, so I can interact with a collection of services in the same Supervisor as one

Background

We currently have one kind of plan which is represented by a plan.sh or a plan.ps1. A plan outputs a package which, when installed, can be started by a Supervisor to start a service. It can be said that a plan/package is a 1:1 mapping to a service.

Some services require another to run and we need a way to represent this "sidecar" pattern. An example of this is an API server and an API proxy.

Changes to Plan-Build

We need a way to represent that the plan is a composite plan. We could do this in a few different ways such as:

• Add a new key/value setting which plan-build reads such as pkg_composite = true
• Use a different filename than plan.sh such as composite-plan.sh

We also do not need to allow any additional artifacts aside from the plan.sh in a composite plan. There's no need for the additional artifacts - read on for details.

Promise Mapping

Once we've indicated that we're building a composite plan we also need a way to configure some mappings for any promises that the package exposes (aka, exports/binds). We also need a way to note that an export of a service of the composite plan satisfies a bind of another. For example, builder-api needs to satisfy the bind for builder-api-proxy.

We need to verify that there are no collisions in the key space of exports or binds. If one package in the composite exports hello and a second also does, we will need to fail the build (and should fail before any work is done)

We will want to record this in a new METADATA file indicating:

• That this is a composite package
• What the promise mappings are
• Which promise mappings self satisfy service members

Changes to Hab CLI

All operations with the CLI should be atomic across all service members of the composite package/service. For example: installing a composite package should install all packages it points to. Another way to think about a composite package is as a "pointer package" (sometimes referred to in other systems as a virtual package).

We may want to consider representing a composite service in the status output of running services as something like:

$ hab svc status
core/builder-api/4567/20170727171959, state:up, time:PT1.870780743S, pid:9199, group:builder-api.acceptance, style:persistent
  - core/builder-api-srv/4567/20170727171959
  - core/builder-api-proxy/4355/20170706225642

All hab pkg and hab svc commands should be refreshed to support these changes

Config Apply

Since we want a composite service to be addressable as a service group itself, we will also need to make some alterations to hab config apply.

Currently we run hab config apply <SERVICE_GROUP> <VERSION_NUMER> [file] which takes a toml file representing a single service's configuration. We should allow the toml file to be split into multiple sections, one for each service in the composite service:

[builder-api-srv]
  [http]
  listen = "0.0.0.0"
  port = 9636
[builder-api-proxy]
  worker_rlimit_nofile = 8192
  worker_processes = "auto"

  [events]
  worker_connections = 8000

Changes to Butterfly / Supervisor

With all of the above changes we may also need to add a new rumor type, CompositeServiceRumor, which contains the metadata for promise mappings.

The config apply changes described above may also result in the need for us to generate a new type of rumor for CompositeServiceConfigRumor. This would contain the combined configuration. If we decide to go this route, we will need to make sure that we reject config changes to the individual service groups which make up the composite service group - contains a combined

Lastly, we should not generate a service directory at /hab/svc/{name} for a composite service.

[christophermaier]

Ongoing notes:

• Ensure parity between Windows and Linux, particularly w/r/t any changes to plan building.

[adamhjk]

Do not create a separate plan file, is my strong feeling.

[christophermaier]

@adamhjk Not that I necessarily disagree, but could you elaborate on your reasons?

[reset]

My preference right now is the same file as well but I hadn't thought about it long enough to really double down on that opinion. The concerns that I have so far are what happens if there's both a plan.sh and composite-plan.sh file and the changes to any tooling that we create which will think of plan.sh as main() but now we may have two main()s

[chrisortman]

I'm not sure that I fully understand the API / API proxy scenario, but I've been trying to conceive how something like this might work for a rails application that has application sever processes and delayed_job processes. I keep thinking I'd want it to work just the way everything does today except that the run hook sees a proc file and starts processes based on what's in there

[bodymindarts]

I don't see why this should be in the same file. The plan.sh is already complecting build instructions with runtime behavior. This would be adding another responsibility overloading it even more. In nomad the are distinct concepts of job, group and task. A single service would correspond to a task and a composite plan would be a group. In kubernetes the pod is also a distinct concept that aggregated multiple sub containers. My point is since other systems consider a group of things to be something different than an element in the group I'm not sure if this approach is right. Using the composite pattern which treats a plan or an aggregation of multiple sub plans identically from the pov of collaborators might make sense but even so I'm not sure if the definition of the aggregation belongs in the Dame file. /brain dump out

…

On Aug 10, 2017 8:32 PM, "Jamie Winsor" ***@***.***> wrote: My preference right now is the same file as well but I hadn't thought about it long enough to really double down on that opinion. The concerns that I have so far are what happens if there's both a plan.sh and composite-plan.sh file and the changes to any tooling that we create which will think of plan.sh as main() but now we may have two main()s — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2902 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AII1MEw33iAr0PgtWNfZWACkTnMa-aFIks5sW0y0gaJpZM4Oyz86> .

[adamhjk]

Right now, we have a 1:1 mapping between a given package and a service it can run.

What we want to be able to say is that we want a 1:N mapping between a packages and services it can run?

Lets say I have a package adam/francisco, which has two services - francisco-api and francisco-db. When I say:

hab start adam/francisco

Is the expected behavior here that I would get two services, one named adam/francisco-api and another named adam/francisco-db? Or would you need to launch them separately?

Another way to look at this is that you want to have a completely different animal, akin to a procfile. All it would do is list a set of packages you want run, and let you get the UX behavior I say above. In that case, you would have:

• adam/francisco-core
• adam/francisco-api -> adam/francisco-core
• adam/francisco-db -> adam/francisco-core
• adam/francisco -> composite(adam/francisco-api adam/francisco-db)

Are we in the same spot? Which is it?

[christophermaier]

@adamhjk My understanding is that it's more like the first scenario you describe. When you load your adam/francisco "package", that's largely saying "load up adam/francisco-api and adam/francisco-db and run both of those services on the same machine". You interact with them in terms of the top-level adam/francisco "service", as opposed to manipulating the API and DB services individually.

[chrisortman]

In this example api & db feel like two separate unrelated things? I haven't had any experiences yet where I've wanted this. I think I only want this sort of behavior when the services are all running off the same codebase. With a rails app and delayed_job it doesn't make much sense to have two packages because it's all the same executables, just different startup params. I think the reality of needing to copy all the app scripts & config files and get them set up in /hab/svc really ?highlights? this

For this instead of a composite plan I'd be happy with parameterized run hooks and maybe some more smarts in init....

hab start adam/francisco-api --run web
hab start adam/fancisco-api --run jobs

ls /hab/svc/francisco-api/hooks
init
run -> web/run
web\run
jobs\run

This would still leave you the capability to run different pieces on different servers.

[eeyun]

Yep I think @chrisortman is right. I was halfway through drafting a similar post. The pain is much more apparent for that sidecar pattern or the rails foreman pattern.

We don't really have an issue with wrapping things up, or with starting different things and getting them to talk to one another. What we DO have pain with is when your app has a dependency on itself or on a sidecar service that functions in support of your running thing. This is why php users will see this a lot.

The examples I've run into recently are with the nomad package and with some work I did with one of our community members in atlanta. Nomad for example - the binary runs in multiple varied states but theres only a single binary. In one case the nomad binary can run in agent mode, but doing so means that dockerd must be running on the underlying system. It's trivial to add a line to my runhook that starts and backgrounds dockerd, but then I lose control over that process.

With the folks in atlanta they were simply trying to package up a very lightweigt webapp with nginx + node to serve static content. The challenge they ran into was pretty similar.

I also find myself wondering if focusing on the way we read/handle run-hooks is the appropriate path. For example: if I could have run-dockerd, and run-nomad or run_1 and run_2 and the supervisor could start and watch each process correctly without a need for some extra layer on the plan that might be suitable enough.

[eeyun]

Or another idea: a clean way for the user to specify in the run hook itself: start this other package first. or start this package in this separate context first before executing the rest of this run hook

[robbkidd]

I have the use case described most-closely by @chrisortman: a single Rails codebase with both web and jobs services. The only difference between the services is the run hook. What's shared between the two services includes configuration: their templates and computed final configs should be identical.

[mwrock]

It seems like there are a couple patterns at play here and I'm sure there are even more in the world and all are very valid. A couple things that come to my mind here:

• We can currently declare runtime and build time deps. What if we added a svc_deps that would be similar to runtime deps but in addition to grabbing the dep, basically wrapped a hab svc start around that package. I'd imagine you would end up with a tiny "meta" package that would typically just have a name, origin and svc_deps.

• For cases where you would want multiple services that allm run the same binary but with different args. Would it suffice to have a single plan for building that binary and then different plans that tyook a runtime dep on the binary and used it with its own specific args in the run hook

[chrisortman]

@mwrock I think I'm doing something like what you are suggesting in your 2nd bullet right now, but not with a binary

https://github.com/ui-icts/sparc-request/tree/master/habitat

I think my primary gripe about the set up is that when the dependent packages need / get restarted doesn't feel deterministic. I think it works most of the time for me by coincidence, but one of my first troubleshooting steps is to stop/start the web & jobs packages

[reset]

@eeyun this feature is exactly that, a way to represent the sidecar pattern 😄

[eeyun]

Thats what I figured @reset. I guess from my perspective just the DB example is a little misleading if only because lots of times having a DB connection is trivial to orchestrate and they aren't usually to hard to manage with the current habitat behaviors. Those paired services with web-server+static content, or service + supporting service are harder to compose currently.