Skip to content

Deployment Basics

Jump to bottom
Andrew MacGaffey edited this page Aug 16, 2026 · 14 revisions

Deployment: Basics

The Quick Start brought a whole system up with a couple of copy-paste commands. This page explains what was actually underneath that - what an Elastic MDS deployment is made of, and how you build and configure one for your own environment.

Audience: operators standing up a deployment (and architects who want the model). This page covers the deployment model and how you bring up and operate a deployment; production-grade topologies, networking modes, and scaling are covered in Deployment: Advanced.

What a deployment is made of

A deployment is a set of services you run - the same base images and the same deployment-config/, whether you run them conventionally as native host processes, under Docker, or under Kubernetes. Three key concepts:

  • Base images. MetaFluent assembles various combinations of software features into container images and publishes them to the GitHub Container Registry (ghcr.io). A base image is stock and unconfigured - it contains the software but knows nothing about your deployment: your hosts, your data sources, your tuning. It is not meant to be run as-is. The Image Catalog lists the available images and what each is for.
  • Deployment projects. A deployment project (deploy-mf-*) is a Git repository you clone. It holds a default configuration for a base image, along with the scripts to operate it; you adapt that configuration for your needs. This is the unit you operate - the deployment project is how a stock base image becomes a configured, runnable service.
  • Compositions. A composition describes how a set of services is wired together and brought up as a running system. The Docker Compose file behind the Demo entry in the Deployment Cookbook - the one the Quick Start used - is a Docker Compose composition. Compositions are worked templates for the deployment topologies - a starting point you adapt for development, evaluation, and single-host use; production at scale runs the same service catalog on Kubernetes.

Which base images - and therefore which deployment projects - you need depends on the topology you are building, and a topology is driven by a few key decisions: whether it needs to scale, whether it uses host networking, and how resilient it must be and at what granularity. Those decisions, and the production-grade (including Kubernetes) topologies that follow from them, are the subject of Deployment: Advanced.

Anatomy of a deployment project

Clone a deployment project and you get a small, predictable layout:

deploy-mf-<service>/
  deployment-config/
    deployment.properties     <- your primary configuration
    resources/                <- configuration resources; add or adapt files here
    overrides/                <- generated; do not edit
  scripts/
    start                     <- run the configured service
    stop                      <- stop it
    bake                      <- build a self-contained, configured image
    config-merge              <- reconcile your config with an upstream release
  README.md

The file you edit most often is deployment-config/deployment.properties - the settings that make the stock image your deployment. You may also add or adapt files under resources/, for configuration resources your deployment needs. The overrides/ directory is generated by the build - leave it alone; it is rewritten when you take an upstream update.

A base image ships with sensible defaults, and it is a deliberate goal that as much as possible you can run on stock configuration - adapting deployment.properties (and resources/) only where your environment genuinely differs. What the settings mean and how configuration works is the subject of Configuration: Basics; this page is about operating and packaging a deployment, not the configuration model itself.

Setting up a DNS entry for your API Gateway

Everything you do to monitor and control a running system goes through its API gateway, addressed throughout the documentation - and by built-in tools such as sim-ctl - by the convention name mf-api-gateway on port 9090. The Quick Start mapped that name to 127.0.0.1 in /etc/hosts because everything ran on one machine. A real deployment needs the name to resolve from wherever your clients and operators are.

The gateway listens on port 9090 on its host. Make mf-api-gateway resolve to that host's address in one of two ways:

  • A DNS A record (preferred). Add mf-api-gateway to your internal DNS, pointing at the gateway host's IP address. Every client and operator on the network then resolves it automatically, and every http://mf-api-gateway:9090/... command in these docs works unchanged from any machine.

  • Per-client /etc/hosts. Where you cannot (or need not) touch DNS, add the same mapping to /etc/hosts on each machine that talks to the gateway:

    echo "<gateway-host-ip> mf-api-gateway" | sudo tee -a /etc/hosts

Either way, the point is that the convention name - not localhost, not a raw IP - is what the docs and tools expect, so using it keeps every command portable across machines and deployments. (A topology with more than one gateway is a Deployment: Advanced concern.)

Running a deployment

A deployment project runs three ways - conventionally as a native host process, under Docker, or under Kubernetes - the same base image and the same deployment-config/ in each case, only the launch mechanism differs. The Static MDS and Demo cookbook pages walk through all three for a concrete deployment; the conventional route is the installation-package launcher covered in Deployment: Without Docker. What follows is the deploy-mf-<service> / scripts/start workflow, one way to bring a service up under Docker.

The operator workflow is clone, configure, start. For each deployment project your topology calls for:

git clone git@github.com:MetaFluent/deploy-<service>.git
cd deploy-<service>

# Adapt deployment-config/ for your environment (deployment.properties, resources/)
scripts/start

scripts/start pulls the stock base image from ghcr.io (the first time) and runs it with your deployment configuration applied; the image itself stays stock. Stop it with:

scripts/stop

Your configuration lives in the deployment project and is applied each time you start - edit it, scripts/start again, and your changes take effect. Repeat for each deployment project in your topology.

Building a self-contained image

Running the stock image with your deployment configuration mounted as a volume is one way to operate. The deployment project also provides tools to build your own, fully configured and self-contained image. We call this baking a configuration.

Baking produces a derived image - the stock base image with your configuration built into it - as a single, self-contained artifact. Instead of a stock image plus a mounted configuration, you get one image that already is your configured service.

scripts/bake --tag <your-tag>

<your-tag> is a tag of your own choosing that identifies this build. scripts/start will then run the baked image rather than the stock base.

You would bake when you want an immutable, portable artifact - for example to promote one reviewed build unchanged through test and into production, or to run in an environment that should not fetch and configure at start time.

A baked image is yours. It goes to a registry you control (see below).

Running without Docker

You are not required to run Elastic MDS under Docker. A base image is just a filesystem plus a startup script, so it can be installed and run as a conventional, native application - and packaged (for example as an RPM) for a Docker-free environment. If that is how your organisation operates, see Deployment: Without Docker.

Setting up your own container registry

Baked images - and, if you prefer, mirrored copies of the stock base images - belong in a container registry you control. For many organisations this is not optional: banks and other regulated operators are reluctant to have any operational element depend on a service outside their control, so pulling from an internal registry rather than directly from ghcr.io at deploy time is often a requirement.

Any standard OCI container registry works. Common choices:

The pattern is the same in every case: authenticate Docker to your registry, then docker tag and docker push the image - one you built, or a mirror of a stock base - into it, under the same image names. scripts/start pulls from MetaFluent's registry (ghcr.io/metafluent) by default; point it at your own with --registry:

scripts/start --registry registry.example.internal/metafluent

We will expand this section with worked, per-vendor steps.

Staying current

Two things can change upstream, and they are separate:

  • A new base image - updated software. You pull the newer image; on its own this does not change anything else.
  • An updated deployment project - new or changed default configuration, resources, or scripts from upstream.

A new base image does not necessarily imply a new deployment project - often you simply run a newer image against the same project. This section is about the second case: taking an updated deployment project into your clone while preserving your own edits, both deployment.properties and anything you added or changed under resources/. Because the project ships defaults that you have adapted, a straight merge can collide. The project includes config-merge for exactly this:

scripts/config-merge --upgrade . --upstream <release-ref>

config-merge --upgrade fetches the upstream release and performs an interactive three-way merge into your project - preserving your edits while folding in new or changed upstream configuration, and refreshing the build-generated scripts/. It does the git fetch and applies the result for you; you review what it proposes. --upstream names the release to reconcile against. If you bake, re-bake afterwards.

Each deployment project ships the release notes for the version it delivers - what changed and any compatibility notes - in the project repository. Check them before taking an upgrade.

Where to go next

Elastic MDS documentation - (c) MetaFluent LLC - Confidential. Tracked in IssueTracking#586.

Elastic MDS

Home

Getting Started

Deployment Cookbook

Concepts

Configuration

Configuration Cookbook

Deployment

Operations

Diagnostic Cookbook

Developing Applications

Reference

Clone this wiki locally