Skip to content

Latest commit

 

History

History
303 lines (233 loc) · 7.17 KB

README.md

File metadata and controls

303 lines (233 loc) · 7.17 KB
Raw

HashiCorp Nomad Hetzner Cloud csi-driver

Preconditions

Getting Started

CSI Setup

  1. Create a read+write API token in the Hetzner Cloud Console.

  2. Create a Nomad Variable for the HCLOUD token:

Note

Consider using HashiCorp Vault for secrets management, see https://developer.hashicorp.com/nomad/docs/job-specification/template#vault-kv-api-v2

export HCLOUD_TOKEN="..."
nomad var put secrets/hcloud hcloud_token=$HCLOUD_TOKEN
  1. Create a CSI Controller Job:
# file: hcloud-csi-controller.hcl

job "hcloud-csi-controller" {
  datacenters = ["dc1"]
  namespace   = "default"
  type        = "service"

  group "controller" {

    ### NOTE
    # We define (at least) 2 allocations to increase the availability in case of a node failure with
    # a controller allocation running on that node. On a "Single Node Cluster", the group stanzas
    # might need modification or should be removed.
    count = 2

    constraint {
      distinct_hosts = true
    }

    update {
      max_parallel     = 1
      canary           = 1
      min_healthy_time = "10s"
      healthy_deadline = "1m"
      auto_revert      = true
      auto_promote     = true
    }

    task "plugin" {
      driver = "docker"

      config {
        # Get the latest version on https://hub.docker.com/r/hetznercloud/hcloud-csi-driver/tags
        image   = "hetznercloud/hcloud-csi-driver:v2.5.1"
        command = "bin/hcloud-csi-driver-controller"
      }

      env {
        CSI_ENDPOINT   = "unix://csi/csi.sock"
        ENABLE_METRICS = true
      }

      template {
        data        = <<EOH
HCLOUD_TOKEN="{{ with nomadVar "secrets/hcloud" }}{{ .hcloud_token }}{{ end }}"
EOH
        destination = "${NOMAD_SECRETS_DIR}/hcloud-token.env"
        env         = true
      }

      csi_plugin {
        id        = "csi.hetzner.cloud"
        type      = "controller"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}
  1. Create a CSI Node Job:
# file: hcloud-csi-node.hcl
job "hcloud-csi-node" {
  datacenters = ["dc1"]
  namespace   = "default"
  type        = "system"

  group "node" {
    task "plugin" {
      driver = "docker"

      config {
        # Get the latest version on https://hub.docker.com/r/hetznercloud/hcloud-csi-driver/tags
        image      = "hetznercloud/hcloud-csi-driver:v2.5.1"
        command    = "bin/hcloud-csi-driver-node"
        privileged = true
      }

      env {
        CSI_ENDPOINT   = "unix://csi/csi.sock"
        ENABLE_METRICS = true
      }

      template {
        data        = <<EOH
HCLOUD_TOKEN="{{ with nomadVar "secrets/hcloud" }}{{ .hcloud_token }}{{ end }}"
EOH
        destination = "${NOMAD_SECRETS_DIR}/hcloud-token.env"
        env         = true
      }

      csi_plugin {
        id        = "csi.hetzner.cloud"
        type      = "node"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}
  1. Deploy the Jobs:

The following commands deploy the job resources created previously on your Nomad cluster:

nomad job run hcloud-csi-controller.hcl
nomad job run hcloud-csi-node.hcl
  1. Verify the status:

To ensure the plugin is running and healthy, check the web UI on path /ui/csi/plugins/csi.hetzner.cloud or by using the CLI:

nomad plugin status

Volumes Setup

  1. Define a Volume:

Create a file db-vol.hcl for the volume resource:

Note

See Nomad Volume Specification for more information.

# file: db-vol.hcl

type      = "csi"
id        = "db-vol"
name      = "db-vol"
namespace = "default"
plugin_id = "csi.hetzner.cloud"

# Default minimum capacity for Hetzner Cloud is 10G
capacity_min = "10G"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

mount_options {
  fs_type     = "ext4"
  mount_flags = ["discard", "defaults"]
}

Important

The volume will be created in the same Hetzner Cloud Location as the controller is deployed into.

To define the Hetzner Cloud Location (CLI: hcloud location list) you would like to create the volume into, append the following snippet into the volume resource definition:

topology_request {
  required {
   # Use your desired location name here
    topology { segments { "csi.hetzner.cloud/location" = "fsn1" } }
  }
}
  1. Create a Volume:
nomad volume create db-vol.hcl

Note

The hcloud cli provides a convenient way to verify if the volume was created: hcloud volume list.

Make use of the Volume

  1. Create a Job definition:

The following example describes how to mount the volume in a Docker Nomad job definition (especially see the parts commented with ### THIS!):

# file: mariadb.hcl

job "mariadb" {
  datacenters = ["dc1"]
  namespace   = "default"
  type        = "service"

  group "mariadb" {
    network {
      port "mariadb" {
        to           = 3306
      }
    }

    ### THIS!
    volume "db-volume" {
      type            = "csi"
      read_only       = false
      source          = "db-vol"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
      per_alloc       = false
    }

    task "mariadb" {
      driver = "docker"
      config {
        image = "mariadb:10.11"
        ports = [
          "mariadb",
        ]
      }

      ### THIS!
      volume_mount {
        volume      = "db-volume"
        destination = "/var/lib/mysql"
      }

      env {
        MARIADB_ROOT_PASSWORD = "<...>"
        MARIADB_DATABASE      = "<...>"
        MARIADB_USER          = "<...>"
        MARIADB_PASSWORD      = "<...>"
      }

      service {
        name = "db"
        # Uses nomad native service discovery. To use consul once configured, set it to "consul".
        # Also see https://developer.hashicorp.com/nomad/docs/job-specification/service#service-block
        provider = "nomad"
        port = "mariadb"
      }

      resources {
        cpu    = 300
        memory = 256
      }

    }
  }
}
  1. Create the Job:
nomad job run mariadb.hcl

Volumes encryption with LUKS

To add encryption with LUKS you have to provide a secret containing the encryption passphrase as part of the volume definition. The secret must be named encryption-passphrase. The volume will then be LUKS encrypted on first use.

# file: db-vol.hcl

secrets {
  "encryption-passphrase" = "<your_encryption_value>"
}

Note

Consider using HashiCorp Vault for secrets management, see https://developer.hashicorp.com/nomad/docs/job-specification/template#vault-kv-api-v2