Luigi Scheduler

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

TL;DR;

$ helm install incubator/luigi

Introduction

This chart bootstraps a Luigi deployment on a Kubernetes cluster using the Helm package manager.

Installing the Chart

To install the chart with the release name my-release:

$ helm install --name my-release stable/luigi

Uninstalling the Chart

To uninstall/delete the my-release deployment:

$ helm delete my-release --purge

The command removes all the Kubernetes components associated with the chart and deletes the release.

Luigi Configuration

Luigi configs are set as a block of text through a configmap and mouted as a file in /etc/luigi. Any value in this text block should match the defined luigi configuration. There are several values here that will have to match our kubernetes configuration.

Configuration

The following table lists the configurable parameters of the Luigi chart and their default values.

Parameter	Description	Default
image.repository	Luigi image	getpolymorph/luigi
image.tag	Luigi image tag	2.7.2
image.pullPolicy	Luigi image pull policy	IfNotPresent
service.name	Luigi service name	luigi
service.type	The kube service type	LoadBalancer
service.externalPort	The service external port	3000
service.internalPort	The service internal port	8082
service.config	The luigi service configs	View this default in the values.yaml file
persistence.enabled	Persistence flag	false
ingressUI.enabled	UI Ingress Flag	false
ingressAPI.enabled	API Ingress Flag	false

Dependent charts can also have values overwritten. Preface values with postgresql.* or redis.*

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

$ helm install --name my-release \
  --set persistence.enabled=false,email.host=email \
    stable/luigi

Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example,

$ helm install --name my-release -f values.yaml stable/luigi

Read through the values.yaml file. It has several commented out suggested values.

Persistence

Luigi requires a pickled state file. To maintain state after a restart you'll need to enable persistence. --set persistence.enabled=true

Ingress

This chart provides support for two Ingress resources. This is to allow authentication in the ui via reverse proxy with something like oauth-proxy and a separate form of authentication for luigi worker access.