Evaluating TESK (via director-wfs) as an alternative to Funnel for 5S-TES

[bp1183]

Context

The Task Execution Service (TES) API aims to provide a standard mechanism for executing batch computing tasks. Rather than implement a TES server ourselves we instead use one of the several off-the-shelf implementations available.

In our federated analysis experiments, we have been using (docker-based) Funnel as the task execution engine for our Five-Safes-TES deployments. Funnel is simple to install for the default Docker backend, but has a number of security concerns when interacting with sensitive data on TRE's.

An alternative implementation is TESK which mitigates the security problems of docker-based Funnel due to its Kubernetes-based implementation.

Funnel

Funnel is a Go implementation of TES that translates task messages directly into docker run commands. Its main appeal is simplicity - it's a single binary that runs anywhere Docker is available, with all logs in one place. Note that Funnel also supports Kubernetes as a compute backend, but our deployment uses the default Docker backend.

The core security concern for TRE use with the Docker backend is that the worker process needs to talk to the Docker daemon to launch task containers. The Docker daemon runs as root, so any process that can issue commands to it has root-equivalent reach on the host.

Also, Funnel's default RunCommand template doesn't add --user, --cap-drop, --security-opt, or user-namespace flags, so containers inherit Docker's defaults - root inside the container, full default capability set, no userns remapping, as you can see in the default run template in config/default.go of the Funnel source code.

// `docker run` command flags
// https://docs.docker.com/reference/cli/docker/container/run/
Container: &ContainerConfig{
	DriverCommand: "docker",
	RunCommand: "run -i --read-only " +
		// Remove container after it exits
		"{{if .RemoveContainer}}--rm{{end}} " +
		
		// Environment variables
		"{{.GetEnvArgs}} " +
		
		// Tags/Labels
		"{{range $k, $v := .Tags}}--label {{$k}}={{$v}} {{end}} " +
		
		// Container Name
		"{{if .Name}}--name {{.Name}}{{end}} " +
		
		// Workdir
		"{{if .Workdir}}--workdir {{.Workdir}}{{end}} " +
		
		// Volumes
		"{{range .Volumes}}--volume {{.HostPath}}:{{.ContainerPath}}:{{if .Readonly}}ro{{else}}rw{{end}} {{end}} " +
		
		// Image and Command
		"{{.Image}} {{.Command}}",
	PullCommand: "pull {{.Image}}",
	StopCommand: "rm -f {{.Name}}",
}

These defaults can be overridden via Funnel's config - including switching to the Kubernetes backend - but hardening requires deliberate per-deployment configuration, so the security burden falls entirely on correct operator configuration.

TESK

TESK is a Kubernetes-native TES implementation where each job runs as a Kubernetes pod. This makes it straightforward to enforce pod-level security policies via Kubernetes admission controllers - in contrast to Funnel's default Docker model.

It is composed of two components:

• tesk-api - receives the TES HTTP request, converts it to a Kubernetes Job spec for the taskmaster and submits it. Written in Java.
• tesk-core - contains the taskmaster and filer images. The taskmaster runs inside the first Kubernetes Job and dynamically spawns the filer and executor Jobs as separate Kubernetes Jobs. Written in Python.

Diagram from the TESK repository, Apache 2.0

Director-WFS

The downside of TESK is the operational overhead of running a Kubernetes cluster. The director-wfs from Swansea University Medical School is an opinionated full-stack deployment of TESK on KIND (Kubernetes IN Docker), with hardening enforced by Gatekeeper admission control, runtime detection via Falco, network policies, monitoring, and ingress management all active by default. A single bootstrap script stands up the whole cluster on macOS or Ubuntu.

KIND runs each Kubernetes node as a Docker container rather than a physical or virtual machine, keeping the infrastructure footprint minimal - a single Linux server or macOS host is enough. Note that director-wfs targets single-server deployments rather than horizontally-scalable production clusters.

High-level Security Comparison

Category	Funnel (Docker backend, default config)	Director-WFS (TESK)
Container root filesystem	Read-only, with a writable /tmp provided by Funnel	Read-only, enforced by Gatekeeper
Container user	Whatever the image specifies - usually root	Non-root by default, enforced by Gatekeeper
Linux capabilities	Docker’s default set is granted	All dropped by default, enforced by Gatekeeper
Privileged containers	Possible if the worker config allows it	Rejected by Gatekeeper
Host filesystem access	Whatever volumes the worker is configured to mount	Restricted to a single path used for storage, enforced by Gatekeeper
Host Docker socket	Required by the worker	Not used
Runtime monitoring	None by default	Falco watches container activity at runtime
Hardening enforcement	Relies on the operator configuring Funnel correctly	Enforced at the cluster level; insecure pod specs are rejected before they run

Integrating with 5STES via deployment on Azure VM

VM Specs

Property	Value
Operating System	Ubuntu 24.04 LTS
VM Generation	V2
Architecture	x64
Azure VM Size	Standard D4s v3
vCPUs	4
RAM	16 GiB
Threads per Core	2
OS Disk Type	Standard SSD LRS
OS Disk Size	128 GiB

Installation

We wrapped the script with our environment-specific variables. Note that we disable SSL as the TRE-agent communicates with TESK over a private network:

#!/bin/bash

DNS_NAME=tesk.internal
DEV_PASSWORD=<your-password>
S3_ENDPOINT="http://<minio-ip>:9002"
S3_ACCESS_KEY="<your-access-key>"
S3_SECRET_KEY="<your-secret-key>"

# Deletes any currently running cluster and builds from scratch
FORCE_RECREATE=1 bash <(curl -s https://raw.githubusercontent.com/SwanseaUniversityMedical/director-wfs/feat/extra-install/director-wfs.sh) \
        "$DNS_NAME" \
        "$DEV_PASSWORD" \
        "$S3_ENDPOINT" \
        "$S3_ACCESS_KEY" \
        "$S3_SECRET_KEY"

# Disable SSL — communicating over private network only
kubectl --context kind-director-wfs -n tesk-stack \
annotate ingress ingress-rules \ 
nginx.ingress.kubernetes.io/ssl-redirect="false"

On a fresh install you'll also need to add your user to the docker group:

sudo usermod -aG docker $USER
newgrp docker

Pointing the TRE-Agent at TESK

Update TesAPIUrl in your TRE-Agent .env:

TesAPIUrl=http://<DNS_NAME>/ga4gh/tes/v1/tasks

And add the TESK VM's IP to /etc/hosts on the TRE-Agent machine so the hostname resolves correctly:

# /etc/hosts
<tesk-vm-ip>    <DNS_NAME>

Testing

Successfully validated the following containerised applications / TES tasks against an Azure based deployment of 5S-TES and an Azure VM with director-wfs deployed on it

• Hello World
• SQL Runner
• Python Analysis - computes simple statistical analyses (mean, contingency tables, etc)
• Bunny

Behavioural difference: TESK does not honour container ENTRYPOINT

Affects: any container whose Docker image declares an ENTRYPOINT and relies on it being run automatically.

When migrating workloads from Funnel to TESK, the most important thing to know is that TESK ignores a container image's ENTRYPOINT and execs command[0] from the TES task payload directly. This is by design — see the comment in TesKubernetesConverter.java in the tesk-api source:

//It will be command == ENTRYPOINT, because of shell requirement and no custom entrypoints in bio tools.
new ExecutorCommandWrapper(executor).getCommandsWithStreamRedirects().forEach(container::addCommandItem);

For example, our SQL Runner container previously worked with Funnel using just the arguments:

"command": [
  "--Output=/outputs/output.csv",
  "--Query=..."
]

On TESK this must become:

"command": [
  "dotnet",
  "SQLRunnerPostgres.dll",
  "--Output=/outputs/output.csv",
  "--Query=..."
]

Looking at the TES-API schema definition it defines command as argv, with command[0] being the program to execute:

command:
	type: array
	description: |-
		A sequence of program arguments to execute, where the first argument
		is the program to execute (i.e. argv). Example:
		```
		{
		  "command" : ["/bin/md5", "/data/file1"]
		}
		```
    items:
	    type: string
	example: ["/bin/md5", "/data/file1"]

TESK follows this strictly. Funnel's default behaviour effectively treats command as arguments to the image's existing ENTRYPOINT — which diverges from the spec, but is convenient when working with bio tools that ship their own entrypoints. The behaviour comes from Funnel passing the command directly to docker run without overriding --entrypoint, rather than from a deliberate design choice; it can be changed by customising Funnel's RunCommand template to add --entrypoint "", but it's not the default. Containers written against Funnel that rely on ENTRYPOINT being honoured will break on TESK until their TES payloads are updated to include the program name as command[0].

Bug: .NET containers fail with a misleading OOM error under Gatekeeper

Affects: any .NET container running under a Kubernetes setup that enforces readOnlyRootFilesystem: true (which director-wfs does by default via Gatekeeper). Not specific to the SQL Runner — any .NET image will hit this.

When running our SQL Runner container without the DOTNET_EnableDiagnostics=0 environment variable, the executor pod failed immediately with what looked like an OOM error:

kubectl -n tesk-stack logs task-cd221ff3-ex-00-hrvm8

Failed to create CoreCLR, HRESULT: 0x8007000E

The pod also reported exit code 137, which typically signals an OOM kill. However, inspection of the pod showed no memory pressure - this is a red herring.

What's actually happening: at startup, the .NET runtime tries to create a Unix socket at /tmp/dotnet-diagnostic-<pid> for its diagnostic pipe (used by debuggers and profilers). The director-wfs TESK stack enables Gatekeeper with readOnlyRootFilesystem: true, which makes the entire container filesystem read-only - including /tmp. CoreCLR fails to create the socket very early in initialisation, before any user code runs, and reports 0x8007000E (E_OUTOFMEMORY - a misleading error code for what is actually a filesystem write failure).

This doesn't surface under Funnel because Funnel's worker mounts a writable host directory at /tmp for every task, so the diagnostic socket has somewhere to land.

The fix is to set DOTNET_EnableDiagnostics=0 in your executor environment, which tells the runtime not to create the diagnostic pipe at all:

"env": {"DOTNET_EnableDiagnostics": "0"}

Alternatively, DOTNET_EnableDiagnostics=0 can be baked directly into the container image via ENV DOTNET_EnableDiagnostics=0 in the Dockerfile, which avoids having to include it in every TES task payload. This is the recommended approach for containers intended for TRE deployment, where diagnostic tooling is unlikely to be needed.

[AndyRae]

Thanks Ben - this is really great.

Specifically on the entrypoint issue:

My conclusion was the guidance for anyone writing a TES complaint container needs some guidance like:

Write your containers as if ENTRYPOINT doesn't exist, and always include the program name in command[0].

I think that would be true? I don't think anyone provides guidance on writing TES containers, and it's annoying because as @beforan mentioned, entrypoint is a useful feature of the OCI standard, that anyone who writes containers uses all the time.

As we care deeply about analysis on containers, I would say we could do the job of how to write one that is compliant, to save this issue arising. Because you can see the headache now - we sell the future on building around a standard, and it turns out it's implemented differently and my TRE can't run your analysis.

So - is it ambitious we ask for change? From the GA4GH TES standard, or Funnel as implementers?

This also feels generally interesting - I don't know if there is a paper in this?
@tb143 @Grazziela @prquinlan

[prquinlan]

@alee-x and @tekkisse I am sure you also have thoughts comments on this one

[mcraig-ibme]

Hi, one thing that struck me about this comparison is the fact that TESK makes /tmp read-only.

I know a lot of tools and libraries assume they can write to /tmp (including FSL). Is there a workaround for this, for example by defining a TMPDIR environment variable pointing to a writable folder? Most tools should respect TMPDIR.

[tekkisse]

its not TESK its GATEKEEPER that doing this. Coz security first

would need extra in TES volume: "volumes": ["/tmp"]

Which could be added by the TRE Agent as a sensible balance

[tekkisse]

the other difference is TESK will obay the RESOURCES section, Funnel is try - no big deal

Bigger one is TESK tasks can be subject to Network Policies allowing finer grained control on access to other systems on the network / cluster