Skip to content

Add simple options for enabling Gremlin GPU access #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ZachNo merged 10 commits into from
Aug 4, 2026
Merged

Add simple options for enabling Gremlin GPU access #147

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
70bb987
Add easy GPU driver setup
ZachNo Jul 20, 2026
0281768
Add runtimeclass creation
ZachNo Jul 20, 2026
c5e802d
Fix ICD not mounting
ZachNo Jul 21, 2026
a5526c8
Simplify ICD loader config
ZachNo Jul 21, 2026
ed9b656
Support Nvidia CDI config, clean up docs
ZachNo Jul 21, 2026
cd043d5
Remove orphaned variable
ZachNo Jul 22, 2026
9048fe0
Add README changes
ZachNo Jul 22, 2026
7be99b5
bump chart version
ZachNo Jul 22, 2026
9bbff57
Address comments, add multi-daemon mode for GPU
ZachNo Jul 29, 2026
457a84c
Address comments about upgrade path
ZachNo Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gremlin/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: gremlin
version: 0.26.0
version: 0.27.0
description: The Gremlin Inc client application
apiVersion: v1
home: https://www.gremlin.com
Expand Down
38 changes: 38 additions & 0 deletions gremlin/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ their default values. See values.yaml for all available options.
| `gremlin.proxy.url` | Specifies the http proxy the agent should use to communicate with api.gremlin.com. | `""` (ignored) | |
| `gremlin.extraEnv` | Specify any arbitrary environment variables to pass to the Gremlin Agent daemonset. | `[]` |
| `gremlin.features.discoverDestinationService.enabled` | Enable discovery of a destination service in a service mesh to resolve hostnames | `false` |
| `gremlin.gpu.enabled` | Expose host GPU/OpenCL drivers to the agent for the GPU attack | `false` |
| `gremlin.gpu.cdiDevice` | CDI device to inject via pod annotation (for CDI-based runtimes) | `""` |
| `gremlin.gpu.projectOpenclIcd` | Project a vendor's OpenCL ICD registry file into the container | `true` |
| `gremlin.gpu.vendors` | Vendor blocks to target; one DaemonSet is created per entry | `[nvidia, amd]` |
| `gremlin.gpu.<vendor>` | Per-vendor config block: `nodeSelector`, `runtimeClassName`, `env`, `volumes`, `volumeMounts`, `openclIcd` | see `values.yaml` |
| `ssl.certFile` | Add a certificate file to Gremlin's set of certificate authorities. This argument expects a file containing the certificate(s) you wish to add. When set, this chart creates secret (`ssl-cert-file`) with the contents and passes it to both agents. This value is ignored when blank or absent. | `""` (ignored) |
| `ssl.certDir` | sets the SSL_CERT_DIR environment variable on the both agents. Unlike ssl.certFile, this value accepts only a path to an existing directory on the Kubernetes nodes. This value is ignored when blank or absent. | `""` (ignored) |

Expand Down Expand Up @@ -213,6 +218,39 @@ helm install gremlin gremlin/gremlin \
--set-file ssl.certFile=$HOME/Workspace/proxy/ca.pem
```

### With GPU Support

To let the GPU attack enumerate and target GPUs, enable `gremlin.gpu` and list the vendors your cluster has in `gremlin.gpu.vendors` (both `nvidia` and `amd` by default).

```shell
helm install gremlin gremlin/gremlin \
--namespace gremlin \
--set gremlin.secret.managed=true \
--set gremlin.secret.teamID=$GREMLIN_TEAM_ID \
--set gremlin.secret.clusterID=$GREMLIN_CLUSTER_ID \
--set-file gremlin.secret.certificate=/path/to/gremlin.cert \
--set-file gremlin.secret.key=/path/to/gremlin.key \
--set gremlin.gpu.enabled=true \
--set gremlin.gpu.vendors={nvidia}
```

_note_: The `nvidia` preset runs the agent under the `nvidia` RuntimeClass. If the Gremlin pod fails to start (for example, a `RuntimeClass not found` error), the RuntimeClass likely doesn't exist on your cluster. This chart does not create RuntimeClass objects. Ensure the RuntimeClass named by the vendor block exists (it is normally provided by the NVIDIA GPU Operator or your platform), or set the vendor's `runtimeClassName` to `""`.

#### One DaemonSet per vendor

Only some nodes have GPUs, and different nodes may have different GPU vendors, so a single GPU DaemonSet cannot run cluster-wide: nodes lacking the vendor's RuntimeClass or device mounts would fail to start the Gremlin pod. So whenever `gremlin.gpu.enabled` is set, the chart renders:

- one GPU DaemonSet per entry in `gremlin.gpu.vendors` (`<release>-gremlin-gpu-<vendor>`), scheduled via node affinity onto that vendor's nodes and carrying that vendor's GPU configuration, and
- one DaemonSet (`<release>-gremlin-gpu-none`) for every node that belongs to none of those vendors.

Each vendor's DaemonSet is rendered whether or not the cluster currently has nodes of that vendor; a vendor with no matching nodes simply schedules no pods. Trim `gremlin.gpu.vendors` to the vendors you care about to avoid the extra DaemonSets.

Scheduling uses each vendor block's `nodeSelector` (node labels such as `nvidia.com/gpu.present` / `amd.com/gpu.present`, set by the NVIDIA GPU Operator / Node Feature Discovery and the AMD GPU labeller): its DaemonSet requires those labels, and the `gpu-none` DaemonSet requires their absence. GPU nodes must therefore carry the vendor's `nodeSelector` labels for its DaemonSet to schedule. Any `affinity` you set is preserved — the vendor requirement is ANDed into it.

This chart does not create RuntimeClass objects; any RuntimeClass named by a vendor block must already exist on the cluster.

_note_: When GPU support is disabled the agent DaemonSet keeps its original `<release>-gremlin` name. Enabling GPU support replaces it with the per-vendor DaemonSets above (including `<release>-gremlin-gpu-none`), so Helm deletes the old DaemonSet and its pods are recreated by the new ones. The per-vendor DaemonSets also carry an extra `gremlin.com/gpu` pod-selector label so each only manages its own pods.

## Uninstallation

```shell
Expand Down
299 changes: 299 additions & 0 deletions gremlin/templates/_daemonset.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,299 @@
{{/*
gremlin.daemonset renders one Gremlin DaemonSet from an already-resolved context. Deciding how many
DaemonSets exist and what goes in each is the caller's job (see daemonset.yaml); this template only
renders what it is handed.

Context: dict
"root" $ # the chart root context
"name" <string> # DaemonSet name
"selectorLabels" <dict> # extra selector/pod labels, so DaemonSets sharing a cluster do not
# fight over each other's pods (optional; default none)
"gpu" <dict> # a gremlin.gpu.<vendor> config block (optional; default no GPU config)
"affinity" <yaml> # rendered affinity (optional; defaults to .Values.affinity)
*/}}
{{- define "gremlin.daemonset" -}}
{{- $root := .root -}}
{{- $name := .name -}}
{{- $selectorLabels := default (dict) .selectorLabels -}}
{{- $gpu := default (dict) .gpu -}}
{{- $affinity := .affinity -}}
{{- if and (not $affinity) $root.Values.affinity -}}{{- $affinity = toYaml $root.Values.affinity -}}{{- end -}}
{{- /* CDI device: global gremlin.gpu.cdiDevice, optionally overridden per vendor block */ -}}
{{- $cdiDevice := "" -}}
{{- with $gpu -}}{{- $cdiDevice = default $root.Values.gremlin.gpu.cdiDevice .cdiDevice -}}{{- end -}}
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: {{ $name }}
namespace: {{ $root.Release.Namespace }}
labels:
app.kubernetes.io/name: {{ include "gremlin.name" $root }}
helm.sh/chart: {{ include "gremlin.chart" $root }}
app.kubernetes.io/instance: {{ $root.Release.Name }}
app.kubernetes.io/managed-by: {{ $root.Release.Service }}
version: v1
{{- with $selectorLabels }}
{{- toYaml . | nindent 4 }}
{{- end }}
{{- if $root.Values.gremlin.podLabels }}
{{- toYaml $root.Values.gremlin.podLabels | nindent 4 }}
{{- end }}
spec:
selector:
matchLabels:
app.kubernetes.io/name: {{ include "gremlin.name" $root }}
{{- with $selectorLabels }}
{{- toYaml . | nindent 6 }}
{{- end }}
{{- if $root.Values.gremlin.updateStrategy }}
updateStrategy:
{{- toYaml $root.Values.gremlin.updateStrategy | nindent 4 }}
{{- end }}
template:
metadata:
labels:
app.kubernetes.io/name: {{ include "gremlin.name" $root }}
helm.sh/chart: {{ include "gremlin.chart" $root }}
app.kubernetes.io/instance: {{ $root.Release.Name }}
app.kubernetes.io/managed-by: {{ $root.Release.Service }}
version: v1
{{- with $selectorLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if $root.Values.gremlin.podLabels }}
{{- toYaml $root.Values.gremlin.podLabels | nindent 8 }}
{{- end }}
{{- if or $root.Values.gremlin.apparmor $root.Values.gremlin.installApparmorProfile $root.Values.gremlin.podSecurity.seccomp.enabled $root.Values.gremlin.podSecurity.securityContextConstraints.create $root.Values.gremlin.podAnnotations $cdiDevice }}
annotations:
{{- if $root.Values.gremlin.apparmor }}
container.apparmor.security.beta.kubernetes.io/{{ $root.Chart.Name }}: {{ $root.Values.gremlin.apparmor }}
{{- else if $root.Values.gremlin.installApparmorProfile }}
container.apparmor.security.beta.kubernetes.io/{{ $root.Chart.Name }}: {{ "localhost/gremlin-agent" }}
{{- end }}
{{- if $root.Values.gremlin.podSecurity.seccomp.enabled }}
container.seccomp.security.alpha.kubernetes.io/{{ $root.Chart.Name }}: {{ $root.Values.gremlin.podSecurity.seccomp.profile }}
{{- end }}
{{- if $root.Values.gremlin.podSecurity.securityContextConstraints.create }}
openshift.io/required-scc: "gremlin"
{{- end }}
{{- if $cdiDevice }}
cdi.k8s.io/gremlin-gpu: {{ $cdiDevice | quote }}
{{- end }}
{{- if $root.Values.gremlin.podAnnotations }}
{{- toYaml $root.Values.gremlin.podAnnotations | nindent 8 }}
{{- end }}
{{- end }}
spec:
serviceAccountName: gremlin
{{- with $gpu.runtimeClassName }}
runtimeClassName: {{ . }}
{{- end }}
{{- with $affinity }}
affinity: {{ . | trimSuffix "\n" | nindent 8 }}
{{- end }}
{{- if $root.Values.nodeSelector }}
nodeSelector: {{ toYaml $root.Values.nodeSelector | trimSuffix "\n" | nindent 8 }}
{{- end }}
{{- if $root.Values.tolerations }}
tolerations: {{ toYaml $root.Values.tolerations | trimSuffix "\n" | nindent 8 }}
{{- end }}
dnsPolicy: {{ $root.Values.gremlin.dnsPolicy }}
hostPID: {{ $root.Values.gremlin.hostPID }}
hostNetwork: {{ $root.Values.gremlin.hostNetwork }}
{{- if $root.Values.image.pullSecret }}
imagePullSecrets:
- name: {{ $root.Values.image.pullSecret }}
{{- end }}
{{- if and $root.Values.gremlin.podSecurity.seccomp.enabled (eq "localhost/gremlin" $root.Values.gremlin.podSecurity.seccomp.profile) }}
initContainers:
- name: seccomp-init
image: {{ $root.Values.image.repository }}:{{ $root.Values.image.tag }}
imagePullPolicy: {{ $root.Values.image.pullPolicy }}
volumeMounts:
- mountPath: {{ $root.Values.gremlin.podSecurity.seccomp.root }}
name: seccomp-root
- mountPath: /gremlin
name: seccomp-profile
command:
- cp
- /gremlin/seccomp.json
- {{ $root.Values.gremlin.podSecurity.seccomp.root }}/gremlin
{{- end }}
containers:
- name: {{ $root.Chart.Name }}
image: {{ $root.Values.image.repository }}:{{ $root.Values.image.tag }}
args: [ "daemon" ]
imagePullPolicy: {{ $root.Values.image.pullPolicy }}
{{- if $root.Values.gremlin.resources }}
resources: {{ toYaml $root.Values.gremlin.resources | nindent 10 }}
{{- end }}
securityContext:
privileged: {{ $root.Values.gremlin.podSecurity.privileged }}
allowPrivilegeEscalation: {{ $root.Values.gremlin.podSecurity.allowPrivilegeEscalation }}
capabilities:
add: {{ toYaml $root.Values.gremlin.podSecurity.capabilities | nindent 14 }}
{{- if $root.Values.gremlin.podSecurity.seLinuxOptions }}
seLinuxOptions: {{ toYaml $root.Values.gremlin.podSecurity.seLinuxOptions | nindent 12 }}
{{- end }}
readOnlyRootFilesystem: {{ $root.Values.gremlin.podSecurity.readOnlyRootFilesystem }}
env:
- name: GREMLIN_TEAM_ID
{{- /* If we aren't managing this secret and a teamID was supplied, assume teamID is not in the external secret */}}
{{- if (and (not $root.Values.gremlin.secret.managed) (default $root.Values.gremlin.teamID $root.Values.gremlin.secret.teamID)) }}
value: {{ default $root.Values.gremlin.teamID $root.Values.gremlin.secret.teamID | quote }}
{{- else }}
valueFrom:
secretKeyRef:
name: {{ include "gremlin.secretName" $root }}
key: GREMLIN_TEAM_ID
{{- end }}

{{- if (eq (include "gremlin.secretType" $root) "secret") }}
- name: GREMLIN_TEAM_SECRET
valueFrom:
secretKeyRef:
name: {{ include "gremlin.secretName" $root }}
key: GREMLIN_TEAM_SECRET
{{- else }}
- name: GREMLIN_TEAM_CERTIFICATE_OR_FILE
{{- /* If managed outside of this chart, or if the value is a literal, reference the secret as a file */}}
{{- if or (not $root.Values.gremlin.secret.managed) (hasPrefix "-----BEGIN" $root.Values.gremlin.secret.certificate) }}
value: file:///var/lib/gremlin/cert/gremlin.cert
{{- else }}
value: {{ $root.Values.gremlin.secret.certificate }}
{{- end }}
- name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE
{{- /* If managed outside of this chart, or if the value is a literal, reference the secret as a file */}}
{{- if or (not $root.Values.gremlin.secret.managed) (hasPrefix "-----BEGIN" $root.Values.gremlin.secret.certificate) }}
value: file:///var/lib/gremlin/cert/gremlin.key
{{- else }}
value: {{ $root.Values.gremlin.secret.key }}
{{- end }}
{{- end }}
- name: GREMLIN_IDENTIFIER
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: GREMLIN_CLIENT_TAGS
value: {{ $root.Values.gremlin.client.tags }}
- name: GREMLIN_COLLECT_DNS
value: {{ $root.Values.gremlin.collect.dns | quote }}
- name: GREMLIN_SERVICE_URL
value: {{ include "gremlinServiceUrl" $root }}
{{- if not $root.Values.gremlin.features.pushCIDRTags.enabled }}
- name: GREMLIN_PUSH_POD_CIDR_TAGS
value: "false"
- name: GREMLIN_PUSH_ZONE_CIDR_TAGS
value: "false"
{{- end }}
{{- if $root.Values.gremlin.proxy.url }}
- name: https_proxy
value: {{ $root.Values.gremlin.proxy.url }}
{{- end }}
{{- if $root.Values.ssl.certFile }}
- name: SSL_CERT_FILE
value: /etc/gremlin/ssl/certfile.pem
{{- end }}
{{- if $root.Values.ssl.certDir }}
- name: SSL_CERT_DIR
value: {{ $root.Values.ssl.certDir }}
{{- end }}
{{- if include "gremlinTlsIdentityEnv" $root }}
{{- include "gremlinTlsIdentityEnv" $root | nindent 10 }}
{{- end }}
{{- with $gpu.env }}
{{- toYaml . | nindent 10 }}
{{- end }}
{{- with $root.Values.gremlin.extraEnv }}
{{- toYaml . | nindent 10 }}
{{- end }}
volumeMounts:
- name: gremlin-state
mountPath: /var/lib/gremlin
readOnly: false
- name: gremlin-executions
mountPath: /var/lib/gremlin/executions
readOnly: false
- name: gremlin-logs
mountPath: /var/log/gremlin
readOnly: false
- name: cgroup-root
mountPath: /sys/fs/cgroup
readOnly: false
{{- if include "containerMounts" $root }}
{{- include "containerMounts" $root | nindent 10 }}
{{- end }}
{{- if (eq (include "gremlin.secretType" $root) "certificate") }}
- name: gremlin-cert
mountPath: /var/lib/gremlin/cert
readOnly: true
{{- end }}
{{- if $root.Values.ssl.certFile }}
- name: ssl-cert-file
mountPath: /etc/gremlin/ssl
readOnly: true
{{- end }}
{{- if include "gremlinTlsIdentityVolumeMounts" $root }}
{{- include "gremlinTlsIdentityVolumeMounts" $root | nindent 10 }}
{{- end }}
{{- with $gpu.volumeMounts }}
{{- toYaml . | nindent 10 }}
{{- end }}
{{- if include "gremlinGpuOpenclIcdActiveFromSpec" (dict "root" $root "gpu" $gpu) }}
- name: gremlin-opencl-icd
mountPath: /etc/OpenCL/vendors/{{ $gpu.openclIcd.filename }}
subPath: {{ $gpu.openclIcd.filename }}
readOnly: true
{{- end }}
volumes:
- name: cgroup-root
hostPath:
path: {{ $root.Values.gremlin.cgroup.root }}
{{- if include "containerVolumes" $root }}
{{- include "containerVolumes" $root | nindent 8 }}
{{- end }}
# The Gremlin daemon communicates with Gremlin sidecars via its state directory.
- name: gremlin-state
emptyDir:
medium: Memory
- name: gremlin-executions
hostPath:
path: /var/lib/gremlin/executions
# The Gremlin daemon forwards logs from the Gremlin sidecars to the Gremlin control plane
# These logs should be shared with the host
- name: gremlin-logs
hostPath:
path: /var/log/gremlin
{{- if (eq (include "gremlin.secretType" $root) "certificate") }}
- name: gremlin-cert
secret:
secretName: {{ include "gremlin.secretName" $root }}
{{- end }}
{{- if and $root.Values.gremlin.podSecurity.seccomp.enabled (eq "localhost/gremlin" $root.Values.gremlin.podSecurity.seccomp.profile) }}
- name: seccomp-root
hostPath:
path: {{ $root.Values.gremlin.podSecurity.seccomp.root }}
- name: seccomp-profile
configMap:
name: {{ template "gremlin.fullname" $root }}-seccomp
{{- end }}
{{- if $root.Values.ssl.certFile }}
- name: ssl-cert-file
secret:
secretName: ssl-cert-file
{{- end }}
{{- if include "gremlinTlsIdentityVolumes" $root }}
{{- include "gremlinTlsIdentityVolumes" $root | nindent 8 }}
{{- end }}
{{- with $gpu.volumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- if include "gremlinGpuOpenclIcdActiveFromSpec" (dict "root" $root "gpu" $gpu) }}
- name: gremlin-opencl-icd
configMap:
name: {{ include "gremlin.fullname" $root }}-opencl-icd
{{- end }}
{{- if $root.Values.gremlin.priorityClassName }}
priorityClassName: {{ $root.Values.gremlin.priorityClassName }}
{{- end }}
{{- end -}}
Loading
Loading