Cannot pass env vars to k6 tests via k6-operator resource

[danielfigueiredo]

Brief summary

Hey team, I've been trying to get my k6 tests running via k8s using the k6-operator, but I'm unable to pass any environment variables.

I've looked in detail at the community forum and demo app, and whatever syntax is there does not seem to work for me.
The runnerSpec go file seems to forward the env vars declared within the spec.runner. However, my tests keep failing right when bootstrapping because the vars provided are unavailable.

I've tried several syntax combinations with different indentations, some of them failed to be parsed, and all others do not throw errors, but they don't seem to work either.

k6-operator version or image

make deploy executed from commit sha 151472705d2e35d1c6d9158affed2ab127c305ba (latest in the repo main branch at time of writing)

K6 YAML

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-distributed
spec:
  parallelism: 4
  script:
    configMap:
      name: my-test-file-dev
      file: my-test-file-dev.js
  runner:
    image: danielfigueiredoc/demo-k6-operator:latest
    env:
      - name: REMOTE_ENV
        value: staging

Other environment details (if applicable)

No response

Steps to reproduce the problem

My docker file is the same as the demo / recommended in k6-operator docs

FROM golang:1.19-alpine as builder
WORKDIR $GOPATH/src/go.k6.io/k6
ADD . .
RUN apk --no-cache add git
RUN CGO_ENABLED=0 go install go.k6.io/xk6/cmd/xk6@latest
# TODO - Want more extensions?! Provide additional `--with ...` lines to the following command:
RUN CGO_ENABLED=0 xk6 build \
    --output /tmp/k6

FROM alpine:3.16
RUN apk add --no-cache ca-certificates \
    && adduser -D -u 12345 -g 12345 k6
COPY --from=builder /tmp/k6 /usr/bin/k6

USER 12345
WORKDIR /home/k6

ENTRYPOINT ["k6"]

It is all very simple; tests run fine without env var assertions. But I added this check when my test bootstraps:

import { fail } from 'k6';
//  ...

const env = __ENV.REMOTE_ENV;

if (!env) {
  fail('no REMOTE_ENV given');
}

When I run kubectl apply -f ./path/to/k6-resource.yml the k6-distributed-initializer job kicks in, I can see the pod being created and immediately failing with:

time="2023-03-22T15:05:41Z" level=error msg="GoError: no REMOTE_ENV given\n\tat go.k6.io/k6/js/modules/k6.(*K6).Fail-fm (native)\n\tat getRemoteEnv (file:///....)

If I swap the script with a sample test requiring no env vars, it works fine.
Everybody seems to provide the env vars as I do, so I think something is up with my setup that I can't figure out yet.

I've also tried to provide env vars via config map but got the same behaviour: there are no failures when running the command, but tests crash as env vars are unavailable. I swapped the runner.env with this:

  envFrom:
  - configMapRef:
      name: trade-mobile-sessions-dev-env-vars

Any help would be much appreciated; I'm not sure what's going on. The only difference between my code and https://github.com/javaducky/demo-k6-operator is that I create my local cluster using minikube, which should not matter theoretically.

Expected behaviour

Be able to provide env vars somehow to the k6 tests being executed.

Actual behaviour

Env vars are not available.

[danielfigueiredo]

I finally discovered the issue; it appears that when running k6 through the operator + k8s the environment variables are only available at a different time (unsure when exactly, but I'll get back here with more info).
The test case I was trying to run had the static options const referencing environment variables such as:

// common file
function getRemoteEnv() {
  const env = __ENV.REMOTE_ENV;

  if (!env) {
    fail('no REMOTE_ENV given');
  }

  return env;
}

// test file imports getRemoteEnv from common file
export const options = {
  vus: 10,
  duration: '10s',
  thresholds: {
    'http_reqs{expected_response:true}': ['rate>10'],
  },
  tags: {
    env: getRemoteEnv(),
  }
};

We used to run k6 via nomad, and this evaluation works fine, but running this way, this env var is not yet available. It is available in later phases because I can see it when logging correct values within the test case or the setup function call.

[yorugac]

Hi @danielfigueiredo, thanks for opening the issue! Could you please try the following simple example:

---
apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
spec:
  parallelism: 1
  script:
    configMap:
      name: "env-vars"
      file: "env-var.js"
  cleanup: "post"
  runner:
    env:
      - name: REMOTE_ENV
        value: staging

and

import { sleep } from 'k6';

export let options = {
    vus: 10,
    duration: '30s',
    discardResponseBodies: true,
};

// Run with:
// k6 run -e REMOTE_ENV=something env-var.js

const testvar = __ENV.REMOTE_ENV;

export default function () {
    console.log("testvar is", testvar);
    console.log("env var is", __ENV.REMOTE_ENV);
    sleep(1);
}

So without custom images and complex outputs. The above works for me. Would these work in your setup? I think it makes sense to go step by step to figure out at which point the issue appears: right now it's not clear 😅

[danielfigueiredo]

Hi @yorugac , thanks for the response! I was finally able to narrow it down, it is what I mentioned here indeed.

Your example runs just fine for me, but when' options' are evaluated (es6 module bootstrap in standard JS lands), the env vars are not available when running via k6-operator. They certainly are if you run them via docker or directly via k6.

I made a slight tweak in your script that should allow you to reproduce this issue, given you run it via k6-operator in a k8s cluster:

import { sleep, fail } from 'k6';

function getRemoteEnv() {
  const env = __ENV.REMOTE_ENV;

  if (!env) {
    fail('no REMOTE_ENV given');
  }

  return env;
}

export let options = {
    vus: 10,
    duration: '30s',
    discardResponseBodies: true,
    tags: {
      env: getRemoteEnv(),
    },
};


export default function () {
    console.log("env var is", __ENV.REMOTE_ENV); // would still log ok if the assertion above is removed
    sleep(5);
}

To sum it up:

• k6 run -e REMOTE_ENV=staging test-scripts/simple.js works fine 👍
• docker run -v $PWD:/scripts --env-file .env -it --rm danielfigueiredoc/demo-k6-operator:latest run /scripts/simple.js 👍 because .env has the REMOVE_ENV defined

So far so good, now to the error:

k6-basic.yml

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-basic
spec:
  parallelism: 2
  script:
    configMap:
      name: test-scripts
      file: simple.js
  runner:
    image: danielfigueiredoc/demo-k6-operator:latest
    env:
      - name: REMOTE_ENV
        value: staging

And finally the execution results:

demo-k6-operator on main [!?] on ☁️  (us-east-1) 
❯ kubectl apply -n k6-demo -f ./resources/k6-basic.yaml
k6.k6.io/k6-basic created

demo-k6-operator on main [!?] on ☁️  (us-east-1) 
❯ kubectl get pods --namespace k6-demo
NAME                         READY   STATUS   RESTARTS   AGE
k6-basic-initializer-s5xkc   0/1     Error    0          3s

demo-k6-operator on main [!?] on ☁️  (us-east-1) 
❯ kubectl logs k6-basic-initializer-s5xkc --namespace k6-demo
time="2023-03-23T14:11:11Z" level=error msg="GoError: no REMOTE_ENV given\n\tat go.k6.io/k6/js/modules/k6.(*K6).Fail-fm (native)\n\tat getRemoteEnv (file:///test/simple.js:17:9(11))\n\tat file:///test/simple.js:10:7(49)\n" hint="script exception"

That proves that the env var is not yet defined; the value is only available run setup or the default function is running, but not when the module is bootstrapping. I believe if you were to log the value outside of the fn, you'd get the same result.

[yorugac]

OK, I think its a bit clearer now what happens here. Thanks for the write up!

This script likely fails on k6 archive command here. k6 archive has a different behaviour than k6 run and by default it tries to omit passing env vars in options for security reasons. This is somewhat hinted at in this doc. IMHO, these small differences could be explained better to be less obscure. By the way, there is apparently an open k6 issue to improve this UX: grafana/k6#2744

Now what we can do with this from k6-operator's side:

1. avoid using such scripts where env var must be evaluated in options 🙂
2. add --include-system-env-vars option to k6 archive command in initializer job as default.
   • even though k6 archive avoids env vars by default as a security measure; in theory, initializer job is a small background job, which is not sending anything anywhere and it is meant to be executed on user's infra. So this change might be acceptable in this case.
3. we might want to go with the changes proposed in this issue instead: Make reading logs by initializer a more reliable operation #193

This could use some more thinking and evaluating. Opinions are welcome!

[danielfigueiredo]

Thank you for all the help
I can try these but this issue should be resolved as it wasn't a real bug. It is behaving as per design

[yorugac]

Yes, @danielfigueiredo you're right, it works as expected now. Even if it also could be improved 😄 I'm fine either way: we'll need to get to #193 some day and your issue is already linked there so we'll re-visit this case then. Let's close this one then 👍

Thank you again for reporting this relatively rare case!

[patenvy1234]

hey you can do something like this if you want to pass the environment variables in the test.js

use the arguments like this :

argument : -e FOR_RPS: {{ .Values.for_rps }}

FOR_RPS can now be used to pass in the options argument

pls dont remove this functionality