Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No longer compatible with Istio in version 0.0.8 #172

Closed
13013SwagR opened this issue Dec 7, 2022 · 6 comments · Fixed by #175
Closed

No longer compatible with Istio in version 0.0.8 #172

13013SwagR opened this issue Dec 7, 2022 · 6 comments · Fixed by #175
Labels
bug Something isn't working

Comments

@13013SwagR
Copy link
Contributor

13013SwagR commented Dec 7, 2022 •
edited

Brief summary

The annotations from the runner are no longer copied to the initializer,
which makes it impossible to add the annotation sidecar.istio.io/inject: "false" which is required, the sidecar prevents the job to complete.
Also tested with scuttle, but the controller is unable to parse the initializer logs because scuttle does log even if scuttleLogging is set to false.

k6-operator version or image

0.0.8

K6 YAML

apiVersion: k6.io/v1alpha1
kind: K6
metadata:
  name: k6-sample
spec:
  parallelism: 4
  runner:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
  starter:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
  scuttle:
    enabled: "true"
    waitForEnvoyTimeout: "20"
    scuttleLogging: false
  script:
    configMap:
      name: testfile
      file: demo.js

Other environment details (if applicable)

istio/proxyv2:1.14.5, kubernetes 1.22

Steps to reproduce the problem

kubectl apply -f k6-sample

Expected behaviour

The initializer pod completes gracefully and the k6-operator starts the starter and runner pods.

Actual behaviour

With scuttle:
The operator is unable to parse the initializer:k6 container logs and the following stack trace is logged

2022-12-07T16:11:35Z scuttle: Sent quitquitquit to Istio, status code: 200
`       {"k6": "ops/k6sample", "error": "invalid character '-' after top-level value"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/grafana/k6-operator/controllers.InitializeJobs.func1
        /workspace/controllers/k6_initialize.go:132
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:211
k8s.io/apimachinery/pkg/util/wait.WaitFor
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:541
k8s.io/apimachinery/pkg/util/wait.pollInternal
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:427
k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:452
k8s.io/apimachinery/pkg/util/wait.PollImmediate
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:441
github.com/grafana/k6-operator/controllers.InitializeJobs
        /workspace/controllers/k6_initialize.go:64
github.com/grafana/k6-operator/controllers.(*K6Reconciler).Reconcile
        /workspace/controllers/k6_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90
2022-12-07T16:12:12.916Z        ERROR   controllers.K6  Failed to initialize the script {"k6": "ops/k6sample", "error": "invalid character '-' after top-level value"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/grafana/k6-operator/controllers.InitializeJobs
        /workspace/controllers/k6_initialize.go:153
github.com/grafana/k6-operator/controllers.(*K6Reconciler).Reconcile
        /workspace/controllers/k6_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90
2022-12-07T16:12:12.916Z        ERROR   controller      Reconciler error        {"reconcilerGroup": "k6.io", "reconcilerKind": "K6", "controller": "k6", "name": "k6sample", "namespace": "ops", "error": "invalid character '-' after top-level value"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:237
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90
2022-12-07T16:12:13.916Z        INFO    controllers.K6  Reconcile(); stage = initialization     {"k6": "ops/k6sample"}
2022-12-07T16:12:13.916Z        DEBUG   controller      Successfully Reconciled {"reconcilerGroup": "k6.io", "reconcilerKind": "K6", "controller": "k6", "name": "k6sample", "namespace": "ops"}

or without scuttle enabled, the initializer job never finishes because the sidecar doesn't terminate on its own.
@13013SwagR 13013SwagR added the bug Something isn't working label Dec 7, 2022
@javaducky
Copy link
Contributor

Sorry about this @yorugac.

Looks like the update from #166 fixed Dapr while breaking Istio injection due to competing needs 😞. Sounds like adding initializer-specific configuration options would be needed for the K6Spec.

@yorugac yorugac mentioned this issue Dec 8, 2022
Merged
@yorugac
Copy link
Collaborator

yorugac commented Dec 8, 2022

@13013SwagR Thank you for spotting the issue!

@javaducky 👍 yes, I see; thanks for giving a quick hint. I'd like to double-check dapr logic but probably initializer's options should be put separately indeed. I'm reverting the removal of annotations in #173 for now.

@yorugac
Copy link
Collaborator

yorugac commented Dec 8, 2022

@13013SwagR annotations should be back and the new image was built. If you could please double-check your use case with the latest image, that'd be very helpful 🙂

@yorugac yorugac mentioned this issue Dec 8, 2022
Open
@13013SwagR
Copy link
Contributor Author

13013SwagR commented Dec 12, 2022 •
edited

Thanks for the revert @yorugac, it does fix the initializer issue with the Istio injected sidecar.
Note: I had to point directly to the latest commit 5cec748534424741fbea0e1875218b56bf5af670. It seems the latest tag wasn't updated.

@13013SwagR
Copy link
Contributor Author

Hey @yorugac,
I just opened a small PR to separate the initializer pod spec under its own key, see #175.

@13013SwagR 13013SwagR mentioned this issue Dec 13, 2022
Merged
3 tasks
@yorugac
Copy link
Collaborator

yorugac commented Dec 14, 2022

@13013SwagR thanks for confirming Istio case! I'll be closing this issue then. (Initializer pod spec is more about another issue.)

Note: I had to point directly to the latest commit 5cec748534424741fbea0e1875218b56bf5af670. It seems the latest tag wasn't updated.

I'm pretty sure I updated latest... Though IIRC, this can happen sometimes when using latest: it could be using the cached version of image so you might want to pull it manually or remove the cache, etc. Either way, I'll make an RC tag in a bit.

@yorugac yorugac closed this as completed Dec 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add initializer spec 13013SwagR/k6-operator
3 participants
@javaducky @yorugac @13013SwagR