Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Admission webhook for VMAlertmanagerConfig does not handle all invalid configs #825

Closed
bh-tt opened this issue Dec 19, 2023 · 2 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@bh-tt
Copy link
Contributor

bh-tt commented Dec 19, 2023

Applying the following VMAlertmanagerConfig to a cluster with the operator installed and admission webhooks enabled does not result in a failure from the admission webhook but a failure in the operator log. It should result in a validation admission webhook failure.

The failure in this case is a missing require_tls setting in the email_configs.

I'm not sure why the admission webhook would use a different set of validation rules than the actual reconciliation, would that make the validation webhook dependent on other resources present in the cluster?

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAlertmanagerConfig
metadata:
  name: metric-target-down-test
spec:
  route:
    receiver: email
    group_by: ["service","alertname"]
    routes:
      - receiver: email
        continue: false
  receivers:
    - name: email
      email_configs:
        - to: test@example.com
{"level":"error","ts":1702980403.351607,"logger":"factory","msg":"bad configs found during alertmanager config building","alertmanager":"alertmanager","error":"errors: incorrect email configuration, tls is required, but no config provided at
 spec in object: it/metric-target-down-test, will ignore vmalertmanagerconfig metric-target-down-test","stacktrace":"github.com/VictoriaMetrics/operator/controllers/factory.buildAlertmanagerConfigWithCRDs\n\tgithub.com/VictoriaMetrics/operat
or/controllers/factory/alertmanager.go:657\ngithub.com/VictoriaMetrics/operator/controllers/factory.createDefaultAMConfig\n\tgithub.com/VictoriaMetrics/operator/controllers/factory/alertmanager.go:547\ngithub.com/VictoriaMetrics/operator/con
trollers/factory.CreateOrUpdateAlertManager\n\tgithub.com/VictoriaMetrics/operator/controllers/factory/alertmanager.go:90\ngithub.com/VictoriaMetrics/operator/controllers.(*VMAlertmanagerReconciler).Reconcile\n\tgithub.com/VictoriaMetrics/op
erator/controllers/vmalertmanager_controller.go:86\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-ru
ntime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsig
s.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.13.1/pkg/internal/controller/controller.go
:234"}
@Haleygo Haleygo added the bug Something isn't working label Dec 20, 2023
@f41gh7 f41gh7 self-assigned this Jul 30, 2024
f41gh7 added a commit that referenced this issue Aug 5, 2024
* adds more strict validation to VMAlertmanagerConfig
* skip broken configurations from VMAlertmanager during config build
* adds new status fields to the VMalertmanager config:
  * lastSyncError - that defines last error occured to the config during config generation
  * status - generic CRD status for health track
  * additional fields for sync error context, like last error timestamp and name of alertmanager
* adds stack trace only to the panic level errors. It should reduce log noise

#825
Signed-off-by: f41gh7 <nik@victoriametrics.com>
@f41gh7
Copy link
Collaborator

f41gh7 commented Aug 6, 2024

Starting with next release VMAlertmanager config must properly validate fields and given error must be fixed.

@f41gh7
Copy link
Collaborator

f41gh7 commented Aug 15, 2024

released in v0.47.0

@f41gh7 f41gh7 closed this as completed Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants