Alerting: Allow none provenance alert rule creation from provisioning API

[alexmobo]

What is this feature?

There has been some pain around being able to modify alerts created via API lately. Since our Alerting API is not stable just yet we are enabling a workaround within the Provisioning API to do it by adding the x-disable-provenance header to the following requests:

• POST /api/v1/provisioning/alert-rules
• PUT /api/v1/provisioning/alert-rules/{UID}

This will create alert rules as if they were created via UI and make the editable.

Why do we need this feature?

Many clients are asking to be able to create editable alert rules from the API.

Who is this feature for?

Users that want to edit API-created alert rules using the UI.

Which issue(s) does this PR fix?:

Fixes #57911

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41319

[JohnnyQQQQ]

I just saw that you edited a generated file (pkg/services/ngalert/api/generated_base_api_provisioning.go). This should not be done. You will need to look at another way to do this.

You could i.e. extend the generation template to consider headers when generating handlers. The codegen code can be found in grafana/pkg/services/ngalert/api/tooling

[alexmobo]

I just saw that you edited a generated file (pkg/services/ngalert/api/generated_base_api_provisioning.go). This should not be done. You will need to look at another way to do this.

You could i.e. extend the generation template to consider headers when generating handlers. The codegen code can be found in grafana/pkg/services/ngalert/api/tooling

I didn't notice it was generated even though it says in the name of the file. Thanks.

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41708

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41715

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41742

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41749

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41754

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41772

[grobinson-grafana]

The changes look good to me!

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41812

[grafanabot]

Drone build failed: https://drone.grafana.net/grafana/grafana-enterprise/41883

[JohnnyQQQQ]

LGTM

[greatvovan]

Please explain, how it will help with Terraformed alerts?

[alexmobo]

There is no workaround to do it directly from terraform. This is API only.

[alexweav]

@greatvovan @alexmobo Actually, this is possible. The grafana terraform provider allows you to provide any set of custom headers via the http_headers provider config option. So you can add the header there, and all the requests will carry the header, so they will all be UI-editable.

See the option docs here:
https://registry.terraform.io/providers/grafana/grafana/latest/docs#optional

[phyzical]

hey @alexweav any chance you could confirm this is how it should be set?

http_headers = {
    "X-Disable-Provenance" = "disabled"
  }

running 9.3.0 i tried remove some alerts and recreating but sadly still complained it was provisioned still

[nitesh8860]

i just tried it with grafana v9.4.3 and terraform provider v1.35.0, both are latest, and it doesn't work

provider "grafana" {
  url          = var.grafana_url
  auth         = var.auth
  http_headers = var.http_headers
}

variable "http_headers" {
  type = map(string)
  default = {
    "X-Disable-Provenance" = "disabled"
  }
}

while working with api directly works /api/v1/provisioning/alert-rules/ and providing http_header

Please review and let me know if there is some other way of doing this.

Thanks
Nitesh

[phyzical]

@nitesh8860 thanks for trying also, i think this might actually be an issue to be raised for the terraform provider instead

[joegoggins]

I'm using GET /api/v1/provisioning/alert-rules/export to extract alert rules from an existing instance with working rules, then adding them to provisioning/alerting/alert-rules.yaml. I'm wondering if there is a way to use this X-Disable-Provenance header in this context? (see docs)

For dashboards, I use allowUiUpdates: true in the provisioning/dashboards/default.yaml to allow tinkering/refinements . I was hoping there was a similar flag I could enable in the alert-rules.yaml to make alert tinkering possible post provisioning.