[Regression]: prometheus/prom_rules/prometheus.rules.yml: severities are inverted

[vladzcloudius]

Installation details
Scylla-Monitoring Version: 4.4.2

Description
Patch 2e3d0c7

commit 2e3d0c7280599f2d09b9eea102f522defdd05db6
Author: Amnon Heiman <amnon@scylladb.com>
Date:   Wed Mar 15 10:22:20 2023 +0200

    prometheus.rules.yml: Use severity as string instead of numbers

set wrong severities values.

For example he set "info" severity to a "DiskFull with less than 15% free" alert and "error" for a "DiskFull 35% disk left"

  - alert: DiskFull
    expr: node_filesystem_avail_bytes{mountpoint="/var/lib/scylla"} / node_filesystem_size_bytes{mountpoint="/var/lib/scylla"}
      * 100 < 35
    for: 30s
    labels:
      severity: "error"
    annotations:
      description: '{{ $labels.instance }} has less than 35% free disk space.'
      summary: Instance {{ $labels.instance }} low disk space
  - alert: DiskFull
    expr: node_filesystem_avail_bytes{mountpoint="/var/lib/scylla"} / node_filesystem_size_bytes{mountpoint="/var/lib/scylla"}
      * 100 < 25
    for: 30s
    labels:
      severity: "warn"
    annotations:
      description: '{{ $labels.instance }} has less than 25% free disk space.'
      summary: Instance {{ $labels.instance }} low disk space
  - alert: DiskFull
    expr: node_filesystem_avail_bytes{mountpoint="/var/lib/scylla"} / node_filesystem_size_bytes{mountpoint="/var/lib/scylla"}
      * 100 < 15
    for: 30s
    labels:
      severity: "info"
    annotations:
      description: '{{ $labels.instance }} has less than 15% free disk space.'
      summary: Instance {{ $labels.instance }} low disk space

[vladzcloudius]

cc @igorribeiroduarte @ebenzecri @AdamStawarz @noellymedina @tomer-sandler @harel-z

[vladzcloudius]

@amnonh as trivial the change is - it's very critical. Could you, please, try to fix this ASAP?

[vladzcloudius]

@amnonh #2030 doesn't fix this issue unfortunately.
The offending patch messed up many other alerts - not just DiskFull.

Please, go other the whole https://github.com/scylladb/scylla-monitoring/commit/2e3d0c7280599f2d09b9eea102f522defdd05db6 patch.

The translation was supposed to be

1 - info
2 - warn
3 - error
4 - critical

Just as a sample - look at this screenshot of the part of the patch in question.
All 4 alerts are translated wrong there (and there are a lot more like these in this patch):

You may consider asking for a review for following patches in this context.

[amnonh]

@vladzcloudius thanks for the update, I miss-understand the original issue and wanted to provide a patch release asap, I've removed 4.4.3 and will take the longer route

[vladzcloudius]

@amnonh Could you, please, clarify which releases have the fix for this?
Is it only 4.5.0 or some 4.4.x too?

[amnonh]

It was backported to 4.4.3 and after