Unified Alerting: Set max_attempts to 1 by default

[gotjosh]

The retry logic for unified alerting has been broken as far as v9.4.x, rather than fixing it in one go and causing a headache to our users with rules putting extra load on their datasources - I think a better approach is to simply set 1 as a default and then let our users change it.

I see two cons with this approach:

• Configuration for legacy to unified alerting cannot be ported over automatically, users will have to manually set max_attempts to 3 when migrating.
• Users expecting to get any sort of retrying (as with legacy alerting) will not have it out of the box and will have to manually edit the configuration.

I think this is the best compromise we can make with the introduction of #79037

Signed-off-by: gotjosh josue.abreu@gmail.com

[alexweav]

LGTM as it appears this config option has no effect currently.

[JacobsonMT]

LGTM

[gotjosh]

We discussed this offline extensively and all agreed that it was a fine change to merge.

Please find the conversation in: https://raintank-corp.slack.com/archives/C028MCV4R7C/p1701445758479449

[grafana-delivery-bot]

The backport to v9.4.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-79095-to-v9.4.x origin/v9.4.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 0c9356a3c78b83b8bb5abf99263053b10de84d48

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-79095-to-v9.4.x
# Create the PR body template
PR_BODY=$(gh pr view 79095 --json body --template 'Backport 0c9356a3c78b83b8bb5abf99263053b10de84d48 from #79095{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title "[v9.4.x] Unified Alerting: Set `max_attempts` to 1 by default" --body-file - --label "type/docs" --label "area/backend" --label "add to changelog" --label "product-approved" --label "backport" --base v9.4.x --milestone 9.4.x --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-79095-to-v9.4.x

# Create a pull request where the `base` branch is `v9.4.x` and the `compare`/`head` branch is `backport-79095-to-v9.4.x`.

# Remove the local backport branch
git switch main
git branch -D backport-79095-to-v9.4.x

[grafana-delivery-bot]

The backport to v9.5.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-79095-to-v9.5.x origin/v9.5.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 0c9356a3c78b83b8bb5abf99263053b10de84d48

When the conflicts are resolved, stage and commit the changes:

git add . && git cherry-pick --continue

If you have the GitHub CLI installed:

# Push the branch to GitHub:
git push --set-upstream origin backport-79095-to-v9.5.x
# Create the PR body template
PR_BODY=$(gh pr view 79095 --json body --template 'Backport 0c9356a3c78b83b8bb5abf99263053b10de84d48 from #79095{{ "\n\n---\n\n" }}{{ index . "body" }}')
# Create the PR on GitHub
echo "${PR_BODY}" | gh pr create --title "[v9.5.x] Unified Alerting: Set `max_attempts` to 1 by default" --body-file - --label "type/docs" --label "area/backend" --label "add to changelog" --label "product-approved" --label "backport" --base v9.5.x --milestone 9.5.x --web

Or, if you don't have the GitHub CLI installed (we recommend you install it!):

# Push the branch to GitHub:
git push --set-upstream origin backport-79095-to-v9.5.x

# Create a pull request where the `base` branch is `v9.5.x` and the `compare`/`head` branch is `backport-79095-to-v9.5.x`.

# Remove the local backport branch
git switch main
git branch -D backport-79095-to-v9.5.x