Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade prom and alert manager to 2.7.1 and 0.16.1 #1098

Conversation

LucasRoesler
Copy link
Member

Description

Motivation and Context

  • I have raised an issue to propose this change (required)

Resolves #1095

How Has This Been Tested?

Tested manually on my Mac

  1. deploy via the script
./deploy_stack.sh
  1. deploy the go-echo function https://github.com/alexellis/echo-fn
  2. create constant load using hey
hey -c 2 -q 10 -n 100000 http://127.0.0.1:31112/function/go-echo
  1. observe that the function scales in steps relatively quickly after the alert stops and that the function then scales quickly to 1 replica when the alert stops, this is shown in the below graphs

image

http://localhost:9090/graph?g0.range_input=15m&g0.expr=rate(gateway_function_invocation_total%5B10s%5D)&g0.tab=0&g1.range_input=15m&g1.expr=ALERTS%7Balertname%3D%22APIHighInvocationRate%22%7D&g1.tab=0&g2.range_input=15m&g2.expr=gateway_service_count&g2.tab=0

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I've read the CONTRIBUTION guide
  • I have signed-off my commits with git commit -s
  • I have added tests to cover my changes.
  • All new and existing tests passed.

**What**
- Removes the `alert` label in the scale-up alert
- Updates the annotaitons to use the `function_name` label instead of
the `instance` label that was removed.
- Per prometheus/prometheus#4836 and the related mailing list discussion
https://groups.google.com/d/msg/prometheus-users/7Ul6ngc7Ogs/j_YDszV5BwAJ
the alert value should not be included in the alert labels otherwise
each calculation of the alert is treated like a new alert and then the
use of `for 5s` will not behave as expected.
- Ports the resoltuion openfaas/faas-netes#372

Signed-off-by: Lucas Roesler <roesler.lucas@gmail.com>
Copy link
Member

@alexellis alexellis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together. More unrelated changes than I expected around formatting and quotes, but if this still works as before I'll merge.

@alexellis alexellis merged commit 3bcc10a into openfaas:master Feb 21, 2019
@LucasRoesler LucasRoesler deleted the fix-upgrade-to-latest-prometheus-and-alertmanager branch April 22, 2019 10:24
@alexellis
Copy link
Member

Derek lock

@derek derek bot locked and limited conversation to collaborators Jul 5, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update AlertManager & Prometheus to match faas-netes
2 participants