Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alert clients repeatedly post the alerts of ended. #1945

Closed
barnettZQG opened this Issue Sep 5, 2016 · 7 comments

Comments

Projects
None yet
4 participants
@barnettZQG
Copy link

barnettZQG commented Sep 5, 2016

I implement a alertmanager and listens for alerts on an API endpoint at /api/v1/alerts.
image
but,I found that Prometheus would repeat send already the end of the alert information, as shown in figure can see.
Why is this?

@fabxc

This comment has been minimized.

Copy link
Member

fabxc commented Sep 5, 2016

We want to notify the Alertmanager about resolved alerts. For reliability
we do not rely on an alerts successfully being sent once to mean that they
were actually processed. After all Alertmanager could have been restarted
right after. Thus, we re-send alerts as long as they are firing.
The same is true for resolved alerts. There we keep resending them for 15
more minutes before actually removing them from the internal state. After
15 minutes you should no longer see them.

On Mon, Sep 5, 2016 at 1:29 PM barnettZQG notifications@github.com wrote:

I implement a alertmanager and listens for alerts on an API endpoint at
/api/v1/alerts.
[image: image]
https://cloud.githubusercontent.com/assets/18493394/18246544/cc2ef3c6-739e-11e6-96f1-e357a5346e5b.png
but,I found that Prometheus would repeat send already the end of the alert
information, as shown in figure can see.
Why is this?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1945, or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuA8jMU7eYFa_HKaJf6V2ay7pujDJCcks5qm_0tgaJpZM4J0_IT
.

@barnettZQG

This comment has been minimized.

Copy link
Author

barnettZQG commented Sep 5, 2016

Understand.So I should according to the startsAt and endsAt time to determine whether the same alert again already exists.

@fabxc

This comment has been minimized.

Copy link
Member

fabxc commented Sep 5, 2016

Yes, generally the semantics are very much geared towards Alertmanager, which has a merging and deduplication procedure for incoming alerts.
If you want to consume them as an end user, you should do so via Alertmanager's notification rather than writing your own layer. If you want to run your own thing, you'll have to deal with duplication.

Closing the issue as it's expected behavior but we can discuss further here if you have any questions.

@barnettZQG

This comment has been minimized.

Copy link
Author

barnettZQG commented Sep 5, 2016

yes,thank you

@barnettZQG barnettZQG closed this Sep 5, 2016

@rajendrakumara

This comment has been minimized.

Copy link

rajendrakumara commented Apr 28, 2017

Hi,

"After 15 minutes you should no longer see them".

Is 15 minutes configurable, in the above context.

Thanks,
Rajendra Kumar. A

@juliusv

This comment has been minimized.

Copy link
Member

juliusv commented Apr 29, 2017

@lock

This comment has been minimized.

Copy link

lock bot commented Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.