Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Track objects that differ for a longer period separately #9

Merged
merged 2 commits into from
Jul 21, 2021
Merged

Track objects that differ for a longer period separately #9

merged 2 commits into from
Jul 21, 2021

Conversation

ties
Copy link
Collaborator

@ties ties commented Jun 22, 2021 •
edited
Loading

Use case: track how long it takes for VRPs in one endpoint (could be an internal source of truth or a RP) in another.

For example:

# HELP vrp_diff Number of VRPS in [lhs_url] that are not in [rhs_url] that were first seen [visibility_seconds] ago in lhs.
# TYPE vrp_diff gauge
vrp_diff{lhs_url="https://routinator-demo.aws.nlnetlabs.nl/json",rhs_url="tcp://rtr.rpki.cloudflare.com:8282",visibility_seconds="0"} 2241
vrp_diff{lhs_url="https://routinator-demo.aws.nlnetlabs.nl/json",rhs_url="tcp://rtr.rpki.cloudflare.com:8282",visibility_seconds="256"} 2240
vrp_diff{lhs_url="https://routinator-demo.aws.nlnetlabs.nl/json",rhs_url="tcp://rtr.rpki.cloudflare.com:8282",visibility_seconds="56"} 2241
vrp_diff{lhs_url="tcp://rtr.rpki.cloudflare.com:8282",rhs_url="https://routinator-demo.aws.nlnetlabs.nl/json",visibility_seconds="0"} 5
vrp_diff{lhs_url="tcp://rtr.rpki.cloudflare.com:8282",rhs_url="https://routinator-demo.aws.nlnetlabs.nl/json",visibility_seconds="256"} 5
vrp_diff{lhs_url="tcp://rtr.rpki.cloudflare.com:8282",rhs_url="https://routinator-demo.aws.nlnetlabs.nl/json",visibility_seconds="56"} 5

What you see here is that one of the VRPs not present in the routinator endpoint is "recent".

When you monitor multiple instances (for example, one rsync vs rrdp, or stayrtr vs its input) a time-lag between various instances is logical. This time lag is not guaranteed to "go to 0" when there are continuous updates. My experience is that alerting on objects that have been seen for a while prevents spurious alerts.

This is available in a container for testing at https://hub.docker.com/repository/docker/tiesdekock/rtrmon

ties added 2 commits June 22, 2021 08:39
@ties
Track objects that differ for a longer period separately
79a006c 
  * Track when the VRP was first seen for each VRP
  * Add a metric which tracks how many objects that were first seen
    [duration] ago are not present in the other endpoint.
    * Use current naming convention for the metric to be consistent (i.e. no
      prefix with application name).
@ties
Visibility thresholds can be overridden via args
71a7648 
  * Visibility thresholds can be overridden via command line arguments.
  * Format the default thresholds in help output
@ties ties force-pushed the feature/track-long-term-diffs branch from 91a57ca to 71a7648 Compare June 22, 2021 08:52
@ties ties requested a review from lspgn July 1, 2021 10:06
@job job force-pushed the master branch 4 times, most recently from da402a1 to d74e626 Compare July 18, 2021 14:27
@ties ties mentioned this pull request Jul 20, 2021
Closed
@job job merged commit 8547db1 into bgp:master Jul 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lspgn lspgn Awaiting requested review from lspgn

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ties @job