Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Query/Ruler: Add metric for number of endpoints resolved for each configured address. #1260

Merged
merged 1 commit into from Jun 18, 2019
Merged

Query/Ruler: Add metric for number of endpoints resolved for each configured address. #1260

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 11 additions & 5 deletions pkg/discovery/dns/provider.go
View file
Open in desktop
Expand Up @@ -20,6 +20,7 @@ type Provider struct {
resolved map[string][]string
logger log.Logger

resolverAddrs *prometheus.GaugeVec
resolverLookupsCount prometheus.Counter
resolverFailuresCount prometheus.Counter
}
Expand Down Expand Up @@ -52,6 +53,10 @@ func NewProvider(logger log.Logger, reg prometheus.Registerer, resolverType Reso
resolver: NewResolver(resolverType.ToResolver(logger)),
resolved: make(map[string][]string),
logger: logger,
resolverAddrs: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "dns_provider_results",
Help: "The number of resolved endpoints for each configured address",
}, []string{"addr"}),
resolverLookupsCount: prometheus.NewCounter(prometheus.CounterOpts{
Name: "dns_lookups_total",
Help: "The number of DNS lookups resolutions attempts",
Expand All @@ -63,6 +68,7 @@ func NewProvider(logger log.Logger, reg prometheus.Registerer, resolverType Reso
}

if reg != nil {
reg.MustRegister(p.resolverAddrs)
reg.MustRegister(p.resolverLookupsCount)
reg.MustRegister(p.resolverFailuresCount)
}
Expand Down Expand Up @@ -91,6 +97,7 @@ func (p *Provider) Resolve(ctx context.Context, addrs []string) {
p.resolverLookupsCount.Inc()
if err != nil {
// The DNS resolution failed. Continue without modifying the old records.
p.resolved[addr] = p.resolved[addr] // Ensure metrics capture the result even if empty.
Copy link
Member

@bwplotka bwplotka Jun 18, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really get this part is, it intentional? Isn't this noop?
It's tricky what should be there TBH ;p Should we set results for both query to empty and metrics to 0? should we only adjust metrics? Not sure yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ensures that the map contains an entry for the address. So when we iterate over the map keys for the metric, we report that there was an address configured regardless of whether we got some results here, got no results, or continue using previous results.

p.resolverFailuresCount.Inc()
level.Error(p.logger).Log("msg", "dns resolution failed", "addr", addr, "err", err)
continue
Expand All @@ -99,15 +106,14 @@ func (p *Provider) Resolve(ctx context.Context, addrs []string) {
}

// Remove stored addresses that are no longer requested.
var entriesToDelete []string
for existingAddr := range p.resolved {
if !contains(addrs, existingAddr) {
entriesToDelete = append(entriesToDelete, existingAddr)
delete(p.resolved, existingAddr)
p.resolverAddrs.DeleteLabelValues(existingAddr)
} else {
p.resolverAddrs.WithLabelValues(existingAddr).Set(float64(len(p.resolved[existingAddr])))
}
}
for _, toDelete := range entriesToDelete {
delete(p.resolved, toDelete)
}
}

// Addresses returns the latest addresses present in the Provider.
Expand Down