Add Dedicated Recursive RTT Metrics alongside General RTT

[JF62]

Currently, tdns-stats polls performance metrics via poll.perfInterval and displays general RTT data. However, when troubleshooting DNS performance, it is critical to distinguish between general network/cached response times and the actual time the Technitium DNS Server takes to perform recursive lookups via upstream authorities.

I would like to see a dedicated metric/chart for Recursive RTT right next to the existing RTT metrics in both the Single Server and Cluster Overview dashboards.

Specifically:

• Dashboard Widget: A new metric card or an extension of the existing RTT card showing the Average Recursive RTT (and ideally p95/p99 if provided by the Technitium API).
• Chart Integration: An option to overlay the Recursive RTT on the real-time chart, allowing administrators to correlate high query volumes with upstream delays.
• Live Feed Column (Optional): A distinct indicator in the live feed for queries that triggered a recursive lookup, showing their individual recursive RTT.

Technitium's Query Log Apps already log recursive resolution times. The backend of tdns-stats would just need to extract the recursive-specific RTT fields from the Technitium Dashboard API during the poll.perfInterval cycle and push them to the frontend via the existing Server-Sent Events (SSE) pipeline.

[Hemsby]

Thanks for the detailed writeup. Good news on most of this: the RTT metrics you're seeing are already recursive-only, so there isn't a separate "general RTT" to sit next to.

What's already in place

• The RTT card is recursive RTT. _pollPerformance in poller.js pulls a sample from the query log app with responseType=Recursive and computes median / mean / p99 / jitter purely from the responseRtt of recursive queries. Cached, blocked, and authoritative responses never enter the math. The same card also shows a "Recursive %" figure. So the dashboard widget you're describing is essentially what's already rendering, just not labelled in a way that makes the "these are upstream resolution times" point obvious.
• The live feed already does the optional part. Every recursive query row shows its individual responseRtt with a low/mid/high latency colour band, and fmtMs deliberately blanks the latency for Cached / Blocked / Authoritative / Refused / Dropped since those are effectively instant from the server's side. Recursive rows also carry a distinct "Recursive" type badge.
• One clarification on the data source: these numbers come from the Query Logs app, not the Dashboard API. The Technitium Dashboard API doesn't expose RTT fields, which is why a query log app is a hard requirement for any of this.

What's genuinely new here

1. p95 on the card. p99 is already computed and shown; adding p95 alongside it is a small change.
2. Recursive RTT overlaid on the real-time chart. This is the real feature request and it's a bigger lift. That chart is rendered straight from Technitium's mainChartData (query counts per time bucket) with no tdns-stats-side time series behind it. The perf poll only produces one aggregate value per cycle (every poll.perfInterval seconds, default 30). To draw an RTT line we'd need tdns-stats to keep its own rolling buffer of mean/p95 recursive RTT keyed to poll cycles and plot it on a secondary y-axis. Contained mostly to poller.js and charts.js, but it's new persistent state and worth its own issue.

One caveat worth knowing either way: the percentiles are computed over the last 500 recursive entries per cycle, not a full server-wide history, so they reflect recent behaviour rather than a long-run distribution. Fine for troubleshooting, just not a lifetime stat.

Would p95 on the card plus a mean recursive RTT overlay on the chart cover what you're after, or do you also want per-cycle p95/p99 bands on the chart itself?

[JF62]

Thank you so much for the clarification and the quick response!

Ah, that makes total sense. I didn't realize the existing RTT card was already filtering out cache hits and focusing strictly on recursive upstream queries. A simple UI label tweak to make that clearer would be awesome.

Regarding your question about the chart:
Overlaying just the Mean Recursive RTT on the real-time chart (via a secondary y-axis) plus adding the p95 metric to the card would perfectly cover what I'm after! Having too many percentile bands directly in the chart might make it look too cluttered. Seeing the average latency trend to spot spikes, while having the exact p95/p99 values in the card, sounds like the ideal balance.

One additional idea regarding recursive servers:
Since we are troubleshooting upstream performance here, would it be possible to also display a "Top Forwarders / Upstream Servers" list (showing the IP/Name, query count, and average RTT per forwarder)?

Technitium exposes these forwarder stats via its Dashboard API. Having this in tdns-stats would be incredibly valuable, especially for cluster setups, to immediately see which upstream DNS (e.g., Cloudflare vs. Quad9) is actually answering, how the traffic is distributed, and which one responds the fastest so we can optimize our forwarder settings.

Thanks again for looking into this, I really appreciate your work on this project!

[Hemsby]

Perfect, that settles the chart scope. Mean recursive RTT on a secondary y-axis plus p95 on the card, no percentile bands on the chart itself. I'll also relabel the card so it's clear those figures are upstream resolution times, not general response times. I'll split this into two tracked issues: the label tweak plus p95 (quick), and the chart overlay (needs the rolling buffer, so it gets its own).

On the Top Forwarders / Upstream Servers idea

I went looking for this because it would be genuinely useful, but the data isn't available anywhere tdns-stats can reach.

Technitium does use per-forwarder timing internally. With multiple forwarders it runs latency-based selection, and with concurrent forwarding it races all of them and takes the fastest response. So at the moment of resolution it knows exactly which forwarder answered and how quickly. The problem is that this is transient state inside the resolver. It's used for the selection decision and then discarded.

Specifically:

• The Query Logs app schema is timestamp, client_ip, protocol, response_type, response_rtt, rcode, qname, qtype, qclass, answer. response_rtt is the total recursive time with no record of which forwarder produced it.
• The Dashboard API getTop endpoint only supports TopClients, TopDomains, TopBlockedDomains. There's no forwarder breakdown anywhere in the Dashboard API, and no RTT fields at all.
• The daily server log doesn't record the winning forwarder per query either.

The only place an upstream server and its RTT show up is the DNS Client tool (api/dnsClient/resolve), which reports the responding nameserver and round-trip time for a single manual lookup. That's a diagnostic one-shot, not something that's logged or aggregated.

So a per-forwarder query count and average RTT table isn't buildable today. It would need Technitium itself to expose the data first, either a forwarder field added to the query log apps or a resolver stats endpoint. Might be worth raising as a feature request on the main DnsServer repo. If that lands, adding the view here would be straightforward.

[JF62]

Perfect, thank you! I'm really looking forward to the changes. Adding the p95 metric, updating the label, and overlaying the Mean Recursive RTT on the chart sounds like an amazing upgrade for the dashboard.

As per your suggestion, I went ahead and opened a feature request on the official Technitium repository to see if they can expose the upstream forwarder metrics in the future:
TechnitiumSoftware/DnsServer#2127

Please feel free to jump into that issue and add any technical details, endpoint preferences, or database schema insights from a developer's perspective if you'd like!

Thanks again for your awesome work and for splitting this into tracked issues. Much appreciated!

[Hemsby]

Shipped in v2.2.14:

• Recursive RTT overlay on the Last Hour chart (Chart: optional mean recursive RTT overlay on the Last Hour view #30). A new "RTT" toggle in the chart controls plots mean recursive RTT on a right-hand ms axis, so upstream latency can be lined up against query volume and spikes. Mean line only, no percentile bands on the chart, as you suggested. tdns-stats keeps its own rolling per-minute history covering roughly the last hour, bucketed by each query's own timestamp, so the line is populated as soon as the page loads. Works for individual servers and the cluster overview; available on the Last Hour range only.
• P95 on the perf card (Perf card: label RTT as recursive-only, add P95 #29), shown next to the existing P99.
• Perf card RTT section relabelled "Recursive RTT (ms)" (Perf card: label RTT as recursive-only, add P95 #29), with a tooltip noting that cache hits, blocked, and authoritative responses are excluded, so it is clear these are upstream resolution times.

#29 and #30 are left open for your feedback once you have had it running against your own setup.

Still outstanding: the Top Forwarders / Upstream Servers list. As covered earlier in this thread, Technitium does not expose per-forwarder query counts or RTT through the Dashboard API, and the query log app does not record which forwarder answered a recursive query, so that one is not buildable in tdns-stats without an upstream change to Technitium (a forwarder field on the query log apps, or a resolver stats endpoint). Might be worth raising on the main DnsServer repo if it would be valuable for you.

Thanks again for the detailed writeup.

[JF62]

Wow, thank you so much for the incredibly fast implementation and for releasing this in v2.2.14!

I have just updated my setup and tested it thoroughly against my high-availability cluster. It works absolutely beautifully!

• The Chart Overlay: The new "RTT" toggle on the Last Hour chart is exactly what I was hoping for. Having the secondary y-axis makes it incredibly easy to immediately correlate query spikes with upstream latency. The rolling history loading instantly on page load is a fantastic touch.
• The Perf Card: The "Recursive RTT (ms)" relabeling, the new tooltip, and the addition of the P95 metric make the dashboard much more intuitive and precise for troubleshooting.

Since everything is running perfectly and behaves exactly as expected, you can absolutely close issues #29 and #30 from my side.

Regarding the Top Forwarders: I already opened the feature request over at the Technitium repo (#2127) as we discussed. Hopefully, they will consider exposing those metrics in a future API/schema update so we can revisit it here.

Thanks again for your amazing work and responsiveness on this!