-
Notifications
You must be signed in to change notification settings - Fork 0
Observability
Harpyx instruments three pillars: tracing (OpenTelemetry), metrics (OpenTelemetry), and logs (Serilog). In addition, each service exposes health endpoints for liveness and readiness probes.
Configured identically in the WebApp and Worker via ConfigureOpenTelemetry in their respective extension classes. Exporters target OTLP when OpenTelemetry:Otlp:Endpoint is set — otherwise instrumentation still runs and populates System.Diagnostics.Activity/Meter, just without external export. Useful for local dev where you can attach dotnet trace or dotnet counters without standing up a collector.
Instrumented sources:
| Source | Emitted by |
|---|---|
| ASP.NET Core (WebApp only) | Incoming HTTP requests |
| HttpClient | Outbound HTTP (LLM providers, URL fetches) |
| SqlClient | SQL Server queries |
HarpyxObservability.BillingActivitySource |
Subscription / billing feature checks |
HarpyxObservability.JobsActivitySource |
ParseDocumentJob execution spans |
HarpyxObservability.WebhooksActivitySource |
Outbound webhook deliveries |
Spans carry tags like document.id and job.type to make correlation with jobs and documents trivial in the exporter.
Built-in instrumentation:
| Source | Emits |
|---|---|
| ASP.NET Core (WebApp) | Request duration, active requests |
| HttpClient | Outbound latency, failures |
| Runtime | GC, thread pool, exceptions |
Custom meters:
| Meter | Key instruments |
|---|---|
HarpyxObservability.BillingMeterName |
Billing-feature grants/denials |
HarpyxObservability.JobsMeterName |
JobProcessedCounter with tags job_type, result ∈ completed/failed/skipped_security |
HarpyxObservability.WebhooksMeterName |
Delivery success/failure counters |
The job_type + result breakdown on JobProcessedCounter is the fastest way to spot ingestion regressions (e.g., spike in result=failed).
Each service announces its name and assembly version to the exporter:
- WebApp:
service.name = Harpyx.WebApp - Worker:
service.name = Harpyx.Worker -
service.version=Assembly.GetName().Version
This keeps traces and metrics cleanly segmented in the collector.
Configured in both hosts via ReadFrom.Configuration(builder.Configuration) pulling from the Serilog:* section, with a console sink always attached. Default minimum level is Information; Microsoft and System namespaces are gated at Warning to reduce framework noise.
Structured properties (e.g. DocumentId, TenantId) are consistent with OTel span tags, so log/trace correlation is straightforward in sinks that support both (Seq, Datadog, Grafana Loki).
Adding extra sinks (Seq, file, Elasticsearch, Loki) is a matter of dropping the corresponding NuGet package and extending the Serilog section — no code changes required.
The WebApp exposes two endpoints:
-
GET /health/live— responds 200 while the process is alive (no dependencies). -
GET /health/ready— 200 only when every critical backing service is reachable.
Registered checks:
| Tag | Check |
|---|---|
live |
self (always healthy) |
ready |
SqlServerReadinessHealthCheck |
ready |
RedisReadinessHealthCheck |
ready |
OpenSearchReadinessHealthCheck |
ready |
RabbitMqReadinessHealthCheck |
ready |
MinioReadinessHealthCheck |
ready |
ClamAvReadinessHealthCheck |
Admins can also open /Admin/Health for a human-readable status page. It runs the same checks, labels each dependency as required or optional, and shows the operational impact of failures without changing probe behavior.
A HealthAlertPublisher emits alerts for any ready regression every 60 seconds after a 15-second warm-up, so production incidents surface independently of probe traffic.
| Symptom | First place to look |
|---|---|
Uploads queue but never land Ready
|
JobProcessedCounter{result=failed} spike; dead-letter queue depth |
| Auth failures spike |
access_denied audit rate; auth rate-limit rejections |
| Slow chat responses | OTel spans on HttpClient to the LLM provider |
| Readiness probe flapping | Correlate with the tagged ready health check that's failing |
| Unexplained OS errors | OpenSearch logs + OpenSearch:AllowInsecureTls, cert expiry |
| Silent data protection breakage | Shared keys path mounted and writable by all WebApp instances |