Skip to content

v2.0.0

Choose a tag to compare

@github-actions github-actions released this 21 May 06:58
· 208 commits to main since this release
5fa0a0b

What's Changed

Bug fixes

  • Zero-value metrics now emitted (#49) — tdarr_library_transcodes and tdarr_library_health_checks now emit 0 for known statuses regardless of whether the API includes them. No more disappearing series mid-scrape, no more Grafana gaps. (58015cb, 7d7289c)
  • Cache invalidation expanded — pie data now refreshes on queue/hold/cancellation transitions via table0Counttable6Count from Tdarr's StatisticsJSONDB, not just file/transcode/healthcheck total changes. Previously, queued items stayed invisible until a transcode completed. (7d7289c)
  • Per-collection-cycle failures now signaled as a scrapable gauge — replaces the broken tdarr_collector_error (which used prometheus.NewInvalidMetric and produced unscrapable "error collecting metric" text in /metrics) with tdarr_up (1=success, 0=failure). Includes per-library pie fetch failures via partial-failure flag. Operators can now alert directly on tdarr_up == 0. (58015cb, this branch)
  • tdarr_scrape_duration_seconds now reports real time — was always 0 because start := time.Now() was inside the defer. (7d7289c)
  • tdarr_stream_stats_num_frames now registered in Describe() — pre-existing bug. (2bb6d03)

New metrics

  • tdarr_node_worker_count{worker_type, compute_type} — active workers per role. worker_type{transcode, healthcheck}, compute_type{cpu, gpu}. Unknown API values pass through as worker_type=<raw> with compute_type="unknown".
  • tdarr_node_worker_limit{worker_type, compute_type} — configured worker cap per role
  • tdarr_node_queue_length{worker_type, compute_type} — queued jobs per role
  • tdarr_node_paused, tdarr_node_max_gpu_workers, tdarr_node_schedule_enabled — node state gauges
  • tdarr_node_worker_percentage, _fps, _original_file_size_gb, _output_file_size_gb, _est_file_size_gb, _job_start_timestamp_seconds, _start_timestamp_seconds, _status_timestamp_seconds, _eta_seconds, _pid — per-worker progress gauges
  • tdarr_unknown_status_total{job_kind, status} — counter for pie statuses outside the known enum (alert via increase(tdarr_unknown_status_total[24h]) > 0 for API drift detection)

Breaking changes

⚠️ This release drops support for Tdarr versions older than v2.24.01. The old pies array parsing path has been removed in favor of /api/v2/stats/get-pies exclusively. Users on older Tdarr versions should not upgrade until they upgrade Tdarr.

  • tdarr_node_info labels stripped — numeric labels removed (node_gpu_health_check_limit, node_cpu_health_check_limit, node_gpu_transcode_limit, node_cpu_transcode_limit, node_health_check_gpu_queue, node_health_check_cpu_queue, node_transcode_gpu_queue, node_transcode_cpu_queue). Use the new gauges instead. (acbd3e5)
  • tdarr_node_worker_info labels restructured — numeric fields (worker_percentage, worker_fps, worker_original_file_size_gb, worker_output_size_gb, worker_est_size_gb, worker_status_ts, worker_job_start_ts, worker_start_ts) removed. Use the new per-worker gauges. New label: flow_worker="true|false". (acbd3e5)
  • tdarr_node_worker_flow_info removed — unified into tdarr_node_worker_info with flow_worker="true" filter. Plugin labels (worker_plugin_id, worker_plugin_position) emit empty strings for flow workers. (acbd3e5)
  • tdarr_node_busy removed — use sum(tdarr_node_worker_count) by (node_name) > 0 instead. (acbd3e5)
  • tdarr_collector_error removed, replaced by tdarr_up — polarity flipped to match Prometheus exporter convention (haproxy_up, mysqld_up, postgres_up). 1 = success, 0 = failure. Old metric was broken anyway (emitted via NewInvalidMetric, never appeared as scrapable series). Alerts must migrate: tdarr_collector_error == 1tdarr_up == 0. (df57645)
  • Label hygiene renames (9bd1343) — descriptive label names per Prometheus naming guidance:
    • typeworker_type on tdarr_node_worker_count, tdarr_node_worker_limit, tdarr_node_queue_length (aligns with the same-named label on tdarr_node_worker_info, enables clean group_left joins)
    • kindjob_kind on tdarr_unknown_status_total (disambiguates against the sibling status label)
    • allow_gpu_do_cpugpu_can_do_cpu on tdarr_node_info (reads as English)
    • node_paused label dropped from tdarr_node_info — was duplicating the existing tdarr_node_paused gauge. Use the gauge.
  • Compound worker_type split into two orthogonal labels (5f6ae4c) — previously worker_type carried compound values (transcodecpu / transcodegpu / healthcheckcpu / healthcheckgpu) smashing two dimensions into one string. Now split:
    • worker_type{transcode, healthcheck}
    • compute_type{cpu, gpu} (sentinel "unknown" for parse failures)
    • Applies to tdarr_node_worker_count, tdarr_node_worker_limit, tdarr_node_queue_length, tdarr_node_worker_info.
    • Follows the standard Prometheus single-dimension-per-label convention (cf. node_exporter mode + cpu). sum by(compute_type) and sum by(worker_type) now answer their respective questions without regex.
  • Synthetic library_id="all_libraries" aggregate series removed (bb19f19) — the collector previously emitted a per-metric tdarr_library_*{library_id="all_libraries", library_name="all"} aggregate alongside the per-library series. Removed: it duplicated information derivable via sum(), forced every dashboard aggregation query to carry a defensive library_id!="all_libraries" filter to avoid double-counting, and polluted label_values() lookups. The collector now skips (with warn log) any library row returned with empty id or name rather than relabeling it as the synthetic aggregate. Queries / alerts that addressed the aggregate directly must rewrite to sum(...) form.

Internal cleanup

  • Dropped old API parsing path (TdarrMetric.Pies field + getPieMetricsFields + loadKeyValue helpers) — exclusively uses /api/v2/stats/get-pies now. (post-merge refactor in 27a1b01)
  • Simplified status label cleanup function. (e3ada65)
  • Go format + lint passes. (e2277cc, 656af3a)
  • Collector refactoredCollect() extracted to inner collect(ch) error helper; outer wrapper always emits the tdarr_up gauge exactly once via deferred unified emission. partialFailure atomic flag now reset unconditionally before the OR check so stale true cannot leak across consecutive scrapes. Describe() now includes the up-metric desc (pre-existing omission). (this branch)
  • Unit tests added — new internal/collector/tdarr_test.go with 9 tests using httptest.NewServer and isolated prometheus.NewRegistry(). Covers happy path, every failure path (stats fetch, score parse, health-score parse, library list fetch, partial pie failure, node fetch), consecutive-scrape partial-flag reset, and Describe() completeness. (this branch)
  • Makefile test targets addedtest, test_verbose, test_race, test_cover, test_collector. (this branch)
  • go.mod tidygithub.com/prometheus/client_model promoted to direct dependency (used by tests). (this branch)

Dashboard — v2 rewrite

examples/dashboard.json was rewritten from scratch for v2.0.0 against the new metric surface. The v1 dashboard is not forward-compatible — import the new one fresh. The previous version is preserved in-repo as examples/dashboard.v1.backup.json for reference / fallback.

Detecting failures

This release tightens failure-detection signals:

Failure mode Detection Example PromQL
Exporter unreachable / dead up == 0 (prometheus-native) up{job="tdarr"} == 0
Tdarr collection cycle failed (API, parse, partial pie) tdarr_up == 0 tdarr_up == 0 for 5m
Exporter handler errors (panics, misroutes) non-2xx scrape request rate rate(tdarr_scrape_requests_total{code!~"2.."}[5m]) > 0
Tdarr API drift (new status) tdarr_unknown_status_total increase(tdarr_unknown_status_total[24h]) > 0
Worker idle vs broken gated by tdarr_up sum(tdarr_node_worker_count) == 0 and on() tdarr_up == 1

Migration

Dashboards / alerts keying on removed labels need adjustments. Examples:

# OLD: tdarr_node_info{node_cpu_transcode_limit="2"}
# NEW: tdarr_node_worker_limit{worker_type="transcode", compute_type="cpu"} == 2

# OLD: tdarr_node_info{node_paused="true"}
# NEW: tdarr_node_paused == 1                    # gauge; node_paused label dropped from node_info

# OLD: tdarr_node_info{allow_gpu_do_cpu="true"}
# NEW: tdarr_node_info{gpu_can_do_cpu="true"}    # renamed for readability

# OLD: count(tdarr_node_worker_flow_info)
# NEW: sum(tdarr_node_worker_count) by (node_name)
# OR:  count(tdarr_node_worker_info{flow_worker="true"})

# OLD: tdarr_node_busy == 1
# NEW: sum(tdarr_node_worker_count) by (node_name) > 0

# OLD: tdarr_collector_error == 1                # never actually scrapable in v1
# NEW: tdarr_up == 0                              # polarity flipped

# Label hygiene renames (9bd1343)
# OLD: tdarr_node_worker_count{type="transcodecpu"}
# NEW: tdarr_node_worker_count{worker_type="transcode", compute_type="cpu"}

# OLD: tdarr_unknown_status_total{kind="transcode"}
# NEW: tdarr_unknown_status_total{job_kind="transcode"}

# Compound → split (5f6ae4c)
# OLD: tdarr_node_worker_count{worker_type="transcodegpu"}                # compound single label
# NEW: tdarr_node_worker_count{worker_type="transcode", compute_type="gpu"}

# All GPU workers, no regex needed:
# OLD: sum(tdarr_node_worker_count{worker_type=~".*gpu"})
# NEW: sum(tdarr_node_worker_count{compute_type="gpu"})

# All transcode load regardless of accelerator:
# OLD: sum(tdarr_node_worker_count{worker_type=~"transcode.*"})
# NEW: sum(tdarr_node_worker_count{worker_type="transcode"})

# Synthetic all_libraries aggregate dropped (bb19f19) — use sum() instead
# OLD: tdarr_library_files_total{library_id="all_libraries"}
# NEW: sum(tdarr_library_files_total) by (tdarr_instance)

# OLD: tdarr_library_transcodes{library_id="all_libraries", status="error"}
# NEW: sum(tdarr_library_transcodes{status="error"}) by (tdarr_instance)

# OLD: tdarr_library_health_checks{library_id="all_libraries", status="error"}
# NEW: sum(tdarr_library_health_checks{status="error"}) by (tdarr_instance)

# Per-library queries unchanged — only the synthetic aggregate row is gone.
# Dashboards that carried the defensive `library_id!="all_libraries"` filter
# in aggregation queries can drop the filter (it's a no-op now).

Full diff: #52

Dashboard migration: v1 panel transformations referenced removed numeric labels and would render empty columns under v2 metrics. Rather than patching, the dashboard was rewritten — import examples/dashboard.json fresh. Forked / customized v1 dashboards should be re-forked from the v2 baseline (or kept on tdarr-exporter v1.x if customizations are extensive).

Commit references

  • acbd3e5 — node metrics overhaul (worktree feature commit)
  • 4053bb5 — node merge
  • 58015cb — pie status enum + partial failure signaling (worktree feature commit)
  • da90d10 — pie merge
  • 2bb6d03 — final review findings (partialFailure scoping, dead code, missing Describe)
  • 7d7289c — cache invalidation expansion, status enum additions, scrape duration fix
  • e3ada65 — simplify clean label function
  • e2277cc — go fmt
  • 656af3a — lint fixes (close body)
  • 27a1b01 — drop old API parsing path + orphaned struct field, dead comments
  • f26449e — Grafana dashboard v2 rewrite against new metric surface
  • df57645tdarr_collector_errortdarr_up polarity flip; inner collect() refactor; Describe() completeness; partialFailure flag reset fix; new tdarr_test.go with 9 tests; Makefile test targets; go.mod tidy
  • 8b9c99f — dashboard Scrape Health rework (Tdarr Collector Success + Handler Errors panels, scrape duration filter, tdarr_up-sourced tdarr_instance variable); Fleet Activity row restored to 6 panels; worker table fixes (duplicate node_name 1..7 columns, category leak via outer join, threshold coloring on Progress %/FPS)
  • 9bd1343 — label hygiene: typeworker_type, kindjob_kind, allow_gpu_do_cpugpu_can_do_cpu; dropped duplicate node_paused label from tdarr_node_info
  • 5f6ae4c — split compound worker_type value into orthogonal worker_type + compute_type labels across all 4 affected metrics; dashboard queries/legends/tables updated accordingly
  • bb19f19 — drop synthetic library_id="all_libraries" aggregate emission from collector; rewrite 2 Errors stat panels to sum() by (tdarr_instance); strip library_id!="all_libraries" filter from 15 dashboard panels; clear $library template-var regex hack; defensive skip + warn log on empty library id/name
  • 621156e — unify all template variable queries (tdarr_instance, library, node_name) on the user-facing ${datasource} picker; previously library alone tracked the picker, the other two were pinned to ${DS_PROMETHEUS} so switching the picker left the dropdowns stale