Skip to content

v2.3.0

Latest

Choose a tag to compare

@hao022 hao022 released this 13 Aug 12:29
· 697 commits to main since this release

Features

  • Added a unified profiler for CPU and memory profiling of native applications, Java, and Python
  • Added native Off-CPU profiling to distinguish blocking from run queue scheduling latency
  • Added native virtual allocation, physical allocation, and physical usage profiling
  • Added huatuo-apiserver and continuous profiling task APIs
  • Added aggregation and persistence of continuous profiling data and a Pyroscope-compatible query API
  • Added an interactive terminal flame graph to the perf tool
  • Added a standalone dropwatch tool with pcap filtering, device filtering, and event rate limiting
  • Added devlink hardware packet drop tracing and deduplication of software and hardware drops
  • Added the tcpshark tool and TCP retransmission events with suspected packet drop correlation
  • Added an Agent SSE API for real-time kernel event subscriptions
  • Added Agent APIs for node task and tracer control
  • Added Ascend NPU metrics for chips, HBM, PCIe, RoCE, and optical modules
  • Added basic disk I/O metrics and I/O latency metrics for block devices and containers
  • Added memory limits, usage, and system/container memory snapshots to OOM events
  • Added ARM GHES processor error detection
  • Added a CPU external wait ratio metric
  • Added container filtering, combined issue rules, and delta thresholds to AutoTracing
  • Added Elasticsearch 7/8, OpenSearch, and SQLite storage backends
  • Added per-tracer local data logs with rotation support
  • Added a Helm Chart, Prometheus Kubernetes service discovery, and deployment resource baselines
  • Added Grafana dashboards for host/container continuous profiling and disk I/O
  • Added version output to all major binaries and HTTP /healthz and /version endpoints
  • Added a typed toolstream event channel for diagnostic subprocess output
  • Added BPF ABI generation and BPF object lease management

Bug Fixes & Improvements

  • Improved net_rx_latency compatibility with multiple skb timestamp layouts and non-ESTABLISHED TCP receive paths
  • Improved structured RAS event output and kernel tracepoint capability detection
  • Improved collector dependency checks to gracefully degrade when procfs or memcg capabilities are unavailable
  • Improved AutoTracing trigger rules, iotracing collection limits, and result storage
  • Improved asynchronous storage queue draining on shutdown and insert-only write semantics
  • Improved compatibility with the Kubernetes Pod cgroup lifecycle
  • Improved CPU event batch reads, perf reader lost-sample recovery, and kretprobe concurrency configuration
  • Improved compatibility with ARM GHES, folios, skb timestamps, and tracefs kprobe symbols
  • Improved and enriched documentation at https://docs.huatuo.tech/en/latest/

What's Changed

  • feat: add iolatency metrics feature by @alahaiyo in #167
  • fix: use PT_REGS_PARM1 macro for kprobe args by @alahaiyo in #168
  • feat: refactor: modularize configuration structure by @alahaiyo in #169
  • refactor: split services into server and handlers packages by @alahaiyo in #171
  • feat: add container filter for CPUIdle and optimize IssuesList by @fanzu8 in #173
  • feat: add job manager for managing huatuo-bamai agent tasks by @alahaiyo in #174
  • docs: unify Included/Excluded filter logic across config and docs by @fanzu8 in #177
  • docs: add datasource.md by @admin-dong in #179
  • docs: update hardware-errors examples by @redsky110 in #181
  • fix: trim spaces from KubeletClientCertPath parts and include err in panic by @huhong-web in #183
  • fix(tracing): fix typo and improve lock scope in Dump() by @Redamency in #184
  • fix: avoid nil store in SetTaskStore and propagate Close error by @Redamency in #185
  • fix: kernel 7.0+ compatibility using CO-RE compat structs by @zy84338719 in #187
  • feat(huatuo-apiserver): add new module for managing agent tasks by @alahaiyo in #191
  • feat(toolstream, pcapfilter, dropwatch): add toolsstream, pcapfilter, extend dropwatch to support various filter expression by @huhong-web in #192
  • fix: fix previous deps and config error by @huhong-web in #195
  • docs(apiserver): clean up huatuo-apiserver.conf by @alahaiyo in #200
  • fix: initStorage 传递 nil esStore 到 SetTaskStore 防止错误 by @enkilee in #188
  • fix: some typo errors by @enkilee in #199
  • Add Ascend NPU monitoring and metrics by @mingchang93 in #198
  • fix: dload.go ctx 取消响应延迟 by @enkilee in #197
  • fix: updateContainerBlkDisk 吞掉 pod.Containers() 的 error by @enkilee in #190
  • fix(ci): wait for dpkg lock before apt-get install by @huhong-web in #201
  • fix: checkAndRecordMemoryUsage 吞掉 readMemInfo 的 error by @enkilee in #189
  • fix: append container scope name to systemd cgroup path by @sisttri in #226
  • perf: eliminate cross-CPU atomic contention in cpu_runqlat_tracing by @sisttri in #235
  • fix(cpu): avoid cpu stat hierarchy wait sum underflow by @pkurunner in #205
  • feat(oom): report memory limit and usage on OOM kill by @sisttri in #239
  • fix(bpf): restore pre-7.0 kernel compat broken by 85e6aed by @alahaiyo in #244
  • feat(iotracing): resolve block device name from major/minor by @sisttri in #241
  • fix(cpuidle): handle cgroup manager init errors by @pkurunner in #210
  • feat(server): add healthz endpoint by @pkurunner in #212
  • fix(netdev): validate ieee pfc attr length by @pkurunner in #208
  • fix(flamegraph): use root name index by @pkurunner in #252
  • feat(perf): add terminal flamegraph viewer by @emmmleo in #251
  • fix(events): reserve watch clients atomically by @pkurunner in #249
  • chore(sync): align codebase with upstream open version by @huhong-web in #255
  • test(qemu): add ubuntu26.04 by @fanzu8 in #248
  • chore(deps): upgrade grpc to v1.79.3 and related deps to fix CVE-2026-33186 by @sisttri in #242
  • feat(metrics): add wait_sum based exter wait rate by @zzzhang1127 in #245
  • docs(pkg): add package-level godoc comments by @Redamency in #233
  • fix(memburst): validate sampling config by @pkurunner in #231
  • fix(iotracing): preserve full minor device number by @pkurunner in #230
  • fix(iotracing): validate device filter bounds by @pkurunner in #229
  • test(kernaddr): cover kernel address encoding by @pkurunner in #222
  • test(request): cover request helpers by @pkurunner in #220
  • fix(iotracing): export max block latencies by @pkurunner in #227
  • feat: add BPF debug/print utility with perf event export by @tong493 in #259
  • fix(pod): css miss due to container cgroup path check errors, and unified resource release by @sisttri in #253
  • fix(vendor): restore capnp internal schema package by @jasen11 in #274
  • fix(iolatency): attach freeze_queue probe on renamed kernel symbol by @lanenson in #270
  • feat: collect memory snapshot for oom events by @JerryJQ in #290
  • docs: populate contribute guide and symlink CONTRIBUTING.md by @greyyi in #297
  • feat(netdev): allow regex device lists by @pkurunner in #298
  • fix: bond lacp the WaitDetachByBreaker call is missing by @sisttri in #295
  • fix(bpf): support arm ghes event trace by @sisttri in #291
  • fix(job): return node agent body read errors by @jasen11 in #266
  • docs: add Linux 7.0 kernel support to README by @JesseW-17 in #254
  • fix(profiler): escape container_id query value by @zzzhang1127 in #288
  • feat(task): add task list endpoint by @jasen11 in #272
  • feat(server): add HTTP version endpoint by @jasen11 in #278
  • fix(pcapfilter): pin pcap stub operands across NOP region by @huhong-web in #300
  • fix(config): return errors for invalid config updates by @zzzhang1127 in #280
  • fix(container): include response body for non-OK errors by @zzzhang1127 in #285
  • fix(container): URL-escape container_id query value by @wc4440222 in #301
  • fix: ticker goroutine delayed release and chan reuse by @sisttri in #309
  • fix: iotracing ticker goroutine leak and add alarm info by @sisttri in #296
  • feat: compatible net_rx_latency with kernel 5.15 and 6.8 by @sisttri in #292
  • test(strutil): add unit tests for SplitCommaList by @qiyu-lu in #318
  • Security: Unsafe Type Conversion in Config Handler by @tomaioo in #319
  • test(matcher): add unit tests for ValueMatcher by @qiyu-lu in #321
  • docs: add comprehensive CONTRIBUTING.md by @wc4440222 in #304
  • test: add unit tests for fileutil, response, and flamegraph (#223, #215, #217) by @wc4440222 in #310
  • docs: complete Chinese translations for autotracing and metrics dev docs by @wc4440222 in #305
  • chore: add hungtask timeout to es data by @sisttri in #243
  • fix: systemd cgroup driver with V2 should use dbus connection by @sisttri in #207
  • feat(profiling): add GET /v1/profiles/capabilities endpoint by @wc4440222 in #315
  • fix(localfile): propagate MkdirAll error and log JSON formatting fallback by @wc4440222 in #313
  • fix(job): prevent nil pointer panic in monitorJob defer on shutdown by @wc4440222 in #312
  • test(localfile): discard os.Chmod return in cleanup to satisfy errcheck by @qiyu-lu in #338
  • fix(profiler/java): correct misleading retry delay log and comment by @qiyu-lu in #337
  • fix(pod): replace sync.Mutex with sync.RWMutex to protect containers map by @wc4440222 in #341
  • docs(metrics): document Didi-kernel-only memory_others metrics by @messere1 in #358
  • feat(metrics): probe Didi memcg interfaces before enabling memory_others by @messere1 in #357
  • fix(metrics,profiler): propagate cgroup init error and fix silent error returns by @wc4440222 in #352
  • fix(metric): avoid duplicate default labels by @neu-hsc in #355
  • feat(profiling): continuous profiling infrastructure & Grafana dashboards by @fanzu8 in #347
  • fix(dropwatch): correct packet metadata layout by @messere1 in #362
  • fix(ci): remove unused code by @fanzu8 in #369
  • docs: define canonical timestamp format by @Loyal-Young in #410
  • test(bytesutil): cover binary struct encoding by @Loyal-Young in #407
  • fix(pod): propagate kubelet response body read errors by @123123213weqw in #394
  • test(profiler): cover time parse fallback by @Loyal-Young in #404
  • fix(profiling): preserve profile query matchers by @messere1 in #405
  • chore(request): remove unused internal request package by @wegjgwioj in #411
  • refactor(tracing): remove log dependency from pkg/tracing by @wc4440222 in #412
  • fix(cpusys): skip zero cpu stat deltas by @pkurunner in #247
  • fix(metrics): protect cpu_util and cpu_stat from bare type assertion panics by @wc4440222 in #351
  • fix(profiler): account folio rmap page count by @sbtger in #364
  • fix(autotracing): extend buildDiskMetric counter reset guard to cover all delta fields by @wc4440222 in #375
  • fix(metrics): guard netdev_hw rx_dropped underflow and protect prog field with mutex by @wc4440222 in #377
  • feat(grafana): update autotracing-event by @fanzu8 in #477
  • feat(docs): add deployment guides by @huhong-web in #478
  • fix(toolstream): recover stale Unix sockets by @huhong-web in #480
  • feat(metrics): add disk IO continuous monitoring by @Wuzheyuan456 in #348
  • docs(profiling): add quick start guide for continuous profiling by @fanzu8 in #479
  • fix(docs): img link failed by @fanzu8 in #484
  • fix(docs): rename Chinese datasource guide by @huhong-web in #483
  • fix(bpf): reject unknown map lookups by @messere1 in #373
  • fix(profiling): prevent output file overwrite by @twei43846-afk in #486
  • fix(pod): replace kubelet cgroup driver panic with graceful fallback by @wc4440222 in #340
  • fix(docs): use repository links for docs by @huhong-web in #493
  • fix(metrics): tcp_memory NaN panic when memoryLimit is zero by @wc4440222 in #421
  • fix(flamegraph): keep symbols out of inline scripts by @lizhenhao123 in #462
  • fix(bpf): discover kprobe symbols from tracefs by @lizhenhao123 in #470
  • chore: update dashbroad and fix metrics by @fanzu8 in #497
  • fix(profiler): resolve valid zero stack ids by @lizhenhao123 in #471
  • feat(bpf): configure kretprobe concurrency by @lizhenhao123 in #475
  • fix(profiler,dashboards): enable region filtering and simplify dashboards by @fanzu8 in #501
  • fix(bpf): disable perf events until BPF attachment by @huhong-web in #502
  • feat(profiler): add native off-CPU delay profiling by @btlqql in #503
  • feat(tcpshark): tighten TCP retransmit tracing by @huhong-web in #494
  • fix(dashboards): fix container metric variables and update docs by @fanzu8 in #510
  • docs(readme): add autotrcing-event img by @fanzu8 in #511
  • fix: repair container cpuset count error by @sisttri in #496
  • fix(docs): polish tcpshark doc rendering by @huhong-web in #512
  • fix(ci): repair lint and distro integration failures by @huhong-web in #513
  • fix(bpf): use stable TCP retransmit tracepoint context by @huhong-web in #514
  • test(types): cover watch event JSON contract by @Loyal-Young in #408
  • fix(metrics): cpu_stat deltaWaitRunSum uint64 underflow on cgroup counter reset by @wc4440222 in #415
  • test(types): cover iotracing wire schema by @Loyal-Young in #409
  • fix(metrics): preserve qdisc packet counters by @sisttri in #464
  • refactor(cgroup): disable cgroup qouta by default by @sisttri in #548
  • fix(integration): wait for included metrics by @messere1 in #481
  • fix(matcher): validate issue filter expressions by @yyqdbngt in #534
  • fix(perf): preserve flamegraph stack order by @cxyyy66 in #574

New Contributors

Full Changelog: v2.2.0...v2.3.0