Repository navigation
Features
- Added a unified profiler for CPU and memory profiling of native applications, Java, and Python
- Added native Off-CPU profiling to distinguish blocking from run queue scheduling latency
- Added native virtual allocation, physical allocation, and physical usage profiling
- Added huatuo-apiserver and continuous profiling task APIs
- Added aggregation and persistence of continuous profiling data and a Pyroscope-compatible query API
- Added an interactive terminal flame graph to the perf tool
- Added a standalone dropwatch tool with pcap filtering, device filtering, and event rate limiting
- Added devlink hardware packet drop tracing and deduplication of software and hardware drops
- Added the tcpshark tool and TCP retransmission events with suspected packet drop correlation
- Added an Agent SSE API for real-time kernel event subscriptions
- Added Agent APIs for node task and tracer control
- Added Ascend NPU metrics for chips, HBM, PCIe, RoCE, and optical modules
- Added basic disk I/O metrics and I/O latency metrics for block devices and containers
- Added memory limits, usage, and system/container memory snapshots to OOM events
- Added ARM GHES processor error detection
- Added a CPU external wait ratio metric
- Added container filtering, combined issue rules, and delta thresholds to AutoTracing
- Added Elasticsearch 7/8, OpenSearch, and SQLite storage backends
- Added per-tracer local data logs with rotation support
- Added a Helm Chart, Prometheus Kubernetes service discovery, and deployment resource baselines
- Added Grafana dashboards for host/container continuous profiling and disk I/O
- Added version output to all major binaries and HTTP
/healthzand/versionendpoints - Added a typed toolstream event channel for diagnostic subprocess output
- Added BPF ABI generation and BPF object lease management
Bug Fixes & Improvements
- Improved net_rx_latency compatibility with multiple skb timestamp layouts and non-ESTABLISHED TCP receive paths
- Improved structured RAS event output and kernel tracepoint capability detection
- Improved collector dependency checks to gracefully degrade when procfs or memcg capabilities are unavailable
- Improved AutoTracing trigger rules, iotracing collection limits, and result storage
- Improved asynchronous storage queue draining on shutdown and insert-only write semantics
- Improved compatibility with the Kubernetes Pod cgroup lifecycle
- Improved CPU event batch reads, perf reader lost-sample recovery, and kretprobe concurrency configuration
- Improved compatibility with ARM GHES, folios, skb timestamps, and tracefs kprobe symbols
- Improved and enriched documentation at https://docs.huatuo.tech/en/latest/
What's Changed
- feat: add iolatency metrics feature by @alahaiyo in #167
- fix: use PT_REGS_PARM1 macro for kprobe args by @alahaiyo in #168
- feat: refactor: modularize configuration structure by @alahaiyo in #169
- refactor: split services into server and handlers packages by @alahaiyo in #171
- feat: add container filter for CPUIdle and optimize IssuesList by @fanzu8 in #173
- feat: add job manager for managing huatuo-bamai agent tasks by @alahaiyo in #174
- docs: unify Included/Excluded filter logic across config and docs by @fanzu8 in #177
- docs: add datasource.md by @admin-dong in #179
- docs: update hardware-errors examples by @redsky110 in #181
- fix: trim spaces from KubeletClientCertPath parts and include err in panic by @huhong-web in #183
- fix(tracing): fix typo and improve lock scope in Dump() by @Redamency in #184
- fix: avoid nil store in SetTaskStore and propagate Close error by @Redamency in #185
- fix: kernel 7.0+ compatibility using CO-RE compat structs by @zy84338719 in #187
- feat(huatuo-apiserver): add new module for managing agent tasks by @alahaiyo in #191
- feat(toolstream, pcapfilter, dropwatch): add toolsstream, pcapfilter, extend dropwatch to support various filter expression by @huhong-web in #192
- fix: fix previous deps and config error by @huhong-web in #195
- docs(apiserver): clean up huatuo-apiserver.conf by @alahaiyo in #200
- fix: initStorage 传递 nil esStore 到 SetTaskStore 防止错误 by @enkilee in #188
- fix: some typo errors by @enkilee in #199
- Add Ascend NPU monitoring and metrics by @mingchang93 in #198
- fix: dload.go ctx 取消响应延迟 by @enkilee in #197
- fix: updateContainerBlkDisk 吞掉 pod.Containers() 的 error by @enkilee in #190
- fix(ci): wait for dpkg lock before apt-get install by @huhong-web in #201
- fix: checkAndRecordMemoryUsage 吞掉 readMemInfo 的 error by @enkilee in #189
- fix: append container scope name to systemd cgroup path by @sisttri in #226
- perf: eliminate cross-CPU atomic contention in cpu_runqlat_tracing by @sisttri in #235
- fix(cpu): avoid cpu stat hierarchy wait sum underflow by @pkurunner in #205
- feat(oom): report memory limit and usage on OOM kill by @sisttri in #239
- fix(bpf): restore pre-7.0 kernel compat broken by 85e6aed by @alahaiyo in #244
- feat(iotracing): resolve block device name from major/minor by @sisttri in #241
- fix(cpuidle): handle cgroup manager init errors by @pkurunner in #210
- feat(server): add healthz endpoint by @pkurunner in #212
- fix(netdev): validate ieee pfc attr length by @pkurunner in #208
- fix(flamegraph): use root name index by @pkurunner in #252
- feat(perf): add terminal flamegraph viewer by @emmmleo in #251
- fix(events): reserve watch clients atomically by @pkurunner in #249
- chore(sync): align codebase with upstream open version by @huhong-web in #255
- test(qemu): add ubuntu26.04 by @fanzu8 in #248
- chore(deps): upgrade grpc to v1.79.3 and related deps to fix CVE-2026-33186 by @sisttri in #242
- feat(metrics): add wait_sum based exter wait rate by @zzzhang1127 in #245
- docs(pkg): add package-level godoc comments by @Redamency in #233
- fix(memburst): validate sampling config by @pkurunner in #231
- fix(iotracing): preserve full minor device number by @pkurunner in #230
- fix(iotracing): validate device filter bounds by @pkurunner in #229
- test(kernaddr): cover kernel address encoding by @pkurunner in #222
- test(request): cover request helpers by @pkurunner in #220
- fix(iotracing): export max block latencies by @pkurunner in #227
- feat: add BPF debug/print utility with perf event export by @tong493 in #259
- fix(pod): css miss due to container cgroup path check errors, and unified resource release by @sisttri in #253
- fix(vendor): restore capnp internal schema package by @jasen11 in #274
- fix(iolatency): attach freeze_queue probe on renamed kernel symbol by @lanenson in #270
- feat: collect memory snapshot for oom events by @JerryJQ in #290
- docs: populate contribute guide and symlink CONTRIBUTING.md by @greyyi in #297
- feat(netdev): allow regex device lists by @pkurunner in #298
- fix: bond lacp the WaitDetachByBreaker call is missing by @sisttri in #295
- fix(bpf): support arm ghes event trace by @sisttri in #291
- fix(job): return node agent body read errors by @jasen11 in #266
- docs: add Linux 7.0 kernel support to README by @JesseW-17 in #254
- fix(profiler): escape container_id query value by @zzzhang1127 in #288
- feat(task): add task list endpoint by @jasen11 in #272
- feat(server): add HTTP version endpoint by @jasen11 in #278
- fix(pcapfilter): pin pcap stub operands across NOP region by @huhong-web in #300
- fix(config): return errors for invalid config updates by @zzzhang1127 in #280
- fix(container): include response body for non-OK errors by @zzzhang1127 in #285
- fix(container): URL-escape container_id query value by @wc4440222 in #301
- fix: ticker goroutine delayed release and chan reuse by @sisttri in #309
- fix: iotracing ticker goroutine leak and add alarm info by @sisttri in #296
- feat: compatible net_rx_latency with kernel 5.15 and 6.8 by @sisttri in #292
- test(strutil): add unit tests for SplitCommaList by @qiyu-lu in #318
- Security: Unsafe Type Conversion in Config Handler by @tomaioo in #319
- test(matcher): add unit tests for ValueMatcher by @qiyu-lu in #321
- docs: add comprehensive CONTRIBUTING.md by @wc4440222 in #304
- test: add unit tests for fileutil, response, and flamegraph (#223, #215, #217) by @wc4440222 in #310
- docs: complete Chinese translations for autotracing and metrics dev docs by @wc4440222 in #305
- chore: add hungtask timeout to es data by @sisttri in #243
- fix: systemd cgroup driver with V2 should use dbus connection by @sisttri in #207
- feat(profiling): add GET /v1/profiles/capabilities endpoint by @wc4440222 in #315
- fix(localfile): propagate MkdirAll error and log JSON formatting fallback by @wc4440222 in #313
- fix(job): prevent nil pointer panic in monitorJob defer on shutdown by @wc4440222 in #312
- test(localfile): discard os.Chmod return in cleanup to satisfy errcheck by @qiyu-lu in #338
- fix(profiler/java): correct misleading retry delay log and comment by @qiyu-lu in #337
- fix(pod): replace sync.Mutex with sync.RWMutex to protect containers map by @wc4440222 in #341
- docs(metrics): document Didi-kernel-only memory_others metrics by @messere1 in #358
- feat(metrics): probe Didi memcg interfaces before enabling memory_others by @messere1 in #357
- fix(metrics,profiler): propagate cgroup init error and fix silent error returns by @wc4440222 in #352
- fix(metric): avoid duplicate default labels by @neu-hsc in #355
- feat(profiling): continuous profiling infrastructure & Grafana dashboards by @fanzu8 in #347
- fix(dropwatch): correct packet metadata layout by @messere1 in #362
- fix(ci): remove unused code by @fanzu8 in #369
- docs: define canonical timestamp format by @Loyal-Young in #410
- test(bytesutil): cover binary struct encoding by @Loyal-Young in #407
- fix(pod): propagate kubelet response body read errors by @123123213weqw in #394
- test(profiler): cover time parse fallback by @Loyal-Young in #404
- fix(profiling): preserve profile query matchers by @messere1 in #405
- chore(request): remove unused internal request package by @wegjgwioj in #411
- refactor(tracing): remove log dependency from pkg/tracing by @wc4440222 in #412
- fix(cpusys): skip zero cpu stat deltas by @pkurunner in #247
- fix(metrics): protect cpu_util and cpu_stat from bare type assertion panics by @wc4440222 in #351
- fix(profiler): account folio rmap page count by @sbtger in #364
- fix(autotracing): extend buildDiskMetric counter reset guard to cover all delta fields by @wc4440222 in #375
- fix(metrics): guard netdev_hw rx_dropped underflow and protect prog field with mutex by @wc4440222 in #377
- feat(grafana): update autotracing-event by @fanzu8 in #477
- feat(docs): add deployment guides by @huhong-web in #478
- fix(toolstream): recover stale Unix sockets by @huhong-web in #480
- feat(metrics): add disk IO continuous monitoring by @Wuzheyuan456 in #348
- docs(profiling): add quick start guide for continuous profiling by @fanzu8 in #479
- fix(docs): img link failed by @fanzu8 in #484
- fix(docs): rename Chinese datasource guide by @huhong-web in #483
- fix(bpf): reject unknown map lookups by @messere1 in #373
- fix(profiling): prevent output file overwrite by @twei43846-afk in #486
- fix(pod): replace kubelet cgroup driver panic with graceful fallback by @wc4440222 in #340
- fix(docs): use repository links for docs by @huhong-web in #493
- fix(metrics): tcp_memory NaN panic when memoryLimit is zero by @wc4440222 in #421
- fix(flamegraph): keep symbols out of inline scripts by @lizhenhao123 in #462
- fix(bpf): discover kprobe symbols from tracefs by @lizhenhao123 in #470
- chore: update dashbroad and fix metrics by @fanzu8 in #497
- fix(profiler): resolve valid zero stack ids by @lizhenhao123 in #471
- feat(bpf): configure kretprobe concurrency by @lizhenhao123 in #475
- fix(profiler,dashboards): enable region filtering and simplify dashboards by @fanzu8 in #501
- fix(bpf): disable perf events until BPF attachment by @huhong-web in #502
- feat(profiler): add native off-CPU delay profiling by @btlqql in #503
- feat(tcpshark): tighten TCP retransmit tracing by @huhong-web in #494
- fix(dashboards): fix container metric variables and update docs by @fanzu8 in #510
- docs(readme): add autotrcing-event img by @fanzu8 in #511
- fix: repair container cpuset count error by @sisttri in #496
- fix(docs): polish tcpshark doc rendering by @huhong-web in #512
- fix(ci): repair lint and distro integration failures by @huhong-web in #513
- fix(bpf): use stable TCP retransmit tracepoint context by @huhong-web in #514
- test(types): cover watch event JSON contract by @Loyal-Young in #408
- fix(metrics): cpu_stat deltaWaitRunSum uint64 underflow on cgroup counter reset by @wc4440222 in #415
- test(types): cover iotracing wire schema by @Loyal-Young in #409
- fix(metrics): preserve qdisc packet counters by @sisttri in #464
- refactor(cgroup): disable cgroup qouta by default by @sisttri in #548
- fix(integration): wait for included metrics by @messere1 in #481
- fix(matcher): validate issue filter expressions by @yyqdbngt in #534
- fix(perf): preserve flamegraph stack order by @cxyyy66 in #574
New Contributors
- @admin-dong made their first contribution in #179
- @Redamency made their first contribution in #184
- @zy84338719 made their first contribution in #187
- @enkilee made their first contribution in #188
- @mingchang93 made their first contribution in #198
- @sisttri made their first contribution in #226
- @emmmleo made their first contribution in #251
- @zzzhang1127 made their first contribution in #245
- @tong493 made their first contribution in #259
- @jasen11 made their first contribution in #274
- @lanenson made their first contribution in #270
- @JerryJQ made their first contribution in #290
- @greyyi made their first contribution in #297
- @JesseW-17 made their first contribution in #254
- @wc4440222 made their first contribution in #301
- @qiyu-lu made their first contribution in #318
- @tomaioo made their first contribution in #319
- @messere1 made their first contribution in #358
- @neu-hsc made their first contribution in #355
- @Loyal-Young made their first contribution in #410
- @123123213weqw made their first contribution in #394
- @wegjgwioj made their first contribution in #411
- @sbtger made their first contribution in #364
- @Wuzheyuan456 made their first contribution in #348
- @twei43846-afk made their first contribution in #486
- @btlqql made their first contribution in #503
- @yyqdbngt made their first contribution in #534
- @cxyyy66 made their first contribution in #574
Full Changelog: v2.2.0...v2.3.0