feat(health): add structured OTLP health-report attributes - #4510
feat(health): add structured OTLP health-report attributes#4510nvrzeznik wants to merge 1 commit into
Conversation
Summary by CodeRabbit
WalkthroughHealth-report OTLP logs now include versioned structured attributes for report metadata, successes, alerts, counts, classifications, and timestamps. Target identifiers use stable serialized strings. Documentation defines the attribute contract. ChangesHealth report OTLP serialization
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant HealthReport
participant OtlpConverter
participant OtlpLogRecord
HealthReport->>OtlpConverter: provide report metadata and evidence
OtlpConverter->>OtlpConverter: serialize structured AnyValue attributes
OtlpConverter->>OtlpLogRecord: emit attributes and selected timestamp
Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/health/src/otlp/convert.rs`:
- Around line 880-965: Extend health report conversion coverage around
health_report_event_time or convert_event with table-driven cases for both a
supplied observed_at and an absent observed_at. Use a deterministic export
timestamp, assert the event timestamp uses observed_at when present and the
export time as fallback when absent, and verify observed_time_unix_nano remains
the supplied export timestamp in both cases.
In `@docs/architecture/health_aggregation.md`:
- Around line 287-295: Expand the OTLP health-report contract section to specify
requiredness, OTLP types, accepted values, and fallback behavior for every
attribute, including event.type="health_report", valid
source/target/classification values, and matching success_count/alert_count with
their nested arrays. Mark health_report.target and health_report.observed_at
optional, and document nested entry fields and schema_version="v1" gating. Add
timestamp semantics: time_unix_nano uses the valid observation time when
available, observed_time_unix_nano uses export time, and observed_at is RFC 3339
nanosecond UTC.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 28e2049c-c6c5-42cc-95d2-7793edce4345
📒 Files selected for processing (3)
crates/health/src/otlp/convert.rscrates/health/src/sink/events.rsdocs/architecture/health_aggregation.md
a9343ce to
0344d33
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/architecture/health_aggregation.md`:
- Around line 289-291: Document the consumer behavior for missing or unsupported
health_report.schema_version values in the OTLP health-report contract: retain
the existing human-readable summary body, ignore the structured health_report
attributes, and specify whether any warning is emitted; do not reject the
record.
- Around line 295-312: Update the health report attribute documentation for
health_report.success_count and health_report.alert_count to specify OTLP
int_value signed 64-bit integers bounded to 0..=i64::MAX. Document that the
exporter retains the first 64 alerts in report order when describing
health_report.alerts and its dropped count.
- Around line 292-293: Document that include_alert_details defaults to false, so
health_report.alerts and health_report.alerts.dropped are absent unless enabled;
clarify that health_report.alerts.dropped appears only when enabled details
exceed 64 alerts. Update docs/architecture/health_aggregation.md lines 292-293
and repeat or link the same guidance in docs/operations/monitoring-health.md
lines 277-283.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0316a17d-8461-4c7a-ad19-c08d1b8d4549
📒 Files selected for processing (4)
crates/health/src/otlp/convert.rscrates/health/src/sink/events.rsdocs/architecture/health_aggregation.mddocs/operations/monitoring-health.md
🚧 Files skipped from review as they are similar to previous changes (2)
- crates/health/src/otlp/convert.rs
- crates/health/src/sink/events.rs
0344d33 to
88a0e79
Compare
|
@nvrzeznik you should check the DCO requirements. Guidance lives in CONTRIBUTING.md |
Health reports reached OTLP as a prose summary body and a single event.type attribute, so consumers had to parse free text to recover the source, probe results, and alert classifications behind a report. Emit a versioned health_report.* attribute contract alongside the unchanged summary body: scalar routing fields plus nested successes and alerts that preserve probe IDs, targets, messages, and classifications. The record now carries the report's own observation time when it has one, leaving observed_time_unix_nano as the export time. Signed-off-by: Andrew Rzeznik <arzeznik@nvidia.com>
88a0e79 to
5c9176d
Compare
|
/ok to test 5c9176d |
|
🌿 Preview your docs: https://nvidia-preview-pull-request-4510.docs.buildwithfern.com/infra-controller |
kensimon
left a comment
kensimon
left a comment
There was a problem hiding this comment.
Haven't reviewed the whole thing yet but it's starting to stand out that vendoring our own copy of the otlp protos was the wrong choice... I filed #4712 to address this. (Feel free to ignore this feedback, since even if we made our own From impls it's still not the right approach, we need to just use the opentelemetry crate instead.)
|
|
||
| fn string_value(s: String) -> Option<AnyValue> { | ||
| Some(AnyValue { | ||
| fn string_any_value(s: String) -> AnyValue { |
There was a problem hiding this comment.
This and all these little one-off conversion functions should be removed and replaced with proper From impls on the AnyValue/etc types. We're code-generating them in this crate, so they're owned by this crate: we can add any impl we want to them.
For instance:
impl From<String> for AnyValue {
fn from(value: String) -> Self {
Self {
value: Some(any_value::Value::StringValue(value)),
}
}
}
impl From<i64> for AnyValue {
fn from(value: i64) -> Self {
Self {
value: Some(any_value::Value::IntValue(value)),
}
}
}
impl From<Vec<AnyValue>> for AnyValue {
fn from(values: Vec<AnyValue>) -> Self {
Self {
value: Some(any_value::Value::ArrayValue(ArrayValue { values })),
}
}
}
impl From<Vec<KeyValue>> for AnyValue {
fn from(values: Vec<KeyValue>) -> Self {
Self {
value: Some(any_value::Value::KvlistValue(KeyValueList { values })),
}
}
}
Then building a key/value can be done from a real new function:
impl KeyValue {
pub fn new(key: impl Into<String>, value: impl Into<AnyValue>) -> Self {
Self {
key: key.into(),
value: Some(value.into()),
}
}
}
Then you don't need special int_kv vs kv functions, building a KeyValue becomes KeyValue::new(some_string, some_int) or KeyValue::new(some_string, some_vec), etc.
But to be honest (and this is a bigger issue) I have no idea why we're vendoring in the opentelemetry protobufs right into our own crate here. There's already an opentelemetry crate, and it already has its own KeyValue type, and it already has all these conversion impls... we're doing things that way all over the rest of the codebase, only the health crate seems to be doing things this weird way.
3 participants
Health reports reached OTLP as a prose summary body and a single event.type attribute, so consumers had to parse free text to recover the source, probe results, and alert classifications behind a report.
Emit a versioned health_report.* attribute contract alongside the unchanged summary body: scalar routing fields plus nested successes and alerts that preserve probe IDs, targets, messages, and classifications. The record now carries the report's own observation time when it has one, leaving observed_time_unix_nano as the export time.
Related issues
#4508
Type of Change
Breaking Changes
Testing
Additional Notes
AI generated PR with tweaks and reading by a human