feat: column roles — skip identifier/sequence columns in point detection (0.7.0)

[copyleftdev]

What

Every scanned column is classified into a role — measurement / identifier / categorical / sequence / constant — and the full map ships in the envelope's new roles array. The point detector skips identifier and sequence columns (a big process-id or a counter's endpoint is not an anomaly), attacking noise at the detection layer. --no-column-roles disables the skipping; roles are still reported.

Impact (dogfooded)

corpus	point findings (roles on)	(roles off)	note
journald 20k	~240	~12,500	skips _PID/_UID/_GID/JOB_ID/TID/SYSLOG_PID/_AUDIT_SESSION/INVOCATION_ID/MESSAGE_ID/N_RESTARTS/TIMESTAMP_*/__SEQNUM
MSHA parquet 127k	32,893 (unchanged)	32,893	DAYS_LOST stays measurement — genuine signal preserved

A 98% cut of the journald noise; zero impact on the real measurement.

Design — honoring "never guess"

• Identifiers by name (*_id, uid, gid, pid, tid, session, uuid, …) — the only reliable signal, because a process-id column is statistically indistinguishable from a discrete measurement (_PID is mid-cardinality with repeats, not near-unique, not monotonic).
• Cardinality is deliberately NOT used to call a numeric column categorical: a near-constant column with a few wild outliers has low cardinality yet is exactly what point detection should catch. (This is the trap that broke the first cut — caught by the existing point tests.)
• Heuristic, but never silent: every column's role is in the envelope (auditable) and --no-column-roles turns skipping off. Setting is in config_version (cr=). Constant columns still flow through scan_column (self-no-op), so they aren't spuriously marked absent.

Contract

Additive: new ax_core::roles module (Role/ColumnRole/Column::role), roles array in the envelope + schema. PROTOCOL unchanged.

Gate

proptest + cargo-mutants 0 missed on all changed files: roles.rs (35 caught), envelope.rs, point.rs+config.rs (32 caught), main.rs+schema.rs (68 caught). Direct tests for the name tokenizer, is_strictly_monotonic boundaries, role classification, and role-based skipping incl. --no-column-roles.

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes - v0.7.0

• New Features

  • Automatic column role classification: Each column is now classified as measurement, identifier, categorical, sequence, or constant based on statistical properties and naming patterns.
  • Column roles included in scan output: Role assignments are reported in the results envelope.
  • --no-column-roles flag: Disable role-based detector filtering while preserving role reporting.

• Documentation

  • Added guidance on column roles and how detectors use them to skip non-measurement columns.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces deterministic column role classification to anomalyx, enabling the detection system to skip columns where statistical analysis is not applicable (e.g., identifiers and monotonic sequences). The feature includes role reporting in the output envelope, a --no-column-roles flag to disable skipping, and integration into the point detector's column selection logic.

Changes

Column Role Classification and Detection

Layer / File(s)	Summary
Core column role classification system crates/ax-core/src/roles.rs, crates/ax-core/src/lib.rs	Role enum distinguishes measurement, identifier, categorical, sequence, and constant columns. Classification uses name tokenization for identifier detection, distinct-count thresholds for constants, and strict monotonicity checks for sequences. Unit tests validate tokenization, identifier matching, monotonic edge cases, and classification precedence.
Envelope wire contract and output schema crates/ax-core/src/envelope.rs, crates/anomalyx/src/schema.rs	Envelope struct and EnvelopeBuilder extended with roles: Vec<ColumnRole> field and roles() builder method. JSON Schema updated to require roles as an array of column-role objects with role enum constraints.
Detection configuration and role-based skipping control crates/ax-detect/src/config.rs	DetectConfig gains column_roles: bool field (default true) to enable role-based column filtering; configuration fingerprint updated to include cr=true/cr=false marker and config version bumped from anomalyx-cfg/6 to anomalyx-cfg/7.
Point detector role-aware column selection crates/ax-detect/src/point.rs	Point detector now counts eligible and scanned columns separately; skips Identifier and Sequence roles when column_roles is enabled; reports distinct absence semantics for zero-eligible vs. all-skipped cases. New tests verify identifier column skipping and measurement column precedence when mixed column types present.
CLI flag and envelope role output crates/anomalyx/src/main.rs	ScanArgs gains no_column_roles flag; flag parsing wires negation into DetectConfig.column_roles; cmd_scan() computes and attaches per-column roles to envelope via EnvelopeBuilder.roles(). Unit test verifies flag toggles config and default behavior.
Release version and documentation updates Cargo.toml, CHANGELOG.md, docs/src/modes.md	Workspace version bumped to 0.7.0 with synchronized path dependencies; new v0.7.0 changelog entry documents column roles, detector skipping, and --no-column-roles flag; modes.md explains role classification, envelope reporting, detector behavior, and flag usage.

Sequence Diagram

sequenceDiagram
    participant User as User
    participant CLI as anomalyx CLI
    participant Parser as parse_scan_args()
    participant ConfigBuilder as config_for()
    participant Detector as Point Detector
    participant Column as Column
    participant Builder as EnvelopeBuilder
    participant Output as Envelope JSON
    
    User->>CLI: scan --no-column-roles
    CLI->>Parser: parse CLI args
    Parser->>Parser: set no_column_roles = true
    Parser-->>ConfigBuilder: ScanArgs
    ConfigBuilder->>ConfigBuilder: column_roles = !no_column_roles<br/>(= false)
    ConfigBuilder-->>Detector: DetectConfig
    
    Detector->>Column: iterate numeric columns
    alt column_roles = false
        Detector->>Detector: assess all numeric columns
    else column_roles = true (default)
        Detector->>Column: get role
        alt role is Identifier or Sequence
            Detector->>Detector: skip column
        else
            Detector->>Detector: assess column
        end
    end
    
    Note over CLI,Output: Envelope construction
    CLI->>Column: extract role() for each column
    Column-->>Builder: ColumnRole { column, role }
    CLI->>Builder: roles(vec![...])
    Builder->>Output: roles field in JSON

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 Roles align the data stream,
Identifiers step aside with grace,
Measurement shines where stats belong,
Constants whisper "no variance here,"
And sequences march in perfect order—
All accounted for in the envelope's embrace.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main feature: column roles classification and identifier/sequence skipping in point detection, with the version bump. It is specific, concise, and directly addresses the primary change.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/column-roles

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/ax-core/src/envelope.rs`:
- Around line 115-118: The wire-format was changed by adding the `roles:
Vec<ColumnRole>` field to the `tq1` envelope but `envelope::PROTOCOL` was not
incremented; update `envelope::PROTOCOL` to a new protocol version to signal the
breaking change, then update the `anomalyx schema` to reflect the new `tq1`
shape and run/update the contract tests that validate envelope compatibility
(including any tests referencing `tq1` layout or dense row layout) so consumers
can detect the new payload format.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f59d1b16-e0e8-4a46-bf8a-13d354243389

📥 Commits

Reviewing files that changed from the base of the PR and between cc9affd and c1d04c8.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (10)

• CHANGELOG.md
• Cargo.toml
• crates/anomalyx/src/main.rs
• crates/anomalyx/src/schema.rs
• crates/ax-core/src/envelope.rs
• crates/ax-core/src/lib.rs
• crates/ax-core/src/roles.rs
• crates/ax-detect/src/config.rs
• crates/ax-detect/src/point.rs
• docs/src/modes.md

[coderabbitai]

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Bump envelope::PROTOCOL for this wire-format change.

Adding roles changes the tq1 envelope shape, but PROTOCOL still advertises the old contract. That leaves old consumers unable to distinguish the new payload from the previous schema-compatible version. As per coding guidelines, "The tq1 envelope is an API. Changing a field, the dense row layout, an exit code, or a handle form is a breaking change: bump envelope::PROTOCOL, update anomalyx schema, and update the contract tests."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/ax-core/src/envelope.rs` around lines 115 - 118, The wire-format was
changed by adding the `roles: Vec<ColumnRole>` field to the `tq1` envelope but
`envelope::PROTOCOL` was not incremented; update `envelope::PROTOCOL` to a new
protocol version to signal the breaking change, then update the `anomalyx
schema` to reflect the new `tq1` shape and run/update the contract tests that
validate envelope compatibility (including any tests referencing `tq1` layout or
dense row layout) so consumers can detect the new payload format.