feat(state): segments / tokens iterators (#3)

[uqio]

Closes #3.

Adds two ergonomic iterator entry points to the safe API and the standard adapter trait impls so they compose with flat_map / rev / for ... in &state.

Surface

state.segments_iter()                    // Segments<'_>
seg.tokens_iter()                        // Tokens<'_>

for seg in &state { ... }                // IntoIterator for &State
for tok in seg { ... }                   // IntoIterator for Segment<'a>  (Copy)
for tok in &seg { ... }                  // IntoIterator for &Segment<'a>

state.segments_iter().rev()              // DoubleEndedIterator
seg.tokens_iter().rev()
state.segments_iter().flat_map(|s| s.tokens_iter())  // flat token stream

Both iterators implement Iterator + ExactSizeIterator + DoubleEndedIterator + FusedIterator. Yielded items: Segments::Item = Segment<'a> (borrows from the source &'a State); Tokens::Item = Token (owned, value-typed snapshot, no further borrow).

Lifetime shape

• Segments<'a> borrows &'a State. Multiple iterators alive simultaneously is sound — the underlying whisper_state is immutable for the borrow's duration (State::full requires &mut self, ruled out by the borrow checker while any &self iterator exists).
• Tokens<'state> owns a Copy of the Segment (which is itself a thin index + pointer projection, derive-Copy). This makes

  state.segments_iter().flat_map(|seg| seg.tokens_iter())

  compile — the inner iterator does not borrow the closure-local seg value. The 'state lifetime still ties yielded item pointer projections to the parent State.
• Auto-traits: !Send + !Sync on both, inherited from &State (State is !Sync) / NonNull<whisper_state> (foreign C type with no impl). Matches the existing State threading model.

Performance

Segments::next and Tokens::next inline the construction / pointer projection rather than calling back through State::segment(i) / Segment::token(j). Those accessors do their own n_segments() / n_tokens() FFI bounds-check, but the iterator already captured end from the same call at construction — and the borrow chain pins the underlying state, so the bounds-check is redundant. One FFI call per yielded item saved on Tokens (which dominates: hundreds to low thousands of tokens per state).

Tests

13 new unit tests in whispercpp/src/state.rs's #[cfg(test)] mod tests:

• segments_iter_empty_state_yields_zero_items
• segments_iter_count_matches_n_segments
• segments_iter_exact_size_len_matches_count
• segments_iter_size_hint_is_exact
• segments_iter_fused_after_exhaustion
• multiple_segments_iter_alive_concurrently
• segments_iter_composes_with_adapters
• nested_segments_and_tokens_iter_compiles
• iterator_type_bounds_are_correct
• tokens_iter_composes_with_flat_map
• into_iter_for_state_ref_yields_segments
• into_iter_for_segment_compiles
• segments_iter_double_ended_compiles_and_empty_yields_none
• tokens_iter_double_ended_compiles

Two #[cfg(test)] pub(crate) test fixtures make this possible without a model file:

• Context::dangling_for_test() — unsafe fn. Returns a Context with NonNull::dangling(). The unsafe contract: the caller must ensure the value (or any Arc<Context> derived from it) is core::mem::forget'd before its Drop runs (which would otherwise call whisper_free on the dangling pointer).
• State::poisoned_for_test(ctx: Arc<Context>) — safe. Returns a State with ptr: None (the Drop impl already short-circuits for poisoned states; no FFI runs).

The test fixture (poisoned_state_for_test) wraps the unsafe call in a single unsafe block with a SAFETY comment explaining the Arc-refcount-leak invariant: Arc::new + Arc::clone brings refcount to 2, then mem::forget on the original handle keeps it at ≥1 forever, so Context::drop is unreachable.

The runtime exercise of Tokens requires a real model (the poisoned fixture yields zero segments, so we can't construct a Segment). The compile-time bound assertions (assert_token_iter, assert_dei) cover the type-system contract instead — every adapter the docs claim should compile is type-checked at the call site.

Audit

whispercpp/src/safety_audit.rs gains a segments_iter / tokens_iter row in the per-method coverage matrix, walking all ten safety axes. Highlights:

• Initial release: safe Rust bindings for whisper.cpp (0.1.0) #1 throw: inlined FFI projection (whisper_full_get_token_data_from_state) is a pure C accessor; bounds verified at iterator construction.
• Add small accessors to the safe wrapper (tracking) #2 sync: State is !Sync; &self permits multiple iterators alive simultaneously, sound because the buffers are immutable for the borrow's duration.
• Token-stream iterators: Segment / Token #3 alloc: iterator state is two i32s + a borrow / Copy of Segment; no Rust-side allocation.
• Async-friendly State::full (tokio / blocking task pool) #4 lifetime: Tokens<'state> owns a copied Segment<'state> so adapter composition works; 'state ties pointer projections to the parent State.
• Tracking: optional whispercpp wrapper expansions #6 sentinels: next / end index counters; the converged-cursor case (next == end) is rejected at the top of every direction's call.

Verification

• cargo build -p whispercpp clean (debug + release).
• cargo test -p whispercpp --lib — 62 passed (was 48, +14 new across the four commits).
• cargo clippy -p whispercpp -p whispercpp-sys --all-targets -- -D warnings clean.
• cargo doc -p whispercpp --no-deps clean.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds ergonomic, adapter-friendly iteration over whisper.cpp decode outputs by introducing segment and token iterator entry points on the safe wrapper types, plus standard IntoIterator impls to enable idiomatic for ... in loops.

Changes:

• Add State::segments_iter() and Segment::tokens_iter() returning Segments<'_> / Tokens<'_> with ExactSizeIterator + DoubleEndedIterator + FusedIterator.
• Implement IntoIterator for &State, Segment<'a>, and &Segment<'a> to compose with standard iterator adapters.
• Add test-only fixtures (Context::dangling_for_test, State::poisoned_for_test) and unit tests validating iterator shape/composability; extend the safety audit.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

File	Description
whispercpp/src/state.rs	Introduces Segments/Tokens iterators, segments_iter/tokens_iter, IntoIterator impls, plus iterator-focused unit tests and test fixtures.
whispercpp/src/context.rs	Adds a #[cfg(test)] unsafe dangling_for_test() constructor used by iterator unit tests without requiring a real model.
whispercpp/src/lib.rs	Re-exports the new iterator types (Segments, Tokens) as part of the public surface.
whispercpp/src/safety_audit.rs	Documents safety/audit coverage for the new iterator methods and their IntoIterator delegations.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.