feat(litellm,llamaindex): LiteLLM dual instrumentation + LlamaIndex streaming safety - #6
Merged
manas-fortifyroot merged 1 commit intoMar 24, 2026
Conversation
…treaming safety LiteLLM — dual instrumentation design: - Add _FortifyRootCompletionLogger (duck-typed CustomLogger) registered at litellm.callbacks[0] so it fires before LiteLLM's native OTel callback. Masks response_obj in-place for non-streaming calls; streaming handled per-chunk by streaming_safety.py. - FR safety span (fortifyroot.litellm.safety, role=safety_wrapper) set as ambient OTel context via set_span_in_context so LiteLLM's litellm_request span is automatically created as a child — no native OTel config change needed. - streaming_safety.py: accept token= param; detach in finally so FR span stays ambient for the full stream including end-of-stream callbacks. - Bug fix: _finalize_awaitable_response now sets FR span as ambient AND suppress key (was suppress-only; callbacks on awaitable path fired on wrong span). - New test_logger_integration.py: 10 FR-marked tests covering logger skip on stream, non-stream masking, finding emission on current span, callback registration at position 0, non-list callbacks, span role attribute. - test_safety_hooks.py: updated for new span name, event counts, role attribute; new test_sync_wrapper_awaitable_response_fr_span_is_ambient. LlamaIndex — streaming safety: - Add streaming_safety.py with holdback-buffer streaming masking for LlamaIndex dispatcher_wrapper async streams. - Update dispatcher_wrapper.py and safety.py to wire streaming safety through the LlamaIndex instrumentation layer. - New test_streaming_safety.py with FR-marked tests for streaming safety behavior.
manas-fortifyroot
deleted the
feature/phase-1h-litellm-dual-instrumentation
branch
March 24, 2026 10:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LiteLLM — dual instrumentation:
_FortifyRootCompletionLogger(duck-typedCustomLogger) registered atlitellm.callbacks[0]so it masksresponse_objin-place before LiteLLM's native OTel callback fires — native OTel sees already-masked content, andlitellm_requestbecomes a child of FR's span automatically via ambient OTel context.fortifyroot.litellm.safety,fortifyroot.span.role=safety_wrapper) set as ambient context viaset_span_in_context+SUPPRESS_LANGUAGE_MODEL_INSTRUMENTATION_KEY; token passed through to streaming wrappers and detached infinallyonly after stream exhaustion._finalize_awaitable_responsenow sets FR span as ambient context (was suppress-only; callbacks on awaitable path emitted findings on wrong span).test_logger_integration.py(10 FR tests): logger skip on stream, non-stream masking, finding emission on correct span, callback registration, non-list callbacks, span role attribute.test_safety_hooks.py: updated for new span name/event counts/role attribute; newtest_sync_wrapper_awaitable_response_fr_span_is_ambient.LlamaIndex — streaming safety:
streaming_safety.py: holdback-buffer streaming masking for LlamaIndex async streams.dispatcher_wrapper.pyandsafety.pyto wire streaming safety through the instrumentation layer.test_streaming_safety.pywith FR-marked tests.Test plan
./scripts/run-tests.sh --fr --package '*litellm*'— 39 passed, 0 failed./scripts/run-tests.sh --fr --package '*llamaindex*'— passes including new streaming safety tests./scripts/run-tests.sh --fr— 338 passed, 0 failed across all packages