Skip to content

GSH v1.4.0 — Real LangChain Telemetry Adapter

Choose a tag to compare

View all tags
@sunilgentyala sunilgentyala released this 15 Jul 22:37
· 15 commits to main since this release
v1.4.0
3cb1900

What's in this release

  • A real LangChain callback adapter (adapters/langchain_callback.py). GSHCallbackHandler attaches to any LangChain Runnable, tool, or agent via the standard config={"callbacks": [handler]} mechanism and evaluates:
    • Tool-call rate and token velocity over a rolling window against Hunt-001 thresholds.
    • Unauthorized tool invocations against an optional allowlist (Hunt-004 style).
    • Suspicious call parameters (credential patterns, path traversal, suspicious encoding) by reusing inspect_parameters() from the MCP proxy adapter.
    • Sends findings to Splunk/Elastic via the existing adapters/siem_dispatch.py, falling back to local file output.
  • Closes repo issue #9.

Important, verified limitation: alert-only, not enforcement

A LangChain callback handler cannot block a tool call. I read LangChain's actual callbacks/manager.py source to confirm this rather than assume it: handle_event() wraps every handler invocation in a try/except and only re-raises if the handler's raise_error is explicitly set (default False) — an exception raised inside on_tool_start is caught, logged as a warning by LangChain itself, and swallowed.

Every finding this adapter emits is therefore explicitly marked enforcement_mode: "alert_only" and action_taken: "ALERTED" — never "BLOCKED" — regardless of policy mode. A dedicated test (test_all_findings_are_alert_only_never_blocked) asserts this holds even when aggressive/block mode is requested. GSH's one real enforcement point remains the MCP proxy (adapters/mcp_proxy.py), which sits as a true man-in-the-middle and can actually refuse a call.

This adapter also has no visibility into DNS queries (Hunt-002) — that requires DDI-AI Fusion telemetry, wired separately.

Verified, not just written

Tested against real langchain-core 1.4.8 objects (FakeListLLM, the @tool decorator) invoked through LangChain's actual callback propagation — not hand-mocked callback calls. 13 new tests in tests/test_langchain_callback.py; 43/43 across the full suite.

Known limitations

  • Only langchain-core's BaseCallbackHandler interface is targeted; other agent frameworks (AutoGen, CrewAI, Haystack, DSPy) are not covered.
  • Alert-only by design (see above) — do not rely on this adapter for enforcement.
  • No DNS-query visibility (Hunt-002).

Full Changelog

v1.3.0...v1.4.0

Assets 2
Loading