fix(agentic_rag): honor similarity_threshold in deduplicate_chunks

[AbirAbbas]

Summary

deduplicate_chunks in the agentic_rag example accepted a similarity_threshold parameter but never read it. Deduplication was a plain exact-match lookup against a set of normalized chunk texts, so only byte-identical (after lowercase + whitespace normalization) chunks were ever dropped. Near-duplicates — exactly what overlapping chunk windows and the multi-pass retrieval loop in main.py produce — always survived, no matter what threshold the caller passed.

The fix adds a small text_similarity helper (word-overlap / Jaccard, in the style of the neighbouring keyword_match_score) and compares each candidate chunk against the chunks already kept, dropping it when the score meets the threshold. Identical text scores 1.0, so the default 0.9 still removes exact duplicates exactly as before — the change is a strict superset of the old behavior.

Independently implements the fix reported in #829 by @andrewwhitecdw (whose patch we couldn't accept because the CLA was not signed). Closes the underlying issue.

Validation contract

• Two chunks whose similarity is >= similarity_threshold are treated as duplicates; the later one is removed.
• Two chunks whose similarity is below the threshold are both kept.
• The comparison is inclusive at the boundary: similarity exactly equal to the threshold counts as a duplicate.
• Default behavior stays sensible: with the default 0.9, exact/normalized-identical chunks are still removed and distinct chunks are still kept.
• First occurrence wins, input order is preserved, and all chunk keys (chunk_id, score, text) survive — main.py re-hydrates the result with RankedChunk(**c), so dropping keys would break the caller.
• Empty input returns an empty list.

Test plan

examples/** has no pytest infrastructure and no CI gate: sdk-python.yml is path-filtered to sdk/python/**, make lint only lints sdk/python, and coverage.yml explicitly documents that examples-only PRs skip. (The one test_*.py under examples/python_agent_nodes/ is a runnable demo script, not a pytest module.) Rather than add a test file nothing would execute, the contract above was verified by driving the function directly with a throwaway script — embedding_manager is stubbed since deduplicate_chunks needs no embeddings:

similarity(a, b) = 0.778
[PASS] (a) near-duplicate removed at threshold 0.5 -> 1 kept
[PASS] (a) removed at threshold exactly == similarity -> 1 kept
[PASS] (b) near-duplicate kept at threshold 0.95 -> 2 kept
[PASS] (b) unrelated chunks both kept -> 2 kept
[PASS] (c) default threshold removes exact/normalized duplicates, keeps distinct
[PASS] (c) default threshold keeps 0.8-similar chunks -> 2 kept
[PASS] (c) empty input
[PASS] (c) first occurrence wins, all keys preserved -> ['1', '3']

Note the first three cases: the same pair of chunks is kept or dropped purely as a function of the threshold — which is the behavior that was impossible before this change.

Lint, using the ruff version CI pins in sdk-python.yml:

• uvx ruff@0.15.22 check examples/python_agent_nodes/agentic_rag/skills.py — All checks passed!
• uvx ruff@0.15.22 format --check examples/python_agent_nodes/agentic_rag/skills.py — 1 file already formatted
• python3 -m py_compile on the changed file — OK

🤖 Generated with Claude Code