GSoC 2026 Module B — Week 3: Stage 2 LLM relevance classifier

[manshusainishab]

Summary

Adds Stage 2 of Module B (Noise/Relevance Filter): an LLM classifier that labels
each content chunk as KNOWLEDGE, NOISE, or UNCERTAIN under the
recall-first rule. Builds on the Week 1 schemas and Week 2 regex/sanitize stages.

This PR is self-contained (classifier + prompt + config + tests). Pipeline
wiring, the queue/DB model, and the CLI entry point come in later weeks.

What's added

• application/utils/noise_filter/config_loader.py — loads Module B settings
  from CRE_NOISE_FILTER_* environment variables into a typed NoiseFilterConfig
  (model, batch size, per-chunk char cap, confidence threshold), with defaults.
• application/utils/noise_filter/prompts.py — the recall-first system prompt
  and a few-shot block (5 KNOWLEDGE / 3 NOISE / 2 UNCERTAIN worked examples), plus
  a helper that renders a numbered batch of chunks into the user prompt.
• application/utils/noise_filter/llm_classifier.py — LLMClassifier, which
  classifies a list of ChangeRecords and returns one ClassifyResult per record.
• application/tests/noise_filter/llm_classifier_test.py — 14 unit tests,
  fully mocked (no network calls).
• .env.example — documents the four new CRE_NOISE_FILTER_* variables.

How the classifier works

• Sends each chunk's heading_path + text to a dedicated lightweight model via
  LiteLLM (default gemini/gemini-2.5-flash-lite).
• Processes records in batches (CRE_NOISE_FILTER_BATCH_SIZE, default 10), one
  request per batch, and maps results back to input order by index.
• Requests a strict JSON-schema response; if the provider doesn't support strict
  mode, falls back to JSON-object mode.
• Truncates each chunk to CRE_NOISE_FILTER_MAX_CHARS (default 1500) before sending.
• Retries on rate-limit/quota errors using the existing CRE_LLM_MAX_RETRIES /
  CRE_LLM_RETRY_SLEEP_SECONDS settings.
• Returns UNCERTAIN (confidence 0.0) for any unparseable, malformed, or invalid
  output, and marks a whole batch UNCERTAIN if the LLM call fails — so a bad
  response never aborts a run.

Configuration

Variable	Default	Purpose
CRE_NOISE_FILTER_LLM_MODEL	gemini/gemini-2.5-flash-lite	Classification model (LiteLLM string)
CRE_NOISE_FILTER_BATCH_SIZE	10	Chunks per LLM request
CRE_NOISE_FILTER_MAX_CHARS	1500	Per-chunk character cap before sending
CRE_NOISE_FILTER_CONFIDENCE_THRESHOLD	0.8	Minimum confidence to enqueue a KNOWLEDGE verdict

The model is Gemini, so it authenticates with the existing GEMINI_API_KEY;
no new credential is required.

Testing

• 14 new unit tests covering prompt content, batch ordering and splitting,
  malformed/invalid/empty output handling, the JSON-schema fallback, rate-limit
  retry and exhaustion, and truncation.
• Full suite: 369 passing, 1 skipped, 0 failures.
• black --check clean across the repo.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 0295bf71-c815-4406-918b-dfe2164b0696

📥 Commits

Reviewing files that changed from the base of the PR and between cbb0ca3 and 8c2286a.

📒 Files selected for processing (1)

• .env.example

💤 Files with no reviewable changes (1)

• .env.example

---

Summary by CodeRabbit

• New Features
  • Added Module B noise/relevance filter configuration in the environment example, including model, batch size, max characters, and confidence threshold.
  • Introduced a recall-first LLM relevance classifier with JSON-only output rules, truncation of overly long inputs, and batch processing.
• Bug Fixes
  • Improved resilience to malformed LLM outputs with safe UNCERTAIN fallbacks and validation.
  • Added retry behavior for transient rate-limit/quota errors, with graceful failure after exhausted retries.
• Tests
  • Added unit tests for config loading/validation plus end-to-end classifier behavior covering prompt correctness, ordering, batching, retries, malformed outputs, truncation, and strict-output fallback.

Walkthrough

Adds Module B noise-filter configuration, prompt construction, and an LLM classifier that batches ChangeRecord inputs, retries rate-limited calls, truncates long text, and parses ordered verdicts. Includes unit tests for prompt content, batching, malformed output, fallback behavior, retries, truncation, and config loading.

Changes

Noise Filter Module B

Layer / File(s)	Summary
Configuration surface .env.example, application/utils/noise_filter/config_loader.py	Module B env defaults, typed config loading, and the new settings block are added together.
Prompt contract application/utils/noise_filter/prompts.py	Defines the base system prompt, few-shot examples, JSON renderers, and user prompt builder for the classifier.
Batch classification flow application/utils/noise_filter/llm_classifier.py	Adds batched LLM calls, strict-schema fallback, rate-limit retries, truncation, and result parsing into ordered verdicts.
Classifier test coverage application/tests/noise_filter/llm_classifier_test.py, application/tests/noise_filter/config_loader_test.py	Adds tests for prompt text, ordering, batching, malformed output, response-format fallback, retries, truncation, and config loading/validation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• OWASP/OpenCRE#913: Introduces the Module B contract models that the new LLMClassifier consumes and validates.

Suggested reviewers: Pa04rth

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.78% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: adding Stage 2 of the Module B LLM relevance classifier.
Description check	✅ Passed	The description is directly aligned with the implemented config, prompts, classifier, tests, and .env example changes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@application/utils/noise_filter/config_loader.py`:
- Around line 29-36: NoiseFilterConfig currently allows invalid values to be
constructed directly, so add invariant checks inside its dataclass
initialization path. Update NoiseFilterConfig to validate batch_size >= 1,
max_chars >= 1, and confidence_threshold between 0.0 and 1.0 in __post_init__,
so every construction route fails fast before llm_classifier uses these fields.
Keep the checks centralized in NoiseFilterConfig so direct callers and
load_config() both get the same validation behavior.

In `@application/utils/noise_filter/llm_classifier.py`:
- Around line 151-163: The fallback in llm_classifier.py is too broad: the
try/except around self._completion_with_retry in the strict_schema path retries
on every exception, even non-capability failures. Update the logic in the
classifier method that builds messages and calls _completion_with_retry so only
schema- მხარდაჭာ unsupported capability errors (for example the provider’s
BadRequestError for strict schema) trigger the json_object retry, and let all
other exceptions propagate without a second attempt. Keep the warning/logging
specific to the capability fallback path so the retry only happens when strict
schema is genuinely unsupported.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 6ed01a44-6a87-42ec-9e1b-c7220c2dfc79

📥 Commits

Reviewing files that changed from the base of the PR and between 4485936 and 02d4bd4.

📒 Files selected for processing (5)

• .env.example
• application/tests/noise_filter/llm_classifier_test.py
• application/utils/noise_filter/config_loader.py
• application/utils/noise_filter/llm_classifier.py
• application/utils/noise_filter/prompts.py

[northdpole]

Good job on B-W3. The recall-first prompt and few-shots are clear, fail-safe parsing (UNCERTAIN on bad output / failed batches) is the right call for this gate, and the json_schema fallback only on capability gaps is a nice improvement over the older pattern. Fine to merge as the Week 3 slice — looking forward to queue + pipeline wiring in W4/W5.

[manshusainishab]

updated