feat(ai-suggestions): optional qsv describegpt stage for LLM-derived metadata

[jqnatividad]

Summary

Re-implements the AI suggestions slice of PR #253 against the current Prefect pipeline. Last of the three replacement PRs for #253 (after #298 spatial extent and #299 resubmit-CLI robustness).

Stays narrowly backend-scoped: qsv subprocess wrapper, new pipeline stage, Prefect task wiring, a template helper, and unit tests. The UI port from #253 (~420 lines of JS + form-snippet plumbing) is a deliberate follow-up.

What's added

File	Change
qsv_utils.py	New QSVCommand.describegpt() wrapper, patterned after stats() / frequency(). qsv owns the LLM round-trip (endpoint, model, prompt, API key all in qsv's own config / env, NOT in ckan.ini).
jobs/stages/ai_suggestions.py (NEW)	AISuggestionsStage. Opt-in via ENABLE_AI_SUGGESTIONS (default False). Non-blocking by design: every failure mode (subprocess, qsv timeout, malformed JSON, package-patch error) logs a warning and returns context unchanged.
jobs/prefect_flow.py	New ai_suggestions_task chained between analyze_task and the database transaction() block. Outside the transaction so a (rare) failure can't poison @on_rollback hooks. Returns input AnalyzeResult unchanged.
config.py + config_declaration.yaml	Three new keys: enable_ai_suggestions (default False), describegpt_config_path (qsv prompt/config file), describegpt_timeout_seconds (default 120 — distinct from QSV_COMMAND_TIMEOUT so a hung LLM doesn't pin a worker for 30 min).
helpers.py + plugin.py	scheming_get_ai_suggestion helper parallel to scheming_get_suggestion_value. Direct top-level lookup (description/tags) → walks dictionary for {"field": ..., "summary": ...} match (with description as alt of summary).
tests/test_ai_suggestions.py (NEW)	22 unit tests covering wrapper arg assembly, stage gate, happy + failure paths, helper lookup shapes.

Output shape

When the stage succeeds, the package picks up:

{
  "dpp_suggestions": {
    "ai_suggestions": {
      "description": "A dataset of widgets.",
      "tags": ["widgets", "production"],
      "dictionary": [
        {"field": "sku", "summary": "Stock keeping unit"}
      ],
      "generated_at": "2026-05-17T03:55:00Z"
    }
  }
}

Sits alongside (NOT inside) dpp_suggestions["package"] (formula-derived suggestions) — they co-exist by design.

What we deliberately skipped

Per the #253 re-land plan:

• Bespoke Python LLM client (+449 lines in Added dpp spatial extent #253) — qsv describegpt already speaks any OpenAI-compatible endpoint (Ollama, OpenRouter, OpenAI, vLLM).
• The 10 PR-Added dpp spatial extent #253 config keys (OPENROUTER_API_KEY, AI_TEMPERATURE, AI_MAX_TOKENS, …) — those live in qsv's describegpt config file.
• The PR-Added dpp spatial extent #253 fallback heuristic — stage is opt-in + non-blocking, so a failed describegpt IS the heuristic.
• The 420-line JS port + form snippets from Added dpp spatial extent #253 — follow-up PR. The backend exposes everything needed (helper, populated ai_suggestions shape, opt-in flag); the JS is just UX plumbing.

Test plan

• 22 new unit tests pass in dpp-test container (mocked qsv + dsu calls; no real subprocess / CKAN required).
• Full unit suite: 149 passed (was 127, +22 new). No regressions.
• ENABLE_AI_SUGGESTIONS=False (default): should_skip short-circuits; pipeline unchanged.
• Stage syntax / Prefect task imports parse cleanly.
• End-to-end verification against Ollama / OpenRouter — needs an operator with a live LLM endpoint configured. Stage is opt-in so unverified-but-merged is safe.

Tracking

Closes the AI-suggestions sub-task on #259 ("DP+ 3.0 tracking issue"). #253 stays open for the UI-port follow-up.

Co-Authored-By: Minhajuddin Mohammed minhajuddin2510@gmail.com

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds an opt-in “AI suggestions” step to the DataPusher Plus Prefect pipeline by wrapping qsv describegpt, persisting the returned JSON onto the CKAN package under dpp_suggestions.ai_suggestions, and exposing a scheming helper to read those suggestions in templates.

Changes:

• Introduces QSVCommand.describegpt() and a new AISuggestionsStage that shells out to qsv describegpt and patches the CKAN package with the results.
• Wires a new Prefect task (ai_suggestions_task) between analysis and the transactional database stages; adds config keys to gate/parameterize the feature.
• Adds scheming_get_ai_suggestion helper + helper registration, plus a new unit-test suite for the feature.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_ai_suggestions.py	New unit tests for qsv wrapper args, stage gating/failure behavior, and helper lookup logic.
ckanext/datapusher_plus/qsv_utils.py	Adds QSVCommand.describegpt() wrapper with config-default timeout support.
ckanext/datapusher_plus/plugin.py	Registers the new scheming_get_ai_suggestion template helper.
ckanext/datapusher_plus/jobs/stages/ai_suggestions.py	New optional pipeline stage that runs describegpt and persists results to the package.
ckanext/datapusher_plus/jobs/prefect_flow.py	Adds a Prefect task and flow wiring to run the AI suggestions stage after analysis.
ckanext/datapusher_plus/helpers.py	Adds scheming_get_ai_suggestion helper to read AI suggestions from package metadata.
ckanext/datapusher_plus/config.py	Adds config keys for enabling AI suggestions, config path, and a dedicated describegpt timeout.
ckanext/datapusher_plus/config_declaration.yaml	Documents the new configuration keys for operators.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.