feat(ai-suggestions): scheming UI for AI-derived metadata (PR #253 follow-up)

[jqnatividad]

Summary

Completes the AI-suggestions feature from PR #253 by porting the scheming form-layer UI on top of the backend that landed in #301. The push-time pipeline already populates package["dpp_suggestions"]["ai_suggestions"] — this PR adds the form-side affordances to surface and apply those suggestions.

What's added

File	Notes
assets/js/scheming-ai-suggestions.js (NEW, 419 lines)	Ported verbatim from #253. Polls package_show every 2.5s, reads dpp_suggestions.ai_suggestions, shows per-field AI buttons when suggestions exist, opens popovers with text + source, applies on click. Terminates when STATUS ∈ [DONE, ERROR, FAILED].
templates/scheming/snippets/ai_suggestions_{button,asset}.html (NEW)	Dataset-level "Get AI Suggestions" CTA gated on scheming_has_ai_suggestion_fields(schema) + the asset loader. Verbatim from #253.
templates/scheming/form_snippets/markdown.html	Per-field AI button block gated on scheming_field_supports_ai_suggestion(field).
assets/webassets.yml	Registers the new datapusher_plus/ai-suggestions bundle.
helpers.py + plugin.py	Four helpers the templates call: _value, _source, _supports_ai_suggestion, _has_ai_suggestion_fields. PR #301's scheming_get_ai_suggestion renamed to _value to match the name templates reference.
jobs/stages/ai_suggestions.py	New _reshape_for_ui transforms qsv's verbatim output into the per-field {value, source} schema the JS reads. Sets STATUS=DONE so polling terminates early.
tests/test_ai_suggestions.py	34 tests (was 23). New coverage for the 3 new helpers + JSON-string decoding + bad-input defensiveness.

Output shape (UI contract)

{
  "dpp_suggestions": {
    "ai_suggestions": {
      "description": {"value": "A dataset of widgets.", "source": "qsv describegpt"},
      "tags":        {"value": "widgets, production", "source": "qsv describegpt"},
      "sku":         {"value": "Stock keeping unit",  "source": "qsv describegpt"},
      "STATUS": "DONE",
      "generated_at": "2026-05-17T04:55:00Z"
    }
  }
}

tags comma-joined (scheming's tag field accepts that natively); per-column entries come from qsv's dictionary[i].field / .summary; STATUS=DONE terminates the JS polling loop early (without it the loop runs out the 100s maxPollAttempts clock — works, but burns redundant package_show calls).

Field schema convention

To opt a scheming field into AI suggestions, add ai_suggestion: true to its config:

- field_name: notes
  label: Description
  preset: markdown
  ai_suggestion: true

scheming_field_supports_ai_suggestion(field) reads this; scheming_has_ai_suggestion_fields(schema) walks both dataset_fields and resource_fields to drive the dataset-level CTA.

Backwards compat

The on-disk schema for ai_suggestions changes (verbatim qsv shape → per-field {value, source}). Practical blast radius is zero: enable_ai_suggestions is off by default and #301 merged minutes ago, so production packages with the old shape effectively don't exist. PR #301's helper was renamed in the same edit (scheming_get_ai_suggestion → _value).

Test plan

• 34 new/updated unit tests pass (mocked qsv, mocked dsu, no real subprocess / CKAN required).
• Full unit suite: 161 passed (was 149, +12 net after the rewrite swap). No regressions.
• Stage reshape correctness verified by test_process_happy_path_persists_suggestions (asserts the per-field {value, source} shape, comma-joined tags, STATUS=DONE).
• JS not unit-tested (no JS test infrastructure in this codebase). Needs manual UI verification on a scheming form with ai_suggestion: true fields once an operator has a working qsv describegpt config + opt-in flag enabled.

Tracking

This is the deferred UI port called out in the close comment on #253 and the #259 checkbox. With this landed, the original PR #253 feature set is fully re-implemented across #298 (spatial extent), #301 (AI backend), and this PR (AI UI).

Co-Authored-By: Minhajuddin Mohammed minhajuddin2510@gmail.com

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds the scheming-form UI layer for DataPusher Plus “AI suggestions” and aligns the persisted dpp_suggestions.ai_suggestions shape with what the UI expects (per-field {value, source} plus polling termination bookkeeping).

Changes:

• Persist AI suggestions in a UI-friendly per-field {value, source} map and stamp STATUS=DONE + generated_at.
• Introduce a new Webassets bundle + JS polling/popover UX for applying AI suggestions in scheming forms.
• Add new template helpers and extend unit tests for helper behavior and reshape logic.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_ai_suggestions.py	Expands coverage for the new {value, source} contract and new helpers.
ckanext/datapusher_plus/templates/scheming/snippets/ai_suggestions_button.html	Adds dataset-level CTA markup (currently mismatched with JS module behavior).
ckanext/datapusher_plus/templates/scheming/snippets/ai_suggestions_asset.html	Adds asset loader snippet for AI suggestions bundle.
ckanext/datapusher_plus/templates/scheming/form_snippets/markdown.html	Adds per-field AI button rendering for markdown preset fields.
ckanext/datapusher_plus/plugin.py	Registers new template helpers for AI suggestions value/source and schema gating.
ckanext/datapusher_plus/jobs/stages/ai_suggestions.py	Reshapes qsv output into {value, source} entries and adds STATUS=DONE.
ckanext/datapusher_plus/helpers.py	Adds helper functions to fetch AI suggestion value/source and schema opt-in checks.
ckanext/datapusher_plus/assets/webassets.yml	Registers the new datapusher_plus/ai-suggestions bundle.
ckanext/datapusher_plus/assets/js/scheming-ai-suggestions.js	Implements polling + popover UI for AI suggestions.

Comments suppressed due to low confidence (2)

ckanext/datapusher_plus/assets/js/scheming-ai-suggestions.js:135

• Polling termination checks dppSuggestionsData.STATUS, but the stage writes STATUS under dpp_suggestions.ai_suggestions.STATUS (and dpp_suggestions.STATUS is used by the formula-suggestions pipeline). As written, the AI polling loop won’t terminate early and will keep making redundant package_show calls until maxPollAttempts. Update the status lookup to read from the AI suggestions payload (e.g., dppSuggestionsData.ai_suggestions.STATUS).

              // Check if processing is complete
              var currentStatus = dppSuggestionsData.STATUS ? dppSuggestionsData.STATUS.toUpperCase() : null;
              
              if (currentStatus && self.options.terminalStatuses.includes(currentStatus)) {
                console.log("AI Suggestions: Processing complete with status " + currentStatus);

ckanext/datapusher_plus/assets/js/scheming-ai-suggestions.js:269

• suggestionSource is interpolated directly into popoverContent HTML without escaping. Since this string ultimately comes from dataset data, it can become an XSS vector in the editor UI. Escape suggestionSource before concatenating, or inject it via .text() after creating the DOM nodes (similar to how escapedDisplayValue is handled).

  var popoverContent = 
    '<div class="suggestion-popover-content">' +
      '<div class="ai-suggestion-header">' +
        '<strong>AI Suggestion</strong>' +
        '<div class="ai-suggestion-source">' + suggestionSource + '</div>' +
      '</div>' +

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.