feat: data agent connector for lumid.data

[timzsu]

Purpose

This PR supports data access via our data plane (lumid.data), which supports natural language-based data retrieval via a data agent. A workflow node describes what it wants in natural language, optionally constrains the search to a schema scope, and gets back the materialized rows alongside the canonical SQL chain the agent emitted — so a downstream node can bind to either.

Changes

• src/worker/connectors/agent_connector.py (new) + connectors/__init__.py — thin wrapper around lumid_data.sdk.Client.retrieve_to_file.
• src/worker/executors/data_retrieval_executor.py — adds the data.type == "agent" branch and emits per-row items[] with table + access chain + run metadata; schema_scope is optional.
• tests/worker/test_agent_connector.py (new) — connector tests with mocked SDK.
• pyproject.toml, uv.lock, src/worker/requirements/requirements.txt — add lumid-data-sdk to the analytics extra; mypy follow_untyped_imports for lumid_data.sdk[.*].
• scripts/dev/sync_requirements.py — accept the name @ git+url form in the package-name regex.
• .github/workflows/security.yml — strip @ git+ deps before pip-audit --strict; rename the temp file to /tmp/requirements-worker-cpu-audit.txt; ignore two new vllm advisories (GHSA-x368-4g9h-fvv4, GHSA-83vm-p52w-f9pw) blocked by the same transformers 4.57 pin.
• templates/data_retrieval_agent.yaml (new) — two-stage e2e: agent retrieval → Qwen 1.5B summary.
• src/shared/utils/manifest.py (new) + deletions of src/{server,worker}/utils/manifest.py — single home for prepare_output_dir / sync_manifest. prepare_output_dir creates each directory at 0o0777, and sync_manifest chmods the manifest to 0o0666 after each write, so single-node deployments where the server (root) and worker (appuser) share the results volume can both overwrite the manifest from either UID.
• src/worker/executors/utils/graph_templates.py — wrap the to_markdown sites in pd.option_context(max_columns=None, width=None, max_colwidth=None) so nested DataFrame cells render in full instead of pandas' default ...-clipped layout.

Test Plan

• Live e2e on a GPU worker pointed at a running lumid.data:

flowmesh stack build
flowmesh stack up
flowmesh stack worker up gpu -t 1
flowmesh workflow submit templates/data_retrieval_agent.yaml
flowmesh result fetch <tsk_id>

Test Result

Output from the first node:

{
  "ok": true,
  "type": "agent",
  "items": [
    {
      "index": 0,
      "description": "Show me NVDA's last 10 quarters of fundamentals and key financial-health ratios.",
      "params": {
        "symbol": "NVDA"
      },
      "table": {
        "df": "***" # Serialized Pandas Dataframe
      },
      "rows": 10,
      "access_chain": [
        {
          "op": "sql",
          "query": "WITH *** SELECT ***", # The SQL query
          "bucket": null,
          "key": null,
          "rows_or_bytes": 10
        }
      ],
      "run_id": "run-***",
      "transcript_url": "http://127.0.0.1:9102/v1/admin/runs/run-***",
      "tokens_in": 20451,
      "tokens_out": 8861,
      "steps_taken": 8,
      "replay_latency_ms": 13,
      "materialized_uri": "s3://lumid-data/***/result.jsonl"
    }
  ],
  "count": 1,
  "_artifacts": {
    "base_dir": "/var/lib/flowmesh-results/tsk-***",
    "base_url": null
  }
}

Output of the second node:

{
  "ok": true,
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "items": [
    {
      "output": [
        "The headline trend across the rows for the leading metric is a significant increase in revenue and earnings per share (EPS) from Q4 FY2025 to Q1 FY2025, followed by a decline in Q2 FY2025. This dataset enables analysts to assess NVDA's financial health and performance trends over the past 10 quarters, providing insights into its revenue growth, profitability, and market valuation."
      ]
    }
  ],
  "usage": {
    "prompt_tokens": 1518,
    "completion_tokens": 89,
    "total_tokens": 1607,
    "latency_sec": 0.30570387840270996,
    "num_requests": 1
  },
  "_artifacts": {
    "base_dir": "/var/lib/flowmesh-results/tsk-***",
    "base_url": null
  }
}

[kaiitunnz]

A few minor comments.

[kaiitunnz]

A few minor comments.

[kaiitunnz]

A few more comments.

[kaiitunnz]

LGTM. Fix the pip-audit issues, and then you can merge.