Dspy playground

[abossard]

This pull request introduces significant improvements to the repository's onboarding, documentation, and developer experience, especially for LLM-powered features and CSV ticket data workflows. The changes clarify project boundaries, add environment and VS Code setup guides, and provide more comprehensive instructions for LLM and KBA Drafter integration. The most important changes are grouped below.

Documentation and Onboarding Improvements:

• Expanded the README.md with detailed instructions for usecase demo development, KBA Drafter setup, OpenAI API key configuration, and a new DSPy prompt tuning notebook section. Added links to new and existing documentation for LLM, KBA, and CSV AI workflows. [1] [2] [3] [4]
• Added a comprehensive .env.example file with explanations for LLM, OpenAI, and KBA Drafter environment variables, including future SharePoint and ITSM adapters.
• Added CLAUDE.md with clear setup, run, test, and architecture instructions for Claude Code, covering both backend and frontend workflows, LLM configuration, and testing strategies.
• Rewrote .github/copilot-instructions.md to clarify the repository's learning-focused scope, CSV data structure, and boundaries for Copilot/AI contributions, emphasizing what should and should not be changed.

Developer Experience:

• Added .vscode/extensions.json and .vscode/launch.json to recommend key extensions and provide launch configurations for backend, frontend, and notebook workflows, supporting full stack and agentic development. [1] [2]

Environment and Tooling:

• Updated .claude/settings.local.json and .dockerignore to align with the .venv Python environment, ensuring consistent activation and Docker context. [1] [2]
• Added .gitattributes to strip outputs from Jupyter notebooks and enable proper notebook diffs, supporting clean version control for notebook-based workflows.

These changes collectively make the repository easier to set up, contribute to, and use for LLM-powered CSV ticket analysis and KBA generation.

[Copilot]

Pull request overview

Adds an “DSPy/LLM playground” developer experience layer (docs + editor config) while also introducing substantial backend functionality for agent tooling, auto-generation scheduling, and persistence.

Changes:

• Added extensive documentation for KBA Drafter, Agent Builder, CSV AI guidance, and Ubuntu setup.
• Introduced Agent Builder module + workbench integration, including REST routes, tool registry, evaluation, and SSE event streaming.
• Added auto-generation scheduler/service/models and migrated tasks.py storage from in-memory to SQLite/SQLModel, plus expanded backend dependencies and tests.

Reviewed changes

Copilot reviewed 89 out of 227 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
docs/KBA_DRAFTER.md	New KBA Drafter architecture + API + ops documentation.
docs/INSTALL_UBUNTU.md	New Ubuntu 22.04 prerequisite install guide.
docs/CSV_AI_GUIDANCE.md	New guidance for agents using CSV ticket tools.
docs/AGENT_BUILDER.md	New Agent Builder module documentation with diagrams.
backend/workbench_integration.py	Wires project operations into Agent Builder ToolRegistry + services.
backend/usecase_demo.py	Adds in-memory background runner with markdown→JSON row extraction.
backend/tests/test_workbench_integration_e2e.py	E2E-style REST verification for workbench integration with mocked LLM.
backend/tests/test_usecase_demo.py	Async tests for UsecaseDemoRunService execution + timeout handling.
backend/tests/test_search_questions.py	Adds tests for search-questions schema + cleaning behavior.
backend/tests/test_litellm_integration.py	Adds opt-in live LiteLLM/Copilot integration tests (skipped if unauthenticated).
backend/tests/test_kba_schema.py	Adds JSON Schema validation tests for legacy KBA schema.
backend/tests/test_agents.py	Adds basic tests for operation registry + langchain tool conversion.
backend/tests/conftest.py	Ensures backend import path is set up for tests.
backend/test_auto_gen.py	Adds a manual “quick test” script for auto-generation components.
backend/tasks.py	Migrates tasks to SQLModel/SQLite; adds engine/session helpers and DB init on import.
backend/scheduler.py	Adds APScheduler-based auto-generation scheduler + manual trigger.
backend/requirements.txt	Expands backend dependencies for agents, SQLModel, schedulers, LLM tooling.
backend/pytest.ini	Adds pytest config for async mode and discovery.
backend/mcp_handler.py	Adds MCP JSON-RPC handler routing to unified operations.
backend/kba_schemas.py	Adds legacy Draft-07 JSON Schema + example for KBA output.
backend/kba_prompts.py	Adds prompt builders for KBA generation, retries, markdown fallback, search questions.
backend/kba_output_models.py	Adds Pydantic models/validators for structured KBA output + search questions.
backend/kba_exceptions.py	Adds explicit KBA exception hierarchy.
backend/kba_audit.py	Adds audit logging service for KBA draft lifecycle.
backend/kb_published/KB-81C44260-gerätewechsel-für-vpn-probleme-im-efd-durchführen.md	Adds a published KBA markdown artifact.
backend/auto_gen_service.py	Implements ticket selection + sequential draft generation and settings updates.
backend/auto_gen_models.py	Adds settings DTOs + SQLModel persistence table + run result model.
backend/api_decorators.py	Extends Operation with MCP arg parsing, JSON serialization, LangChain tool conversion.
backend/agent_workbench/init.py	Adds backward-compat shim re-exporting Agent Builder API.
backend/agent_builder/tools/schema_converter.py	Converts JSON Schema → Pydantic args models (pure).
backend/agent_builder/tools/registry.py	Tool registry for dependency-injected StructuredTools.
backend/agent_builder/tools/mcp_adapter.py	Adapts external MCP tools into LangChain StructuredTools.
backend/agent_builder/tools/init.py	Re-exports tool helpers.
backend/agent_builder/tests/test_service.py	Adds WorkbenchService CRUD/tool introspection tests.
backend/agent_builder/tests/test_schema_converter.py	Adds schema converter tests.
backend/agent_builder/tests/test_registry.py	Adds ToolRegistry tests.
backend/agent_builder/tests/test_prompt_builder.py	Adds prompt builder tests (schema/defaults/efficiency mode).
backend/agent_builder/tests/test_persistence.py	Adds repository tests with real temp SQLite DB.
backend/agent_builder/tests/test_models.py	Adds extensive model validation/roundtrip tests.
backend/agent_builder/tests/test_evaluator.py	Adds evaluator tests (criteria types + scoring + llm_judge guard).
backend/agent_builder/tests/test_engine.py	Adds extract_tools_used tests.
backend/agent_builder/routes.py	Adds Quart blueprint routes for workbench CRUD/runs/eval + SSE stream.
backend/agent_builder/persistence/repository.py	Adds persistence repository for agents/runs/evaluations.
backend/agent_builder/persistence/database.py	Adds engine builder + lightweight SQLite migrations.
backend/agent_builder/persistence/init.py	Re-exports persistence public API.
backend/agent_builder/models/run.py	Adds SQLModel run table + JSON property accessors + DTO.
backend/agent_builder/models/evaluation.py	Adds criteria/evaluation models + SQLModel table.
backend/agent_builder/models/chat.py	Adds chat request/response models.
backend/agent_builder/models/agent.py	Adds agent definition models + JSON-backed fields + create/update DTOs.
backend/agent_builder/models/init.py	Re-exports models API.
backend/agent_builder/evaluator.py	Implements criteria evaluation + optional llm_judge + scoring.
backend/agent_builder/engine/react_runner.py	Implements LLM construction, ReAct agent execution, tool usage extraction.
backend/agent_builder/engine/prompt_builder.py	Implements structured output schema prompting + efficiency mode prompt.
backend/agent_builder/engine/event_bus.py	Adds in-process pub/sub event bus with history for SSE.
backend/agent_builder/engine/callbacks.py	Adds logging + SSE publishing callbacks for tools/LLM calls.
backend/agent_builder/engine/init.py	Re-exports engine API.
backend/agent_builder/chat_service.py	Adds one-shot chat service using shared ReAct runner.
backend/agent_builder/init.py	Defines Agent Builder public API surface.
backend/=3.10.4	Adds a file containing pip install output (likely accidental).
CLAUDE.md	Adds Claude Code guidance: setup/run/test/architecture notes.
.vscode/launch.json	Adds VS Code launch configs for backend/frontend/notebooks/browser.
.vscode/extensions.json	Adds recommended VS Code extensions.
.github/copilot-instructions.md	Rewrites Copilot guidance for CSV-ticket learning scope and boundaries.
.gitattributes	Enables nbstripout + notebook diff driver configuration.
.env.example	Adds documented environment variable template for LLM/KBA features.
.dockerignore	Updates ignore rules to align with .venv naming.
.claude/settings.local.json	Updates Claude allowed command to source .venv.

[Copilot]

run_now() assumes _run_auto_generation() always returns an object with model_dump(), but _run_auto_generation() returns None when auto-generation is disabled (and also on exceptions). This will raise an AttributeError on manual runs; return a consistent result shape (e.g., an AutoGenRunResult with success=False + reason) or handle None explicitly in run_now().

Suggested change

	Result dictionary with generation stats
	"""
	logger.info("Manual trigger of auto-generation")
	result = await self._run_auto_generation()
	Result dictionary with generation stats. If auto-generation
	did not run (e.g., disabled or failed), returns a failure
	dictionary with a reason message.
	"""
	logger.info("Manual trigger of auto-generation")
	result = await self._run_auto_generation()
	if result is None:
	# Ensure we always return a dictionary, even when auto-generation
	# is disabled or fails with an exception.
	return {
	"success": False,
	"reason": "Auto-generation did not run; see logs for details.",
	}

[Copilot]

run_now() assumes _run_auto_generation() always returns an object with model_dump(), but _run_auto_generation() returns None when auto-generation is disabled (and also on exceptions). This will raise an AttributeError on manual runs; return a consistent result shape (e.g., an AutoGenRunResult with success=False + reason) or handle None explicitly in run_now().

[Copilot]

The SSE tool_end event reports tool_name using kwargs.get("name"), but on_tool_end typically doesn’t receive the tool name in kwargs; this will often publish empty tool names and break UI/telemetry correlation. Persist the tool name from on_tool_start keyed by callback run_id (similar to _start_times) and reuse it in on_tool_end/on_tool_error.

[Copilot]

delete_all_runs() loads all runs into memory just to compute the count. Prefer a COUNT(*) query (or use a delete statement with rowcount if supported by the driver) to avoid O(n) memory/time for large tables.

Suggested change

	count = session.exec(select(AgentRun)).all().__len__()
	session.exec(sql_delete(AgentRun))
	session.commit()
	return count
	result = session.exec(sql_delete(AgentRun))
	session.commit()
	# result.rowcount may be None if not supported; normalize to int
	return int(result.rowcount or 0)

[Copilot]

This looks like accidental pip output committed as a file (backend/=3.10.4). It will confuse contributors and pollute the repo; remove it and consider adding a guard (e.g., .gitignore patterns) to prevent similar artifacts from being committed.

Suggested change

	Collecting APScheduler
	Downloading apscheduler-3.11.2-py3-none-any.whl.metadata (6.4 kB)
	Collecting tzlocal>=3.0 (from APScheduler)
	Downloading tzlocal-5.3.1-py3-none-any.whl.metadata (7.6 kB)
	Downloading apscheduler-3.11.2-py3-none-any.whl (64 kB)
	Downloading tzlocal-5.3.1-py3-none-any.whl (18 kB)
	Installing collected packages: tzlocal, APScheduler
	Successfully installed APScheduler-3.11.2 tzlocal-5.3.1

[Copilot]

The comment says the DB path can be set via environment variable, but the implementation always uses Path(__file__).parent / "data" / "tasks.db". Either read an env var (e.g., TASKS_DB_PATH) or adjust the comment so behavior is accurate.

[Copilot]

get_stats() materializes full result sets just to count rows, which will scale poorly as tasks grow. Use COUNT(*) aggregation queries to compute total and completed without loading rows into Python.

[Copilot]

Initializing/creating the SQLite DB on module import introduces side effects during import (creates directories/files, runs DDL) which can be problematic in test discovery, tooling, or multi-process deployments. Consider moving initialization to an explicit app startup hook (or a lazy “init once” path) so imports remain side-effect free.

Suggested change

	DB_PATH.parent.mkdir(parents=True, exist_ok=True)
	# Create SQLAlchemy engine
	DATABASE_URL = f"sqlite:///{DB_PATH}"
	engine = create_engine(DATABASE_URL, echo=False)
	def init_db():
	"""Initialize database - create all tables."""
	SQLModel.metadata.create_all(engine)
	def get_session():
	"""Get database session."""
	return Session(engine)
	# Initialize database on module import
	init_db()
	# Create SQLAlchemy engine
	DATABASE_URL = f"sqlite:///{DB_PATH}"
	engine = create_engine(DATABASE_URL, echo=False)
	_DB_INITIALIZED = False
	def init_db() -> None:
	"""Initialize database - ensure directory exists and create all tables.
	This function is idempotent and safe to call multiple times.
	"""
	global _DB_INITIALIZED
	if _DB_INITIALIZED:
	return
	# Create parent directory for the SQLite file if it does not exist.
	DB_PATH.parent.mkdir(parents=True, exist_ok=True)
	# Create all tables defined on SQLModel metadata.
	SQLModel.metadata.create_all(engine)
	_DB_INITIALIZED = True
	def get_session() -> Session:
	"""Get database session, initializing the database on first use."""
	init_db()
	return Session(engine)
	# Database initialization is performed lazily when a session is first requested.

[Copilot]

serialize_result() returns a non-JSON string for bool results ("Success: True"), while other branches serialize JSON. For MCP clients expecting consistent JSON text (especially if they parse tool outputs), this inconsistency can cause failures; prefer serializing booleans as JSON (e.g., true/false with json.dumps).

Suggested change

	return f"Success: {result}"
	return json.dumps(result, indent=2)

[Copilot]

The PR description emphasizes onboarding/documentation improvements, but the diff includes major functional/backend changes (e.g., migrating tasks persistence to SQLite/SQLModel, adding agent builder modules, scheduler/auto-generation). Consider updating the PR description to explicitly call out these runtime-affecting changes (or splitting into separate PRs) so reviewers can scope risk and release impact accurately.

[Copilot]

Pull request overview

Copilot reviewed 68 out of 79 changed files in this pull request and generated 2 comments.

[Copilot]

run_baseline is defined as run_baseline(task_id: str, *, max_eval: Optional[int] = None) and uses the currently configured DSPy LM. These calls pass a model as a positional arg, which will raise TypeError: run_baseline() takes 1 positional argument but 2 were given. Configure the model via configure_dspy(model=...) (or dspy.context) and call run_baseline("sentiment", max_eval=...) instead.

[Copilot]

run_optimization is defined as run_optimization(task_id: str, optimizer: str = "BootstrapFewShot", *, max_eval=None, instructions=None) and does not take a model argument. Passing get_default_model() as the second positional argument will be interpreted as the optimizer string, and passing "BootstrapFewShot" as the third positional argument will raise a TypeError. Configure the LM separately (e.g., configure_dspy(get_default_model())) and call run_optimization("sentiment", optimizer="BootstrapFewShot", max_eval=5) (or pass instructions= if intended).

[Copilot]

Pull request overview

Copilot reviewed 74 out of 85 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 76 out of 87 changed files in this pull request and generated 19 comments.

[Copilot]

Same issue as above: run_baseline is keyword-only after task_id, so passing get_default_model() positionally will raise TypeError. Configure the model via configure_dspy(model=...) before calling run_baseline(...), or change the action signature to accept a model.

[Copilot]

Same issue as above: run_baseline doesn’t accept a positional model argument (only task_id + keyword args). This call will raise a TypeError unless you configure the model separately via configure_dspy(model=...).

[Copilot]

This loop calls run_baseline("sentiment", model, max_eval=3) but run_baseline does not accept a positional model argument. This will raise TypeError; configure the model with configure_dspy(model) before each baseline run (or change run_baseline to accept a model).

[Copilot]

These tests are explicitly making real LLM calls (network + cost + rate limits), which makes the default test suite flaky and non-deterministic in CI. Consider gating them behind an env var (e.g. RUN_LIVE_LLM_TESTS=1) and pytest.skip by default, or marking them (e.g. @pytest.mark.live) and excluding that marker in CI.

[Copilot]

This script uses an absolute, developer-specific file path (/Users/.../csv/data.csv), which makes it non-portable. Use a repo-relative Path(...) to csv/data.csv instead.

[Copilot]

signal.SIGALRM is not available on Windows and alarm-based timeouts only work reliably in the main thread. As written, this helper will crash on platforms without SIGALRM. Consider guarding with hasattr(signal, "SIGALRM") and falling back to no timeout (or another timeout mechanism) when unavailable.

[Copilot]

This notebook cell contains an incomplete statement (lm =) which will cause a SyntaxError when executed. Remove it or replace it with the intended model configuration (e.g. call configure_dspy(...) and assign the returned LM if you need it).

[Copilot]

This cell uses dspy and TranslateEnDe but neither is imported in the notebook prior to use, so execution will fail with NameError. Add the missing imports (e.g. import dspy and an import for TranslateEnDe).

[Copilot]

This script uses an absolute, developer-specific file path (/Users/.../csv/data.csv), which makes it non-portable. Use a repo-relative Path(...) to csv/data.csv instead.

[Copilot]

This script uses an absolute, developer-specific file path (/Users/.../csv/data.csv), which makes it non-portable. Use a repo-relative Path(...) to csv/data.csv instead.