LLM-powered agents: From scaffolding to verified projects

[Akshay1018]

Proposal: Build a Verified AI Agent Workflow for Trove

Overview

I would like to explore an AI agent workflow that helps developers go beyond generating project files.

The proposed agent would plan a backend configuration, use Trove to generate the project, run appropriate validation checks, diagnose failures, and suggest or apply bounded repairs.

The objective is not to promise that every application can be built fully autonomously. It is to reduce repetitive setup and debugging while making the result more reliable and easier to verify.

Proposed Workflow

Given a request such as:

Generate a NestJS API with PostgreSQL, authentication, Docker, and automated tests.

The workflow could:

• Convert the request into a structured project specification.
• Check the specification against Trove's supported capabilities.
• Present the plan for review.
• Generate the project using existing templates.
• Run supported build, lint, type-check, and test commands.
• Analyze failures and propose targeted fixes.
• Apply approved changes and rerun the relevant checks.
• Summarize the result, including failures and outstanding manual steps.

Proposed Capabilities

1. Structured planning

Convert natural-language requirements into a typed project configuration and explain unresolved decisions.

2. Deterministic execution

Use Trove's generation engine and supported templates rather than allowing the LLM to invent an arbitrary project structure.

3. Automated verification

Run relevant checks and report their actual results.

4. Bounded repair

Allow the agent to propose focused changes when validation fails, with configurable retry limits and appropriate approval.

5. Transparent results

Provide a summary of generated files, validation results, modifications, and unresolved errors.

6. Safe execution

Restrict filesystem operations, protect secrets, and require approval for destructive or externally consequential actions.

Proposed Initial Scope

• Define the agent workflow and its execution states.
• Reuse the existing generator and validation capabilities.
• Implement structured error reporting.
• Support build and test verification.
• Add a limited repair workflow.
• Establish permission boundaries and execution limits.
• Create repeatable test cases to measure generation success.

Advanced features such as autonomous deployment, cloud provisioning, and unrestricted code modification would remain outside the initial scope.

Measuring Success

Rather than measuring success by the number of AI features, I would propose evaluating:

• First-pass build success rate.
• Test pass rate for generated projects.
• Accuracy of stack recommendations.
• Percentage of failures successfully diagnosed.
• Repair success rate.
• Time taken to generate and validate a project.
• Frequency of unsupported recommendations or unsafe proposed actions.

Questions for the Community

1. Should the initial agent only generate and verify projects, or should it also attempt automatic repairs?
2. Which checks should run by default?
3. Should repairs always require approval, or should safe, narrowly scoped repairs be optionally automatic?
4. What should happen when the agent repeatedly fails to fix a problem?
5. How should the agent communicate partial success and unresolved issues?
6. Which security boundaries are essential for an AI agent that can write files and execute commands?
7. What metrics would convince you that the generated project is reliable?

Suggested First Milestone

Build a workflow that plans a project, validates its configuration, generates it, and runs a defined set of checks.

Only after this workflow is reliable should Trove introduce automatic repair attempts.

I would appreciate feedback on the proposed workflow, execution safeguards, evaluation criteria, and the features that would make this useful in real development environments.