Feature Proposal: Leverage Declarative Build Systems like makepkg or nix

[tcztzy]

Hello Sola team,

First off, this is a very interesting project! The ability to automatically set up and test arbitrary GitHub projects is a significant challenge, and your approach is very promising.

I'm opening this issue to propose and discuss the possibility of integrating more formal and declarative build systems, specifically Arch Linux's makepkg and nix from NixOS, as a core part of the agent's execution strategy.[1][2]

Currently, ExecutionAgent uses a large language model to autonomously figure out the necessary steps to build and test a project in an isolated container (Correct me if I misundersood). While this is incredibly flexible, it might also lead to less reproducible and potentially fragile build processes, as the LLM has to rediscover the build logic for each project.

Proposal

Instead of, or in addition to, the current approach of generating a sequence of shell commands, the ExecutionAgent could attempt to generate a declarative build definition, such as a PKGBUILD file for makepkg or a Nix derivation.[3] Arhlinux has very detailed instruction to help user create new packages.[4] The agent could then use makepkg or nix-build to perform the actual build and dependency installation.

Potential Benefits

• Increased Reliability and Reproducibility: Both makepkg and nix are designed to create reproducible builds.[1][2] By targeting these systems, the agent would produce a build process that is more robust and less prone to environment-specific failures.
• Leveraging Existing Knowledge: The Arch User Repository (AUR) and the Nixpkgs repository are vast collections of build scripts for a wide variety of software. The agent could potentially learn from these existing package definitions to generate new ones for projects that are not yet packaged.
• Clearer Dependency Management: PKGBUILD and Nix derivations have explicit dependency declarations. This would allow the agent to clearly define and install build-time and run-time dependencies, leading to cleaner build environments.
• Structured Build Process: These build systems provide a well-defined structure for the build process (e.g., prepare(), build(), check(), package() functions in a PKGBUILD). This could provide a clearer framework for the LLM to work within, potentially improving its accuracy.

Discussion Points

• Do you think this is a viable direction for the ExecutionAgent?
• Would a hybrid approach be feasible, where the agent falls back to the current shell command generation if it fails to create a valid PKGBUILD or Nix derivation?
• Could the agent start by trying to find an existing PKGBUILD or Nix expression for the project (or a similar one) and adapt it?

I believe that integrating these powerful build systems could significantly enhance the robustness and capabilities of the ExecutionAgent.

Thanks for your time and consideration