From 3ba0ce25d368d9946b2f232b1d91ec6143a5761e Mon Sep 17 00:00:00 2001 From: Sasha Date: Wed, 25 Sep 2024 14:42:27 -0400 Subject: [PATCH 1/2] . --- docs/agent.md | 30 +++++++++++++ docs/index.md | 7 ++++ docs/setupdist.md | 39 +++++++++++++++++ docs/{setup.md => setuplocal.md} | 72 +------------------------------- mkdocs.yml | 3 +- 5 files changed, 79 insertions(+), 72 deletions(-) create mode 100644 docs/agent.md create mode 100644 docs/setupdist.md rename docs/{setup.md => setuplocal.md} (50%) diff --git a/docs/agent.md b/docs/agent.md new file mode 100644 index 0000000..b39f80d --- /dev/null +++ b/docs/agent.md @@ -0,0 +1,30 @@ +## Running an Agent + +Next we will see how this can be run with an AI agent system. +We will use [Aider](https://aider.chat/) which is a nice +command-line oriented agent system. + +To setup Aider first set your api key. +We recommend using Claude Sonnet. + +```bash +# Work with Claude 3.5 Sonnet on your repo +export ANTHROPIC_API_KEY=your-key-goes-here +``` + +Once this is setup you can run Aider with the following command. +This will edit the files locally in your branch, but +run the tests inside the environment. + +```bash +aider --model sonnet --file repos/minitorch/operators.py --message "fill in" \ + --auto-test --test \ + --test-cmd 'commit0 test minitorch branch=mychange tests/test_operators.py::test_relu' \ + --yes +``` + +This will run an LLM agent that will try to fill in the +functions in one file of the minitorch library. + +For a full example baseline system that tries to solve +all the tests in the library see the [baseline](baseline) documentation. diff --git a/docs/index.md b/docs/index.md index 5b9f22d..e49b2bd 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,6 +19,13 @@ The [commit0 tool](setup) allows you to: * Distribute testing and development across cloud systems * Track and log all changes made throughout. +To install run: + +```bash +pip install commit0 +``` + + | | Name | Repo | Commit0 | Tests | | |--|--------|-------|----|----|------| | | [minitorch](https://minitorch.github.io/) | [[orig](http://github.com/minitorch/minitorch)] | [[commit0](http://github.com/commit-0/minitorch)] | 230 | | diff --git a/docs/setupdist.md b/docs/setupdist.md new file mode 100644 index 0000000..5ac175f --- /dev/null +++ b/docs/setupdist.md @@ -0,0 +1,39 @@ +## Distributed Mode + +Commit0 is a command-line tool that allows you to run unit-tests on a +variety of libraries in isolated environments. + +Commit0 uses [modal](https://modal.com/) as a distributed +test runner. + +```bash +pip install modal +modal token new +``` + +To get started, run the `setup` command with the dataset +split that youare interested in working with. +We'll start with the `lite` split. + +```bash +commit0 setup lite +``` + +This will clone a set of skeleton libraries in your `repos/` directory. +Commiting changes to branches in this directory + + + + + +You can pass this configuration file as an argumnet to clone. + +```bash +commit0 setup lite +``` + +Next to run tests you can run the standard test command. + +```bash +commit0 test simpy tests/test_event.py::test_succeed +``` diff --git a/docs/setup.md b/docs/setuplocal.md similarity index 50% rename from docs/setup.md rename to docs/setuplocal.md index e7ffd93..ce4b9e9 100644 --- a/docs/setup.md +++ b/docs/setuplocal.md @@ -1,6 +1,5 @@ -# Quickstart +## Local Mode -## Install First be sure that you have docker tools installed. @@ -10,11 +9,6 @@ apt install docker To install the benchmark run, -```bash -pip install commit0 -``` - -## Commands The system is a command-line tool that allows you to run unit-tests on a variety of libraries in isolated environments. To get started with the full @@ -73,67 +67,3 @@ a branch and the environment will sync and run. ```bash commit0 test minitorch branch=mychange tests/test_operators.py::test_relu ``` - -## Running an Agent - -Next we will see how this can be run with an AI agent system. -We will use [Aider](https://aider.chat/) which is a nice -command-line oriented agent system. - -To setup Aider first set your api key. -We recommend using Claude Sonnet. - -```bash -# Work with Claude 3.5 Sonnet on your repo -export ANTHROPIC_API_KEY=your-key-goes-here -``` - -Once this is setup you can run Aider with the following command. -This will edit the files locally in your branch, but -run the tests inside the environment. - -```bash -aider --model sonnet --file repos/minitorch/operators.py --message "fill in" \ - --auto-test --test \ - --test-cmd 'commit0 test minitorch branch=mychange tests/test_operators.py::test_relu' \ - --yes -``` - -This will run an LLM agent that will try to fill in the -functions in one file of the minitorch library. - -For a full example baseline system that tries to solve -all the tests in the library see the [baseline](baseline) documentation. - - -## Distributed Tests - -One of the main advantages of `commit0` is that it can run -a range of unit tests in distributed environments. - -By default, the library is configured to work with [modal](https://modal.com/). - -```bash -pip install modal -modal token new -``` - -To enable distributed run, first -create a file called `distributed.yaml` - -```yaml -backend: modal -base_dir: repos.dist/ -``` - -You can pass this configuration file as an argumnet to clone. - -```bash -commit0 clone lite --cfg=distributed.yaml -``` - -Next to run tests you can run the standard test command. - -```bash -commit0 test simpy master tests/test_event.py::test_succeed --cfg=distributed.yaml -``` diff --git a/mkdocs.yml b/mkdocs.yml index 67cf1fb..f6e64f8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -3,7 +3,8 @@ site_name: "" docs_dir: docs nav: - Home: index.md - - Setup: setup.md + - Distributed: setupdist.md + - Local: setuplocal.md - Extending: repos.md - About: about.md theme: From 2b0cc7dc3c7f8433adf04ae649163151941a8a02 Mon Sep 17 00:00:00 2001 From: Sasha Date: Wed, 25 Sep 2024 14:49:30 -0400 Subject: [PATCH 2/2] . --- docs/agent.md | 30 --------------------------- docs/setupdist.md | 23 +++++++++++++-------- docs/setuplocal.md | 51 ++++++++-------------------------------------- 3 files changed, 24 insertions(+), 80 deletions(-) delete mode 100644 docs/agent.md diff --git a/docs/agent.md b/docs/agent.md deleted file mode 100644 index b39f80d..0000000 --- a/docs/agent.md +++ /dev/null @@ -1,30 +0,0 @@ -## Running an Agent - -Next we will see how this can be run with an AI agent system. -We will use [Aider](https://aider.chat/) which is a nice -command-line oriented agent system. - -To setup Aider first set your api key. -We recommend using Claude Sonnet. - -```bash -# Work with Claude 3.5 Sonnet on your repo -export ANTHROPIC_API_KEY=your-key-goes-here -``` - -Once this is setup you can run Aider with the following command. -This will edit the files locally in your branch, but -run the tests inside the environment. - -```bash -aider --model sonnet --file repos/minitorch/operators.py --message "fill in" \ - --auto-test --test \ - --test-cmd 'commit0 test minitorch branch=mychange tests/test_operators.py::test_relu' \ - --yes -``` - -This will run an LLM agent that will try to fill in the -functions in one file of the minitorch library. - -For a full example baseline system that tries to solve -all the tests in the library see the [baseline](baseline) documentation. diff --git a/docs/setupdist.md b/docs/setupdist.md index 5ac175f..e04679b 100644 --- a/docs/setupdist.md +++ b/docs/setupdist.md @@ -3,7 +3,7 @@ Commit0 is a command-line tool that allows you to run unit-tests on a variety of libraries in isolated environments. -Commit0 uses [modal](https://modal.com/) as a distributed +The defaul tool uses [modal](https://modal.com/) as a distributed test runner. ```bash @@ -20,20 +20,27 @@ commit0 setup lite ``` This will clone a set of skeleton libraries in your `repos/` directory. -Commiting changes to branches in this directory - - +Commiting changes to branches in this directory is how you send changes +to the test runner. +Next to run tests you can run the standard test command. +This command will run a reference unit test for the `simpy` repo. +```bash +commit0 test simpy tests/test_event.py::test_succeed --reference +``` -You can pass this configuration file as an argumnet to clone. +To run a test in your codebase you can run with no args. +This one will fail. ```bash -commit0 setup lite +commit0 test simpy tests/test_event.py::test_succeed ``` -Next to run tests you can run the standard test command. +To run a test in your codebase with a specific branch +you can commit to the branch and call with the --branch command. + ```bash -commit0 test simpy tests/test_event.py::test_succeed +commit0 test simpy tests/test_event.py::test_succeed --branch my_branch ``` diff --git a/docs/setuplocal.md b/docs/setuplocal.md index ce4b9e9..c238b39 100644 --- a/docs/setuplocal.md +++ b/docs/setuplocal.md @@ -1,30 +1,29 @@ ## Local Mode - -First be sure that you have docker tools installed. +To run in local mode you first be sure that you have [docker tools](https://docs.docker.com/desktop/install/mac-install/) +installed. On Debian systems: ```bash apt install docker ``` -To install the benchmark run, - +To get started, run the `setup` command with the dataset +split that you are interested in working with. +We'll start with the `lite` split. -The system is a command-line tool that allows you to run unit-tests on a -variety of libraries in isolated environments. To get started with the full -setup run the `setup` command which will install a clone the code of a subset -of libraries to your `repos/` directory. ```bash commit0 setup lite ``` +This will install a clone the code for subset of libraries to your `repos/` directory. + Next run the `build` command which will configure Docker containers for each of the libraries with isolated virtual environments. The command uses the [uv](https://github.com/astral-sh/uv) library for efficient builds. ```bash -commit0 build lit +commit0 build ``` The main operation you can do with these enviroments is to run tests. @@ -34,36 +33,4 @@ Here we run [a test](https://github.com/commit-0/simpy/blob/master/tests/test_ev commit0 test simpy tests/test_event.py::test_succeed ``` -This test should run and pass, but others will fail. - -```bash -commit0 test minitorch tests/test_operators.py::test_relu -``` - -Let's now manually go in and change that repo. -This is all just standard shell commands. - -```bash -cd repos/minitorch/ -git checkout -b mychange -``` - -And apply and commit this patch. - -``` ---- a/minitorch/operators.py -+++ b/minitorch/operators.py -@@ -81,7 +81,7 @@ def relu(x: float) -> float: - (See https://en.wikipedia.org/wiki/Rectifier_(neural_networks) .) - """ - # TODO: Implement for Task 0.1. -- raise NotImplementedError('Need to implement for Task 0.1') -+ return 1. if x > 0. else 0. -``` - -Once this is done we can run `test` with -a branch and the environment will sync and run. - -```bash -commit0 test minitorch branch=mychange tests/test_operators.py::test_relu -``` +See [distributed setup](setupdist) for more commands.