Curie: Automate Rigorous Scientific Experimentation

Curie is the first AI-agent framework designed for automated and rigorous scientific experimentation. Curie helps answer your curiosity through end-to-end experimentation automation, ensuring that every step—from hypothesis formulation to result interpretation—is conducted with precision, reliability, and reproducibility.

Key Features

• 🚀 Automated Experimentation – End-to-end workflow management: hypothesis formulation, experiment setup, experiment execution, result analysis and finding reflection.
• 📊 Rigor Enhancement - Built-in verification modules enforce methodical procedure, reliability and interpretability.
• 🔬 Broad Applicability – Supports ML research, system analysis, and scientific discovery.
• 📖 Experimentation Benchmark - Provide 46 questions from 4 Computer Science domains, based on influential papers and open-source projects (benchmark/experimentation_bench).

Table of Contents

• Installation
• Quick Start
• Use Cases
• Tutorial
• Customize Your Experiment Agents

Installation

1. Install docker: https://docs.docker.com/engine/install/ubuntu/.

• Grant permission to docker via sudo chmod 666 /var/run/docker.sock.

• If you encounter an error that /var/run/docker.sock doesn’t exist, you may find the actual path to docker.sock and create a soft link. For example, Docker Desktop stores this file at ~/.docker/desktop/docker.sock, in which case you may use:

  sudo chmod 666 ~/.docker/desktop/docker.sock
  sudo ln -s ~/.docker/desktop/docker.sock /var/run/docker.sock

• Run docker ps to check that permission has been granted with the Docker daemon.

2. Clone the repository:

git clone https://github.com/Just-Curieous/Curie.git
cd Curie

3. Put your LLM API credentials under curie/setup/env.sh. Example:

export MODEL="gpt-4o" 
export OPENAI_API_KEY="sk-xxx" 

4. Build the container image. This will take a few minutes. Note: you may need to setup a virtual environment before running pip install.

pip install -e .
docker images -q exp-agent-image | xargs -r docker rmi -f # remove any existing conflict image
cd curie && docker build --no-cache --progress=plain -t exp-agent-image -f ExpDockerfile_default .. && cd -

Quick Start

Use the following command to input your research question or problem statement: python3 -m curie.main -q "<Your research question>".

Example 1: Understanding Sorting Algorithm Efficiency

python3 -m curie.main \
  -q "How does the choice of sorting algorithm impact runtime performance across different \
  input distributions (random, nearly sorted, reverse sorted)?" --report

• Estimated runtime: ~5 minutes
• Sample log file: Available here
• Experiment report: Available here.
• Log monitoring:
  • Real-time logs are streamed to the console.
  • Logs are also stored in:
    • logs/research_question_<ID>.log
    • logs/research_question_<ID>_verbose.log
  • Experiment report details:
    • Stored in: logs/research_question_<ID>.md
    • Will only be produced when the --report flag is used.
• Reproducibility: The full experimentation process is saved in workspace/research_<ID>/.

Example 2: How does the choice of activation function (e.g., ReLU, sigmoid, tanh) impact the model training convergence rate?

python3 -m curie.main -f benchmark/junior_ml_engineer_bench/q1_activation_func.txt --report

• Detailed question: q1_diffusion_step.txt
• Sample log file: Available here
• Sample report file: Available here

More example questions can be found here.

Tutorial

• How to let Curie work on your own starter files?
• How to reproduce the results in `Large Language Monkeys'.

Use Cases

Curie is designed for scientific discovery across multiple domains:

• 🔬 Machine Learning & AI Research – Hyperparameter tuning and algorithm behavior
  • How does the optimal learning rate change with the increase of model size?
  • How does repeated sampling in LLM inference affect the quality of response?
• 💻 System Performance Analysis – Benchmarking systems, optimizing configurations, investigating system trade-offs.
  • What configurations affects the energy consumption of LLM serving?
  • How does the request bursty arrival pattern affects the user experience in LLM serving?
• 🧪 Algorithmic & Scientific Discovery – Validating hypotheses, automating computational simulations.

Customize Your Experimentation Agents

Config curie/configs/base_config.json to adapt to your own tasks:

• Add your domain-specific instructions by customizing supervisor_system_prompt_filename for the supervisor, control_worker_system_prompt_filename for the experimentation worker, and so on.
• Human interruption in the experiment design phase can be activated by setting the is_user_interrupt_allowed key to true.
• Configure timeouts and maximum number of steps (global, and coding agent specific).

Community and Support

For any issues or feature requests, please open an issue on our GitHub Issues page.

License

Curie is released under the Apache 2.0 License. See LICENSE for more details.