Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update doc #449

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
14a7faf
fixed some typos
musslick Jun 15, 2023
8d4feec
added additional info
musslick Jun 15, 2023
64849db
fixed typos
musslick Jun 15, 2023
7943462
added hint and pointer to forum
musslick Jun 15, 2023
2faf248
added pointer
musslick Jun 15, 2023
331b94c
some minor fixes
musslick Jun 15, 2023
278afc6
added some more details on how to make a module contribution
musslick Jun 15, 2023
c116c57
fixed theorist contribution (and cross-referenced to module contribut…
musslick Jun 15, 2023
b479be7
increased figure resolution
musslick Jun 15, 2023
69cde78
added experimentalist module doc
musslick Jun 15, 2023
bd2364b
fixed broken link
musslick Jun 15, 2023
ec70d6f
renaming title
musslick Jun 15, 2023
b118866
added experiment runner image
musslick Jun 15, 2023
b055166
replaced fig with higher resolution
musslick Jun 15, 2023
f51649f
Update theorist-module.md
benwandrew Jun 16, 2023
c719b83
Update docs/contribute/core.md
musslick Jun 16, 2023
b0fe554
Update docs/contribute/core.md
musslick Jun 16, 2023
8fae546
fixed synthetic link
musslick Jun 16, 2023
05d78e6
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
7cd2ec3
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
6ef3c80
Update docs/contribute/module.md
musslick Jun 16, 2023
7b6a3ba
Update docs/contribute/module.md
musslick Jun 16, 2023
d98e786
Update docs/contribute/module.md
musslick Jun 16, 2023
1242b1b
Update docs/contribute/module.md
musslick Jun 16, 2023
3ca5e10
included conda
musslick Jun 16, 2023
946e8ab
Merge branch 'main' into doc/exp-module
benwandrew Jun 16, 2023
54b9ebb
addressed younes' suggestions
musslick Jun 16, 2023
3af9643
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
735ec46
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
8032c33
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
ae7fe20
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
02b5f39
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
5a76b3c
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
83a9dfb
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
674ca8e
Update docs/contribute/index.md
musslick Jun 16, 2023
f5b868f
Update docs/contribute/index.md
musslick Jun 16, 2023
ac00888
Update docs/contribute/index.md
musslick Jun 16, 2023
efa7ae2
Update docs/contribute/index.md
musslick Jun 16, 2023
67b3934
Update docs/contribute/module.md
musslick Jun 16, 2023
c08f709
Update docs/contribute/module.md
musslick Jun 16, 2023
c741b6c
Update docs/contribute/module.md
musslick Jun 16, 2023
4a2e8d1
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
b55927a
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
fe86e85
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
83a8d0d
Update docs/contribute/experimentalist-module.md
musslick Jun 16, 2023
b91e51d
Update docs/contribute/module.md
musslick Jun 16, 2023
a09699e
Update docs/contribute/module.md
musslick Jun 16, 2023
b59cad7
Update docs/contribute/module.md
musslick Jun 16, 2023
235bfb3
Update docs/contribute/module.md
musslick Jun 16, 2023
6977843
Update docs/contribute/module.md
musslick Jun 16, 2023
133a72d
Update docs/contribute/module.md
musslick Jun 16, 2023
55aa4bb
Update docs/contribute/module.md
musslick Jun 16, 2023
a4e730c
Update docs/contribute/module.md
musslick Jun 16, 2023
0fa4744
Update docs/contribute/module.md
musslick Jun 16, 2023
fae97e6
Update docs/contribute/module.md
musslick Jun 16, 2023
d28ff97
Update docs/contribute/module.md
musslick Jun 16, 2023
486340e
docs: add experimentalist-module.md to nav in mkdocs.yml
benwandrew Jun 16, 2023
efaf4fb
docs: sentence case for all headings in experimentalist-module.md
benwandrew Jun 16, 2023
ca638aa
docs: sentence case for headings throughout contributor docs
benwandrew Jun 16, 2023
286b676
Merge pull request #448 from AutoResearch/docs/quick-fix-to-theorist-…
benwandrew Jun 16, 2023
1fe43d0
Merge pull request #445 from AutoResearch/doc/exp-module
musslick Jun 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@
![PyPI - Downloads](https://img.shields.io/pypi/dm/autora)

<b>[AutoRA](https://pypi.org/project/autora/)</b> (<b>Auto</b>mated <b>R</b>esearch <b>A</b>ssistant) is an open-source framework for
automating multiple stages of the empirical research process, including model discovery, experimental design, data collection, and documentation for open science.
automating multiple stages of the empirical research process, including model discovery, experimental design, data collection, and documentation for open science.

AutoRA was initially intended for accelerating research in the behavioral and brain sciences. However, AutoRA is designed as a general framework that enables automation of the research processes in other empirical sciences, such as material science or physics.

![Autonomous Empirical Research Paradigm](https://github.com/AutoResearch/autora/raw/main/docs/img/overview.png)

Expand All @@ -18,7 +20,6 @@ Check out the documentation at

This project is in active development by
the [Autonomous Empirical Research Group](http://empiricalresearch.ai),
led by [Sebastian Musslick](https://smusslick.com),
in collaboration with the [Center for Computation and Visualization at Brown University](https://ccv.brown.edu).

The development of this package is supported by Schmidt Science Fellows, in partnership with the Rhodes Trust, as well as the Carney BRAINSTORM program at Brown University.
Expand Down
10 changes: 7 additions & 3 deletions docs/contribute/core.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
# Contribute to the Core
# Contribute to core packages

Core contributions are changes to AutoRA which aren't experimentalists, (synthetic) experiment runners and theorists.
Core contributions are changes to AutoRA which aren't experimentalists, (synthetic) experiment runners, or theorists.
The primary purpose of the core is to provide utilities for:

- describing experiments (in the [`autora-core` package](https://github.com/autoresearch/autora-core))
- handle workflows for automated experiments
(currently in the [`autora-workflow` package](https://github.com/autoresearch/autora-workflow))
- run synthetic experiments (currently in the [`autora-synthetic` package](https://github.com/autoresearch/autora-synthetic). Synthetic experiment runners may be submitted as pull requests to the
[`autora-synthetic`](https://github.com/AutoResearch/autora-synthetic/blob/main/CONTRIBUTING.md) package, providing they
require no additional dependencies. However, if your contribution requires additional dependencies, you can submit it as a full package following
the [module contribution guide](module.md).

Suggested changes to the core should be submitted as follows, depending on their content:

Expand All @@ -26,4 +30,4 @@ Core packages should as a minimum:
- Be compatible with all current AutoRA packages
- Have comprehensive test suites
- Use the linters and checkers defined in the `autora-core`
[.pre-commit-config.yaml](https://github.com/AutoResearch/autora-core/blob/main/.pre-commit-config.yaml)
[.pre-commit-config.yaml](https://github.com/AutoResearch/autora-core/blob/main/.pre-commit-config.yaml) (see [pre-commit hooks](pre-commit-hooks.md))
127 changes: 127 additions & 0 deletions docs/contribute/experimentalist-module.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Contribute an experimentalist

AutoRA experimentalists are meant to return novel experimental conditions based on prior experimental conditions, prior
observations, and/or prior models. Such conditions may serve as a basis for new, informative experiments conducted
by an experiment runner. Experimentalists are generally implemented as functions that can be integrated into an
[Experimentalist Pipeline](https://autoresearch.github.io/autora/core/docs/pipeline/Experimentalist%20Pipeline%20Examples/).

![Experimentalist Module](../img/experimentalist.png)

Experimentalists can be implemented as *poolers* or as *samplers*.
- **Poolers** return a pool of candidate experimental conditions, which can be passed to a sampler that selects
a subset of conditions from the pool to be used in the next experiment.
- **Samplers** directly return a subset of experimental conditions from a pool of candidate experimental conditions that already exist.

## Repository setup

We recommend using the [cookiecutter template](https://github.com/AutoResearch/autora-template-cookiecutter) to set up
a repository for your experimentalist. Alternatively, you can use the
[unguided template](https://github.com/AutoResearch/autora-template). If you choose the cookiecutter template, you can set up your repository using

```shell
cookiecutter https://github.com/AutoResearch/autora-template-cookiecutter
```

Make sure to select the `experimentalist` option when prompted. You may also select whether you want to implement an experimentalist as a sampler, pooler, or custom function. You can skip all other prompts pertaining to other modules
(e.g., experiment runners) by pressing enter.

## Implementation

Irrespective of whether you are implementing a pooler or a sampler,
you should implement a function that returns a set of experimental conditions. This set may be
a numpy array, iterator variable or other data format.

!!! hint
We generally **recommend using 2-dimensional numpy arrays as outputs** in which
each row represents a set of experimental conditions. The columns of the array correspond to the independent variables.

### Implementing poolers

Once you've created your repository, you can implement your experimentalist pooler by editing the `init.py` file in
``src/autora/experimentalist/pooler/name_of_your_experimentalist/``.
You may also add additional files to this directory if needed.
It is important that the `init.py` file contains a function called `name_of_your_experimentalist`
which returns a pool of experimental conditions (e.g., as an iterator object or numpy array).

The following example ``init.py`` illustrates the implementation of a simple experimentalist pooler
that generates a grid of samples within the specified bounds of each independent variable (IV):

```python

"""
Example Experimentalist Pooler
"""

from itertools import product
from typing import List
from autora.variable import IV


def grid_pool(ivs: List[IV]):
"""
Creates exhaustive pool from discrete values using a Cartesian product of sets

Arguments:
ivs {List[IV]}: List of independent variables

Returns:
pool: An iterator over all possible combinations of IV values
"""

l_iv_values = []
for iv in ivs:
assert iv.allowed_values is not None, (
f"gridsearch_pool only supports independent variables with discrete allowed values, "
f"but allowed_values is None on {iv=} "
)
l_iv_values.append(iv.allowed_values)

# Return Cartesian product of all IV values
return product(*l_iv_values)


```

### Implementing samplers

Once you've created your repository, you can implement your experimentalist sampler by editing the `init.py` file in
``src/autora/experimentalist/sampler/name_of_your_experimentalist/``.
You may also add additional files to this directory if needed.
It is important that the `init.py` file contains a function called `name_of_your_experimentalist`
which returns a set of experimental conditions (e.g., as a numpy array) given a pool of candidate experimental conditions.

The following example ``init.py`` illustrates the implementation of a simple experimentalist sampler
that uniformly samples without replacement from a pool of candidate conditions.

```python
"""
Example Experimentalist Sampler
"""

import random
from typing import Iterable, Sequence, Union

random_sample(conditions: Union[Iterable, Sequence], n: int = 1):
"""
Uniform random sampling without replacement from a pool of conditions.
Args:
conditions: Pool of conditions
n: number of samples to collect

Returns: Sampled pool

"""

if isinstance(conditions, Iterable):
conditions = list(conditions)
random.shuffle(conditions)
samples = conditions[0:n]

return samples
```


## Next steps: testing, documentation, publishing

For more information on how to test, document, and publish your experimentalist, please refer to the
[general guideline for module contributions](module.md) .
50 changes: 27 additions & 23 deletions docs/contribute/index.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,25 @@
# Contributor Guide
# Contributor guide

The AutoRA project is a collection of packages which together form a framework for closed-loop empirical research.
We invite contributions to all parts of the project, including the [core packages](core.md), and the [modules](module.md). Below is a brief overview of the
We invite contributions to all parts of the project, including the ["core" packages](core.md), and the [modules](module.md). Below is a brief overview of the
project structure, along with pointers to more detailed contribution guides for each part of the project.

## Project Structure
!!! hint
If you run into any issues or have any questions regarding a contribution to AutoRA, please reach out to us on the
[AutoRA forum](https://github.com/orgs/AutoResearch/discussions). We look forward to hearing from you!



## Project structure

Contributions to AutoRA are organized into one "parent" and many "child" packages.
Child packages are generally maintained by individual contributors. The parent package, along with some other
*core* packages, is maintained by [Autonomous Empirical Research Group](https://musslick.github.io/AER_website/Team.html),
*core* packages, is maintained by the [Autonomous Empirical Research Group](https://musslick.github.io/AER_website/Team.html),
as well as external contributors.

![image](../img/package_overview.png)

[`autora`](https://github.com/autoresearch/autora) is the "parent" package which end users are expected to install. The
[`autora`](https://github.com/autoresearch/autora) is the parent package which end users are expected to install. The
parent depends on core packages, such as [`autora-core`](https://github.com/autoresearch/autora-core),
[`autora-workflow`](https://github.com/autoresearch/autora-workflow), and
[`autora-synthetic`](https://github.com/autoresearch/autora-synthetic). It also includes vetted modules (child packages) as optional dependencies which users can choose
Expand All @@ -22,28 +28,28 @@ to install.
You may contribute to any of the core packages or develop your own module as a stand-alone package (see below).


### Module Contributions
## Module contributions

Each theorist, experimentalist, or experiment runner is a child package. Child packages are owned and maintained by you, the contributor, which provides several advantages:
- *Easy setup*: We provide simple [templates](module.md) for child packages, which you can use to get started quickly
Modules include theorists, experimentalists, experiment runners, or other functionalities not covered by the core packages.
All modules are child packages and can become optional dependencies of the `autora` parent package. Modules packages are
owned and maintained by you, the contributor, which provides several advantages:
- *Easy setup*: We provide simple [templates](module.md) for modules, which you can use to get started quickly
- *Independence*: You can develop and maintain your package independently of other child packages (and thereby avoid dependency conflicts)
- *Ownership*: You can publish your package on PyPI, use it in other projects, and get credit for its use.
- *Ownership*: You can publish your package on PyPI or Conda, use it in other projects, and get credit for its use.

For details on how to submit child packages
for inclusion in `autora`, see
[the module contributor guide](./module.md).

Feel free to post questions and feedback regarding core contributions on the
[the module contributor guide](./module.md). Feel free to post questions and feedback regarding module contributions on the
[AutoRA forum](https://github.com/orgs/AutoResearch/discussions/categories/module-contributions).

### Core Contributions
## Core contributions

The following packages are considered "core" packages, and are actively maintained by the
The following packages are considered core packages, and are actively maintained by the
[Autonomous Empirical Research Group](https://musslick.github.io/AER_website/Team.html):

- **autora-core** [`https://github.com/autoresearch/autora-core`](https://github.com/autoresearch/autora-core) This package includes fundamental utilities
- **autora-core** [`https://github.com/autoresearch/autora-core`](https://github.com/autoresearch/autora-core) This package includes fundamental utilities
and building blocks for all the other packages. This is always installed when a user installs `autora` and can be
a dependency of other "child" packages.
a dependency of other child packages.


- **autora-workflow** [`https://github.com/autoresearch/autora-workflow`](https://github.com/autoresearch/autora-workflow): The workflow package includes basic utilities for managing the workflow of closed-loop research processes, e.g., coordinating workflows between the theorists, experimentalists, and experiment runners. Though it currently stands alone, this package will ultimately be merged into autora-core.
Expand All @@ -54,15 +60,13 @@ a dependency of other "child" packages.

We welcome contributions to
these packages in the form of pull requests, bug reports, and feature requests. For more details, see the
[core contributor guide](core.md).

For core contributions, it is possible to set up your python environment in many different ways.
One setup which works for us is described in [the setup guide](./setup.md).

We welcome questions and feedback regarding core contributions on the
[core contributor guide](core.md). Feel free to ask any questions or provide any feedback regarding core contributions on the
[AutoRA forum](https://github.com/orgs/AutoResearch/discussions/categories/core-contributions).

For core contributions, including contributions to [`autora-synthetic`](https://github.com/autoresearch/autora-synthetic), it is possible to set up your python environment in many different ways.
One setup which works for us is described in [the setup guide](./setup.md).

!!! hint
If you would like to become actively involved in the development and maintenance of core AutoRA packages,
we welcome you to join the [Autonomous Empirical Research Group](https://musslick.github.io/AER_website/Team.html).
we welcome you to [join the Autonomous Empirical Research Group](https://musslick.github.io/AER_website/Team.html).