Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: updated docs for installation, theorists, experiment runners, and experimentalists #571

Merged
merged 5 commits into from
Oct 31, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/experiment-runner/index.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Experiment Runner Overview

AutoRA includes tools for running synthetic and real-world experiments.
``autora`` includes tools for running synthetic and real-world experiments.

![Overview](../img/experiment_runner.png)

For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants.
For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between ``autora`` and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants.

Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in AutoRA.
Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in ``autora``.

| Name | Links | Description |
|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. |
| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. |
| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. |
| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between ``autora`` and an experiment on Firebase. |
| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using ``autora``. |
| Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. |

45 changes: 16 additions & 29 deletions docs/experimentalist/index.md
Original file line number Diff line number Diff line change
@@ -1,41 +1,28 @@
# Experimentalist Overview

The primary goal of an experimentalist is to identify experiments that yield
scientific merit. AutoRA implements techniques for automating the identification
scientific merit. ``autora`` implements techniques for automating the identification
of novel experiments.

An experiment consists of a series of **experimental conditions** $\vec{x} \in X$.
The experimental variables manipulated in each experimental condition
are defined as **factors**, and the values of each variable to be sampled
in the experiment are defined as **levels** of the corresponding **factors**.
As an example, consider a visual discrimination tasks in which participants are presented
with two lines of different lengths, and are asked to indicate which line is longer.
There are two factors in this experiment: the length of the first line and
the length of the second line. Instances of the two line lengths
(e.g., 2.0 cm for the first line and 2.1 cm for the second line)
can be considered levels of the two factors, respectively. Thus, *an experimental condition is a vector of values that
corresponds to a specific combination of experiment levels $x_i$,
each of which is an instance of an experiment factor.*

Experimentalists in AutoRA serve to identify novel
experimental conditions $\vec{x} \in X$, where $x_i$ corresponds
to the level of an experimental factor $i$.
An experiment consists of a series of **conditions** $\vec{x} \in X$. The variables manipulated in each condition
are defined as **independent variables**. As an example, consider a visual discrimination task in which participants are presented
with two lines of different lengths, and are asked to indicate which line is longer. There are two independent variables in this experiment: the length of the first line and the length of the second line, which each have values (e.g., 2.0 cm for the first line and 2.1 cm for the second line). Thus, *a condition is a vector of values that corresponds to a specific combination of values of the independent variables $x_i$.*

![Overview](../img/experimentalist.png)

Experimentalists may use information about candidate models $M$ obtained from a theorist,
experimental conditions that have already been probed $\vec{x}' \in X'$, or
respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented
in AutoRA.
conditions that have already been probed $\vec{x}' \in X'$, or
respective observations $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented
in ``autora``.

| Name | Links | Description | Arguments |
|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | |
| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ |
| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ |
| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ |
| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ |
| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ |
| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ |
| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ |
| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ |
| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | |
| Novelty | [Package](https://pypi.org/project/autora-experimentalist-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/novelty/) | Identifies conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ |
| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/uncertainty/) | Identifies conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ |
| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/model-disagreement/) | Identifies conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ |
| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/falsification/) | An experimentalist with pooling and sampling functionality that generates and samples from novel conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ |
| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/mixture/) | Uses a mixture of specified sampling strategies to identify novel conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ |
| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/nearest-value/) | Returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ |
| Leverage | [Package](https://pypi.org/project/autora-experimentalist-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/leverage/) | Identifies conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M, X', Y'$ |
| Inequality | [Package](https://pypi.org/project/autora-experimentalist-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/inequality/) | Uses a pairwise distance metric to compare and select new conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ |