From 38d56f9e7fee028d92eb1c86d20234a7362e4725 Mon Sep 17 00:00:00 2001 From: Chad C Williams Date: Tue, 19 Sep 2023 10:56:34 -0700 Subject: [PATCH 1/5] Updated installation --- docs/installation.md | 49 ++++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 13 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 43f135e4f..626567ccd 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -29,8 +29,7 @@ Type "help", "copyright", "credits" or "license" for more information. ``` ## Step 2: Install `autora` -!!! success - We recommended using a `Python` environment manager like `virtualenv`. You may refer to the Development Guide on how to [set up a virtual environment](https://autoresearch.github.io/autora/contribute/setup/#create-a-virtual-environment). +We recommended using a `Python` environment manager like `virtualenv`. You may refer to the Development Guide on how to [set up a virtual environment](https://autoresearch.github.io/autora/contribute/setup/#create-a-virtual-environment). Before installing the PyPI ``autora`` package, you may [activate your environment](https://autoresearch.github.io/autora/contribute/setup/#activating-and-using-the-environment). To install the PyPI `autora` package, run the following command: @@ -48,13 +47,13 @@ In using AutoRA, it is helpful to be aware of its structure, which is described ## Step 3: Install Optional Dependencies -AutoRA is organized into one "parent" and many "child" packages. +``autora`` is organized into one "parent" and many "child" packages. ![image](img/package_overview.png) The ``autora`` "parent" package depends on core packages, such as [`autora-core`](https://github.com/autoresearch/autora-core), [`autora-workflow`](https://github.com/autoresearch/autora-workflow), and [`autora-synthetic`](https://github.com/autoresearch/autora-synthetic), which are automatically installed with ``autora``. -However, the ``autora`` ecosystem also includes vetted modules (child packages) as **optional dependencies** which users can choose to install. Below, we provide a list of all optional dependencies that are currently vetted by the AutoRA Core Team. +However, the ``autora`` ecosystem also includes vetted modules (child packages) as **optional dependencies** which users can choose to install. Below, we provide a list of all optional dependencies that are currently vetted by the ``autora`` core team. To install any (combination) of optional dependencies, users should run the relevant analogue of the following command, with the name in brackets matching the name as specified in the parent `pyproject.toml` file: @@ -76,7 +75,7 @@ python -c "from autora.theorist.bms import BMSRegressor; BMSRegressor()" ### List of Vetted Dependencies -For a complete list of dependencies that have been vetted by the core AutoRA team, see the `[project.optional-dependencies]` section of the [`pyproject.toml` file](https://github.com/AutoResearch/autora/blob/main/pyproject.toml) in the parent `autora` package. +For a complete list of dependencies that have been vetted by the core ``autora`` team, see the `[project.optional-dependencies]` section of the [`pyproject.toml` file](https://github.com/AutoResearch/autora/blob/main/pyproject.toml) in the parent `autora` package. For your convenience, we provide a list of vetted packages along with their description below. **Required dependencies** are included in the installation of the ``autora`` package. @@ -87,17 +86,41 @@ For your convenience, we provide a list of vetted packages along with their desc | [theorist-bms](https://pypi.org/project/autora-theorist-bms/) | Optional | Theorist | Equation Discovery Method Using the Bayesian Machine Scientist ([GuimerΓ  et al. 2020](https://www.science.org/doi/full/10.1126/sciadv.aav6971)) | [πŸ”—](https://autoresearch.github.io/autora/user-guide/theorists/bms/) | | [theorist-bsr](https://pypi.org/project/autora-theorist-bsr/) | Optional | Theorist | Equation Discovery Method Using the Bayesian Symbolic Regression ([Jin et al. 2020](https://arxiv.org/abs/1910.08892)) | [πŸ”—](https://autoresearch.github.io/autora/user-guide/theorists/bsr/) | | [theorist-darts](https://pypi.org/project/autora-theorist-darts/) | Optional | Theorist | Equation Disovery Method using Differentiable Architecture Search ([Liu et al., 2018](https://arxiv.org/abs/1806.09055); [Musslick et al., 2020](https://escholarship.org/uc/item/9wd571ts) ) | [πŸ”—](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | -| [experimentalist-falsification](https://pypi.org/project/autora-experimentalist-falsification/) | Optional | Experimentalist (Sampler/Pooler) | Identifies experimental conditions method that maximize predicted model loss (Musslick et al., 2023) | [πŸ”—](https://autoresearch.github.io/autora/falsification/docs/pooler/) [πŸ”—](https://autoresearch.github.io/autora/falsification/docs/sampler/) | -| [experimentalist-sampler-inequality](https://pypi.org/project/autora-experimentalist-sampler-inequality/) | Required | Experimentalist (Sampler) | Identifies experimental conditions from a candidate pool based on their inequality to a reference pool | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | -| [experimentalist-sampler-mixture](https://pypi.org/project/mixture-experimentalist/) | Required | Experimentalist (Sampler) | Identifies experimental conditions based on a weighted score of multiple experimentalists. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | -| [experimentalist-sampler-model-disagreement](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/) | Required | Experimentalist (Sampler) | Identifies experimental conditions for which a list of models make different predictions | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | -| [experimentalist-sampler-nearest-value](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/) | Required | Experimentalist (Sampler) | Identifies experimental conditions from a candidate pool that are nearest to experimental conditions in a reference pool. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | -| [experimentalist-sampler-novelty](https://pypi.org/project/autora-experimentalist-sampler-novelty/) | Required | Experimentalist (Sampler) | Identifies experimental conditions from a candidate pool that are most novel with respect to a reference pool. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | -| [experimentalist-sampler-uncertainty](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/) | Optional | Experimentalist (Sampler) | Identifies experimental conditions based on uncertainty sampling. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | +| [experimentalist-falsification](https://pypi.org/project/autora-experimentalist-falsification/) | Optional | Experimentalist | Identifies experimental conditions method that maximize predicted model loss ([Musslick et al., 2023](https://escholarship.org/content/qt5ch569fg/qt5ch569fg.pdf)) | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/falsification/) | +| [experimentalist-inequality](https://pypi.org/project/autora-experimentalist-inequality/) | Required | Experimentalist | Identifies experimental conditions from a candidate pool based on their inequality to a reference pool | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/inequality/) | +| [experimentalist-mixture](https://pypi.org/project/mixture-experimentalist/) | Required | Experimentalist | Identifies experimental conditions based on a weighted score of multiple experimentalists. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/mixture/) | +| [experimentalist-model-disagreement](https://pypi.org/project/autora-experimentalist-model-disagreement/) | Required | Experimentalist | Identifies experimental conditions for which a list of models make different predictions | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/model-disagreement/) | +| [experimentalist-nearest-value](https://pypi.org/project/autora-experimentalist-nearest-value/) | Required | Experimentalist | Identifies experimental conditions from a candidate pool that are nearest to experimental conditions in a reference pool. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/inequality/) | +| [experimentalist-novelty](https://pypi.org/project/autora-experimentalist-novelty/) | Required | Experimentalist | Identifies experimental conditions from a candidate pool that are most novel with respect to a reference pool. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/novelty/) | +| [experimentalist-uncertainty](https://pypi.org/project/autora-experimentalist-uncertainty/) | Optional | Experimentalist | Identifies experimental conditions based on uncertainty sampling. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experimentalists/uncertainty/) | | [autora-experiment-runner-firebase-prolific](https://pypi.org/project/autora-experiment-runner-firebase-prolific/) | Optional | Experiment Runner | This experiment runner provides functionality for running behavioral experiments hosted on Firebase using human participants from Prolific. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | -| [experiment-runner-experimentation-manager-firebase](https://pypi.org/search/?q=experiment-runner-experimentation-manager-firebase) | Optional | Experiment Runner (Experimentation Manager) | This manager provides functionality to manage communication of conditions and observation between AutoRA and a behavioral experiment hosted on Firebase. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | +| [experiment-runner-experimentation-manager-firebase](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/) | Optional | Experiment Runner (Experimentation Manager) | This manager provides functionality to manage communication of conditions and observation between AutoRA and a behavioral experiment hosted on Firebase. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | | [autora-experiment-runner-recruitment-manager-prolific](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/) | Optional | Experiment Runner (Recruitment Manager) | This manager provides functionality for recruiting human participants from Prolific. | [πŸ”—](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | +### List of Bundled Dependencies + + + +``autora`` also provides the ability to install some of these optional dependencies in a bundle. These bundles can also be found in the `[project.optional-dependencies]` section of the [`pyproject.toml` file](https://github.com/AutoResearch/autora/blob/main/pyproject.toml) in the parent `autora` package. + +You can install these bundles just like any other package. For example, to install the ``all-theorists`` bundle: + +```shell +pip install -U "autora[all-theorists]" +``` + +For your convenience, we provide a list of vetted bundled packages and what components they include below. + + + +| Bundle Name | Components Included | +|---|---| +|all|The theorist, experimentalist, and experiment runner bundles| +|all-theorists| [theorist-bms](https://pypi.org/project/autora-theorist-bms/), [theorist-bsr](https://pypi.org/project/autora-theorist-bsr/), [theorist-darts](https://pypi.org/project/autora-theorist-darts/)| +|all-experimentalists| [experimentalist-falsification](https://pypi.org/project/autora-experimentalist-falsification/), [experimentalist-inequality](https://pypi.org/project/autora-experimentalist-inequality/), [experimentalist-mixture](https://pypi.org/project/mixture-experimentalist/), [experimentalist-model-disagreement](https://pypi.org/project/autora-experimentalist-model-disagreement/), [experimentalist-nearest-value](https://pypi.org/project/autora-experimentalist-nearest-value/), [experimentalist-novelty](https://pypi.org/project/autora-experimentalist-novelty/), [experimentalist-uncertainty](https://pypi.org/project/autora-experimentalist-uncertainty/)| +all-experiment_runners | [autora-experiment-runner-firebase-prolific](https://pypi.org/project/autora-experiment-runner-firebase-prolific/)| +all-experiment-runner-experimentation-managers|[experiment-runner-experimentation-manager-firebase](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/)| +all-experiment-runner-recruitment-managers|[autora-experiment-runner-recruitment-manager-prolific](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/) From 6f9b6fbdb7eb3df668f31cae6c8e8dc5152e396b Mon Sep 17 00:00:00 2001 From: Chad C Williams Date: Tue, 19 Sep 2023 11:31:33 -0700 Subject: [PATCH 2/5] Updated theorists docs --- docs/theorist/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/theorist/index.md b/docs/theorist/index.md index d34179db2..a730cd9b6 100644 --- a/docs/theorist/index.md +++ b/docs/theorist/index.md @@ -9,7 +9,7 @@ respective dependent measures $\vec{y}' \in Y'$. The following table includes th | Name | Links | Description | Arguments | |--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| -| Differentiable Architecture Search (DARTS) | [Package](https://pypi.org/project/autora-theorist-darts/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | A theorist that automates the discovery of neural network architectures by making architecture search amenable to gradient descent. | $X', Y'$ | | Bayesian Machine Scientist (BMS) | [Package](https://pypi.org/project/autora-theorist-bms/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bms/) | A theorist that uses one algorithmic Bayesian approach to symbolic regression, with the aim of discovering interpretable expressions which capture relationships within data. | $X', Y'$ | | Bayesian Symbolic Regression (BSR) | [Package](https://pypi.org/project/autora-theorist-bsr/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bsr/) | A theorist that uses another algorithmic Bayesian approach to symbolic regression, with the aim of discovering interpretable expressions which capture relationships within data. | $X', Y'$ | +| Differentiable Architecture Search (DARTS) | [Package](https://pypi.org/project/autora-theorist-darts/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | A theorist that automates the discovery of neural network architectures by making architecture search amenable to gradient descent. | $X', Y'$ | From e3b2dd6e957cffd0a8f44e1ec314a256153a1321 Mon Sep 17 00:00:00 2001 From: Chad C Williams Date: Tue, 19 Sep 2023 11:39:19 -0700 Subject: [PATCH 3/5] Updated experiment runner docs --- docs/experiment-runner/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/experiment-runner/index.md b/docs/experiment-runner/index.md index 8f2e0d25a..5f6243b9b 100644 --- a/docs/experiment-runner/index.md +++ b/docs/experiment-runner/index.md @@ -1,12 +1,12 @@ # Experiment Runner Overview -AutoRA includes tools for running synthetic and real-world experiments. +``autora`` includes tools for running synthetic and real-world experiments. ![Overview](../img/experiment_runner.png) -For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants. +For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between ``autora`` and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants. -Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in AutoRA. +Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in ``autora``. | Name | Links | Description | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| From db3977a7f4a3041b648368c81b91544114014d13 Mon Sep 17 00:00:00 2001 From: Chad C Williams Date: Tue, 19 Sep 2023 12:28:30 -0700 Subject: [PATCH 4/5] Updated experimentalist docs --- docs/experimentalist/index.md | 45 +++++++++++++---------------------- 1 file changed, 16 insertions(+), 29 deletions(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index 52183c1a6..8160d913e 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -1,41 +1,28 @@ # Experimentalist Overview The primary goal of an experimentalist is to identify experiments that yield -scientific merit. AutoRA implements techniques for automating the identification +scientific merit. ``autora`` implements techniques for automating the identification of novel experiments. -An experiment consists of a series of **experimental conditions** $\vec{x} \in X$. -The experimental variables manipulated in each experimental condition -are defined as **factors**, and the values of each variable to be sampled -in the experiment are defined as **levels** of the corresponding **factors**. -As an example, consider a visual discrimination tasks in which participants are presented -with two lines of different lengths, and are asked to indicate which line is longer. -There are two factors in this experiment: the length of the first line and -the length of the second line. Instances of the two line lengths -(e.g., 2.0 cm for the first line and 2.1 cm for the second line) -can be considered levels of the two factors, respectively. Thus, *an experimental condition is a vector of values that -corresponds to a specific combination of experiment levels $x_i$, -each of which is an instance of an experiment factor.* - -Experimentalists in AutoRA serve to identify novel -experimental conditions $\vec{x} \in X$, where $x_i$ corresponds -to the level of an experimental factor $i$. +An experiment consists of a series of **conditions** $\vec{x} \in X$. The variables manipulated in each condition +are defined as **independent variables**. As an example, consider a visual discrimination task in which participants are presented +with two lines of different lengths, and are asked to indicate which line is longer. There are two independent variables in this experiment: the length of the first line and the length of the second line, which each have values (e.g., 2.0 cm for the first line and 2.1 cm for the second line). Thus, *a condition is a vector of values that corresponds to a specific combination of values of the independent variables $x_i$.* ![Overview](../img/experimentalist.png) Experimentalists may use information about candidate models $M$ obtained from a theorist, -experimental conditions that have already been probed $\vec{x}' \in X'$, or -respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented - in AutoRA. +conditions that have already been probed $\vec{x}' \in X'$, or +respective observations $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented + in ``autora``. | Name | Links | Description | Arguments | |--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------| -| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | | -| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ | -| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ | -| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ | -| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ | -| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ | -| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ | -| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ | -| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ | +| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | | +| Novelty | [Package](https://pypi.org/project/autora-experimentalist-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/novelty/) | Identifies conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ | +| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/uncertainty/) | Identifies conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ | +| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/model-disagreement/) | Identifies conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ | +| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/falsification/) | An experimentalist with pooling and sampling functionality that generates and samples from novel conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ | +| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/mixture/) | Uses a mixture of specified sampling strategies to identify novel conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ | +| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/nearest-value/) | Returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ | +| Leverage | [Package](https://pypi.org/project/autora-experimentalist-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/leverage/) | Identifies conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M, X', Y'$ | +| Inequality | [Package](https://pypi.org/project/autora-experimentalist-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/inequality/) | Uses a pairwise distance metric to compare and select new conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ | From 3ec545aca63f63853eff91d6877dbc84cf70d7d4 Mon Sep 17 00:00:00 2001 From: Chad C Williams Date: Tue, 19 Sep 2023 12:28:48 -0700 Subject: [PATCH 5/5] Relabeled AutoRA to autora in docs --- docs/experiment-runner/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/experiment-runner/index.md b/docs/experiment-runner/index.md index 5f6243b9b..d14bb9e1d 100644 --- a/docs/experiment-runner/index.md +++ b/docs/experiment-runner/index.md @@ -11,7 +11,7 @@ Experiment runner tools may take in information about new experimental condition | Name | Links | Description | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. | -| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. | -| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. | +| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between ``autora`` and an experiment on Firebase. | +| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using ``autora``. | | Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. |