Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorporate Estimator and Corrector descriptions into MetaResult objects #724

Merged
merged 50 commits into from Mar 14, 2023
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
03aee76
Initial attempt.
tsalo Jun 20, 2022
f76bc50
Use the MetaResult instead.
tsalo Jun 20, 2022
8abf695
Close some gaps.
tsalo Jun 20, 2022
50629c4
Fix things up some more.
tsalo Jun 20, 2022
71994ac
Add IBMA descriptions.
tsalo Jun 20, 2022
eea1444
Fix?
tsalo Jun 21, 2022
340dead
Add NiMARE version/ref to Estimator descriptions.
tsalo Jun 21, 2022
b9b6e70
Move BibTeX file into NiMARE resources.
tsalo Jun 21, 2022
9289fd7
Remove duplicate text.
tsalo Jul 8, 2022
5f0bf49
Add kernel transformer descriptions.
tsalo Jul 8, 2022
f014fd7
Add some tests.
tsalo Jul 8, 2022
3b97530
Use _generate_description method.
tsalo Jul 8, 2022
e17d6cb
Draft functions to extract references from bib file.
tsalo Jul 11, 2022
e6d296d
Get things working.
tsalo Jul 15, 2022
1b067a4
Update examples.
tsalo Jul 15, 2022
0eff163
Fix example.
tsalo Jul 15, 2022
50c1493
Update utils.py
tsalo Jul 15, 2022
6dba8a7
More stuff.
tsalo Jul 15, 2022
043895c
Fix.
tsalo Jul 15, 2022
e3177dd
Update ale.py
tsalo Jul 15, 2022
9613186
Reorg refs.
tsalo Jul 15, 2022
0f091b0
Expand docstring.
tsalo Jul 15, 2022
a3c0f7b
More improvements.
tsalo Jul 15, 2022
4a3c8b8
Fix some more stuff.
tsalo Jul 15, 2022
9ba367d
Remove ALE workflow boilerplate.
tsalo Jul 16, 2022
eb829cd
Update ale.py
tsalo Jul 16, 2022
b28f59a
Merge remote-tracking branch 'upstream/main' into class-boilerplates
tsalo Aug 2, 2022
a02c96f
Fix merge issues.
tsalo Aug 2, 2022
59db0a2
Fix bugs.
tsalo Aug 2, 2022
cb77696
Update base.py
tsalo Aug 2, 2022
8133c44
Fix the stuff.
tsalo Aug 4, 2022
359602f
Replace mentions of MetaEstimator with Estimator.
tsalo Aug 6, 2022
e73944f
Remove unimplemented SCALE workflow.
tsalo Aug 6, 2022
7716f46
Remove the conperm workflow.
tsalo Aug 6, 2022
1afd17b
Remove the dropped workflows from the CLI.
tsalo Aug 6, 2022
c152450
Use new boilerplates in remaining workflows.
tsalo Aug 6, 2022
7eb3264
Update utils.py
tsalo Aug 6, 2022
93aed9d
Update __init__.py
tsalo Aug 6, 2022
4e45b48
Update api.rst
tsalo Aug 6, 2022
022fffe
Merge remote-tracking branch 'upstream/main' into class-boilerplates
tsalo Aug 6, 2022
744dfd5
Merge remote-tracking branch 'upstream/main' into class-boilerplates
tsalo Feb 2, 2023
4d3dd2f
Remove kernel from kernel name in boilerplate.
tsalo Feb 2, 2023
e38e185
Coerce n_subjects to int in boilerplates.
tsalo Feb 2, 2023
33157f4
Give set attributes trailing underscore.
tsalo Feb 2, 2023
f5966f7
Update docstrings too.
tsalo Feb 2, 2023
907c5e4
Le sigh
tsalo Feb 2, 2023
5d48cc9
merge
jdkent Mar 14, 2023
68668a5
fix conflict
jdkent Mar 14, 2023
c86e68c
display boilerplates in examples
jdkent Mar 14, 2023
780cab3
print fuller information in notebooks
jdkent Mar 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/conf.py
Expand Up @@ -209,7 +209,7 @@
# -----------------------------------------------------------------------------
# sphinxcontrib-bibtex
# -----------------------------------------------------------------------------
bibtex_bibfiles = ["./references.bib"]
bibtex_bibfiles = ["../nimare/resources/references.bib"]
bibtex_style = "unsrt"
bibtex_reference_style = "author_year"
bibtex_footbibliography_header = ""
Expand Down
10 changes: 8 additions & 2 deletions examples/02_meta-analyses/04_plot_estimators.py
Expand Up @@ -32,6 +32,8 @@
###############################################################################
# The Estimator
# -----------------------------------------------------------------------------
from pprint import pprint

from nimare.meta.cbma.ale import ALE

# First, the Estimator should be initialized with any parameters.
Expand All @@ -40,6 +42,12 @@
# Then, the ``fit`` method takes in the Dataset and produces a MetaResult.
results = meta.fit(dset)

# You can also look at the description of the Estimator.
print("Description:")
pprint(results.description)
print("References:")
pprint(results.bibtex)

###############################################################################
# Coordinate-based Estimators allow you to provide a specific KernelTransformer
# -----------------------------------------------------------------------------
Expand Down Expand Up @@ -111,8 +119,6 @@
###############################################################################
# The null distributions are stored within the Estimators
# `````````````````````````````````````````````````````````````````````````````
from pprint import pprint

pprint(meta.null_distributions_)

###############################################################################
Expand Down
8 changes: 8 additions & 0 deletions examples/02_meta-analyses/05_plot_correctors.py
Expand Up @@ -8,6 +8,7 @@

Here we take a look at multiple comparisons correction in meta-analyses.
"""
from pprint import pprint

import matplotlib.pyplot as plt
import seaborn as sns
Expand Down Expand Up @@ -87,6 +88,13 @@

fig.tight_layout()

###############################################################################
# You can also look at the description of the Corrector.
print("Description:")
pprint(cres.description)
print("References:")
pprint(cres.bibtex)

###############################################################################
# Show corrected results
# =============================================================================
Expand Down
11 changes: 8 additions & 3 deletions nimare/base.py
Expand Up @@ -45,7 +45,7 @@ def __repr__(self):
if v.default is not inspect.Parameter.empty
}

# Eliminate any sub-parameters (e.g., parameters for a MetaEstimator's KernelTransformer),
# Eliminate any sub-parameters (e.g., parameters for a Estimator's KernelTransformer),
# as well as default values
params = self.get_params()
params = {k: v for k, v in params.items() if "__" not in k}
Expand Down Expand Up @@ -278,6 +278,11 @@ def _collect_inputs(self, dataset, drop_invalid=True):
)
self.inputs_[k] = v

@abstractmethod
def _generate_description(self):
"""Generate a text description of the Estimator."""
pass

@abstractmethod
def _preprocess_input(self, dataset):
"""Perform any additional preprocessing steps on data in self.inputs_.
Expand Down Expand Up @@ -328,11 +333,11 @@ def fit(self, dataset, drop_invalid=True):
"""
self._collect_inputs(dataset, drop_invalid=drop_invalid)
self._preprocess_input(dataset)
maps, tables = self._fit(dataset)
maps, tables, description = self._fit(dataset)

if hasattr(self, "masker") and self.masker is not None:
masker = self.masker
else:
masker = dataset.masker

return MetaResult(self, mask=masker, maps=maps, tables=tables)
return MetaResult(self, mask=masker, maps=maps, tables=tables, description=description)
56 changes: 44 additions & 12 deletions nimare/correct.py
Expand Up @@ -153,7 +153,7 @@ def transform(self, result):
Returns
-------
result : :obj:`~nimare.results.MetaResult`
MetaResult with new corrected maps and tables added.
MetaResult with new corrected maps, tables, and description added.
"""
correction_method = f"correct_{self._correction_method}_{self.method}"

Expand All @@ -163,7 +163,7 @@ def transform(self, result):
# Also operate on a copy of the estimator
est = result.estimator

# If a correction method with the same name exists in the current MetaEstimator, use it.
# If a correction method with the same name exists in the current Estimator, use it.
# Otherwise fall back on _transform, and the Corrector methods.
# In case a method is present in both the Estimator and the Corrector, the Estimator's
# implementation takes precedence.
Expand All @@ -172,14 +172,17 @@ def transform(self, result):
"Using correction method implemented in Estimator: "
f"{est.__class__.__module__}.{est.__class__.__name__}.{correction_method}."
)
corr_maps, corr_tables = getattr(est, correction_method)(result, **self.parameters)
corr_maps, corr_tables, description = getattr(est, correction_method)(
result, **self.parameters
)
else:
self._collect_inputs(result)
corr_maps, corr_tables = self._transform(result, method=correction_method)
corr_maps, corr_tables, description = self._transform(result, method=correction_method)

# Update corrected map names and add them to maps dict
corr_maps = {(k + self._name_suffix): v for k, v in corr_maps.items()}
result.maps.update(corr_maps)
result.description += " " + description

corr_tables = {(k + self._name_suffix): v for k, v in corr_tables.items()}
result.tables.update(corr_tables)
Expand Down Expand Up @@ -214,6 +217,8 @@ def _transform(self, result, method):
corr_tables : :obj:`dict`
An empty dictionary meant to contain any tables (pandas DataFrames) produced by the
correction procedure.
description : :obj:`str`
A description of the correction procedure.
"""
p = result.maps["p"]

Expand All @@ -223,15 +228,15 @@ def _transform(self, result, method):
p_no_nans = p[nonnan_mask]

# Call the correction method
p_corr_no_nans, tables = getattr(self, method)(p_no_nans)
p_corr_no_nans, tables, description = getattr(self, method)(p_no_nans)

# Unmask the corrected p values based on the NaN mask
p_corr[nonnan_mask] = p_corr_no_nans

# Create a dictionary of the corrected results
corr_maps = {"p": p_corr}
self._generate_secondary_maps(result, corr_maps)
return corr_maps, tables
return corr_maps, tables, description


class FWECorrector(Corrector):
Expand Down Expand Up @@ -284,8 +289,13 @@ def correct_fwe_bonferroni(self, p):

Returns
-------
:obj:`numpy.ndarray`
p_corr : :obj:`numpy.ndarray`
A 1D array of adjusted p values.
tables : :obj:`dict`
A dictionary of DataFrames with summary information from the correction.
This correction method does not produce any tables, so it will be an empty dict.
description : :obj:`str`
A description of the correction procedure.

References
----------
Expand All @@ -295,7 +305,11 @@ def correct_fwe_bonferroni(self, p):
--------
nimare.stats.bonferroni
"""
return bonferroni(p), {}
description = (
"Family-wise error rate correction was performed with the Bonferroni correction "
"procedure \\citep{bonferroni1936teoria,shaffer1995multiple}."
)
return bonferroni(p), {}, description


class FDRCorrector(Corrector):
Expand Down Expand Up @@ -352,8 +366,13 @@ def correct_fdr_indep(self, p):

Returns
-------
:obj:`numpy.ndarray`
p_corr : :obj:`numpy.ndarray`
A 1D array of adjusted p values.
tables : :obj:`dict`
A dictionary of DataFrames with summary information from the correction.
This correction method does not produce any tables, so it will be an empty dict.
description : :obj:`str`
A description of the correction procedure.

References
----------
Expand All @@ -363,7 +382,11 @@ def correct_fdr_indep(self, p):
--------
pymare.stats.fdr
"""
return fdr(p, q=self.alpha, method="bh"), {}
description = (
"False discovery rate correction was performed with the Benjamini-Hochberg procedure "
"\\citep{benjamini1995controlling}."
)
return fdr(p, q=self.alpha, method="bh"), {}, description

def correct_fdr_negcorr(self, p):
"""Perform Benjamini-Yekutieli FDR correction.
Expand All @@ -384,8 +407,13 @@ def correct_fdr_negcorr(self, p):

Returns
-------
:obj:`numpy.ndarray`
p_corr : :obj:`numpy.ndarray`
A 1D array of adjusted p values.
tables : :obj:`dict`
A dictionary of DataFrames with summary information from the correction.
This correction method does not produce any tables, so it will be an empty dict.
description : :obj:`str`
A description of the correction procedure.

Notes
-----
Expand All @@ -403,4 +431,8 @@ def correct_fdr_negcorr(self, p):
--------
pymare.stats.fdr
"""
return fdr(p, q=self.alpha, method="by"), {}
description = (
"False discovery rate correction was performed with the Benjamini-Yekutieli procedure "
"\\citep{benjamini2001control}."
)
return fdr(p, q=self.alpha, method="by"), {}, description
2 changes: 1 addition & 1 deletion nimare/dataset.py
Expand Up @@ -134,7 +134,7 @@ def __repr__(self):
if v.default is not inspect.Parameter.empty
}

# Eliminate any sub-parameters (e.g., parameters for a MetaEstimator's KernelTransformer),
# Eliminate any sub-parameters (e.g., parameters for a Estimator's KernelTransformer),
# as well as default values
params = self.get_params()
params = {k: v for k, v in params.items() if "__" not in k}
Expand Down