Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Indices - add species richness estimators #420

Closed
GoogleCodeExporter opened this issue Mar 27, 2015 · 28 comments
Closed

Indices - add species richness estimators #420

GoogleCodeExporter opened this issue Mar 27, 2015 · 28 comments
Assignees
@shawnlaffan
Labels
auto-migrated enhancement
Milestone
Release_2.0

Comments

@GoogleCodeExporter
Copy link 

It would be useful to add some of the richness estimators as indices.  See the 
EstimateS software for more details.  
http://viceroy.eeb.uconn.edu/estimates/EstimateSPages/EstSUsersGuide/EstimateSUs
ersGuide.htm#Appendices

Original issue reported on code.google.com by shawnlaffan on 19 Nov 2013 at 5:18
@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2012.

Add branch to work on this issue.

Original comment by shawnlaffan on 19 Nov 2013 at 5:20

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2014.

Add initial test code.  Use one of the example files from EstimateS so we can 
be more sure of the answers.  This requires the addition of an extra basedata 
import option to TestHelpers.
Test results are place-holders at the moment (copied across from the lists and 
counts test file).

Original comment by shawnlaffan on 19 Nov 2013 at 6:02

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2015.

Add Chao1 and Chao2 code.  
Cannot replicate results from EstimateS, suspect it is running resampling 
regardless of settings (or my settings are incorrect).  Need to generate 
expectations analytically.

Original comment by shawnlaffan on 20 Nov 2013 at 2:46

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2016.

Update expected values based on worked calculations for the given data.

Original comment by shawnlaffan on 20 Nov 2013 at 3:12

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2017.

More closely follow EstimateS in the conditions.

Original comment by shawnlaffan on 20 Nov 2013 at 4:01

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2019.

Add confidence intervals.

Original comment by shawnlaffan on 20 Nov 2013 at 9:01

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2020.

Refactor the confidence intervals into a sub, add more cases.  
Still incomplete as we need the variances to be calculated for more cases.

Original comment by shawnlaffan on 20 Nov 2013 at 11:24

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2021.

Add more variance estimates for chao1 - need testing.

Original comment by shawnlaffan on 21 Nov 2013 at 12:15

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2022.

Add more variance estimates for chao2 - need testing.  #11 is also actually 
#10, so is for later extraction if needed.

Original comment by shawnlaffan on 21 Nov 2013 at 1:22

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2023.

Add metadata to the result sets.  This allows users to see which formulae are 
used for the calculations.  
Also set confidence intervals to undef when we have only uniques. In the common 
case it has only one site, so there is no variability to work with.

Original comment by shawnlaffan on 21 Nov 2013 at 5:30

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2024.

Initial, untested, code for the ACE estimator.

Original comment by shawnlaffan on 21 Nov 2013 at 6:14

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2025.

Add ICE code as a special case of the ACE code.  
Add test scores.
Add capacity in run_indices_test1 to specify expected results.

Original comment by shawnlaffan on 21 Nov 2013 at 10:37

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2141.

Avoid divide by zero

Original comment by shawnlaffan on 14 Feb 2014 at 2:41

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2364.

Trap negative variances and set them to zero.

Original comment by shawnlaffan on 23 Apr 2014 at 5:31

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2365.

Merge trunk revisions across to the branch.

Original comment by shawnlaffan on 23 Apr 2014 at 6:06

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2525.

Merge all recent trunk revisions across to the branch.

Original comment by shawnlaffan on 24 Jun 2014 at 11:14

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2526.

Make the RichnessEstimation indices metadata use objects to conform with 
changes under issue 473.

Original comment by shawnlaffan on 24 Jun 2014 at 11:22

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2527.

Fix a precedence issue in the basedata generation section of the indices test 
helpers.

Original comment by shawnlaffan on 24 Jun 2014 at 11:46

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

Previous commit also reverted the incidence based indices back to using 
calc_abc3 instead of calc_abc2.

Original comment by shawnlaffan on 24 Jun 2014 at 11:47

  • Changed state: Started
  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2528.

Properly fix the precedence issue in the basedata generation section of the 
indices test helpers.

Original comment by shawnlaffan on 25 Jun 2014 at 1:34

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2529.

Switch back to using calc_abc2 for the chao2 and ICE calculations.

Original comment by shawnlaffan on 26 Jun 2014 at 11:20

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2772.

Merge recent trunk revisions across to the branch.

Original comment by shawnlaffan on 6 Sep 2014 at 6:15

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r2901.

Merge recent trunk revisions across.

Original comment by shawnlaffan on 9 Oct 2014 at 10:09

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r3009.

Merge recent trunk revisions across to the branch

Original comment by shawnlaffan on 4 Nov 2014 at 9:59

  • Added labels: ****
  • Removed labels: ****

@GoogleCodeExporter
Copy link
Author 

This issue was updated by revision r3098.

Merge recent trunk revisions across to the branch

Original comment by shawnlaffan on 3 Dec 2014 at 11:57

  • Added labels: ****
  • Removed labels: ****

@shawnlaffan shawnlaffan self-assigned this Apr 9, 2015
shawnlaffan added a commit that referenced this issue Apr 15, 2015
@shawnlaffan
Update issue #420. Update version numbers in branch to match master. …
63287b9 
…Maybe this will avoid some of the merge conflicts.

Signed-off-by: Shawn Laffan <shawnlaffan@gmail.com>
shawnlaffan added a commit that referenced this issue Apr 10, 2016
@shawnlaffan
Update Chao1 test values to use those generated from the SpadeR packa…
e8009b4 
…ge for R.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 10, 2016
@shawnlaffan
Add Chao2 standard errors
386ac2d 
Updates issue #420
@shawnlaffan
Copy link
Owner

This issue is now close to completed.
Need to decide whether or not to include the ACE and ICE variance estimators. The formulae can be taken from SpadeR.

shawnlaffan added a commit that referenced this issue Apr 20, 2016
@shawnlaffan
Add initial code for ICE variance calculations.
e4a4f27 
Adapted directly from SpadeR, and completely untested at this stage.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 25, 2016
@shawnlaffan
Add more tests for Chao2 calcs.
0e2fb57 
We check for cases (Q1==0 && Q2>0), (Q1 > 0 && Q2==0) and (Q1==0 && Q2==0).

Flush out an error in the confidence interval logic by doing so.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 25, 2016
@shawnlaffan
Add more tests for Chao1 calcs.
d8c9731 
As with the Chao2 calcs in 0e2fb57 we now check for cases (F1==0 && F2>0), (F1 > 0 && F2==0) and (F1==0 && F2==0).

Flushes out some errors with the variance calcs in the process.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 28, 2016
@shawnlaffan
Use a higher precision estimate of z for the confidence interval
7be885e 
Update the test expectations as appropriate.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 28, 2016
@shawnlaffan
Factor out some common variables.
ccaf111 
Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 28, 2016
@shawnlaffan
Initial work on the ACE variance and CI calculations
34bdfd2 
Not yet functional.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 28, 2016
@shawnlaffan
Add confidence intervals for ACE
54370fc 
Needs more testing to shake out the various permutations.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 28, 2016
@shawnlaffan
Make calc_ace use Chao1 if there are no rares or no singletons.
091c64d 
Updates issue #420

Still need to update the ICE code and conditions for the same cases.
shawnlaffan added a commit that referenced this issue Apr 30, 2016
@shawnlaffan
Rename CHAO1, CHAO2, ICE_SCORE and ACE_SCORE indices to use _ESTIMATE…
f7c6e7f 
… suffix for consistency

This enables some code simplification.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 30, 2016
@shawnlaffan
calc_ice passes tests for simplest case.
97b1215 
Next is to test for Q1==0 an Q2==0.

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 30, 2016
@shawnlaffan
Handle more edge cases for ICE and ACE
a26c5ca 
Cases include all singletons, not singletons, single cells.

Still need to add a test for richness = 0

Updates issue #420
shawnlaffan added a commit that referenced this issue Apr 30, 2016
@shawnlaffan
calc_ice now uses non-empty groups
c68f533 
This entailed adding a new method to calculate the number of non-empty groups, and associated tests.

updates issue #420
shawnlaffan added a commit that referenced this issue Apr 30, 2016
@shawnlaffan
Merge remote-tracking branch 'remotes/origin/issue_420_richness_estim…
086cae6 
…ators'

Updates issue #420

# Conflicts were:
#	lib/Biodiverse/GUI/BasedataImport.pm
#	lib/Biodiverse/GUI/Exclusions.pm
#	lib/Biodiverse/GUI/Export.pm
#	lib/Biodiverse/GUI/GUIManager.pm
#	lib/Biodiverse/GUI/MatrixImport.pm
#	lib/Biodiverse/GUI/Overlays.pm
#	lib/Biodiverse/GUI/ParametersTable.pm
#	lib/Biodiverse/GUI/PhylogenyImport.pm
#	lib/Biodiverse/GUI/Popup.pm
#	lib/Biodiverse/GUI/ProgressDialog.pm
#	lib/Biodiverse/GUI/Tabs/Clustering.pm
#	lib/Biodiverse/GUI/Tabs/Labels.pm
#	lib/Biodiverse/GUI/Tabs/Outputs.pm
#	lib/Biodiverse/GUI/Tabs/Randomise.pm
#	lib/Biodiverse/GUI/Tabs/Spatial.pm
#	lib/Biodiverse/GUI/Tabs/SpatialMatrix.pm
#	lib/Biodiverse/GUI/YesNoCancel.pm
#	lib/Biodiverse/Indices.pm
#	lib/Biodiverse/Metadata/Indices.pm
#	lib/Biodiverse/SpatialConditions.pm
#	t/lib/Biodiverse/TestHelpers.pm
shawnlaffan added a commit that referenced this issue Apr 30, 2016
@shawnlaffan
ACE and ICE flag when they fell back to Chao estimators
3f0269f 
Updates issue #420
@shawnlaffan
Copy link
Owner

Now we really are close to completion.

SpadeR also gives coverage estimates and frequencies of rare species, so we could also add those if needed.

There are also the iChao estimators, but those could be added under their own issue.
http://onlinelibrary.wiley.com/doi/10.1111/biom.12200/abstract

@shawnlaffan shawnlaffan reopened this Apr 30, 2016
@shawnlaffan shawnlaffan added this to the Release_2.0 milestone Apr 30, 2016
@shawnlaffan
Copy link
Owner

Mark as closed.

The additional points in the last comment can be dealt with under new issues.

@shawnlaffan shawnlaffan mentioned this issue May 1, 2016
Open
shawnlaffan added a commit that referenced this issue May 1, 2016
@shawnlaffan
Avoid some divide by zero errors.
5575f87 
This also make sthe code consistent with EstimateS in terms of when it switches from ACE/ICE to Chao1/2.

Updates issue #420 (even though it is closed now)
shawnlaffan added a commit that referenced this issue May 3, 2016
@shawnlaffan
Add ACE test where all rares are singletons
20a06b5 
Updates issue #420
@shawnlaffan shawnlaffan mentioned this issue May 10, 2016
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shawnlaffan shawnlaffan

Labels
auto-migrated enhancement
Projects
None yet
Milestone
Release_2.0
Development

No branches or pull requests
2 participants
@shawnlaffan @GoogleCodeExporter