NA's leading to incorrect 0 total values in abundance / density tables

[LHMarshall]

    While your fingers are deep inside this issue, I want to alert you to an oddity in precision measures for abundance/density of individuals in total study area results of differing `er_method` argument values when analysing one of Arthur's replicates (that contains no detections in one of his strata):

runs below conducted using this version of Distance
Distance * 1.0.5.9001 2022-08-10 [1] Github (DistanceDevelopment/Distance@b386d14)

er_method=1

df <- read.csv(file='E:/troubleshoot/arthur/ds_camargue.csv',sep= ';')
df$Area <- df$Area / 1000^2
df$Effort <- df$Effort / 1000
df$distance <- df$distance / 1000
convu <- convert_units("kilometer", "kilometer", "square kilometer")
dist.bins <- c(0,60,160,360)/1000
groups <- ds(df, key="hn", cutpoints = dist.bins, er_method = 1)

Summary for individuals

Summary statistics:
     Region      Area CoveredArea  Effort     n  k        ER     se.ER     cv.ER mean.size   se.mean
1  camargue 439.19964   252.49824 350.692 63401 22 180.78827 42.345959 0.2342296  960.6212 185.85088
2      gard 188.34741   108.64656 150.898 19178 15 127.09247 41.824256 0.3290852  767.1200 167.91072
3     rhone  13.23249     7.64712  10.621     0  3   0.00000  0.000000 0.0000000    0.0000   0.00000
4 vigueirat  18.55230    10.95480  15.215   332  2  21.82057  7.069714 0.3239930  110.6667  84.69816
5     Total 659.33184   379.74672 527.426 82911 42 157.19930 30.729351 0.1954802  882.0319 138.48758

Abundance:
      Label    Estimate         se        cv          lcl       ucl         df
1  camargue 195650.6827 49143.6563 0.2511806 119881.07706 319309.69 117.699918
2      gard  58983.2393 21821.1954 0.3699559  28503.72831 122055.00  34.838134
3     rhone      0.0000     0.0000 0.0000000           NA        NA         NA
4 vigueirat    997.5001   798.1894 0.8001898     72.46542  13730.78   2.362988
5     Total 255631.4221     0.0000 0.0000000      0.00000      0.00   0.000000

Density:
      Label  Estimate        se        cv        lcl      ucl         df
1  camargue 445.47095 111.89366 0.2511806 272.953493 727.0263 117.699918
2      gard 313.16194 115.85610 0.3699559 151.335919 648.0312  34.838134
3     rhone   0.00000   0.00000 0.0000000         NA       NA         NA
4 vigueirat  53.76693  43.02375 0.8001898   3.906008 740.1120   2.362988
5     Total 387.71284   0.00000 0.0000000   0.000000   0.0000   0.000000

er_method=2

groups <- ds(df, key="hn", cutpoints = dist.bins, er_method = 2)
Summary for individuals

Summary statistics:
     Region      Area CoveredArea  Effort     n  k        ER     se.ER     cv.ER mean.size   se.mean
1  camargue 439.19964   252.49824 350.692 63401 22 180.78827 42.345959 0.2342296  960.6212 185.85088
2      gard 188.34741   108.64656 150.898 19178 15 127.09247 41.824256 0.3290852  767.1200 167.91072
3     rhone  13.23249     7.64712  10.621     0  3   0.00000  0.000000 0.0000000    0.0000   0.00000
4 vigueirat  18.55230    10.95480  15.215   332  2  21.82057  7.069714 0.3239930  110.6667  84.69816
5     Total 659.33184   379.74672 527.426 82911 42 157.19930 30.729351 0.1954802  882.0319 138.48758

Abundance:
      Label    Estimate         se        cv          lcl      ucl        df
1  camargue 195650.6827 49978.7900 0.2554491 117034.46749 327076.2 29.469070
2      gard  58983.2393 20320.3754 0.3445110  29078.93897 119640.6 16.792117
3     rhone      0.0000     0.0000 0.0000000      0.00000      0.0  0.000000
4 vigueirat    997.5001   338.8011 0.3396502     59.21283  16803.9  1.207644
5     Total 255631.4221 56178.3864 0.2197632 165027.69422 395978.5 44.244253

Density:
      Label  Estimate        se        cv        lcl      ucl        df
1  camargue 445.47095 113.79515 0.2554491 266.472137 744.7096 29.469070
2      gard 313.16194 107.88774 0.3445110 154.389907 635.2125 16.792117
3     rhone   0.00000   0.00000 0.0000000   0.000000   0.0000  0.000000
4 vigueirat  53.76693  18.26195 0.3396502   3.191671 905.7585  1.207644
5     Total 387.71284  85.20503 0.2197632 250.295352 600.5755 44.244253

Originally posted by @erex in #52 (comment)

[LHMarshall]

Will look for cases where NA's are generated and replace them with 0's then check that Total values are correct.

[LHMarshall]

@erex Can you please send me the dataset. I cannot replicate this behaviors with other datasets.

All the following works fine. I took the minke dataset added a new strata with some transects and no detections on them but it always generates 0's in the tables and so doesn't cause issues in the totals. I tried both methods of estimating ER variance. Then checked it wasn't the call via Distance. All works ok on this data.

library(Distance)
data(minke)
temp <- data.frame(Region.Label = rep("Central",5),
                   Area = rep(60000, 5),
                   Sample.Label = 26:30,
                   Effort = c(25,40,50,50,100),
                   distance = rep(NA,5),
                   object = rep(NA,5))

minke.test <- rbind(minke, temp)
# load development version of mrds
#devtools::load_all()

minke.ddf <- ddf(dsmodel = ~cds(key = 'hn', formula=~1),
                 data = minke.test,
                 meta.data = list(width = 1.5))

plot(minke.ddf)

tables <- Distance::unflatten(minke.test)

minke.dht.VF1 <- dht(minke.ddf,
                     tables$region.table,
                     tables$sample.table,
                     tables$obs.table,
                     options = list(varflag = 1))

minke.dht.VF1

minke.dht.VF2 <- dht(minke.ddf,
                     tables$region.table,
                     tables$sample.table,
                     tables$obs.table,
                     options = list(varflag = 2))

minke.dht.VF2


minke.ds <- ds(minke.test, key="hn", er_method = 1)
summary(minke.ds)

[LHMarshall]

Tried binned analysis also and also failed to replicate issue

dist.bins <- seq(0, 1.5, length = 5)
minke.ds <- ds(minke.test, key="hn", 
                cutpoints = dist.bins, er_method = 1)
Starting AIC adjustment term selection.
Fitting half-normal key function
AIC= 222.4
Fitting half-normal key function with cosine(2) adjustments
AIC= 224.393

Half-normal key function selected.
> summary(minke.ds)

Summary for distance analysis 
Number of observations :  88 
Distance range         :  0  -  1.5 

Model : Half-normal key function 
AIC   : 222.3995 

Detection function parameters
Scale coefficient(s):  
              estimate        se
(Intercept) -0.3246629 0.1154827

                       Estimate          SE         CV
Average p             0.5809851  0.05368923 0.09241068
N in covered region 151.4668705 17.46884297 0.11533111

Summary statistics:
   Region   Area CoveredArea  Effort  n  k         ER      se.ER     cv.ER
1 Central  60000      795.00  265.00  0  5 0.00000000 0.00000000 0.0000000
2   North 630582     4075.14 1358.38 49 12 0.03607238 0.01317937 0.3653591
3   South  84734     1453.23  484.41 39 13 0.08051031 0.01809954 0.2248102
4   Total 775316     6323.37 2107.79 88 30 0.03813742 0.01090007 0.2858103

Abundance:
    Label  Estimate        se        cv      lcl       ucl       df
1 Central     0.000    0.0000 0.0000000    0.000     0.000  0.00000
2   North 13050.589 4918.3059 0.3768647 5917.809 28780.562 12.44601
3   South  3914.019  951.3511 0.2430625 2357.291  6498.791 16.33360
4   Total 16964.608 5095.8047 0.3003786 9043.407 31824.058 14.31337

Density:
    Label   Estimate          se        cv         lcl        ucl       df
1 Central 0.00000000 0.000000000 0.0000000 0.000000000 0.00000000  0.00000
2   North 0.02069610 0.007799629 0.3768647 0.009384677 0.04564127 12.44601
3   South 0.04619184 0.011227502 0.2430625 0.027819900 0.07669638 16.33360
4   Total 0.02188089 0.006572552 0.3003786 0.011664156 0.04104656 14.31337

[LHMarshall]

Managed to replicate. The issue arises for both binned and non binned analyses when cluster size is in the dataset. Tables for clusters are fine but issues arise in the tables for individuals. Suspected cause is NaN in Expected Cluster Size table.

data(minke)
#minke <- minke[minke$distance <= 1.5,]
temp <- data.frame(Region.Label = rep("Central",5),
                   Area = rep(60000, 5),
                   Sample.Label = 26:30,
                   Effort = c(25,40,50,50,100),
                   distance = rep(NA,5),
                   object = rep(NA,5))

minke.test <- rbind(minke, temp)
minke.test$size <- rpois(nrow(minke.test), 30)
minke.test$size <- ifelse(is.na(minke.test$distance), NA, minke.test$size)

minke.ds <- ds(minke.test, key="hn", er_method = 1)
summary(minke.ds)
minke.ds <- ds(minke.test, key="hn", er_method = 2)
summary(minke.ds)

er_method = 1

Abundance:
    Label Estimate        se        cv       lcl      ucl       df
1 Central      0.0      0.00 0.0000000        NA       NA       NA
2   North 408202.9 156448.92 0.3832627 182240.11 914340.9 12.02357
3   South 117905.3  28156.25 0.2388039  71424.43 194634.6 15.24992
4   Total 526108.2      0.00 0.0000000      0.00      0.0  0.00000

Density:
    Label  Estimate        se        cv       lcl      ucl       df
1 Central 0.0000000 0.0000000 0.0000000        NA       NA       NA
2   North 0.6473430 0.2481024 0.3832627 0.2890030 1.449995 12.02357
3   South 1.3914761 0.3322899 0.2388039 0.8429253 2.297008 15.24992
4   Total 0.6785726 0.0000000 0.0000000 0.0000000 0.000000  0.00000

Expected cluster size
   Region Expected.S
1 Central        NaN
2   North   30.23529
3   South   30.30769
4   Total   30.25149

er_method = 2

Abundance:
    Label Estimate        se        cv       lcl      ucl       df
1 Central      0.0      0.00 0.0000000      0.00      0.0  0.00000
2   North 408202.9 155196.38 0.3801942 183247.75 909313.1 11.95289
3   South 117905.3  28180.16 0.2390067  71321.75 194914.8 14.89506
4   Total 526108.2 159523.22 0.3032137 277658.53 996871.4 13.32581

Density:
    Label  Estimate        se        cv       lcl      ucl       df
1 Central 0.0000000 0.0000000 0.0000000 0.0000000 0.000000  0.00000
2   North 0.6473430 0.2461161 0.3801942 0.2906010 1.442022 11.95289
3   South 1.3914761 0.3325721 0.2390067 0.8417135 2.300314 14.89506
4   Total 0.6785726 0.2057525 0.3032137 0.3581230 1.285761 13.32581

Expected cluster size
   Region Expected.S se.Expected.S cv.Expected.S
1 Central    0.00000     0.0000000    0.00000000
2   North   30.23529     0.3896273    0.01288650
3   South   30.30769     0.6192760    0.02043296
4   Total   30.25149     0.3336484    0.01102916

[LHMarshall]

Note to self start at dht.R line 370... looks like I have been poking around here before trying to fix something similar in 2012. Adding NaN checks here doesn't work however.