Set data <MDL to MDL?

[HeatherGranger]

As the EMS database (and likely ECCC database) has a lot of data below MDL, setting all that data to zero (as wqbc is currently set-up) looks weird on a plot and in presenting it in a table. If the data was set to the MDL, it could be presented in a table with the result letter column, illustrating the 'minimum' value was less than the MDL, as opposed to showing up as zero.

Possible for the user to do this with tidy_ems_data ? I can't tell from the documentation.

[ateucher]

This is definitely doable, but with EMS data I think we were having problems with some of the detection limits not being in the same units as the value? Or was that resolved?

[HeatherGranger]

It's resolved:
I found Rob's email from March:

The way to determine MDL is to find any record with a < sign in the RESULT LETTER. The result will be the MDL. Use the < sign as the identifier to find results below detection; then treat these results as you wish (set to zero/half MDL/etc whatever convention is being used.)

The RESULT LETTER of “<” has always been used to indicate “no detect”.

I still struggle to wrap my brain around the fact that converting the result to standard units doesn't mess up it's association with the MDL, but Rob explained this to me in March and it made sense so I'd say we can move forward with this.

[ateucher]

Ok, sounds good! 🚀

[ateucher]

I haven't merged it yet, but I have this working:

library(wqbc)
library(rems)
library(ggplot2)

some_data <- get_ems_data(ask = FALSE)
#> Fetching data from cache...

filtered_data <- filter_ems_data(some_data, emsid = "E307225", 
                                 from_date = "2017-01-01", 
                                 to_date = "2017-04-30", 
                                 param_code = "1106")

## The January reading is a non-detect
filtered_data[, c("COLLECTION_START", "PARAMETER", "RESULT", "RESULT_LETTER")]
#> # A tibble: 4 x 4
#>      COLLECTION_START          PARAMETER RESULT RESULT_LETTER
#>                <dttm>              <chr>  <dbl>         <chr>
#> 1 2017-01-03 13:10:00 Fluoride Dissolved  0.020             <
#> 2 2017-02-01 12:50:00 Fluoride Dissolved  0.022          <NA>
#> 3 2017-03-08 13:40:00 Fluoride Dissolved  0.020          <NA>
#> 4 2017-04-05 15:40:00 Fluoride Dissolved  0.023          <NA>

## Set non-detects to zero
tidied_data <- tidy_ems_data(filtered_data, mdl_action = "zero")

ggplot(tidied_data, aes(x = DateTime, y = Value, colour = ResultLetter)) +
  geom_point()

## Set non-detects to half MDL
tidied_data <- tidy_ems_data(filtered_data, mdl_action = "half")

ggplot(tidied_data, aes(x = DateTime, y = Value, colour = ResultLetter)) +
  geom_point()

## Set non-detects to MDL
tidied_data <- tidy_ems_data(filtered_data, mdl_action = "mdl")

ggplot(tidied_data, aes(x = DateTime, y = Value, colour = ResultLetter)) +
  geom_point()

There is also the function set_non_detects() so you can use it directly on any data frame
without going through tidy_ems_data():

filtered_data <- mutate(filtered_data, 
                        censored_data_zero = set_non_detects(RESULT, 
                                                             mdl_flag = RESULT_LETTER, 
                                                             mdl_action = "zero"), 
                        censored_data_mdl = set_non_detects(RESULT, 
                                                            mdl_flag = RESULT_LETTER, 
                                                            mdl_action = "mdl"), 
                        censored_data_half_mdl = set_non_detects(RESULT, 
                                                                 mdl_flag = RESULT_LETTER, 
                                                                 mdl_action = "half"))

filtered_data %>% 
  select(COLLECTION_START, PARAMETER, RESULT, RESULT_LETTER, starts_with("censored_data")) %>% 
  glimpse()

#> Observations: 4
#> Variables: 7
#> $ COLLECTION_START       <dttm> 2017-01-03 13:10:00, 2017-02-01 12:50:...
#> $ PARAMETER              <chr> "Fluoride Dissolved", "Fluoride Dissolv...
#> $ RESULT                 <dbl> 0.020, 0.022, 0.020, 0.023
#> $ RESULT_LETTER          <chr> "<", NA, NA, NA
#> $ censored_data_zero     <dbl> 0.000, 0.022, 0.020, 0.023
#> $ censored_data_mdl      <dbl> 0.020, 0.022, 0.020, 0.023
#> $ censored_data_half_mdl <dbl> 0.010, 0.022, 0.020, 0.023

[ateucher]

@HeatherGranger I updated the examples to show the use of set_non_detects() as a standalone, without going through tidy_ems_data()

[HeatherGranger]

@ateucher Looks great. I can't test/use until you merge into the master right?

[ateucher]

You can install from the mdl branch with: devtools::install_github("bcgov/rems", ref = "mdl") to test it out. That would be great if you don't mind!

[HeatherGranger]

I get this error when I try to install:

devtools::install_github("bcgov/rems", ref = "mdl")
Downloading GitHub repo bcgov/rems@mdl
from URL https://api.github.com/repos/bcgov/rems/zipball/mdl
Installation failed: 404: Not Found
(404)

Does it work for you?

I can download it, but not sure how to test by integrating my script with the updated tidy function.

[HeatherGranger]

Hey @ateucher I checked out the repo list and the mdl branch isn't in there..did you push?

aka @stephhazlitt mentioned that might be the problem! 🥇

[ateucher]

Ack sorry it's wqbc not rems!

devtools::install_github("bcgov/wqbc", ref = "mdl")