Lifestage samples and their origin

[kellijohnson-NOAA]

@chantelwetzel-noaa can you point me to or quickly summarize the rationale on why we exclude Life Stage samples? I was recently asked why PullCatch.fn led to less catch than getting data from the website directly for a given species and it was at least in part because of YOY samples being excluded.

https://github.com/nwfsc-assess/nwfscSurvey/blob/aed0f490afc6822edbfbdff506d20c4ce35238fd/R/PullCatch.fn.R#L158

[chantelwetzel-noaa]

The initial purpose of this code and package was to pull and provide data that would be used in West Coast groundfish stock assessments. If I remember correctly, these data are excluded because they are non-standard samples (samples collected for a particular research purpose) and may not follow the standard survey protocol. However, as use has expanded beyond the assessment team we should be attempting to provide avenues for people to get particular data for research projects while ensuring that only the proper data are provided for assessment purposes. I think providing additional flexibility for users to extract data that targets their need would be a useful addition to the package and should be added to the to-do list to work on post-assessment season. Another item in this vein would be to create a function to pull data on other biological samples collected outside of the standard survey protocol (e.g., maturity reads).

I am going to leave this issue open for others to comment with ideas.

[kellijohnson-NOAA]

Thanks @chantelwetzel-noaa, the non-standard sample explanation makes perfect sense. And, the following of the package does seem to be increasing; so, it would be good to potentially add features later like you mention.

I am going to add a note in the documentation that special samples are currently excluded but users can get all samples from https://www.webapps.nwfsc.noaa.gov/data/map as a csv file, right @Curt-Whitmire-NOAA? I think all samples are available here, but the metadata that says samples stop in 2017.

[Curt-Whitmire-NOAA]

@kellijohnson-NOAA, that's a good question. I'll have to do some digging about the status of available sample data in the DW. I suspect it's aligned with the available catch data, which currently goes through 2019. Once I confirm, I'll put in a ticket to update the metadata_list values.

[Curt-Whitmire-NOAA]

@kellijohnson-NOAA and @chantelwetzel-noaa , I did a little digging of ETL code on the question about special project samples. They are indeed included in the DW, and when there is any type of standard measurement (length/width, weight, age, maturity), those individuals are given a record_type = 'Standard Survey'.

Also, we have a workflow for updating the metadata information for each trawl table (operation, catch, individual, marine debris) when new records are inserted. This typically happens sometime after each survey cycle, once QA/QC has been completed. The metadata for those four tables is current through the 2021 survey year.

If it's ok with you both, I'll close out this ticket.