Skip to content

Kernel/helper fixes and a few new features#160

Merged
westonslaughter merged 5 commits intoMacroSHEDS:masterfrom
vlahm:master
Oct 14, 2022
Merged

Kernel/helper fixes and a few new features#160
westonslaughter merged 5 commits intoMacroSHEDS:masterfrom
vlahm:master

Conversation

@vlahm
Copy link
Copy Markdown
Collaborator

@vlahm vlahm commented Oct 14, 2022

@westonslaughter, take a look at the changes to any of the Niwot stream_chemistry munge kernels (process_1_108 is a good example). This domain flags measurements that are below detection limit by inserting e.g. "<0.02" into the data columns. I noticed Spencer was referring to every instance of "<0.XX" individually, which is not future proof, so now there's a wildcard that represents any numeric value, including a possible decimal point.

There were two other issues in these kernels. One inconsequential but confusing to potential copy/pasters: variable_flags_dirty = 'dirty') wasn't doing anything here.

More importantly, I forgot to insert d <- qc_hdetlim_and_uncert(d, prodname_ms = prodname_ms) into these kernels! So BDL datapoints weren't being replaced by their half-detection-limit values.

Nvm re: qc_hdetlim_and_uncert! Forgot I added this to the munge engines, so it's not needed inside kernels in most cases. Though note it is required in kernels that write their output, rather than returning it. ftgrep .R qc_hdetlim to see where it's used.

Copy link
Copy Markdown
Contributor

@westonslaughter westonslaughter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, thanks for the detailed heads up on your changes! and for all of your close review of the pkernels lately -- I believe I understand the write/return distinction, and certainly do practically, I will look out for that especially as the data team approaches munging.

@westonslaughter westonslaughter merged commit e2604ab into MacroSHEDS:master Oct 14, 2022
vlahm pushed a commit to vlahm/macrosheds_data_processing that referenced this pull request Jul 18, 2023
Kernel/helper fixes and a few new features
vlahm pushed a commit to vlahm/macrosheds_data_processing that referenced this pull request Dec 5, 2023
Kernel/helper fixes and a few new features
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants