feat(ingestion): wire the loader end to end - #856
Merged
Conversation
san_acacia_observations joins the pieces that existed separately: reconcile the vendor point to a well, choose the deployment its transducer hangs from, ask the database where that series got to, fetch forward, map, upsert, and extend the QC block. The seeding half of 3.2 turned out to be nothing. All 38 wells already have deployments, the parameter exists as `groundwater level` in feet -- the unit the adapter emits -- and existing observations already use it. So the series is chosen rather than created. Choosing it needs a rule, because a well carries several open deployments: a deployment is equipment, not a measured property. SO-0140 has a DiverLink, a Pressure Transducer and a Diver Cable, and only the transducer produces a water level. Picking any other would attribute a reading to a cable. That resolves cleanly for 35 of the 38 wells. Two have two open transducers and one has none; those are skipped and reported. Taking the lower id would be a silent guess about equipment, and a removed transducer is not used as a fallback -- writing current data against retired kit looks like success while being wrong. A well that cannot be resolved costs that well's readings for the run, not the other thirty-seven's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 78.98% total — gate is 75%. Coverage for the Python files changed in this PR
|
Contributor
Coverage✅ 78.08% total — gate is 75%. Coverage for the Python files changed in this PR
|
Diver-HUB serves nothing before late 2024. Probing six points put their earliest reading at 2024-10-08 and 2024-11-10, matching the deployments on these wells being installed 2024-11-25 -- the vendor project was populated then. INITIAL_START was 2015-01-01, chosen before anyone knew what the vendor retains. At a 365-day span that made a first run walk twelve windows per well, ten of them guaranteed empty, against an endpoint that answers 500 when pushed. The floor is now 2024-01-01: three windows per well, 228 requests across the thirty-eight rather than 912. It sits nine months below the earliest observed reading rather than at it, because only six of thirty-eight points were probed and a well with slightly earlier data should not be silently truncated. This also records why the datum comparison could not be completed. Ocotillo's AMPAPI data for these wells ends August 2022 and the vendor starts late 2024, so there are no matching timestamps to compare -- attempted on SO-0125 and SO-0245, zero vendor rows at every reference. The two sources never overlap, which means no datum mixing can occur on a normal run, and roughly twenty-seven months are missing from the record and cannot be recovered from this source. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
san_acacia_observationsjoins the pieces that until now existed separately:reconcile the vendor point to a well → choose the deployment its transducer
hangs from → ask the database where that series got to → fetch forward → map →
upsert → extend the QC block.
The seeding half of 3.2 turned out to be nothing
The plan expected to create wells, a parameter, a sensor and deployments.
Checked against staging — all of it already exists:
groundwater level,default_unit = ft— whichis exactly what the adapter emits, so the centimetre conversion lands in the
right unit
So the series is chosen, not created.
Choosing it needs a rule
A well carries several open deployments because a deployment is a piece of
equipment, not a measured property. SO-0140 has three:
DiverLinkPressure TransducerDiver CableBarometerappears on other wells. Only the transducer produces a water level —picking any other would attribute a reading to a cable.
It does not resolve everywhere, and that is reported
The three are skipped and counted in asset metadata, not guessed at. Taking the
lower id would be a silent decision about equipment records.
A removed transducer is not used as a fallback: writing current readings
against retired kit would look like success while being wrong.
An unresolvable well costs that well's readings for the run, not the other
thirty-seven's — same failure isolation as the rest of the pipeline.
Before running it
Two things from earlier findings still apply, and neither is code:
2022 under an unverified datum. A normal run starts from each series'
watermark so it will not overlap them — but confirm before any backfill
covering 2016–2022.
ngwmn_helper.pyfiltersrelease_status == "public", and this loadswith
release_status="public". Whether San Acacia should reach NGWMN isunchecked.
102 tests pass.
🤖 Generated with Claude Code