Enhance interpolation documentation and update mocked queries#989
Merged
Enhance interpolation documentation and update mocked queries#989
Conversation
- Added detailed warning in `interpolate.py` regarding time gradient buffering for accurate boundary interpolation. - Updated mocked SQL queries in `sdk_test_objects.py` to include a ±5 minute buffer for the requested date range, ensuring accurate interpolation. - Adjusted the structure of the mocked queries to reflect the new logic for handling event times and values. Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
…e return values Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
…ordering Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
…ter and update handling in related functions Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
…e parameter Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
cching95
previously approved these changes
Apr 16, 2026
… sparse data Signed-off-by: Amber-Rigg <amber.l.rigg25@gmail.com>
cching95
approved these changes
Apr 16, 2026
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Summary: Refactor Interpolation Query Architecture - Time Gradient Buffering
Overview
Refactored the interpolation query builder to implement time gradient buffering for accurate boundary-aware linear interpolation. This change improves accuracy at query boundaries by automatically expanding the data range based on the time interval unit.
Key Changes
New Functions Added:
_build_raw_query_for_interpolate() - Raw query with time gradient buffering
Automatically expands query range by time-based buffer (±5 min for minutes, ±60 sec for seconds, ±1 hour for hours, ±1 day for days)
Uses timestampadd() instead of plain to_timestamp() for start/end dates
_build_interpolate_intervals_cte() - Generates regular time intervals using sequence()
_build_interpolate_fill_intervals_cte() - Unions raw data with intervals using SortKey (0 for real, 1 for desired)
_build_interpolate_calculate_cte() - Applies LAG/LEAD window functions to capture previous/next real data points
_build_interpolate_interpolate_cte() - Performs linear interpolation using unix_millis() for precision
_build_interpolate_uom_cte() - Joins interpolated data with metadata for Unit of Measure
_build_interpolate_query() - Orchestrates all CTEs together
Updates to _interpolation_query():
Changed from _build_raw_query() to _build_raw_query_for_interpolate()
Removed old resample/fill_intervals approach
Simplified query structure: raw → intervals → fill_intervals → interpolate_calculate → interpolate → uom
Removed manual UoM query building (now integrated into main flow)
Old Structure:
Used window() function for time grouping
Complex FULL OUTER JOIN between calculate and fill_intervals CTEs
Separate handling of UoM
New Structure:
No window() function
5 dedicated CTEs with clear separation of concerns
Linear interpolation formula: value = prev_value + ((next_value - prev_value) * (current_time - prev_time) / (next_time - prev_time))
Uses unix_millis() for millisecond precision
Time gradient buffering ensures boundary accuracy
UoM integrated into the main CTE chain
Updated all 4 interpolate mock queries:
INTERPOLATE_MOCKED_QUERY
INTERPOLATE_MOCKED_QUERY_CHECK_TAGS
INTERPOLATE_MOCKED_QUERY_PIVOT
INTERPOLATE_MOCKED_QUERY_UOM
Changes:
Replaced window() function with time gradient buffering: timestampadd(minute, -5, ...) to timestampadd(minute, 5, ...)
Updated raw CTE to use 'Good' AS Status instead of actual status column
Restructured CTEs to follow new architecture
Reflected new SortKey-based approach for fill_intervals
Added comprehensive warning section:
Files Modified
[_time_series_query_builder.py](major refactor)
[interpolate.py] (documentation)
[sdk_test_objects.py]((test mocks)