Water Use #195
Merged
Water Use #195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smozely
force-pushed
the
integration/water-use
branch
3 times, most recently
from
5834787
to
a7bf662
Compare
* Improve logging configuration * Update IngestAPI to accept and store allocation messages with usage * Message format tweaks * Make Metered Allocations nullable in API * Add sourceId to ingested water allocation
* Improve logging configuration * Update IngestAPI to accept and store allocation messages with usage * Message format tweaks * Make Metered Allocations nullable in API * Add sourceId to ingested water allocation * Remove unused code * Updated WaterAllocationMessage and Consumer to handle allocations by consent (rather than area) * Fix typo * Ensure Cache manifest for new PlanLimit queries is periodically updated * Fix failing tests by truncating datetime before comparison to avoid precision differences. * Update GeoJsonControllerIntegrationTest to reflect new plan limits controller, and remove redundant test * Changes from PR review
* Manager - Hilltop Crawler Changes * Shuffle files around * martin/aggregate water use (#95) * add water use daily aggregation sql view * optimise sql query performance * Adding Error handling to Observations Consumer --------- Co-authored-by: Martin Peak <martin.peak@gw.govt.nz>
* Manager - Hilltop Crawler Changes * Shuffle files around * Crawler - Updates * Intergration testing * Improve DB naming * Big refactor of test setup * Correct time to wait between quiet polls * Add global error handling around task * Add check to make sure Virtual Measurement matches Measurement Name * Add a bunch to the readme * Add views to aggregate daily allocations and usage by area (#100) * Add views to aggregate consent allocations and usage by day * Start writing tests * Allow easier setup and assertion against test data, start adding more tests * Add tests for various allocation scenarios * Add failing test for aggregating data across different areas * Simplify views * Splitting out the calculation of daily useage by area * Add more assertions for aggregation of different areas * water_allocation.meter should correlate with observation_site.name * Format code * Feedback from PR review --------- Co-authored-by: Steve Mosley <steve@starter4ten.com> * Remove unnesscary `yield` * Adding index on next_fetch_at column * Add some more readme details about the task queue * Adding notes about partition sizes * Improve note about how schedulign tasks works --------- Co-authored-by: Vimal Jobanputra <vim@noodle.io>
* Adding in sample data scripts and way to run from gradle * Update so source id is unique per row * Update to insert observation data for 50 sites * Adding in a simple script to add time to the observed data * Tweak how loading is done to avoid the function * Update the refresh to be relative to current date * Adding in transactions for the scripts
Bumps [msw](https://github.com/mswjs/msw) from 1.2.1 to 1.3.2. - [Release notes](https://github.com/mswjs/msw/releases) - [Changelog](https://github.com/mswjs/msw/blob/main/CHANGELOG.md) - [Commits](mswjs/msw@v1.2.1...v1.3.2) --- updated-dependencies: - dependency-name: msw dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [net.logstash.logback:logstash-logback-encoder](https://github.com/logfellow/logstash-logback-encoder) from 7.3 to 7.4. - [Release notes](https://github.com/logfellow/logstash-logback-encoder/releases) - [Commits](logfellow/logstash-logback-encoder@logstash-logback-encoder-7.3...logstash-logback-encoder-7.4) --- updated-dependencies: - dependency-name: net.logstash.logback:logstash-logback-encoder dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Minor code cleanups/comments * Move non-component files into lib, and move sidebar into a subdir * Remove redundant files * Extract Sidebar components * Wire up basic loading mechanism * Copy across button and indicator components from Interrain * Extract hook to load and transform data, start formatting usage table * Add GroundWater and populate tooltips with the right data * Take into account WaterTakeFilter when showing usage table * Clean up some naming * Ensure WaterAllocationMessage messages match IngestAPI types * Added fields to water usage API required by the front-end * Camelise API properties * Ensure usage dates are sorted * Better handling of errors loading usage data * Feedback from code review
* Moving WaterAllocation View to a Materialised View * Remove spots * Updates to allow water_allocation_and_usage_by_area to be refreshed * Fix up permission issue with materialized view * Turn effective_daily_consents back into a regular migration so we can ensure it is run before water_allocation_and_usage_by_area * Add materialized_views_role definition to LocalInfrastructure init script * Ensure materialized view is refreshed before running tests * Ensure materialized_views_role has the same permissions as eop_manager_app_user * Ensure view is materialized after generating test data * Remove old version of DB config - Has been superceded by LocalInfra folder * Limit materialized views role to Read only --------- Co-authored-by: Vim <vim@noodle.io>
* Update react-router-dom * Update Vite and associated packages * Update typescript and related packages, sort dev and main packages * Update linting and formatting libs And remove pretty-quick which is not compatible with prettier 3: prettier/pretty-quick#164 * Update testing libs * Update autoprefixer, remove unused web-vitals * Update tailwind, postcss and related libs * Update data/state libraries * npm audit fix * Update react-map-gl and maplibre-gl * Update MSW
* Move permissions required for materialized_views to migration * Add RESET ROLE to script - Otherwise the connection goes back to the pool with the wrong role, and causes pain for later threads.
* Changes for Water Use View - Move data to NZST timezone when aggregating daily - Fix issue with Water Meter Volume, previously we were treating it as a l/s value when it is actually a raw M^3 used since the last data point. Which makes the aggregation a simple sum * Update test with new data
* WIP: Detailed usage page with initial weekly usage * WIP: Group heatmap into regions and order * WIP: Tidy up presentation and data * Use full path in link to detailed usage page to fix AWS Amplify weirdness * Display tweaks * Add a link back to the limits viewer * Error handling and code cleanup * Fix bug formatting date in tooltip * Make UsageDisplayGroups part of council data
* Tweak to limit amount of history kept. * Extend measurement list so it will be on month ahead if there are recent measurements * Add new fetch task that is constantly refreshing the latest data * Refactor to make comparsion more explicit * Update README for recent changes
Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 6.9.0 to 6.10.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v6.10.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [maplibre-gl](https://github.com/maplibre/maplibre-gl-js) from 3.5.1 to 3.5.2. - [Release notes](https://github.com/maplibre/maplibre-gl-js/releases) - [Changelog](https://github.com/maplibre/maplibre-gl-js/blob/main/CHANGELOG.md) - [Commits](maplibre/maplibre-gl-js@v3.5.1...v3.5.2) --- updated-dependencies: - dependency-name: maplibre-gl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Extract results to separate components * Display daily usage * Use the same color scheme for daily usage we weekly usage * Format data in tooltip * Add a key to the daily usage chart * Add legend to weekly usage * Tweaks based on feedback * Fix rounding issue * Test alternate visualisation type * Add a key to the heatmapped daily data * Code tidy-ups * Remove TimeRange graph and tidy up code * Remove unused renderCell Override * Ensure top axis is not generated for weekly graphs other than first * Copy tweaks
* fix for bug in daily usage calc * remove date filter. no longer needed as filtering occurs on a downstream view
* Handle missing data for Usage table * Handle missing data in Detailed Daily usage graph * Display tweaks * Handle missing and show more detailed info for weekly usage * Formatting tweaks * Only show extended tooltip data when the d key is held down * Remove console.log * Formatting tweaks * More explicit handling of 0 vs NULL values when displaying usage * Always show daily usage table in weekly tooltip * Show CMU if CMSU is not present
* Update field names to match updated view * Don't default to 0's where NULLs are present in Usage and Allocation data * Update view with showing allocation vs usage * Update view tests --------- Co-authored-by: Vim <vim@noodle.io>
* Add some docs * Docs tweaks
smozely
force-pushed
the
integration/water-use
branch
from
a7bf662
to
2968749
Compare
smozely
force-pushed
the
integration/water-use
branch
from
876a95a
to
fca86c4
Compare
smozely
force-pushed
the
integration/water-use
branch
from
9680d50
to
a8e0202
Compare
* Skip large messages from Hilltop - Some messages will cause issues later in processing because of their size * Fixup off by one day error in test * Update manager so observation updated_at is only changed if the value changes - This is better information, but also should imporve loading performance
* Update Ingest API for new message format * Update manager with new fields * Update Plan Limits with new data * Adding in Measured vs Total Allocation for usage
…ations Multiple values for same day (#235) * Update Ingest API for new message format * Update manager with new fields * Update Plan Limits with new data * Adding in Measured vs Total Allocation for usage * added the .sql file which removes the daily duplicates for each site * fix the where clause position * drop the view and create it as there are structural changes * drop the view and create it as there are structural changes * drop the cascaded view and recreate them * removed the drop command * removed the drop command * changes to ALTER view * changes to ALTER view * added drop and removed alter * added both view script in one migration file * added both view script in one migration file * added both view script in one migration file * added both view in the same migration file and corrected the sql query * keeping the naming conventions same * Adjusting roles to allow removeing the materalised view --------- Co-authored-by: Steve Mosley <steve@starter4ten.com>
* TEAMU-1016 Add FE and BE unit tests * TEAMU-1016 Add FE and BE unit tests * TEAMU-1016 Improve BE exception handling * TEAMU-1016 Teak start scripts * TEAMU-1016 Check error message in test * TEAMU-1016 Lint fixes * TEAMU-1016 Spelling error * TEAMU-1016 Linting and batect run * TEAMU-1016 Resolve merge conflict in readme --------- Co-authored-by: wtcarter-gw <wayne.carter@gw.govt.nz>
…rst (#248) * TEAMU-940 Add new migration 41 * TEAMU-940 Correct migration 41 as per failing test * TEAMU-940 Apply review feedback and add some tests * TEAMU-940 Augment tests * TEAMU-940 Spotless linting --------- Co-authored-by: wtcarter-gw <wayne.carter@gw.govt.nz>
* fix to exclude all combined_meters * Remove the extra space
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.