From 9b8a45985e6820aac1f867edf98b57a09a59bc9e Mon Sep 17 00:00:00 2001 From: Steve Mosley Date: Tue, 26 Mar 2024 06:33:03 +1300 Subject: [PATCH] Water Use (#195) * Update IngestAPI with changes to accommodate usage (#91) * Improve logging configuration * Update IngestAPI to accept and store allocation messages with usage * Message format tweaks * Make Metered Allocations nullable in API * Add sourceId to ingested water allocation * Update water allocation consumer (#92) * Improve logging configuration * Update IngestAPI to accept and store allocation messages with usage * Message format tweaks * Make Metered Allocations nullable in API * Add sourceId to ingested water allocation * Remove unused code * Updated WaterAllocationMessage and Consumer to handle allocations by consent (rather than area) * Fix typo * Ensure Cache manifest for new PlanLimit queries is periodically updated * Fix failing tests by truncating datetime before comparison to avoid precision differences. * Update GeoJsonControllerIntegrationTest to reflect new plan limits controller, and remove redundant test * Changes from PR review * Ensure only Rainfall measurements are used in the Rainfall queries (#96) * Manager - Kafka Consumer for Observations (#97) * Manager - Hilltop Crawler Changes * Shuffle files around * martin/aggregate water use (#95) * add water use daily aggregation sql view * optimise sql query performance * Adding Error handling to Observations Consumer --------- Co-authored-by: Martin Peak * Hilltop Crawler (#98) * Manager - Hilltop Crawler Changes * Shuffle files around * Crawler - Updates * Intergration testing * Improve DB naming * Big refactor of test setup * Correct time to wait between quiet polls * Add global error handling around task * Add check to make sure Virtual Measurement matches Measurement Name * Add a bunch to the readme * Add views to aggregate daily allocations and usage by area (#100) * Add views to aggregate consent allocations and usage by day * Start writing tests * Allow easier setup and assertion against test data, start adding more tests * Add tests for various allocation scenarios * Add failing test for aggregating data across different areas * Simplify views * Splitting out the calculation of daily useage by area * Add more assertions for aggregation of different areas * water_allocation.meter should correlate with observation_site.name * Format code * Feedback from PR review --------- Co-authored-by: Steve Mosley * Remove unnesscary `yield` * Adding index on next_fetch_at column * Add some more readme details about the task queue * Adding notes about partition sizes * Improve note about how schedulign tasks works --------- Co-authored-by: Vimal Jobanputra * Fixup the path being passed to archive build reports (#99) * Adding in sample data scripts and way to run from gradle (#101) * Adding in sample data scripts and way to run from gradle * Update so source id is unique per row * Update to insert observation data for 50 sites * Adding in a simple script to add time to the observed data * Tweak how loading is done to avoid the function * Update the refresh to be relative to current date * Adding in transactions for the scripts * Bump msw from 1.2.1 to 1.3.2 in /packages/PlanLimitsUI (#108) Bumps [msw](https://github.com/mswjs/msw) from 1.2.1 to 1.3.2. - [Release notes](https://github.com/mswjs/msw/releases) - [Changelog](https://github.com/mswjs/msw/blob/main/CHANGELOG.md) - [Commits](https://github.com/mswjs/msw/compare/v1.2.1...v1.3.2) --- updated-dependencies: - dependency-name: msw dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump net.logstash.logback:logstash-logback-encoder (#107) Bumps [net.logstash.logback:logstash-logback-encoder](https://github.com/logfellow/logstash-logback-encoder) from 7.3 to 7.4. - [Release notes](https://github.com/logfellow/logstash-logback-encoder/releases) - [Commits](https://github.com/logfellow/logstash-logback-encoder/compare/logstash-logback-encoder-7.3...logstash-logback-encoder-7.4) --- updated-dependencies: - dependency-name: net.logstash.logback:logstash-logback-encoder dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add query and controller endpoint for usage data (#102) * Weekly overview table (#124) * Minor code cleanups/comments * Move non-component files into lib, and move sidebar into a subdir * Remove redundant files * Extract Sidebar components * Wire up basic loading mechanism * Copy across button and indicator components from Interrain * Extract hook to load and transform data, start formatting usage table * Add GroundWater and populate tooltips with the right data * Take into account WaterTakeFilter when showing usage table * Clean up some naming * Ensure WaterAllocationMessage messages match IngestAPI types * Added fields to water usage API required by the front-end * Camelise API properties * Ensure usage dates are sorted * Better handling of errors loading usage data * Feedback from code review * Moving WaterAllocation View to a Materialised View (#125) * Moving WaterAllocation View to a Materialised View * Remove spots * Updates to allow water_allocation_and_usage_by_area to be refreshed * Fix up permission issue with materialized view * Turn effective_daily_consents back into a regular migration so we can ensure it is run before water_allocation_and_usage_by_area * Add materialized_views_role definition to LocalInfrastructure init script * Ensure materialized view is refreshed before running tests * Ensure materialized_views_role has the same permissions as eop_manager_app_user * Ensure view is materialized after generating test data * Remove old version of DB config - Has been superceded by LocalInfra folder * Limit materialized views role to Read only --------- Co-authored-by: Vim * Update plan limit libs (#127) * Update react-router-dom * Update Vite and associated packages * Update typescript and related packages, sort dev and main packages * Update linting and formatting libs And remove pretty-quick which is not compatible with prettier 3: https://github.com/azz/pretty-quick/issues/164 * Update testing libs * Update autoprefixer, remove unused web-vitals * Update tailwind, postcss and related libs * Update data/state libraries * npm audit fix * Update react-map-gl and maplibre-gl * Update MSW * Moving the materalized view permissions into main migrations (#129) * Move permissions required for materialized_views to migration * Add RESET ROLE to script - Otherwise the connection goes back to the pool with the wrong role, and causes pain for later threads. * Fix permissions so app can query usage view (#137) * Config ErrorHandlingDeserializer to DLQ unparsable messages (#134) * Re-apply changes from PR #130 (#138) * Changes for Water Use View (#139) * Changes for Water Use View - Move data to NZST timezone when aggregating daily - Fix issue with Water Meter Volume, previously we were treating it as a l/s value when it is actually a raw M^3 used since the last data point. Which makes the aggregation a simple sum * Update test with new data * Weekly usage graph (#148) * WIP: Detailed usage page with initial weekly usage * WIP: Group heatmap into regions and order * WIP: Tidy up presentation and data * Use full path in link to detailed usage page to fix AWS Amplify weirdness * Display tweaks * Add a link back to the limits viewer * Error handling and code cleanup * Fix bug formatting date in tooltip * Make UsageDisplayGroups part of council data * Crawler fix start of month (#149) * Tweak to limit amount of history kept. * Extend measurement list so it will be on month ahead if there are recent measurements * Add new fetch task that is constantly refreshing the latest data * Refactor to make comparsion more explicit * Update README for recent changes * Use correct allocation measure (#151) * Bump @typescript-eslint/parser in /packages/PlanLimitsUI (#140) Bumps [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) from 6.9.0 to 6.10.0. - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v6.10.0/packages/parser) --- updated-dependencies: - dependency-name: "@typescript-eslint/parser" dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump maplibre-gl from 3.5.1 to 3.5.2 in /packages/PlanLimitsUI (#132) Bumps [maplibre-gl](https://github.com/maplibre/maplibre-gl-js) from 3.5.1 to 3.5.2. - [Release notes](https://github.com/maplibre/maplibre-gl-js/releases) - [Changelog](https://github.com/maplibre/maplibre-gl-js/blob/main/CHANGELOG.md) - [Commits](https://github.com/maplibre/maplibre-gl-js/compare/v3.5.1...v3.5.2) --- updated-dependencies: - dependency-name: maplibre-gl dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Daily Usage Graph (#158) * Extract results to separate components * Display daily usage * Use the same color scheme for daily usage we weekly usage * Format data in tooltip * Add a key to the daily usage chart * Add legend to weekly usage * Tweaks based on feedback * Fix rounding issue * Test alternate visualisation type * Add a key to the heatmapped daily data * Code tidy-ups * Remove TimeRange graph and tidy up code * Remove unused renderCell Override * Ensure top axis is not generated for weekly graphs other than first * Copy tweaks * Water meter reading view bugfix (#159) * fix for bug in daily usage calc * remove date filter. no longer needed as filtering occurs on a downstream view * Handle missing data in the UI (#165) * Handle missing data for Usage table * Handle missing data in Detailed Daily usage graph * Display tweaks * Handle missing and show more detailed info for weekly usage * Formatting tweaks * Only show extended tooltip data when the d key is held down * Remove console.log * Formatting tweaks * More explicit handling of 0 vs NULL values when displaying usage * Always show daily usage table in weekly tooltip * Show CMU if CMSU is not present * Handle missing data in view (#173) * Update field names to match updated view * Don't default to 0's where NULLs are present in Usage and Allocation data * Update view with showing allocation vs usage * Update view tests --------- Co-authored-by: Vim * Add overview docs (#174) * Add some docs * Docs tweaks * Remove log error message about rack assigner * Adding date check back in because of how it effect materlizing the view (#161) * Lower number of consumers for Kafka to hopefully stabalise things * Update Jackson max message size * Crawler handle large messages (#225) * Skip large messages from Hilltop - Some messages will cause issues later in processing because of their size * Fixup off by one day error in test * Update manager so observation updated_at is only changed if the value changes - This is better information, but also should imporve loading performance * TEAMU-876 - Show what percentage of allocation is measured (#232) * Update Ingest API for new message format * Update manager with new fields * Update Plan Limits with new data * Adding in Measured vs Total Allocation for usage * https://gwrc.atlassian.net/browse/TEAMU-929 : Daily Aggregated Observations Multiple values for same day (#235) * Update Ingest API for new message format * Update manager with new fields * Update Plan Limits with new data * Adding in Measured vs Total Allocation for usage * added the .sql file which removes the daily duplicates for each site * fix the where clause position * drop the view and create it as there are structural changes * drop the view and create it as there are structural changes * drop the cascaded view and recreate them * removed the drop command * removed the drop command * changes to ALTER view * changes to ALTER view * added drop and removed alter * added both view script in one migration file * added both view script in one migration file * added both view script in one migration file * added both view in the same migration file and corrected the sql query * keeping the naming conventions same * Adjusting roles to allow removeing the materalised view --------- Co-authored-by: Steve Mosley * TEAMU-1016-Fix-leap-year-issue (#243) * TEAMU-1016 Add FE and BE unit tests * TEAMU-1016 Add FE and BE unit tests * TEAMU-1016 Improve BE exception handling * TEAMU-1016 Teak start scripts * TEAMU-1016 Check error message in test * TEAMU-1016 Lint fixes * TEAMU-1016 Spelling error * TEAMU-1016 Linting and batect run * TEAMU-1016 Resolve merge conflict in readme --------- Co-authored-by: wtcarter-gw * TEAMU-940 Add new migration 41 replacing MAX and MIN with last and first (#248) * TEAMU-940 Add new migration 41 * TEAMU-940 Correct migration 41 as per failing test * TEAMU-940 Apply review feedback and add some tests * TEAMU-940 Augment tests * TEAMU-940 Spotless linting --------- Co-authored-by: wtcarter-gw * fix to exclude all combined_meters (#249) * fix to exclude all combined_meters * Remove the extra space --------- Signed-off-by: dependabot[bot] Co-authored-by: Vimal Jobanputra Co-authored-by: Martin Peak Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Swapna Josmi Sam <56238575+swapnasam@users.noreply.github.com> Co-authored-by: Wayne Carter Co-authored-by: wtcarter-gw --- .github/workflows/hilltop-crawler.yml | 2 +- .github/workflows/ingest-api.yml | 2 +- .github/workflows/manager.yml | 2 +- .run/Plan Limits UI.run.xml | 13 + README.md | 10 +- packages/HilltopCrawler/README.md | 108 + packages/HilltopCrawler/build.gradle.kts | 19 +- .../govt/eop/hilltop_crawler/Application.kt | 24 +- .../hilltop_crawler/CheckForNewSourcesTask.kt | 34 + .../nz/govt/eop/hilltop_crawler/Constants.kt | 3 + .../govt/eop/hilltop_crawler/TaskScheduler.kt | 128 - .../eop/hilltop_crawler/api/HilltopFetcher.kt | 40 + .../api/parsers/HilltopMeasurementValues.kt | 49 + .../api/parsers/HilltopMeasurments.kt | 28 + .../api/parsers/HilltopSites.kt | 18 + .../api/parsers/HilltopXmlParsers.kt | 32 + .../api/requests/HilltopRequestBuilders.kt | 54 + .../config/JacksonJsonConfig.kt | 17 + .../eop/hilltop_crawler/config/KafkaConfig.kt | 59 +- .../config/RestTemplateConfig.kt | 25 + .../nz/govt/eop/hilltop_crawler/db/DB.kt | 136 +- .../fetcher/FetchTaskProcessor.kt | 171 + .../fetcher/FetchTasksRunner.kt | 48 + .../fetcher/HilltopMessageClient.kt | 22 + .../eop/hilltop_crawler/fetcher/Messages.kt | 101 + .../hilltop_crawler/fetcher/TaskMappers.kt | 280 + .../producer/ObservationMessages.kt | 75 + .../producer/ObservationProducer.kt | 87 + .../support/HilltopRequests.kt | 65 - .../support/HilltopSitesParser.kt | 38 - .../support/HilltopXmlParsers.kt | 101 - .../kotlin/nz/govt/eop/messages/Messages.kt | 11 - .../src/main/resources/application.yml | 29 +- .../db/migration/R__Hilltop_Sources.sql | 14 - .../db/migration/V0001__Hilltop_Indexing.sql | 22 +- .../migration/V0002__Index_on_next_fetch.sql | 1 + .../HilltopCrawlerTestConfiguration.kt | 13 + .../api/parsers/HilltopXmlParsersTest.kt | 181 + .../FetchTaskProcessorIntegrationTest.kt | 680 + .../fetcher/MessagesParsingTest.kt | 399 + .../fetcher/TaskMappersTests.kt | 756 + .../producer/ObservationsProducerTest.kt | 140 + .../HilltopMeasurementListParserTest.kt | 18 - .../support/HilltopSitesParserTest.kt | 62 - .../src/test/resources/application-test.yml | 16 + .../graphql-documents/observation.graphql | 3 - .../resources/hilltop-xml/ErrorResponse.xml | 4 + .../hilltop-xml/MeasurementValuesResponse.xml | 27 + ...list.xml => MeasurementsResponse-list.xml} | 4 +- ...list-empty.xml => SitesResponse-empty.xml} | 0 .../{sitelist.xml => SitesResponse-list.xml} | 0 ...on.xml => SitesResponse-no-projection.xml} | 0 .../src/test/resources/logback-test.xml | 7 +- packages/IngestAPI/README.md | 6 +- packages/IngestAPI/build.gradle.kts | 2 +- .../nz/govt/eop/ingest/api/Controller.kt | 13 +- .../kotlin/nz/govt/eop/messages/Messages.kt | 28 +- .../src/main/resources/application.yml | 2 +- .../nz/govt/eop/ingest/IntegrationTest.kt | 73 +- .../src/test/resources/logback-test.xml | 25 + .../.batect/database/init.sql | 5 + packages/Manager/.batect/database/Dockerfile | 8 - .../Manager/.batect/database/health-check.sh | 21 - packages/Manager/.batect/database/init.sql | 39 - packages/Manager/Dockerfile | 2 +- packages/Manager/batect.yml | 15 + packages/Manager/build.gradle.kts | 49 +- .../Manager/sample-data/allocation_data.sql | 56 + .../Manager/sample-data/observation_data.sql | 761 + .../sample-data/update_observation_dates.sql | 15 + .../main/kotlin/nz/govt/eop/Application.kt | 7 +- .../eop/consumers/WaterAllocationConsumer.kt | 56 +- .../hilltop_crawler/HilltopCrawlerConsumer.kt | 152 - .../hilltop_crawler/HilltopSitesParser.kt | 38 - .../hilltop_crawler/HilltopXmlParsers.kt | 79 - .../observations/ObservationMessages.kt | 75 + .../observations/ObservationStore.kt | 157 + .../observations/ObservationsConsumer.kt | 47 + .../kotlin/nz/govt/eop/messages/Messages.kt | 26 +- .../nz/govt/eop/plan_limits/Controller.kt | 24 + .../nz/govt/eop/plan_limits/Manifest.kt | 6 + .../kotlin/nz/govt/eop/plan_limits/Queries.kt | 48 +- .../nz/govt/eop/rainfall/RainfallQueries.kt | 7 +- .../eop/tasks/WaterAllocationViewUpdater.kt | 29 + .../src/main/resources/application-prod.yml | 12 +- .../src/main/resources/application.yml | 41 +- .../migration/R__effective_daily_consents.sql | 1 + .../db/migration/R__setup_permissions.sql | 9 +- .../R__water_allocation_and_usage_by_area.sql | 1 + .../V0027__update_water_allocations.sql | 20 + ...__create_water_allocation_by_area_view.sql | 11 + .../migration/V0029__observations_changes.sql | 15 + ...served_water_use_aggregated_daily_view.sql | 27 + .../V0031__effective_daily_consents.sql | 30 + ...ocation_and_usage_by_area_materialized.sql | 105 + ...served_water_use_aggregated_daily_view.sql | 22 + ...served_water_use_aggregated_daily_view.sql | 30 + ...and_usage_by_area_materialized_updates.sql | 167 + ...served_water_use_aggregated_daily_view.sql | 30 + .../V0037__update_water_allocations.sql | 15 + .../V0038__effective_daily_consents.sql | 56 + ...ocation_and_usage_by_area_materialized.sql | 170 + ...0__observed_water_use_aggregated_daily.sql | 277 + ...1__observed_water_use_aggregated_daily.sql | 285 + ...ocation_and_usage_by_area_materialized.sql | 168 + ...WaterAllocationConsumerErrorHandlerTest.kt | 49 +- .../WaterAllocationConsumerIntegrationTest.kt | 38 +- .../consumers/WaterAllocationConsumerTest.kt | 84 +- .../observations/ObservationsConsumerTest.kt | 311 + .../eop/plan_limits/DetailedWaterUsageTest.kt | 55 + .../WaterAllocationAndUsageViewsTest.kt | 666 + .../src/test/resources/application-test.yml | 7 - packages/Manager/start.sh | 80 + packages/PlanLimitsUI/.env.local.template | 1 - packages/PlanLimitsUI/.prettierrc.json | 3 +- packages/PlanLimitsUI/package-lock.json | 12642 ++++++++-------- packages/PlanLimitsUI/package.json | 74 +- .../plan-limits-ui-screenshot.png | Bin 0 -> 2329452 bytes packages/PlanLimitsUI/readme.md | 20 + packages/PlanLimitsUI/src/App.tsx | 8 +- packages/PlanLimitsUI/src/api/index.ts | 85 +- .../PlanLimitsUI/src/components/Button.tsx | 55 + .../src/components/Disclaimer/index.tsx | 2 +- .../src/components/Indicators.tsx | 29 + .../{Button => RoundedButton}/index.tsx | 0 .../src/components/map/LayerControl.tsx | 4 +- packages/PlanLimitsUI/src/global.d.ts | 64 +- packages/PlanLimitsUI/src/lib/councilData.tsx | 91 +- .../formatWaterQuantity.spec.tsx | 0 .../Limits => lib}/formatWaterQuantity.tsx | 0 packages/PlanLimitsUI/src/lib/loader.ts | 15 +- .../Limits => lib}/locationString.spec.ts | 0 .../{pages/Limits => lib}/locationString.ts | 0 .../src/{pages/Limits => lib}/useAppState.ts | 123 +- .../src/lib/useDetailedWaterUseData.spec.ts | 46 + .../src/lib/useDetailedWaterUseData.ts | 214 + .../PlanLimitsUI/src/lib/useWaterUseData.ts | 93 + .../src/pages/Limits/RiverTilesSource.tsx | 4 +- .../src/pages/Limits/Sidebar/Footer.tsx | 32 + .../src/pages/Limits/Sidebar/Header.tsx | 17 + .../Limits/{ => Sidebar}/LimitsTable.tsx | 2 + .../src/pages/Limits/Sidebar/Overview.tsx | 121 + .../src/pages/Limits/Sidebar/UsageTable.tsx | 291 + .../src/pages/Limits/Sidebar/index.tsx | 77 + .../Limits/defaultFlowLimitAndSite.spec.tsx | 22 - .../pages/Limits/defaultFlowLimitAndSite.tsx | 48 - .../PlanLimitsUI/src/pages/Limits/index.tsx | 6 +- .../PlanLimitsUI/src/pages/Limits/map.tsx | 36 +- .../PlanLimitsUI/src/pages/Limits/sidebar.tsx | 201 - .../src/pages/Usage/DailyResults.tsx | 160 + .../src/pages/Usage/WeeklyResults.tsx | 164 + .../PlanLimitsUI/src/pages/Usage/index.tsx | 67 + packages/PlanLimitsUI/src/setupTests.ts | 14 +- packages/PlanLimitsUI/start.sh | 3 + packages/PlanLimitsUI/tsconfig.json | 5 +- packages/RainfallViewer/.env.local.template | 2 + packages/RainfallViewer/README.md | 43 +- .../rainfall-viewer-screenshot.png | Bin 0 -> 1113905 bytes start.sh | 31 + 159 files changed, 15729 insertions(+), 8017 deletions(-) create mode 100644 .run/Plan Limits UI.run.xml create mode 100644 packages/HilltopCrawler/README.md create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/CheckForNewSourcesTask.kt delete mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/TaskScheduler.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/api/HilltopFetcher.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/api/parsers/HilltopMeasurementValues.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/api/parsers/HilltopMeasurments.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/api/parsers/HilltopSites.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/api/parsers/HilltopXmlParsers.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/api/requests/HilltopRequestBuilders.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/config/JacksonJsonConfig.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/config/RestTemplateConfig.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/fetcher/FetchTaskProcessor.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/fetcher/FetchTasksRunner.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/fetcher/HilltopMessageClient.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/fetcher/Messages.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/fetcher/TaskMappers.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/producer/ObservationMessages.kt create mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/producer/ObservationProducer.kt delete mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/support/HilltopRequests.kt delete mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/support/HilltopSitesParser.kt delete mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/hilltop_crawler/support/HilltopXmlParsers.kt delete mode 100644 packages/HilltopCrawler/src/main/kotlin/nz/govt/eop/messages/Messages.kt delete mode 100644 packages/HilltopCrawler/src/main/resources/db/migration/R__Hilltop_Sources.sql create mode 100644 packages/HilltopCrawler/src/main/resources/db/migration/V0002__Index_on_next_fetch.sql create mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/HilltopCrawlerTestConfiguration.kt create mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/api/parsers/HilltopXmlParsersTest.kt create mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/fetcher/FetchTaskProcessorIntegrationTest.kt create mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/fetcher/MessagesParsingTest.kt create mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/fetcher/TaskMappersTests.kt create mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/producer/ObservationsProducerTest.kt delete mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/support/HilltopMeasurementListParserTest.kt delete mode 100644 packages/HilltopCrawler/src/test/kotlin/nz/govt/eop/hilltop_crawler/support/HilltopSitesParserTest.kt create mode 100644 packages/HilltopCrawler/src/test/resources/application-test.yml delete mode 100644 packages/HilltopCrawler/src/test/resources/graphql-documents/observation.graphql create mode 100644 packages/HilltopCrawler/src/test/resources/hilltop-xml/ErrorResponse.xml create mode 100644 packages/HilltopCrawler/src/test/resources/hilltop-xml/MeasurementValuesResponse.xml rename packages/HilltopCrawler/src/test/resources/hilltop-xml/{measurementlist.xml => MeasurementsResponse-list.xml} (99%) rename packages/HilltopCrawler/src/test/resources/hilltop-xml/{sitelist-empty.xml => SitesResponse-empty.xml} (100%) rename packages/HilltopCrawler/src/test/resources/hilltop-xml/{sitelist.xml => SitesResponse-list.xml} (100%) rename packages/HilltopCrawler/src/test/resources/hilltop-xml/{sitelist-no-projection.xml => SitesResponse-no-projection.xml} (100%) create mode 100644 packages/IngestAPI/src/test/resources/logback-test.xml delete mode 100644 packages/Manager/.batect/database/Dockerfile delete mode 100755 packages/Manager/.batect/database/health-check.sh delete mode 100644 packages/Manager/.batect/database/init.sql create mode 100644 packages/Manager/sample-data/allocation_data.sql create mode 100644 packages/Manager/sample-data/observation_data.sql create mode 100644 packages/Manager/sample-data/update_observation_dates.sql delete mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/consumers/hilltop_crawler/HilltopCrawlerConsumer.kt delete mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/consumers/hilltop_crawler/HilltopSitesParser.kt delete mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/consumers/hilltop_crawler/HilltopXmlParsers.kt create mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/consumers/observations/ObservationMessages.kt create mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/consumers/observations/ObservationStore.kt create mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/consumers/observations/ObservationsConsumer.kt create mode 100644 packages/Manager/src/main/kotlin/nz/govt/eop/tasks/WaterAllocationViewUpdater.kt create mode 100644 packages/Manager/src/main/resources/db/migration/R__effective_daily_consents.sql create mode 100644 packages/Manager/src/main/resources/db/migration/R__water_allocation_and_usage_by_area.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0027__update_water_allocations.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0028__create_water_allocation_by_area_view.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0029__observations_changes.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0030__observed_water_use_aggregated_daily_view.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0031__effective_daily_consents.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0032__water_allocation_and_usage_by_area_materialized.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0033__observed_water_use_aggregated_daily_view.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0034__observed_water_use_aggregated_daily_view.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0035__water_allocation_and_usage_by_area_materialized_updates.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0036__observed_water_use_aggregated_daily_view.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0037__update_water_allocations.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0038__effective_daily_consents.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0039__water_allocation_and_usage_by_area_materialized.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0040__observed_water_use_aggregated_daily.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0041__observed_water_use_aggregated_daily.sql create mode 100644 packages/Manager/src/main/resources/db/migration/V0042__water_allocation_and_usage_by_area_materialized.sql create mode 100644 packages/Manager/src/test/kotlin/nz/govt/eop/consumers/observations/ObservationsConsumerTest.kt create mode 100644 packages/Manager/src/test/kotlin/nz/govt/eop/plan_limits/DetailedWaterUsageTest.kt create mode 100644 packages/Manager/src/test/kotlin/nz/govt/eop/plan_limits/WaterAllocationAndUsageViewsTest.kt create mode 100755 packages/Manager/start.sh create mode 100644 packages/PlanLimitsUI/plan-limits-ui-screenshot.png create mode 100644 packages/PlanLimitsUI/readme.md create mode 100644 packages/PlanLimitsUI/src/components/Button.tsx create mode 100644 packages/PlanLimitsUI/src/components/Indicators.tsx rename packages/PlanLimitsUI/src/components/{Button => RoundedButton}/index.tsx (100%) rename packages/PlanLimitsUI/src/{pages/Limits => lib}/formatWaterQuantity.spec.tsx (100%) rename packages/PlanLimitsUI/src/{pages/Limits => lib}/formatWaterQuantity.tsx (100%) rename packages/PlanLimitsUI/src/{pages/Limits => lib}/locationString.spec.ts (100%) rename packages/PlanLimitsUI/src/{pages/Limits => lib}/locationString.ts (100%) rename packages/PlanLimitsUI/src/{pages/Limits => lib}/useAppState.ts (80%) create mode 100644 packages/PlanLimitsUI/src/lib/useDetailedWaterUseData.spec.ts create mode 100644 packages/PlanLimitsUI/src/lib/useDetailedWaterUseData.ts create mode 100644 packages/PlanLimitsUI/src/lib/useWaterUseData.ts create mode 100644 packages/PlanLimitsUI/src/pages/Limits/Sidebar/Footer.tsx create mode 100644 packages/PlanLimitsUI/src/pages/Limits/Sidebar/Header.tsx rename packages/PlanLimitsUI/src/pages/Limits/{ => Sidebar}/LimitsTable.tsx (99%) create mode 100644 packages/PlanLimitsUI/src/pages/Limits/Sidebar/Overview.tsx create mode 100644 packages/PlanLimitsUI/src/pages/Limits/Sidebar/UsageTable.tsx create mode 100644 packages/PlanLimitsUI/src/pages/Limits/Sidebar/index.tsx delete mode 100644 packages/PlanLimitsUI/src/pages/Limits/defaultFlowLimitAndSite.spec.tsx delete mode 100644 packages/PlanLimitsUI/src/pages/Limits/defaultFlowLimitAndSite.tsx delete mode 100644 packages/PlanLimitsUI/src/pages/Limits/sidebar.tsx create mode 100644 packages/PlanLimitsUI/src/pages/Usage/DailyResults.tsx create mode 100644 packages/PlanLimitsUI/src/pages/Usage/WeeklyResults.tsx create mode 100644 packages/PlanLimitsUI/src/pages/Usage/index.tsx create mode 100755 packages/PlanLimitsUI/start.sh create mode 100644 packages/RainfallViewer/.env.local.template create mode 100644 packages/RainfallViewer/rainfall-viewer-screenshot.png create mode 100755 start.sh diff --git a/.github/workflows/hilltop-crawler.yml b/.github/workflows/hilltop-crawler.yml index ba3021ab..f911b5f1 100644 --- a/.github/workflows/hilltop-crawler.yml +++ b/.github/workflows/hilltop-crawler.yml @@ -51,7 +51,7 @@ jobs: if: always() with: name: reports - path: format('{0}/build/reports', env.WORKING_DIR) + path: ${{ env.WORKING_DIR }}/build/reports - name: Configure AWS credentials if: ${{ github.actor != 'dependabot[bot]' }} diff --git a/.github/workflows/ingest-api.yml b/.github/workflows/ingest-api.yml index 3e077b1c..993fad73 100644 --- a/.github/workflows/ingest-api.yml +++ b/.github/workflows/ingest-api.yml @@ -51,7 +51,7 @@ jobs: if: always() with: name: reports - path: format('{0}/build/reports', env.WORKING_DIR) + path: ${{ env.WORKING_DIR }}/build/reports - name: Configure AWS credentials if: ${{ github.actor != 'dependabot[bot]' }} diff --git a/.github/workflows/manager.yml b/.github/workflows/manager.yml index 1200345f..ce827a30 100644 --- a/.github/workflows/manager.yml +++ b/.github/workflows/manager.yml @@ -51,7 +51,7 @@ jobs: if: always() with: name: reports - path: format('{0}/build/reports', env.WORKING_DIR) + path: ${{ env.WORKING_DIR }}/build/reports - name: Configure AWS credentials if: ${{ github.actor != 'dependabot[bot]' }} diff --git a/.run/Plan Limits UI.run.xml b/.run/Plan Limits UI.run.xml new file mode 100644 index 00000000..3c62d271 --- /dev/null +++ b/.run/Plan Limits UI.run.xml @@ -0,0 +1,13 @@ + + + + + +