715 716 717 718 observability safety slo - #792
Merged
hman38705 merged 5 commits intoMay 28, 2026
Merged
Conversation
# [1.1.0](v1.0.0...v1.1.0) (2026-05-28) ### Bug Fixes * **solutions-plug#452:** remove double newsletter rate limiting, make policy configurable ([6169065](6169065)), closes [solutions-plug#452](https://github.com/Hexstar-labs/predictIQ/issues/452) [solutions-plug#452](https://github.com/Hexstar-labs/predictIQ/issues/452) * **solutions-plug#453:** Redis-backed newsletter IP rate limiter with atomic counters ([932a14d](932a14d)), closes [solutions-plug#453](https://github.com/Hexstar-labs/predictIQ/issues/453) * **solutions-plug#454:** harden client IP extraction with trusted proxy CIDRs ([04db648](04db648)), closes [solutions-plug#454](https://github.com/Hexstar-labs/predictIQ/issues/454) * **solutions-plug#463:** add TTL and max-size eviction to watched_txs ([428b7a7](428b7a7)), closes [solutions-plug#463](https://github.com/Hexstar-labs/predictIQ/issues/463) * **solutions-plug#485-solutions-plug#488:** sync API_SPEC, add email DLQ, validate templates at startup ([0749bc5](0749bc5)), closes [solutions-plug#485](https://github.com/Hexstar-labs/predictIQ/issues/485) [-solutions-plug#488](https://github.com/-/issues/488) [solutions-plug#485](https://github.com/Hexstar-labs/predictIQ/issues/485) [solutions-plug#486](https://github.com/Hexstar-labs/predictIQ/issues/486) [solutions-plug#487](https://github.com/Hexstar-labs/predictIQ/issues/487) [solutions-plug#488](https://github.com/Hexstar-labs/predictIQ/issues/488) * **solutions-plug#624:** fetch_pyth_price calls on-chain Pyth contract via oracle_address/feed_id ([eb3079f](eb3079f)), closes [solutions-plug#624](https://github.com/Hexstar-labs/predictIQ/issues/624) * **solutions-plug#625:** determine_outcome uses per-market strike_price from OracleConfig ([901f800](901f800)), closes [solutions-plug#625](https://github.com/Hexstar-labs/predictIQ/issues/625) * **solutions-plug#626:** validate_oracle_staleness checks all oracle indices ([bc53975](bc53975)), closes [solutions-plug#626](https://github.com/Hexstar-labs/predictIQ/issues/626) * **solutions-plug#627:** align DISPUTE_WINDOW_SECONDS to 72h (259200) ([4f26453](4f26453)), closes [solutions-plug#627](https://github.com/Hexstar-labs/predictIQ/issues/627) * **api:** stop leaking internal error details to API consumers ([solutions-plug#628](https://github.com/Hexstar-labs/predictIQ/issues/628)) ([e3d8cd9](e3d8cd9)) * contract query optimizations and docs fixes ([solutions-plug#481](https://github.com/Hexstar-labs/predictIQ/issues/481)-484) ([2a0d193](2a0d193)), closes [solutions-plug#481-484](https://github.com/Hexstar-labs/predictIQ/issues/481-484) * Enforce SendGrid webhook signature validation ([d49109a](d49109a)), closes [solutions-plug#24](https://github.com/Hexstar-labs/predictIQ/issues/24) * log cache warming errors and cover all critical endpoints ([bf978d5](bf978d5)), closes [solutions-plug#493](https://github.com/Hexstar-labs/predictIQ/issues/493) * replace single-connection SCAN loop with per-batch connection acquisition ([solutions-plug#458](https://github.com/Hexstar-labs/predictIQ/issues/458)) ([2cfdec4](2cfdec4)) * resolve compile errors from issues [solutions-plug#477](https://github.com/Hexstar-labs/predictIQ/issues/477)-[#480](https://github.com/Hexstar-labs/predictIQ/issues/480) ([81235ac](81235ac)), closes [solutions-plug#478](https://github.com/Hexstar-labs/predictIQ/issues/478) * stop swallowing blockchain RPC errors into silent defaults ([solutions-plug#460](https://github.com/Hexstar-labs/predictIQ/issues/460)) ([fc0ec9c](fc0ec9c)) * structured API errors, webhook sig verification, OpenAPI sync ([2430bfe](2430bfe)) ### Features * **solutions-plug#519:** Enhance accessibility compliance with improved ARIA labels and keyboard navigation ([174c41a](174c41a)), closes [solutions-plug#519](https://github.com/Hexstar-labs/predictIQ/issues/519) * **solutions-plug#520:** Add frontend error boundary components ([6deea3f](6deea3f)), closes [solutions-plug#520](https://github.com/Hexstar-labs/predictIQ/issues/520) * **solutions-plug#521:** Implement frontend route-based code splitting ([e4166bc](e4166bc)), closes [solutions-plug#521](https://github.com/Hexstar-labs/predictIQ/issues/521) * **solutions-plug#522:** Add frontend environment variable validation ([b6dc301](b6dc301)), closes [solutions-plug#522](https://github.com/Hexstar-labs/predictIQ/issues/522) * **solutions-plug#523:** Implement frontend API response caching ([4daf584](4daf584)), closes [solutions-plug#523](https://github.com/Hexstar-labs/predictIQ/issues/523) * **solutions-plug#524:** Add frontend internationalization support ([266a5e3](266a5e3)), closes [solutions-plug#524](https://github.com/Hexstar-labs/predictIQ/issues/524) * **solutions-plug#525:** Implement frontend dark mode support ([e171d57](e171d57)), closes [solutions-plug#525](https://github.com/Hexstar-labs/predictIQ/issues/525) * **solutions-plug#526:** Add k6 load test for newsletter subscribe endpoint ([b31196d](b31196d)), closes [solutions-plug#526](https://github.com/Hexstar-labs/predictIQ/issues/526) * add cache stampede protection (Issue [solutions-plug#47](https://github.com/Hexstar-labs/predictIQ/issues/047)) ([4fff7e7](4fff7e7)) * add configurable database query timeout ([solutions-plug#49](https://github.com/Hexstar-labs/predictIQ/issues/049)) ([522e0f0](522e0f0)) * add Dependabot configuration for automated security updates ([34ed35e](34ed35e)) * add E2E tests for market creation flow with CI and flaky detection ([8778933](8778933)) * add idempotency key support for email sends (Issue [solutions-plug#45](https://github.com/Hexstar-labs/predictIQ/issues/045)) ([54dde3f](54dde3f)) * **api:** add database connection health check ([solutions-plug#495](https://github.com/Hexstar-labs/predictIQ/issues/495)) ([e012013](e012013)) * **frontend:** add form validation with accessible user feedback ([solutions-plug#73](https://github.com/Hexstar-labs/predictIQ/issues/073)) ([f376168](f376168)) * implement cache TTL per key type (Issue [solutions-plug#46](https://github.com/Hexstar-labs/predictIQ/issues/046)) ([8f2bbb1](8f2bbb1)) * implement database migration version tracking ([281e2a2](281e2a2)), closes [solutions-plug#496](https://github.com/Hexstar-labs/predictIQ/issues/496) * implement graceful shutdown for background workers ([0ef02d4](0ef02d4)) * implement issues [solutions-plug#513](https://github.com/Hexstar-labs/predictIQ/issues/513) [solutions-plug#514](https://github.com/Hexstar-labs/predictIQ/issues/514) [solutions-plug#515](https://github.com/Hexstar-labs/predictIQ/issues/515) [solutions-plug#516](https://github.com/Hexstar-labs/predictIQ/issues/516) ([f8cdec2](f8cdec2)) * implement real market resolve write flow ([solutions-plug#457](https://github.com/Hexstar-labs/predictIQ/issues/457)) ([ca9de29](ca9de29)) * introduce targeted cache invalidation tags ([solutions-plug#459](https://github.com/Hexstar-labs/predictIQ/issues/459)) ([e2924e3](e2924e3)) * newsletter-security-issues ([59e2a2d](59e2a2d)) * protect /metrics endpoint with auth and IP allowlist ([044ef94](044ef94)), closes [solutions-plug#456](https://github.com/Hexstar-labs/predictIQ/issues/456) * Redis pooling, circuit breaker, retry, correlation IDs, OTel tracing ([cab2ffb](cab2ffb)) * sanitize email recipients before SendGrid call (Issue [solutions-plug#44](https://github.com/Hexstar-labs/predictIQ/issues/044)) ([13ff88a](13ff88a)) * wire and globalise admin security & validation middleware ([solutions-plug#446](https://github.com/Hexstar-labs/predictIQ/issues/446) [solutions-plug#447](https://github.com/Hexstar-labs/predictIQ/issues/447) [solutions-plug#448](https://github.com/Hexstar-labs/predictIQ/issues/448) [solutions-plug#449](https://github.com/Hexstar-labs/predictIQ/issues/449)) ([38ad7b3](38ad7b3))
- Add prometheus-slo-rules.test.yml with unit tests for SLO recording and alerting rules - Add prometheus-rules-test job to CI pipeline - Check syntax of prometheus-slo-rules.yml and alerts.yaml - Run promtool unit tests for critical alerting rules - Fail CI on rule syntax errors
- Add runbook_url to all alert rules in alerts.yaml - Create comprehensive runbooks in docs/runbooks/ for: - API performance (response time, error rate, throughput) - Cache performance (hit rate) - Database performance (query latency, connection pool) - Contract performance (gas costs) - System health (service down, memory, CPU) - Performance regression detection - Each runbook includes: alert meaning, impact, investigation steps, remediation
- Add check-slo-compliance.js script to query Prometheus for SLO metrics - Script checks error budget remaining for all SLOs - Blocks production deployment if error budget exhausted (< 0%) - Supports CONFIRM_SLO_OVERRIDE=yes for emergency deploys with approval - Integrated into deploy workflow for production environment - Provides clear feedback on SLO status and error budget
… targets - Add deploy-testnet target with confirmation prompt - Add deploy-mainnet target with additional CONFIRM_MAINNET=yes requirement - Separate testnet and mainnet deployment targets for safety - Testnet requires single confirmation: 'Are you sure? [y/N]' - Mainnet requires both confirmation prompt AND CONFIRM_MAINNET=yes env var - Prevents accidental mainnet deployments - Builds optimized contract before deployment
|
@Hexstar-labs Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observability & Safety Enhancements: SLO Testing, Runbooks, Compliance Gates, and Deploy Safety
Overview
Comprehensive improvements to observability, reliability, and deployment safety across the PredictIQ platform. Implements SLO testing, runbook documentation, SLO compliance enforcement, and safe deployment practices.
Changes
#715: Prometheus SLO Rules Testing
prometheus-slo-rules.test.ymlwith unit tests for SLO recording and alerting rulesprometheus-rules-testjob into CI pipelineprometheus-slo-rules.ymlandalerts.yamlusingpromtool check rulespromtool test rules#716: Alert Runbook Documentation
runbook_urlannotations to all 20+ alert rules inalerts.yamldocs/runbooks/:#717: SLO Compliance Enforcement in Deploy Pipeline
performance/scripts/check-slo-compliance.jsto query Prometheus for SLO metricsCONFIRM_SLO_OVERRIDE=yesenvironment variable for emergency deploys with approval#718: Safe Deployment Targets with Confirmation
deploy-testnettarget with interactive confirmation promptdeploy-mainnettarget with dual protection:CONFIRM_MAINNET=yesTesting
promtool check rulesprometheus-rules-testjobFiles Changed
.github/workflows/test.yml- Added prometheus-rules-test job.github/workflows/deploy.yml- Added SLO compliance checkperformance/config/prometheus-slo-rules.test.yml- New test fileperformance/config/alerts.yaml- Added runbook_url annotationsperformance/scripts/check-slo-compliance.js- New compliance check scriptcontracts/predict-iq/Makefile- Added deploy-testnet and deploy-mainnet targetsdocs/runbooks/*.md- 16 new runbook filesCloses
prometheus-slo-rules.ymlare not tested #715alerts.yamldo not have runbook URLs #716SLO_GUIDE.mddefines SLOs but they are not enforced in the deploy pipeline #717contracts/predict-iq/Makefiledeploytarget does not prompt for confirmation #718