feat(aggregation): native MySQL/SQLite bucketing + ad-hoc cache (#1609, #1610)#1646
Merged
Merged
Conversation
Adds the OpenSpec change scaffold proposing an ad-hoc time-bucket aggregation primitive across both REST and GraphQL. - proposal.md: motivates the primitive (every dashboard needs it; openconnector rebuild blocked on it; AggregationQuery.dateBucket already shipped but not wired to Postgres/REST/GraphQL). kind: code, no schema register patches. - design.md: decisions D1-D9 — reuse AggregationQuery, date_trunc native path, field allow-listing via Schema::getProperties(), dedicated /aggregate/timeseries route, GroupByInput on every list query, AggregationRunner::runAdhoc() shared by REST+GraphQL. - specs/aggregation-api/spec.md: new capability for ad-hoc bucketing — validation rules, RBAC integration, Postgres+PHP-fallback contract. - specs/graphql-api/spec.md: delta — adds groupBy/TimeInterval/GroupBucket types + modifies the connection-type requirement. - tasks.md: 6 task groups covering runner, REST, GraphQL, tests, quality gates, and docs. No PR/merge tasks (Hydra coordination owns that). Complementary to the in-flight aggregations-backend-native change which already plumbed dateBucket into Solr+ES translators but not Postgres/REST/ GraphQL.
Implements the `add-time-bucket-aggregation` opsx change: an ad-hoc,
client-controlled aggregation primitive that lets dashboards bucket
register-schema rows by a field (categorical) or by a `date_trunc`
interval with optional `from`/`to` bounds, available on both REST and
GraphQL with the same response shape.
Why
- Every Conduction dashboard chart needs the same primitive ("group
rows by day-of-`created` between from and to, return {key, value}").
- openconnector dashboard rebuild (6 charts) was blocked on it.
- nextcloud-vue CnChartWidget.dataSource can now grow its `bucket: {
field, interval, fromVar, toVar }` shorthand.
- AggregationQuery.dateBucket was shipped by aggregations-backend-native
but never plumbed to Postgres / REST / GraphQL.
Backend
- AggregationRunner: extended tryNativeAggregation() with a dateBucket
parameter that emits `date_trunc(?, "$field")::text AS bucket` SQL
with explicit `WHERE field >= start AND field < end` bounds. ISO
week (Monday) + quarter (first-of-Q1-month) computed explicitly to
match Postgres semantics.
- New public methods runAdhoc(Register, Schema, AggregationQuery) and
runAdhocByRef(string, string, AggregationQuery) for the controller /
resolver entry points. RBAC + multi-tenant gates inherited from
the existing native path.
- bucketInPhp() — PHP fallback for non-Postgres DBs (SQLite tests,
MySQL dev) with truncateTimestamp() polyfill keyed on the gap
vocabulary. backend: "php-fallback" annotated in response.
- coerceBucketKey() produces stable Y-m-d\TH:i:s\Z strings across all
paths so the wire shape stays identical.
REST
- New TimeseriesRequestValidator class — separate testable validator
shared between REST + GraphQL. Validates field allow-list against
Schema::getProperties() + metadata cols (_created, _updated,
_deleted_at); validates sub-day intervals (MINUTE / HOUR) require
format: date-time; validates interval/from/to/metric/metricField
shape; constructs the AggregationQuery via the existing factory.
- AggregationController::timeseries(register, schema) action. 400/
403/404 translation. New route /api/objects/aggregations/{register}/
{schema}/timeseries ordered before the {name} wildcard.
GraphQL
- TypeMapperHandler grew lazy factories for GroupByInput, TimeInterval
enum, AggregationMetric enum, and GroupBucket object type. Each
cached on a private property so introspection-cycle and per-request
schema generation pay the construction cost once.
- getListArgs() gains `groupBy: GroupByInput`; getConnectionType()
gains `groups: [GroupBucket!]` (nullable — null = client did not
request).
- GraphQLResolver::resolveList() now calls resolveGroupBy() when
args.groupBy is present. Reuses TimeseriesRequestValidator for the
same allow-list + sub-day rules. Validation / RBAC failures surface
as GraphQL field-errors on `groups` — the rest of the connection
still resolves.
Tests
- 12 TimeseriesRequestValidatorTest cases covering every spec-defined
400 path.
- 5 AggregationControllerTimeseriesTest cases covering 404 / 400 / 403
translations + happy path body parity.
- AggregationControllerTest constructor updated to inject the new
validator dependency.
- Full Unit Tests suite (12 243 tests, 25 991 assertions) green inside
the dev container.
Quality gates
- PHPCS strict — clean on all touched files.
- PHPMD — clean (targeted SuppressWarnings on bucketInPhp, resolveList,
validate for legitimate branching-density cases).
- Psalm — clean.
- PHPStan level 8 — clean. Baseline reclassified the existing `string
=== null` defensive check to `int === null` after refactoring three
call sites away from the redundant `instanceof` ternary.
Docs
- docs/technical/aggregation-api.md covers both surfaces, validation
rules, status codes, Postgres index recommendations, and the
decision matrix between named-annotation and ad-hoc primitive.
- openspec/platform-capabilities.md grew the ad-hoc primitive
companion sentence on the aggregations row.
Non-goals deferred to issues
- #1606 multi-field groupBy
- #1607 cumulative / running aggregates
- #1608 multi-metric in one request
- #1609 native MySQL / SQLite bucketing
- #1610 caching ad-hoc queries
#1610) Extends the time-bucket primitive added by #1611 with two of the five deferred follow-ups: - **#1609 — Native MySQL/SQLite bucketing.** `AggregationRunner` now detects MySQL/MariaDB and SQLite alongside PostgreSQL and emits the matching native bucketing expression (`DATE_FORMAT` / `strftime`) with the correct identifier quoting per engine. ISO-Monday week-start semantics carry across all three platforms; quarter buckets use the matching CASE / CONCAT form. Backend annotation reports `mysql` / `sqlite` / `postgres` / `php-fallback`. - **#1610 — Read-through cache for ad-hoc queries.** `AggregationCache` gains `getAdhoc()` / `setAdhoc()` wrappers that derive the cache name slot from `'adhoc:'.sha1(json_encode(AggregationQuery::toArray()))`. `AggregationQuery::toArray()` ksort-sorts filter sub-arrays so two structurally-equivalent queries hash identically. `runAdhoc()` wires the cache read-through; the existing `AggregationCacheInvalidationListener` covers ad-hoc entries unchanged because they share the underlying `openregister_aggregations` distributed cache. The remaining three follow-ups (#1606 multi-field groupBy, #1607 cumulative/rolling windows, #1608 multi-metric) are documented at design depth in `openspec/changes/add-aggregation-enhancements/design.md` so the next pickup can file deliberate opsx cycles for them. Tests cover MySQL/SQLite/Postgres SQL emission per gap (including the `%i` vs `%M` minute placeholder divergence), ad-hoc cache hit/miss semantics, key stability under filter reordering, and ad-hoc vs named cache-entry isolation.
…gation-enhancements # Conflicts: # phpstan-baseline.neon
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Picks up two of the five deferred follow-ups filed against the time-bucket aggregation primitive (#1611):
AggregationRunner::tryNativeAggregation()now detects MySQL/MariaDB and SQLite alongside PostgreSQL and emits the matching native bucketing expression (DATE_FORMAT/strftime) with the correct identifier quoting per engine (backticks on MySQL, double-quotes on Postgres/SQLite). ISO-Monday week-start carries across all three platforms; quarter buckets use the matching CASE / CONCAT form. The responsebackendfield reportsmysql/sqlite/postgres/php-fallbackso callers can observe which path served the request.AggregationCachegainsgetAdhoc()/setAdhoc()wrappers that derive the cache name slot from'adhoc:'.sha1(json_encode(AggregationQuery::toArray())).AggregationQuery::toArray()ksort-sorts filter sub-arrays so{a, b}and{b, a}hash identically.runAdhoc()wires the cache read-through; the existingAggregationCacheInvalidationListenercovers ad-hoc entries unchanged because they share the underlyingopenregister_aggregationsdistributed cache.The remaining three follow-ups (#1606 multi-field groupBy, #1607 cumulative/rolling windows, #1608 multi-metric) require a value-object refactor on
AggregationQueryplus knock-on translator changes (Solr / ES / Postgres / PHP fallback / GraphQL types) and are deferred to separate opsx cycles. Design notes are recorded inopenspec/changes/add-aggregation-enhancements/design.mdso the next pickup has the API decisions documented without re-running the discovery work.Depends on
add-time-bucket-aggregation) — this PR extends the runner that lands there. Until feat(aggregation): ad-hoc time-bucket aggregation across REST + GraphQL #1611 merges, the diff here will include feat(aggregation): ad-hoc time-bucket aggregation across REST + GraphQL #1611's commits; the diff collapses naturally once feat(aggregation): ad-hoc time-bucket aggregation across REST + GraphQL #1611 lands and this branch rebases.API decisions
backendis now one ofpostgres/mysql/sqlite/php-fallback(waspostgres/php-fallback). Additive — clients that branch on"postgres"keep working; clients that previously assumed"php-fallback" === !postgreslearn three new values they can also treat as "native fast path".agg:{registerSlug}:{schemaSlug}:adhoc:{sha1(query.toArray())}:{filterHash}:{rbacHash}. Theadhoc:prefix keeps ad-hoc and named entries visually distinct in cache dumps without partitioning the backing store.AggregationCache::TTLconstant.AggregationCacheInvalidationListener. The existing globalICache::clear()on lifecycle events covers ad-hoc entries because they share the cache namespace.AggregationQuery::toArray()ksort-sorts both the top-level filter map and every operator sub-array ({gt, lte}and{lte, gt}hash identically). Cache hit rate stays high under client filter-key reordering.What ships
openspec/changes/add-aggregation-enhancements/tasks.md).AggregationQueryTest.php+AggregationCacheTest.php. All 113 aggregation unit tests pass.docs/technical/aggregation-api.md(backend matrix + cache section) andopenspec/platform-capabilities.md(aggregations row).What stays deferred
Each gets a design-only note in
design.md. No new GitHub issues to file — the existing #1606..#1608 are the long-running trackers.Test plan
/timeseriesendpoint twice against the dev container's MySQL service profile, confirmbackend: "mysql"andcached: trueon the second call.cached: false(eviction listener fired).