Skip to content

feat(aggregation): native MySQL/SQLite bucketing + ad-hoc cache (#1609, #1610)#1646

Merged
rubenvdlinde merged 4 commits into
developmentfrom
feat/add-aggregation-enhancements
May 22, 2026
Merged

feat(aggregation): native MySQL/SQLite bucketing + ad-hoc cache (#1609, #1610)#1646
rubenvdlinde merged 4 commits into
developmentfrom
feat/add-aggregation-enhancements

Conversation

@rubenvdlinde
Copy link
Copy Markdown
Contributor

Summary

Picks up two of the five deferred follow-ups filed against the time-bucket aggregation primitive (#1611):

  • MySQL / SQLite native time-bucket aggregation #1609 — Native MySQL/SQLite time-bucket. AggregationRunner::tryNativeAggregation() now detects MySQL/MariaDB and SQLite alongside PostgreSQL and emits the matching native bucketing expression (DATE_FORMAT / strftime) with the correct identifier quoting per engine (backticks on MySQL, double-quotes on Postgres/SQLite). ISO-Monday week-start carries across all three platforms; quarter buckets use the matching CASE / CONCAT form. The response backend field reports mysql / sqlite / postgres / php-fallback so callers can observe which path served the request.
  • Cache ad-hoc time-bucket aggregations in AggregationCache #1610 — Read-through cache for ad-hoc queries. AggregationCache gains getAdhoc() / setAdhoc() wrappers that derive the cache name slot from 'adhoc:'.sha1(json_encode(AggregationQuery::toArray())). AggregationQuery::toArray() ksort-sorts filter sub-arrays so {a, b} and {b, a} hash identically. runAdhoc() wires the cache read-through; the existing AggregationCacheInvalidationListener covers ad-hoc entries unchanged because they share the underlying openregister_aggregations distributed cache.

The remaining three follow-ups (#1606 multi-field groupBy, #1607 cumulative/rolling windows, #1608 multi-metric) require a value-object refactor on AggregationQuery plus knock-on translator changes (Solr / ES / Postgres / PHP fallback / GraphQL types) and are deferred to separate opsx cycles. Design notes are recorded in openspec/changes/add-aggregation-enhancements/design.md so the next pickup has the API decisions documented without re-running the discovery work.

Depends on

API decisions

  • Backend annotation surface. Response backend is now one of postgres / mysql / sqlite / php-fallback (was postgres / php-fallback). Additive — clients that branch on "postgres" keep working; clients that previously assumed "php-fallback" === !postgres learn three new values they can also treat as "native fast path".
  • Cache key derivation. Ad-hoc cache key is agg:{registerSlug}:{schemaSlug}:adhoc:{sha1(query.toArray())}:{filterHash}:{rbacHash}. The adhoc: prefix keeps ad-hoc and named entries visually distinct in cache dumps without partitioning the backing store.
  • Cache TTL. 60 s, same as the existing named-aggregation cache. Shares the AggregationCache::TTL constant.
  • Invalidation surface. No changes to AggregationCacheInvalidationListener. The existing global ICache::clear() on lifecycle events covers ad-hoc entries because they share the cache namespace.
  • Filter stability. AggregationQuery::toArray() ksort-sorts both the top-level filter map and every operator sub-array ({gt, lte} and {lte, gt} hash identically). Cache hit rate stays high under client filter-key reordering.

What ships

  • 5 of 5 task sections done (see openspec/changes/add-aggregation-enhancements/tasks.md).
  • 9 new tests across the 2 new test files; 7 new tests added to existing AggregationQueryTest.php + AggregationCacheTest.php. All 113 aggregation unit tests pass.
  • Quality gates clean on touched lib files: PHPCS strict / PHPMD strict / Psalm strict / PHPStan level 8.
  • Docs updated in docs/technical/aggregation-api.md (backend matrix + cache section) and openspec/platform-capabilities.md (aggregations row).

What stays deferred

Each gets a design-only note in design.md. No new GitHub issues to file — the existing #1606..#1608 are the long-running trackers.

Test plan

  • CI green on the new branch (PHPCS / PHPMD / PHPStan / Psalm / PHPUnit).
  • After feat(aggregation): ad-hoc time-bucket aggregation across REST + GraphQL #1611 lands and this branch rebases, run the /timeseries endpoint twice against the dev container's MySQL service profile, confirm backend: "mysql" and cached: true on the second call.
  • Insert a row into the test schema, call once more, confirm cached: false (eviction listener fired).
  • Repeat against SQLite via env override on the same dev container.

rubenvdlinde and others added 3 commits May 21, 2026 13:58
Adds the OpenSpec change scaffold proposing an ad-hoc time-bucket aggregation
primitive across both REST and GraphQL.

- proposal.md: motivates the primitive (every dashboard needs it; openconnector
  rebuild blocked on it; AggregationQuery.dateBucket already shipped but not
  wired to Postgres/REST/GraphQL). kind: code, no schema register patches.
- design.md: decisions D1-D9 — reuse AggregationQuery, date_trunc native path,
  field allow-listing via Schema::getProperties(), dedicated
  /aggregate/timeseries route, GroupByInput on every list query,
  AggregationRunner::runAdhoc() shared by REST+GraphQL.
- specs/aggregation-api/spec.md: new capability for ad-hoc bucketing —
  validation rules, RBAC integration, Postgres+PHP-fallback contract.
- specs/graphql-api/spec.md: delta — adds groupBy/TimeInterval/GroupBucket
  types + modifies the connection-type requirement.
- tasks.md: 6 task groups covering runner, REST, GraphQL, tests, quality
  gates, and docs. No PR/merge tasks (Hydra coordination owns that).

Complementary to the in-flight aggregations-backend-native change which
already plumbed dateBucket into Solr+ES translators but not Postgres/REST/
GraphQL.
Implements the `add-time-bucket-aggregation` opsx change: an ad-hoc,
client-controlled aggregation primitive that lets dashboards bucket
register-schema rows by a field (categorical) or by a `date_trunc`
interval with optional `from`/`to` bounds, available on both REST and
GraphQL with the same response shape.

Why
- Every Conduction dashboard chart needs the same primitive ("group
  rows by day-of-`created` between from and to, return {key, value}").
- openconnector dashboard rebuild (6 charts) was blocked on it.
- nextcloud-vue CnChartWidget.dataSource can now grow its `bucket: {
  field, interval, fromVar, toVar }` shorthand.
- AggregationQuery.dateBucket was shipped by aggregations-backend-native
  but never plumbed to Postgres / REST / GraphQL.

Backend
- AggregationRunner: extended tryNativeAggregation() with a dateBucket
  parameter that emits `date_trunc(?, "$field")::text AS bucket` SQL
  with explicit `WHERE field >= start AND field < end` bounds. ISO
  week (Monday) + quarter (first-of-Q1-month) computed explicitly to
  match Postgres semantics.
- New public methods runAdhoc(Register, Schema, AggregationQuery) and
  runAdhocByRef(string, string, AggregationQuery) for the controller /
  resolver entry points. RBAC + multi-tenant gates inherited from
  the existing native path.
- bucketInPhp() — PHP fallback for non-Postgres DBs (SQLite tests,
  MySQL dev) with truncateTimestamp() polyfill keyed on the gap
  vocabulary. backend: "php-fallback" annotated in response.
- coerceBucketKey() produces stable Y-m-d\TH:i:s\Z strings across all
  paths so the wire shape stays identical.

REST
- New TimeseriesRequestValidator class — separate testable validator
  shared between REST + GraphQL. Validates field allow-list against
  Schema::getProperties() + metadata cols (_created, _updated,
  _deleted_at); validates sub-day intervals (MINUTE / HOUR) require
  format: date-time; validates interval/from/to/metric/metricField
  shape; constructs the AggregationQuery via the existing factory.
- AggregationController::timeseries(register, schema) action. 400/
  403/404 translation. New route /api/objects/aggregations/{register}/
  {schema}/timeseries ordered before the {name} wildcard.

GraphQL
- TypeMapperHandler grew lazy factories for GroupByInput, TimeInterval
  enum, AggregationMetric enum, and GroupBucket object type. Each
  cached on a private property so introspection-cycle and per-request
  schema generation pay the construction cost once.
- getListArgs() gains `groupBy: GroupByInput`; getConnectionType()
  gains `groups: [GroupBucket!]` (nullable — null = client did not
  request).
- GraphQLResolver::resolveList() now calls resolveGroupBy() when
  args.groupBy is present. Reuses TimeseriesRequestValidator for the
  same allow-list + sub-day rules. Validation / RBAC failures surface
  as GraphQL field-errors on `groups` — the rest of the connection
  still resolves.

Tests
- 12 TimeseriesRequestValidatorTest cases covering every spec-defined
  400 path.
- 5 AggregationControllerTimeseriesTest cases covering 404 / 400 / 403
  translations + happy path body parity.
- AggregationControllerTest constructor updated to inject the new
  validator dependency.
- Full Unit Tests suite (12 243 tests, 25 991 assertions) green inside
  the dev container.

Quality gates
- PHPCS strict — clean on all touched files.
- PHPMD — clean (targeted SuppressWarnings on bucketInPhp, resolveList,
  validate for legitimate branching-density cases).
- Psalm — clean.
- PHPStan level 8 — clean. Baseline reclassified the existing `string
  === null` defensive check to `int === null` after refactoring three
  call sites away from the redundant `instanceof` ternary.

Docs
- docs/technical/aggregation-api.md covers both surfaces, validation
  rules, status codes, Postgres index recommendations, and the
  decision matrix between named-annotation and ad-hoc primitive.
- openspec/platform-capabilities.md grew the ad-hoc primitive
  companion sentence on the aggregations row.

Non-goals deferred to issues
- #1606 multi-field groupBy
- #1607 cumulative / running aggregates
- #1608 multi-metric in one request
- #1609 native MySQL / SQLite bucketing
- #1610 caching ad-hoc queries
#1610)

Extends the time-bucket primitive added by #1611 with two of the five
deferred follow-ups:

- **#1609 — Native MySQL/SQLite bucketing.** `AggregationRunner` now
  detects MySQL/MariaDB and SQLite alongside PostgreSQL and emits the
  matching native bucketing expression (`DATE_FORMAT` / `strftime`) with
  the correct identifier quoting per engine. ISO-Monday week-start
  semantics carry across all three platforms; quarter buckets use the
  matching CASE / CONCAT form. Backend annotation reports `mysql` /
  `sqlite` / `postgres` / `php-fallback`.

- **#1610 — Read-through cache for ad-hoc queries.** `AggregationCache`
  gains `getAdhoc()` / `setAdhoc()` wrappers that derive the cache name
  slot from `'adhoc:'.sha1(json_encode(AggregationQuery::toArray()))`.
  `AggregationQuery::toArray()` ksort-sorts filter sub-arrays so two
  structurally-equivalent queries hash identically. `runAdhoc()` wires
  the cache read-through; the existing `AggregationCacheInvalidationListener`
  covers ad-hoc entries unchanged because they share the underlying
  `openregister_aggregations` distributed cache.

The remaining three follow-ups (#1606 multi-field groupBy, #1607
cumulative/rolling windows, #1608 multi-metric) are documented at
design depth in `openspec/changes/add-aggregation-enhancements/design.md`
so the next pickup can file deliberate opsx cycles for them.

Tests cover MySQL/SQLite/Postgres SQL emission per gap (including the
`%i` vs `%M` minute placeholder divergence), ad-hoc cache hit/miss
semantics, key stability under filter reordering, and ad-hoc vs named
cache-entry isolation.
@rubenvdlinde rubenvdlinde added ready-for-code-review Build complete — awaiting code reviewer ready-for-security-review Code review complete — awaiting security reviewer labels May 22, 2026
…gation-enhancements

# Conflicts:
#	phpstan-baseline.neon
@rubenvdlinde rubenvdlinde merged commit f0259b5 into development May 22, 2026
1 check failed
@rubenvdlinde rubenvdlinde deleted the feat/add-aggregation-enhancements branch May 22, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-code-review Build complete — awaiting code reviewer ready-for-security-review Code review complete — awaiting security reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant