Skip to content

Releases: zkrebbekx/flexitype

v1.3.0

Choose a tag to compare

@github-actions github-actions released this 28 Jul 13:41
a44620a

Twenty defects an independent reviewer raised against the 1.2 line, closed in
six pull requests. Two were P0: a right-to-erasure purge that could report
success with the data still present, and a change-set publish that a cancelled
request stranded with no API able to move it.

It is a minor release rather than a patch because the fixes add public API:
Service.Context, uow.AccessFromPermissions, changeset.ClaimReclaimer and
changeset.PublishClaimTTL, computed.Materializer.OnFormulaError,
ratelimit.Limiter.Refund, and formula.Members with EvalWithMembers,
EvalRatWithMembers and NumericRefs. Nothing existing changed shape.

BREAKING — check before upgrading

Six behaviour changes are visible from outside. Each is the documented
contract being honoured, and two of them can refuse a request that previously
succeeded — the carve-out docs/api-stability.md states.

  • FLEXITYPE_TIMEZONE now reaches rule evaluation. It previously reached
    nothing, so every today/now dependency rule and dynamic default resolved
    in UTC whatever the setting said. A deployment that set it will see
    date-boundary rules fire on the configured day — which is the point, and is
    a change from what it did yesterday. An embedder must pass its context
    through Service.Context (see docs/embedding.md); the API does it in
    middleware.
  • A formula may only read or fold a numeric attribute. (Refuses what
    previously succeeded.)
    sum, avg, min and max over a string,
    date, json or other non-numeric source, and a bare read of one, are
    refused with 422 at create and update. They previously materialized 0 or
    nothing at all. count() accepts any type, because it folds members. A
    stored formula is not re-validated, so an existing definition keeps working
    until it is next written — the materializer reports it through the
    background-error observer instead.
  • A purge that cannot make progress reports an error. (Refuses what
    previously "succeeded".)
    Completion is decided by a count of the remaining
    rows, not by a chunk that removed nothing, so an erasure that leaves rows
    behind fails loudly instead of returning a receipt.
  • CSV multi-value cells carry a #flexitype-values: prefix. A tool that
    parses an export must handle it. Import still accepts both earlier formats
    for non-json columns, so old files load unchanged.
  • The pre-authentication rate limiter counts failed authentications, not
    every request. A deployment behind a proxy will stop seeing 429 on healthy
    authenticated traffic.
  • leaseWait for migrations is 17 minutes, up from 10, so a runner waits
    out an abandoned lease instead of exiting before it can expire.

Fixed — A migration that is interrupted recovers, and an abandoned lease frees itself

  • A no-transaction migration reaps its own invalid indexes before it
    replays.
    A failed CREATE INDEX CONCURRENTLY leaves an INVALID index
    behind. The name then exists, so IF NOT EXISTS skipped the rebuild for
    ever — and migration 000028 also drops the index it replaces, so a replay
    removed the working index and left only the invalid one. Every by-entity
    revision read then sequentially scanned the table, permanently, while the
    schema version read as current and nothing raised an error. The trigger is
    an ordinary pod lifecycle event during a deploy: an OOMKill, an eviction or
    a lock timeout part-way through the build. The reap covers every
    no-transaction migration (000018, 000021 and 000025 build indexes
    concurrently too), and is a no-op when nothing is invalid.
  • leaseWait now exceeds leaseTTL. A runner that dies holding the
    migration lease leaves a row with a live expires_at that nobody renews.
    With a 10-minute wait against a 15-minute TTL, every replica booting inside
    that window gave up BEFORE the lease could expire — inside a container
    startup path, so the orchestrator restarted it and it repeated, and a whole
    generation could not boot. The wait is the TTL plus a two-minute margin, so
    a survivor outlasts the abandoned lease.
  • The acquire failure names the blocker. It reports the current holder and
    the expires_at it is waiting on, so an operator can correlate the stall
    with the pod that died.
  • The lease is released on a detached context. The commonest reason
    Migrate returns is that the caller's context ended, and releasing on that
    same context was a no-op — so an embedded deployment with a cancellable
    startup context stranded the lease for the full TTL while the process was
    alive and able to free it.

Fixed — Four residual one-sided fixes: the CSV multi-value marker, the redrive ramp, saved-view locking and two documents

  • The multi-value CSV marker is out of band. Any in-band sentinel drawn
    from the JSON grammar can be forged by a JSON payload: the format was a bare
    array of {"value",…} objects, and retagging it as {"values":[…]} moved
    WHICH documents collide rather than whether they can — the tagged shape is
    exactly what an export of a json column looks like, so re-importing this
    tool's own output read one document as two members, wrote both to a
    single-valued attribute, kept the last, and reported one row written with
    zero errors. A cell is now marked with a #flexitype-values: prefix, which
    no JSON document can begin with, so a multi-valued json column round-trips
    too. Both legacy forms are still accepted on import for non-json columns.
  • The redrive ramp is per subscription. ClaimDue takes a subscription's
    single lowest-feed_seq pending row, and only if that row is due, so a
    per-row random offset was head-of-line blocking rather than smoothing:
    measured over 20 revived rows, the head drew +4m26s and nothing was
    claimable for four and a half minutes while 19 later deliveries were
    already due. The offset is now derived from the subscription id — identical
    for every row of one subscription, spread across the window between
    subscriptions.
  • Saved-view optimistic locking is reachable from a client. PATCH
    accepts an optional version: send the one you read and a view someone else
    edited answers 409 instead of being overwritten. Patch re-read the view
    microseconds before writing it, so two users editing the same view both
    passed their own check and the second silently discarded the first. Omitting
    version keeps last-write-wins. The field is in the OpenAPI document and in
    the Go client (SavedViewPatch.Version, SavedView.Version).
  • Two documents now describe the code. docs/design/identity.md no longer
    carries the pre-fix bullet calling a deleted role "the safe direction" — the
    belief the fail-open guard exists to refuse — and both it and
    docs/configuration.md state that auth-cache eviction is PER PROCESS: a
    multi-replica deployment converges over FLEXITYPE_AUTH_CACHE_TTL, so
    during an incident the TTL, not the API response, is when a revocation is in
    force everywhere.

Fixed — Five one-sided guards: DSN quoting, the pre-auth limiter, the GraphQL schema key, the effective-permissions view and the time zone

  • The TLS guard parses the connection string with libpq's grammar. The
    keyword form was split on whitespace and cut at the first =, so a
    single-quoted value kept its quotes: sslmode='disable' was compared as
    'disable', matched nothing and passed a guard that exists to refuse
    exactly that — while lib/pq honoured the quoted form and connected in
    cleartext. host and hostaddr were evaluated unstripped for the same
    reason. Quoted values, backslash escapes and spaces around = are now read
    as libpq reads them.
  • The pre-authentication limiter charges failed authentications, not
    traffic.
    A token is taken up front and refunded unless the response is
    401. Charging every request made the shipped 20 rps default a ceiling on
    ALL traffic behind an ingress or a Cluster-policy LoadBalancer, where
    every request appears to come from one address: healthy authenticated
    clients, well inside their per-account and per-tenant budgets, got 429.
    pkg/ratelimit gains Limiter.Refund.
  • The GraphQL schema-cache key covers Access.Default. An empty Attr
    map signed as "open" whatever the default was, so uow.DenyAll() — what
    an account naming a deleted role resolves to — collided with an
    unrestricted principal. Whichever arrived first warmed the cache for both:
    a deny-all caller was served the unrestricted schema, disclosing every
    restricted attribute name, or an unrestricted caller was served an empty
    one and every query failed. The key is now a hash of the whole policy.
  • The effective-permissions view reports what is enforced. Enforcement
    ignores the merged field-permission map when the account holds admin, and
    ignores it entirely when a role is unresolved, so a reviewer could read
    salary: none off an account with unrestricted field access. The view
    gains field_acl_bypassed and denied_all, both derived by the same
    function the request path uses (uow.AccessFromPermissions), and the
    console shows what applies instead of the map.
  • FLEXITYPE_TIMEZONE reaches rule evaluation. Service.Interactors
    stamped the zone onto a context it then discarded, and the HTTP middleware
    never stamped it at all — an interactor set carries no context of its own,
    so every today/now rule and dynamic default resolved in UTC. The API
    stamps it in middleware; the background loops stamp it too; and embedders
    pass their context through the new Service.Context first.

Fixed — Aggregates: reserved words, non-numeric sources, integer precision and a guard that skipped half its rule

Five defects introduced with the aggregate feature, all of which produced a
plausible number rather than an error.

  • The five aggregate names are no longer reserved words. count, sum,
    min, `max...
Read more

v1.2.0

Choose a tag to compare

@zkrebbekx zkrebbekx released this 18 Jul 07:37
0fb01d8

Three places where the API contradicted itself, corrected. Released as a minor rather than a major: in each case the documented contract was already the new behaviour, so the old behaviour was the defect being fixed.

Upgrade note: the first item below can reject requests that previously succeeded. Check it before upgrading.

Declared numeric types are enforced, not coerced

An integer or float attribute now rejects a quoted number ("5", "1.5") with 422 VALIDATION. Previously encoding/json unmarshalled these straight into a json.Number and they were accepted, so the declared type meant nothing at the boundary.

decimal is deliberately unchanged — it accepts a string form on purpose, to carry exact precision without float rounding.

Clients sending quoted numerics to integer/float attributes must send bare JSON numbers.

DELETE of a missing unit family or match rule is 404, not 204

These two were the only by-id routes reporting success for a record that was never there; saved views, service-account revoke, unlink, value removal and dependency archival all already returned 404. The existence check is tenant-scoped, so another tenant's record stays indistinguishable from a missing one.

Provisioning routes authorize before reporting the feature gate

A caller without the admin scope now gets 403 rather than 501 FEATURE_DISABLED, so an unauthorized caller cannot learn whether provisioning is configured in this deployment. Matches the order the other protected routes already use.


Full changelog: https://github.com/zkrebbekx/flexitype/blob/v1.2.0/CHANGELOG.md
Compare: v1.1.0...v1.2.0

Verify a download against flexitype_1.2.0_checksums.txt.

v1.1.0

Choose a tag to compare

@github-actions github-actions released this 18 Jul 04:06
8bb1e14

A post-1.0 independent review (security, architecture, performance and
coding-standards) plus follow-ups — every issue implemented and merged. The
REST API (/api/v1), the storage schema (forward-only migrations), and the
supported Go facade stay backward-compatible; changes to them are additive. The
only breaking changes are to unsupported Go internals — see below.

Security

  • Media download is tenant-scoped — an object key is served only to the
    tenant that owns it; a mismatch is a 404, so ownership is not probeable
    (was a cross-tenant IDOR).
  • Field-level ACL now covers relationship (link) attributes in the FQL
    binder, closing a binary-search value oracle on restricted link attributes.
  • CSV export neutralises formula injection — a cell starting with =, +,
    -, @, tab or CR is quoted so spreadsheets treat it as text (CWE-1236).
  • Response hardening — media downloads force Content-Disposition: attachment + nosniff + a content-free CSP; a middleware sets a restrictive
    Content-Security-Policy (script-src pinned to 'self' + the console's
    hashed inline theme script, no 'unsafe-inline'), X-Frame-Options: DENY,
    nosniff and Referrer-Policy: no-referrer on every response.
  • Bootstrap admin hardened — the existence check fails closed (a transient
    error no longer mints a fresh credential) and the token is printed to stdout
    once, never through the structured logger. The admin scope is documented as
    a global platform-operator privilege.
  • GraphQL queries have a field-count and execution-time budget; request bodies
    are capped.

Added

  • POST /api/v1/computed/recompute and Service.RecomputeComputed — rebuild a
    tenant's computed attributes (the recovery counterpart to search/reindex).
  • Quantity data type in the admin console — a magnitude + unit editor and
    {magnitude} {unit} rendering; the DataType union now covers quantity.
  • WithCleanupObserver (surface swallowed post-erasure cleanup failures) and a
    context-aware AuthenticatorCtx extension point (the credential lookup now
    honours the request's cancellation/deadline/trace).
  • The first-party Go client defaults to a 30s per-request HTTP timeout.

Performance

  • Entity-summary projection (trigger-maintained) turns entity-list and the
    FQL enumeration base into a bounded keyset index scan instead of
    re-aggregating every value row per page (~313 ms → ~0.3 ms at 200k entities;
    constant rather than linear in entity count).
  • Replica-safe GraphQL schema cache — a persisted per-tenant
    schema_version (trigger-maintained) drives invalidation across replicas,
    with a short memo and an LRU bound.
  • Windowed GraphQL nested connections — each parent fetches only first+1
    children via a row_number() window with the definition filter pushed into
    SQL, routed through a dataloader (no N+1, no full materialisation).
  • Coalesced per-commit search/computed projection maintenance; chunked CSV
    import; a shared-trigram inverted index for duplicate detection; supporting
    indexes for webhook delivery, decimal comparisons and attribute-value scans.
  • A //go:build stress harness seeds up to 10M entities and profiles CRUD /
    FQL / GraphQL.

Changed

  • Internal projections (computed attributes, search index, GraphQL schema
    cache) are maintained in the originating unit of work's post-commit, in both
    delivery modes
    — so a write's own computed values and matches() results
    are visible to that request (read-your-writes) independent of WithOutbox.
    The external event dispatcher is reserved for consumer hooks. A projection
    failure is surfaced to WithDispatchObserver, never silently swallowed;
    recover post-crash staleness with search/reindex / computed/recompute.
  • Erasure is atomic and honest — the revision purge joins the value
    transaction, projection removal uses one consistent post-commit policy, and
    the PurgeReport counts only confirmed blob deletions (new MediaBlobsFailed
    / UnpurgedBlobKeys).
  • Usecase timestamps are normalised to UTC in one place.

Fixed

  • In-memory backend transaction isolation — a per-transaction undo journal
    replaces the whole-store snapshot, so interleaved transactions no longer
    clobber each other's committed writes, and a write is O(touched keys).

Breaking — Go library embedders only

No impact on REST/CLI/Docker consumers, the storage schema, or the supported Go
facade. The supported public Go surface is now explicitly the flexitype
facade, the client module, and the documented extension ports
(events.Handler/Publisher, blob.Store, serviceaccount.Authenticator,
db.Transactor); everything else is internal, with no compatibility promise
(see API stability).

  • Deployment plumbing moved from pkg/ to internal/: config, shutdown,
    telemetry, safedial.
  • application/* and domain/* internals were restructured: an appctx leaf
    package breaks the application-root dependency cycle; an erasure.Interactor
    owns the purge flow and the value interactor's setter injection is replaced
    by constructor config; the domain repository ports were slimmed and the SQL
    executor removed from domain signatures (an opaque db.Tx marker).

Internal

  • Full FQL parity corpus and PostgreSQL behavioural parity test coverage across
    the previously memory-only suites; a completeness guard fails CI if a new FQL
    construct is left uncovered.

v1.0.0

Choose a tag to compare

@github-actions github-actions released this 12 Jul 15:28
93fb365

First stable release. The full feature set is verified against PostgreSQL 16
and covered by the test suite (both the Postgres and in-memory backends, with a
cross-backend FQL parity corpus). SemVer applies from this release.

Added

  • Soft types & attributes — runtime-defined TypeDefinition
    AttributeDefinitionAttributeValue over an opaque entity_id, with 14
    data types and constraints (min/max length, min/max value, RE2 pattern,
    one-of; required / multi-valued / unique flags).
  • Attribute dependencies — cascading picklists and conditional validation
    (equals / in / range / pattern / dynamic-time), resolved as a per-entity
    effective schema.
  • Type inheritance — single-inheritance hierarchies with hierarchy-wide
    no-shadowing, subtype-anchored values, and cross-level dependencies and
    relationships.
  • Relationships — user-defined directed (parent/child, role labels,
    per-side version pinning) and symmetric (unordered peer) relationship types,
    each with their own attributes, constraints, definition inheritance, and
    cardinality limits.
  • Localized & channel-scoped values — a value can vary per locale and
    channel; uniqueness and FQL filtering apply per scope, and a query can pin a
    scope.
  • Computed attributes — read-only attributes derived from a formula over an
    entity's other values, materialized as ordinary FQL-queryable values that
    stay in sync via an event subscriber (with dependency-cycle rejection).
  • Units of measure — quantity attributes backed by tenant unit families;
    values convert to a base unit for comparison (exact rational conversion) with
    the original unit preserved for display, and FQL accepts unit suffixes.
  • Media attributes — file values backed by a pluggable blob store (local
    disk or S3-compatible), with sniffed-MIME and size constraints and
    garbage-collection of superseded/erased blobs.
  • Entity revisions — immutable point-in-time snapshots with as-of reads,
    diff, and restore (scope-aware); history is never mutated.
  • Change management — draft → review → approve → publish change-sets with
    separation-of-duties approval and scheduled publishing.
  • Duplicate detection — per-type match rules (exact, case-insensitive,
    trigram) producing scored, dismissable candidate pairs, scored identically on
    both backends.
  • Faceted grid & saved views — attribute-column projection (no N+1), value
    facets over the current result set, and persisted views.
  • CSV import/export — column-mapped import with dry-run and best-effort /
    transactional modes (required fields enforced); export honours the active FQL
    query.
  • Schema templates & cloning — a lossless portable schema bundle, type
    cloning, and curated go:embedded starter templates.
  • FQL — a schema-aware query language (comparisons, in, range, has,
    length, min/max/count, case-sensitive and insensitive string
    matching, boolean nesting with three-valued NULL logic, type isa,
    child()/parent()/linked() traversals, matches() full-text) executed
    identically over PostgreSQL and the in-memory store.
  • Read-only GraphQL API — a Relay-connection schema generated from the live
    type definitions (edges/node/cursor, pageInfo, on-demand totalCount, FQL
    filter argument), ACL-filtered and free of N+1 loads.
  • Keyset pagination — every listing uses cursor pagination stable under
    concurrent inserts and deletes, with on-demand total counts.
  • Field-level access control — per-attribute read/write permissions on
    service accounts, enforced through the value read/write paths, effective
    schema, grid/facets/export, and the FQL binder (an unreadable attribute is
    invisible, not leaked).
  • Data erasure — audited, admin-scoped hard purge of an entity's or a
    tenant's data (values, revisions, links, media blobs) for right-to-erasure
    compliance.
  • Domain events & delivery — a typed dispatcher fanning a stable JSON
    envelope to consumer hooks; a transactional outbox with gap-free feed
    sequencing, managed HMAC-signed webhook subscriptions (backoff,
    dead-lettering, redrive, SSRF guard), a cursor-paged events feed with SSE
    tail and CAS cursors, and a Google Cloud Pub/Sub publisher.
  • Activity log — every change audited with JSON before/after descriptors,
    written in the same transaction as the change.
  • Search index (optional) — an event-driven per-entity projection powering
    FQL matches(), with trigram-accelerated contains.
  • Admin console — a Vue 3 SPA for modelling types, attributes, dependencies
    and relationships; browsing entities with dependency-aware editing; import,
    revisions, change-sets, duplicates, the faceted grid, a GraphQL explorer, and
    operations — bearer-token sign-in, keyboard-accessible and responsive.
  • WebAssembly playground — the whole service compiled to WASM over the
    in-memory store, hosted on GitHub Pages.
  • First-party Go client SDKgithub.com/zkrebbekx/flexitype/client, a
    standard-library-only module mirroring the embedded usecase surface over
    REST, conformance-tested against the real handler.
  • OpenAPI 3 contract — the complete REST surface documented at
    api/openapi.yaml and served at /api/v1/openapi.{json,yaml}, with a CI
    route-coverage guard.
  • Deployment shapes — embedded Go library and standalone service (versioned
    REST API, service-account auth with runtime provisioning, OpenTelemetry,
    Prometheus metrics, rate limiting, health endpoints), multi-tenant from day
    one, shipped as static binaries and a GHCR container image.

Security

  • Tenant isolation enforced on every by-ID interactor path.
  • Field-level ACL enforced across grid, facets and CSV export (not only the
    single-entity read path).
  • FLEXITYPE_REQUIRE_AUTH refuses to boot without an account source; the
    service stamps the principal's access explicitly rather than defaulting open.
  • FQL parser recursion and query size are bounded (a deeply nested query
    returns a validation error rather than crashing the process).
  • Media uploads are validated against the sniffed content type, not the
    client-declared one.
  • The webhook SSRF guard validates the actual connect-time IP via the dialer
    control hook, closing a DNS-rebinding window; it blocks private, loopback,
    link-local and cloud-metadata targets, overridable for on-prem.
  • sslmode=disable is refused for a non-loopback database host.

Fixed

  • Revision restore/diff preserve locale/channel scope instead of collapsing
    scoped values onto the base value.
  • In-memory keyset pagination compares cursors by value, staying stable when
    the cursor row is updated or deleted between pages.
  • Decimal and JSON uniqueness compare numerically / structurally on both
    backends (Postgres no longer admits 1.5 vs 1.50 as distinct).
  • Committed writes are not failed by a post-commit subscriber error in the
    default delivery mode.
  • Quantity one_of members and defaults are unit-rebased; equal quantities in
    different units compare equal.