Releases: zkrebbekx/flexitype
Release list
v1.3.0
Twenty defects an independent reviewer raised against the 1.2 line, closed in
six pull requests. Two were P0: a right-to-erasure purge that could report
success with the data still present, and a change-set publish that a cancelled
request stranded with no API able to move it.
It is a minor release rather than a patch because the fixes add public API:
Service.Context, uow.AccessFromPermissions, changeset.ClaimReclaimer and
changeset.PublishClaimTTL, computed.Materializer.OnFormulaError,
ratelimit.Limiter.Refund, and formula.Members with EvalWithMembers,
EvalRatWithMembers and NumericRefs. Nothing existing changed shape.
BREAKING — check before upgrading
Six behaviour changes are visible from outside. Each is the documented
contract being honoured, and two of them can refuse a request that previously
succeeded — the carve-out docs/api-stability.md states.
FLEXITYPE_TIMEZONEnow reaches rule evaluation. It previously reached
nothing, so everytoday/nowdependency rule and dynamic default resolved
in UTC whatever the setting said. A deployment that set it will see
date-boundary rules fire on the configured day — which is the point, and is
a change from what it did yesterday. An embedder must pass its context
throughService.Context(seedocs/embedding.md); the API does it in
middleware.- A formula may only read or fold a numeric attribute. (Refuses what
previously succeeded.)sum,avg,minandmaxover astring,
date,jsonor other non-numeric source, and a bare read of one, are
refused with422at create and update. They previously materialized0or
nothing at all.count()accepts any type, because it folds members. A
stored formula is not re-validated, so an existing definition keeps working
until it is next written — the materializer reports it through the
background-error observer instead. - A purge that cannot make progress reports an error. (Refuses what
previously "succeeded".) Completion is decided by a count of the remaining
rows, not by a chunk that removed nothing, so an erasure that leaves rows
behind fails loudly instead of returning a receipt. - CSV multi-value cells carry a
#flexitype-values:prefix. A tool that
parses an export must handle it. Import still accepts both earlier formats
for non-jsoncolumns, so old files load unchanged. - The pre-authentication rate limiter counts failed authentications, not
every request. A deployment behind a proxy will stop seeing429on healthy
authenticated traffic. leaseWaitfor migrations is 17 minutes, up from 10, so a runner waits
out an abandoned lease instead of exiting before it can expire.
Fixed — A migration that is interrupted recovers, and an abandoned lease frees itself
- A no-transaction migration reaps its own invalid indexes before it
replays. A failedCREATE INDEX CONCURRENTLYleaves an INVALID index
behind. The name then exists, soIF NOT EXISTSskipped the rebuild for
ever — and migration 000028 also drops the index it replaces, so a replay
removed the working index and left only the invalid one. Every by-entity
revision read then sequentially scanned the table, permanently, while the
schema version read as current and nothing raised an error. The trigger is
an ordinary pod lifecycle event during a deploy: an OOMKill, an eviction or
a lock timeout part-way through the build. The reap covers every
no-transaction migration (000018, 000021 and 000025 build indexes
concurrently too), and is a no-op when nothing is invalid. leaseWaitnow exceedsleaseTTL. A runner that dies holding the
migration lease leaves a row with a liveexpires_atthat nobody renews.
With a 10-minute wait against a 15-minute TTL, every replica booting inside
that window gave up BEFORE the lease could expire — inside a container
startup path, so the orchestrator restarted it and it repeated, and a whole
generation could not boot. The wait is the TTL plus a two-minute margin, so
a survivor outlasts the abandoned lease.- The acquire failure names the blocker. It reports the current holder and
theexpires_atit is waiting on, so an operator can correlate the stall
with the pod that died. - The lease is released on a detached context. The commonest reason
Migratereturns is that the caller's context ended, and releasing on that
same context was a no-op — so an embedded deployment with a cancellable
startup context stranded the lease for the full TTL while the process was
alive and able to free it.
Fixed — Four residual one-sided fixes: the CSV multi-value marker, the redrive ramp, saved-view locking and two documents
- The multi-value CSV marker is out of band. Any in-band sentinel drawn
from the JSON grammar can be forged by a JSON payload: the format was a bare
array of{"value",…}objects, and retagging it as{"values":[…]}moved
WHICH documents collide rather than whether they can — the tagged shape is
exactly what an export of ajsoncolumn looks like, so re-importing this
tool's own output read one document as two members, wrote both to a
single-valued attribute, kept the last, and reported one row written with
zero errors. A cell is now marked with a#flexitype-values:prefix, which
no JSON document can begin with, so a multi-valuedjsoncolumn round-trips
too. Both legacy forms are still accepted on import for non-jsoncolumns. - The redrive ramp is per subscription.
ClaimDuetakes a subscription's
single lowest-feed_seqpending row, and only if that row is due, so a
per-row random offset was head-of-line blocking rather than smoothing:
measured over 20 revived rows, the head drew +4m26s and nothing was
claimable for four and a half minutes while 19 later deliveries were
already due. The offset is now derived from the subscription id — identical
for every row of one subscription, spread across the window between
subscriptions. - Saved-view optimistic locking is reachable from a client.
PATCH
accepts an optionalversion: send the one you read and a view someone else
edited answers 409 instead of being overwritten.Patchre-read the view
microseconds before writing it, so two users editing the same view both
passed their own check and the second silently discarded the first. Omitting
versionkeeps last-write-wins. The field is in the OpenAPI document and in
the Go client (SavedViewPatch.Version,SavedView.Version). - Two documents now describe the code.
docs/design/identity.mdno longer
carries the pre-fix bullet calling a deleted role "the safe direction" — the
belief the fail-open guard exists to refuse — and both it and
docs/configuration.mdstate that auth-cache eviction is PER PROCESS: a
multi-replica deployment converges overFLEXITYPE_AUTH_CACHE_TTL, so
during an incident the TTL, not the API response, is when a revocation is in
force everywhere.
Fixed — Five one-sided guards: DSN quoting, the pre-auth limiter, the GraphQL schema key, the effective-permissions view and the time zone
- The TLS guard parses the connection string with libpq's grammar. The
keyword form was split on whitespace and cut at the first=, so a
single-quoted value kept its quotes:sslmode='disable'was compared as
'disable', matched nothing and passed a guard that exists to refuse
exactly that — while lib/pq honoured the quoted form and connected in
cleartext.hostandhostaddrwere evaluated unstripped for the same
reason. Quoted values, backslash escapes and spaces around=are now read
as libpq reads them. - The pre-authentication limiter charges failed authentications, not
traffic. A token is taken up front and refunded unless the response is
401. Charging every request made the shipped 20 rps default a ceiling on
ALL traffic behind an ingress or aCluster-policy LoadBalancer, where
every request appears to come from one address: healthy authenticated
clients, well inside their per-account and per-tenant budgets, got 429.
pkg/ratelimitgainsLimiter.Refund. - The GraphQL schema-cache key covers
Access.Default. An emptyAttr
map signed as"open"whatever the default was, souow.DenyAll()— what
an account naming a deleted role resolves to — collided with an
unrestricted principal. Whichever arrived first warmed the cache for both:
a deny-all caller was served the unrestricted schema, disclosing every
restricted attribute name, or an unrestricted caller was served an empty
one and every query failed. The key is now a hash of the whole policy. - The effective-permissions view reports what is enforced. Enforcement
ignores the merged field-permission map when the account holds admin, and
ignores it entirely when a role is unresolved, so a reviewer could read
salary: noneoff an account with unrestricted field access. The view
gainsfield_acl_bypassedanddenied_all, both derived by the same
function the request path uses (uow.AccessFromPermissions), and the
console shows what applies instead of the map. FLEXITYPE_TIMEZONEreaches rule evaluation.Service.Interactors
stamped the zone onto a context it then discarded, and the HTTP middleware
never stamped it at all — an interactor set carries no context of its own,
so everytoday/nowrule and dynamic default resolved in UTC. The API
stamps it in middleware; the background loops stamp it too; and embedders
pass their context through the newService.Contextfirst.
Fixed — Aggregates: reserved words, non-numeric sources, integer precision and a guard that skipped half its rule
Five defects introduced with the aggregate feature, all of which produced a
plausible number rather than an error.
- The five aggregate names are no longer reserved words.
count,sum,
min, `max...
v1.2.0
Three places where the API contradicted itself, corrected. Released as a minor rather than a major: in each case the documented contract was already the new behaviour, so the old behaviour was the defect being fixed.
Upgrade note: the first item below can reject requests that previously succeeded. Check it before upgrading.
Declared numeric types are enforced, not coerced
An integer or float attribute now rejects a quoted number ("5", "1.5") with 422 VALIDATION. Previously encoding/json unmarshalled these straight into a json.Number and they were accepted, so the declared type meant nothing at the boundary.
decimal is deliberately unchanged — it accepts a string form on purpose, to carry exact precision without float rounding.
Clients sending quoted numerics to integer/float attributes must send bare JSON numbers.
DELETE of a missing unit family or match rule is 404, not 204
These two were the only by-id routes reporting success for a record that was never there; saved views, service-account revoke, unlink, value removal and dependency archival all already returned 404. The existence check is tenant-scoped, so another tenant's record stays indistinguishable from a missing one.
Provisioning routes authorize before reporting the feature gate
A caller without the admin scope now gets 403 rather than 501 FEATURE_DISABLED, so an unauthorized caller cannot learn whether provisioning is configured in this deployment. Matches the order the other protected routes already use.
Full changelog: https://github.com/zkrebbekx/flexitype/blob/v1.2.0/CHANGELOG.md
Compare: v1.1.0...v1.2.0
Verify a download against flexitype_1.2.0_checksums.txt.
v1.1.0
A post-1.0 independent review (security, architecture, performance and
coding-standards) plus follow-ups — every issue implemented and merged. The
REST API (/api/v1), the storage schema (forward-only migrations), and the
supported Go facade stay backward-compatible; changes to them are additive. The
only breaking changes are to unsupported Go internals — see below.
Security
- Media download is tenant-scoped — an object key is served only to the
tenant that owns it; a mismatch is a404, so ownership is not probeable
(was a cross-tenant IDOR). - Field-level ACL now covers relationship (link) attributes in the FQL
binder, closing a binary-search value oracle on restricted link attributes. - CSV export neutralises formula injection — a cell starting with
=,+,
-,@, tab or CR is quoted so spreadsheets treat it as text (CWE-1236). - Response hardening — media downloads force
Content-Disposition: attachment+nosniff+ a content-free CSP; a middleware sets a restrictive
Content-Security-Policy (script-src pinned to'self'+ the console's
hashed inline theme script, no'unsafe-inline'),X-Frame-Options: DENY,
nosniffandReferrer-Policy: no-referreron every response. - Bootstrap admin hardened — the existence check fails closed (a transient
error no longer mints a fresh credential) and the token is printed to stdout
once, never through the structured logger. Theadminscope is documented as
a global platform-operator privilege. - GraphQL queries have a field-count and execution-time budget; request bodies
are capped.
Added
POST /api/v1/computed/recomputeandService.RecomputeComputed— rebuild a
tenant's computed attributes (the recovery counterpart tosearch/reindex).- Quantity data type in the admin console — a magnitude + unit editor and
{magnitude} {unit}rendering; theDataTypeunion now coversquantity. WithCleanupObserver(surface swallowed post-erasure cleanup failures) and a
context-awareAuthenticatorCtxextension point (the credential lookup now
honours the request's cancellation/deadline/trace).- The first-party Go client defaults to a 30s per-request HTTP timeout.
Performance
- Entity-summary projection (trigger-maintained) turns entity-list and the
FQL enumeration base into a bounded keyset index scan instead of
re-aggregating every value row per page (~313 ms → ~0.3 ms at 200k entities;
constant rather than linear in entity count). - Replica-safe GraphQL schema cache — a persisted per-tenant
schema_version(trigger-maintained) drives invalidation across replicas,
with a short memo and an LRU bound. - Windowed GraphQL nested connections — each parent fetches only
first+1
children via arow_number()window with the definition filter pushed into
SQL, routed through a dataloader (no N+1, no full materialisation). - Coalesced per-commit search/computed projection maintenance; chunked CSV
import; a shared-trigram inverted index for duplicate detection; supporting
indexes for webhook delivery, decimal comparisons and attribute-value scans. - A
//go:build stressharness seeds up to 10M entities and profiles CRUD /
FQL / GraphQL.
Changed
- Internal projections (computed attributes, search index, GraphQL schema
cache) are maintained in the originating unit of work's post-commit, in both
delivery modes — so a write's own computed values andmatches()results
are visible to that request (read-your-writes) independent ofWithOutbox.
The external event dispatcher is reserved for consumer hooks. A projection
failure is surfaced toWithDispatchObserver, never silently swallowed;
recover post-crash staleness withsearch/reindex/computed/recompute. - Erasure is atomic and honest — the revision purge joins the value
transaction, projection removal uses one consistent post-commit policy, and
thePurgeReportcounts only confirmed blob deletions (newMediaBlobsFailed
/UnpurgedBlobKeys). - Usecase timestamps are normalised to UTC in one place.
Fixed
- In-memory backend transaction isolation — a per-transaction undo journal
replaces the whole-store snapshot, so interleaved transactions no longer
clobber each other's committed writes, and a write is O(touched keys).
Breaking — Go library embedders only
No impact on REST/CLI/Docker consumers, the storage schema, or the supported Go
facade. The supported public Go surface is now explicitly the flexitype
facade, the client module, and the documented extension ports
(events.Handler/Publisher, blob.Store, serviceaccount.Authenticator,
db.Transactor); everything else is internal, with no compatibility promise
(see API stability).
- Deployment plumbing moved from
pkg/tointernal/:config,shutdown,
telemetry,safedial. application/*anddomain/*internals were restructured: anappctxleaf
package breaks the application-root dependency cycle; anerasure.Interactor
owns the purge flow and the value interactor's setter injection is replaced
by constructor config; the domain repository ports were slimmed and the SQL
executor removed from domain signatures (an opaquedb.Txmarker).
Internal
- Full FQL parity corpus and PostgreSQL behavioural parity test coverage across
the previously memory-only suites; a completeness guard fails CI if a new FQL
construct is left uncovered.
v1.0.0
First stable release. The full feature set is verified against PostgreSQL 16
and covered by the test suite (both the Postgres and in-memory backends, with a
cross-backend FQL parity corpus). SemVer applies from this release.
Added
- Soft types & attributes — runtime-defined
TypeDefinition→
AttributeDefinition→AttributeValueover an opaqueentity_id, with 14
data types and constraints (min/max length, min/max value, RE2 pattern,
one-of; required / multi-valued / unique flags). - Attribute dependencies — cascading picklists and conditional validation
(equals / in / range / pattern / dynamic-time), resolved as a per-entity
effective schema. - Type inheritance — single-inheritance hierarchies with hierarchy-wide
no-shadowing, subtype-anchored values, and cross-level dependencies and
relationships. - Relationships — user-defined directed (parent/child, role labels,
per-side version pinning) and symmetric (unordered peer) relationship types,
each with their own attributes, constraints, definition inheritance, and
cardinality limits. - Localized & channel-scoped values — a value can vary per locale and
channel; uniqueness and FQL filtering apply per scope, and a query can pin a
scope. - Computed attributes — read-only attributes derived from a formula over an
entity's other values, materialized as ordinary FQL-queryable values that
stay in sync via an event subscriber (with dependency-cycle rejection). - Units of measure — quantity attributes backed by tenant unit families;
values convert to a base unit for comparison (exact rational conversion) with
the original unit preserved for display, and FQL accepts unit suffixes. - Media attributes — file values backed by a pluggable blob store (local
disk or S3-compatible), with sniffed-MIME and size constraints and
garbage-collection of superseded/erased blobs. - Entity revisions — immutable point-in-time snapshots with as-of reads,
diff, and restore (scope-aware); history is never mutated. - Change management — draft → review → approve → publish change-sets with
separation-of-duties approval and scheduled publishing. - Duplicate detection — per-type match rules (exact, case-insensitive,
trigram) producing scored, dismissable candidate pairs, scored identically on
both backends. - Faceted grid & saved views — attribute-column projection (no N+1), value
facets over the current result set, and persisted views. - CSV import/export — column-mapped import with dry-run and best-effort /
transactional modes (required fields enforced); export honours the active FQL
query. - Schema templates & cloning — a lossless portable schema bundle, type
cloning, and curated go:embedded starter templates. - FQL — a schema-aware query language (comparisons,
in,range,has,
length,min/max/count, case-sensitive and insensitive string
matching, boolean nesting with three-valued NULL logic,type isa,
child()/parent()/linked()traversals,matches()full-text) executed
identically over PostgreSQL and the in-memory store. - Read-only GraphQL API — a Relay-connection schema generated from the live
type definitions (edges/node/cursor,pageInfo, on-demandtotalCount, FQL
filterargument), ACL-filtered and free of N+1 loads. - Keyset pagination — every listing uses cursor pagination stable under
concurrent inserts and deletes, with on-demand total counts. - Field-level access control — per-attribute read/write permissions on
service accounts, enforced through the value read/write paths, effective
schema, grid/facets/export, and the FQL binder (an unreadable attribute is
invisible, not leaked). - Data erasure — audited, admin-scoped hard purge of an entity's or a
tenant's data (values, revisions, links, media blobs) for right-to-erasure
compliance. - Domain events & delivery — a typed dispatcher fanning a stable JSON
envelope to consumer hooks; a transactional outbox with gap-free feed
sequencing, managed HMAC-signed webhook subscriptions (backoff,
dead-lettering, redrive, SSRF guard), a cursor-paged events feed with SSE
tail and CAS cursors, and a Google Cloud Pub/Sub publisher. - Activity log — every change audited with JSON before/after descriptors,
written in the same transaction as the change. - Search index (optional) — an event-driven per-entity projection powering
FQLmatches(), with trigram-acceleratedcontains. - Admin console — a Vue 3 SPA for modelling types, attributes, dependencies
and relationships; browsing entities with dependency-aware editing; import,
revisions, change-sets, duplicates, the faceted grid, a GraphQL explorer, and
operations — bearer-token sign-in, keyboard-accessible and responsive. - WebAssembly playground — the whole service compiled to WASM over the
in-memory store, hosted on GitHub Pages. - First-party Go client SDK —
github.com/zkrebbekx/flexitype/client, a
standard-library-only module mirroring the embedded usecase surface over
REST, conformance-tested against the real handler. - OpenAPI 3 contract — the complete REST surface documented at
api/openapi.yamland served at/api/v1/openapi.{json,yaml}, with a CI
route-coverage guard. - Deployment shapes — embedded Go library and standalone service (versioned
REST API, service-account auth with runtime provisioning, OpenTelemetry,
Prometheus metrics, rate limiting, health endpoints), multi-tenant from day
one, shipped as static binaries and a GHCR container image.
Security
- Tenant isolation enforced on every by-ID interactor path.
- Field-level ACL enforced across grid, facets and CSV export (not only the
single-entity read path). FLEXITYPE_REQUIRE_AUTHrefuses to boot without an account source; the
service stamps the principal's access explicitly rather than defaulting open.- FQL parser recursion and query size are bounded (a deeply nested query
returns a validation error rather than crashing the process). - Media uploads are validated against the sniffed content type, not the
client-declared one. - The webhook SSRF guard validates the actual connect-time IP via the dialer
control hook, closing a DNS-rebinding window; it blocks private, loopback,
link-local and cloud-metadata targets, overridable for on-prem. sslmode=disableis refused for a non-loopback database host.
Fixed
- Revision restore/diff preserve locale/channel scope instead of collapsing
scoped values onto the base value. - In-memory keyset pagination compares cursors by value, staying stable when
the cursor row is updated or deleted between pages. - Decimal and JSON uniqueness compare numerically / structurally on both
backends (Postgres no longer admits1.5vs1.50as distinct). - Committed writes are not failed by a post-commit subscriber error in the
default delivery mode. - Quantity
one_ofmembers and defaults are unit-rebased; equal quantities in
different units compare equal.