Add group storage to the permissions schema

[johnworth]

Migration 000057, the storage half of moving DE group data out of Grouper and
into the permissions schema of the DE database. Groups sit beside the tables
the permissions service owns so that expanding a subject to its groups becomes a
join inside the permission lookup rather than a call to another service.

No service depends on this yet — the groups service reads it, but is not
deployed anywhere. It is safe to apply and roll back on its own.

What it adds

• group_types — reference rows (collaborator_list, team, community,
  system) with an owner_required flag, so adding a kind is an INSERT rather
  than a migration.
• groups — keyed on the internal subjects.id, so a permission granted to
  a group expands to its members without translating between identifier spaces.
  The external identifier stays subjects.subject_id. Identity is structured
  (type / owner / short name) rather than Grouper's colon-delimited paths, and
  legacy_name keeps the Grouper path for provenance.
• group_memberships — direct membership. Members may be users or groups:
  production Grouper has 156 nested memberships, across collaborator lists,
  teams, and communities, so nesting had to be preserved.
• group_effective_members — the materialized transitive closure, restricted
  to users. Permission lookup is the hottest query in the DE, so the expansion is
  precomputed on write instead of recursing at read time.
• group_ancestors() / recompute_group_closure() — one implementation of
  the closure, shared by the service and the delete trigger.
• trigger_groups_detach_before_delete — see below.
• subjects.user_id — a nullable correlation to public.users.

Two changes to subjects worth attention

subject_id widens from varchar(64) to varchar(512), matching
public.users.username. QA already has a 65-character username, and membership
now requires every member to have a subject row, so that user would otherwise
fail a constraint they do not hit today.

subject_id also gains a default of replace(uuid_generate_v1()::text,'-','').
Groups imported from Grouper keep their original 32-hex identifiers so existing
permission grants and iRODS @grouper-<id> names stay valid; groups created
after the migration must be indistinguishable from them, or the DE ends up with
two identifier shapes and two iRODS group-name shapes.

The delete trigger is load-bearing

Deleting a nested group cascades its membership row away, which destroys the path
back to the containing groups before any AFTER trigger or calling service could
react — leaving those groups still granting access to the deleted group's
members. This was verified as a real leak before the fix. A BEFORE DELETE
trigger detaches and recomputes at the one point the containers are still
findable, and covers every caller including a cascade from subjects.

subjects.user_id

subject_id is a bare username while public.users.username carries a domain
suffix, so joining the two means reconstructing the suffixed form everywhere.

It is nullable and best-effort: a subject can be created before the user has
ever logged in to the DE, and 5 of the 19,728 production user subjects correspond
to no DE user at all. A NULL means "not correlated", never "no such user".

ON DELETE SET NULL rather than CASCADE — removing a DE user must not delete the
subject, because that would cascade away their group memberships and every
permission granted to them. A CHECK stops a group subject from naming a user, and
a partial unique index stops two subjects claiming the same one.

It is deliberately not backfilled here: matching needs the username suffix,
which is deployment-specific. Population belongs to the services and the Grouper
importer, which have it configured. Match on the fully suffixed username rather
than the bare name — production public.users holds 128,110 rows of a
doubled-suffix form (user@@iplantcollaborative.org) which are near-inert (34
jobs between all of them, against 1,477,921 for the 142,368 real rows) but would
make a bare-name match ambiguous for almost every user.

Compatibility

• NULLS NOT DISTINCT is avoided in favour of a coalesce-based unique index,
  so the migration does not require PostgreSQL 15+.
• Both recursive CTEs use UNION rather than UNION ALL, so a membership cycle
  terminates instead of recursing forever. The service also rejects cycles at
  write time.

Verification

up / down / up against a fresh PostgreSQL 18 database with all preceding
migrations applied. The down migration restores subject_id to varchar(64),
failing rather than truncating if a longer identifier exists — silently
discarding part of an identity is worse than refusing to roll back.

The constraints were exercised individually against a clean database, and the
delete-trigger behaviour is covered by integration tests in the groups service
(dropping the trigger produces exactly the two expected failures).

group_data_source

The importer reconciles — it removes memberships and grants Grouper no longer
has — which is correct only while Grouper is authoritative. Run it once after
cutover and it deletes whatever was created natively in the meantime.

A single-row table records which system owns group data, seeded to grouper.
The importer will refuse to start unless it says so. There is deliberately no
override flag: the only way to run a destructive reconcile afterwards is to set
this back, which is an attributed, timestamped write rather than something typed
at the end of an argument list. Flipping it to native is an ordered step of
cutover, performed before anything is pointed at the new store.

A BEFORE UPDATE trigger maintains changed_at, so the record of when cutover
happened cannot be left stale — verified by confirming an explicit
changed_at = '2020-01-01' is overridden.

Single-row-ness, the source vocabulary, and a non-blank changed_by are all
enforced by constraints and were exercised individually against a live database.

🤖 Generated with Claude Code