Repository navigation
Replies: 1 comment
|
I feel an issue might be more appropriate for this topic. I will close this for now |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
This discussion details Layer 1 of #9395, the database schema. #9395 proposed to store the common parts of every resource once, under two supertypes,
ResourceandResource Version. Here I propose the tables for that model, and a way to move today's data into them while today's code keeps working.The Model
The model is the one from #9395. Every kind is a subtype of
Resource, and only workflows, datasets, and models have versions, which are subtypes ofResource Version, keyed by(rid, vno). Oneaccessrelationship links a resource to a user with a level, andlineagelinks a version to the versions it relates to.Proposed Tables
resourceridkind,name,description,is_public,created_by,created_at,updated_at,specworkflow,workflow_computing_unit, anduser_warehouse, and all ofdataset,model, andvirtual_environmentsaccess(rid, uid)level: own, write, or readworkflow_of_user, the four*_user_accesstables, and theowner_uidanduidcolumnsresource_version(rid, vno)name,created_by,created_at,content_refdataset_version,model_version, and the common columns ofworkflow_versionlineage(rid, vno, source_rid, source_vno, relation)eidworkflow,computing_unit,warehouseridcontent,uri, orlakekeeper_warehouse_nameworkflow_version(rid, vno)contentworkflow_versionexecutioneid(workflow_rid, vno),computing_unit_rid,warehouse_rid,uid,status,error, and today's other columnsworkflow_executionsLikes, view counts, and upload sessions follow the same rule: one table each, keyed by
rid, instead of one per kind. Tables that do not describe a resource, such asuserandsite_settings, stay as they are. In total, today's 42 tables become 30.Decisions In The Schema
ridis new, global, and random (a UUID), so it does not reveal how many resources exist, as Design of sharing and access control of resources (computing units, datasets, and workflows) #3749 asked. Each kind keeps its old id (wid,did, and so on) until the legacy routes are gone, and afterwards as a mapping torid, because links and stored result paths contain old ids.resource.specand checked against the kind's definition in the API (Layer 2). A kind gets its own table only when its fields need an index, a constraint, or a foreign key, which today means workflows, computing units, and warehouses. Adding a field to a kind then needs no migration.resource_versionrecords who made a version and when, andcontent_refpoints to where the kind keeps the content. For a dataset or a model that is a lakeFS commit, so the files stay in lakeFS, and replacing lakeFS later changes no table.(rid, vno), as Design of sharing and access control of resources (computing units, datasets, and workflows) #3749 decided for workflow versions.usesmeans a version needs another version to execute, such as a workflow version that reads a dataset version.derived_frommeans a version was made from another, such as a dataset version written by an execution, or a cloned workflow.eidnames the execution that made it, if any.execution.errorstores the fatal errors, which today are not stored with the execution. An execution keeps its owneidinstead of the composite key that Design of sharing and access control of resources (computing units, datasets, and workflows) #3749 decided, because lineage and stored results refer to it by one id.resourcetherefore keepscreated_by, and(kind, created_by, name)stays unique for datasets and models, as it is today.Moving The Data Without Breaking Today's Code
The rule is that at any moment each kind has one source of truth, and the other side follows it automatically.
rid, and copies the existing rows. From then on, triggers copy every write to today's tables into the new ones, so new code can read every kind while today's code does not change.Each step is checked the same way: the existing tests run unchanged against it, and a write to a read-only view fails with an error instead of going to the wrong place.
One thing has to come first: today no deployment applies schema updates by itself. Single-node and Helm run only
texera_ddl.sql, on a new database, and the Liquibase changelog lists only updates 23 to 52. Every deployment needs to apply new migrations automatically before the first stage can ship.Questions For This Discussion
resource.spec, or do you prefer a table for every kind?eid, as proposed here, or become the weak entity that Design of sharing and access control of resources (computing units, datasets, and workflows) #3749 decided?usesandderived_fromin one table, or onlyderived_from, with references kept by each kind?Once the schema is agreed, a separate discussion will cover Layer 2, the API definition.
All reactions