Skip to content

Add icebergUpdateSchema for schema evolution#31

Merged
platypii merged 1 commit into
masterfrom
update-schema
Jul 11, 2026
Merged

Add icebergUpdateSchema for schema evolution#31
platypii merged 1 commit into
masterfrom
update-schema

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Fixes #25

Exposes the existing add-schema / set-current-schema commit logic through a stable public entry point, so consumers can add nullable columns (or rename columns, promote types) to an existing table in place — no recreate-and-reload.

Changes

  • icebergStageUpdateSchema({ metadata, schema }) (stage.js) — pure, metadata-only stage emitting add-schema (spec sentinel schema-id: -1) + set-current-schema, guarded by assert-table-uuid, assert-current-schema-id, and assert-last-assigned-field-id CAS requirements. Runs applyUpdates eagerly so evolution violations (field-id reuse, illegal type promotion, required column without defaults) fail at stage time rather than only at commit / server-side.
  • icebergUpdateSchema({ catalog, ..., schema }) (write.js) — top-level commit through commitWithRetry; retry-safe since it writes no data files. Works for file and REST catalogs.
  • StagedCommit supertype (types.d.ts) with optional snapshot for metadata-only commits; StagedUpdate extends it and keeps snapshot required for the snapshot-producing stages, so existing callers and tests are unaffected.
  • Exported from index.js, documented in the README.

API

The caller passes the complete evolved schema with field ids assigned: existing columns keep their ids, new columns use ids above the table's last-column-id (matches HypAware's mergeFieldIdsFromTable). The new schema's id is assigned at commit time.

await icebergUpdateSchema({
  catalog, tableUrl,
  schema: {
    type: 'struct',
    'schema-id': 0, // ignored; assigned at commit
    fields: [
      ...schema.fields,
      { id: 3, name: 'score', required: false, type: 'double' },
    ],
  },
})

Acceptance criteria from #25

  • ✅ Add a nullable column to an existing table and commit it, no recreate
  • ✅ Subsequent appends use the evolved schema (staging reads current-schema-id)
  • ✅ Pre-existing files read the new column as null (existing read-path null-fill)
  • ✅ Field ids preserved for existing columns, assigned by caller for new ones
  • ✅ Round-trip test: create → append → add column → append → read all back, old rows null in the new column

Testing

7 new tests in test/write/update-schema.test.js: the round-trip above, rename-by-field-id, rejection of incompatible type changes and defaultless required columns, and the staged requirements/updates shape. npx tsc, npm run lint, and all 598 tests pass.

Expose the existing add-schema / set-current-schema commit logic through a
stable public entry point so consumers can add nullable columns (or rename
columns, promote types) without recreating the table.

- icebergStageUpdateSchema: pure metadata-only stage with CAS requirements
  (assert-current-schema-id, assert-last-assigned-field-id), validated
  eagerly via applyUpdates
- icebergUpdateSchema: top-level commit through commitWithRetry, works for
  file and REST catalogs
- StagedCommit supertype with optional snapshot for metadata-only commits

Fixes #25
@platypii platypii merged commit ec237ef into master Jul 11, 2026
6 checks passed
@platypii platypii deleted the update-schema branch July 11, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schema evolution (add columns) isn't reachable by consumers

1 participant