Skip to content

docs: bring tech-specs current to Hudi 1.2.0 (table version 9) and promote to default#19179

Merged
vinothchandar merged 1 commit into
apache:asf-sitefrom
vinothchandar:fix-tech-specs
Jul 4, 2026
Merged

docs: bring tech-specs current to Hudi 1.2.0 (table version 9) and promote to default#19179
vinothchandar merged 1 commit into
apache:asf-sitefrom
vinothchandar:fix-tech-specs

Conversation

@vinothchandar

Copy link
Copy Markdown
Member

The Hudi 1.0 tech spec at /learn/tech-specs-1point0 hadn't been touched since Dec 2024 and was scoped only to table version 8. Since then, Hudi 1.1.0 (Nov 2025) and Hudi 1.2.0 (May 2026) both shipped on table version 9, adding new column types, a new base file format, and a new secondary-index encoding without a further version bump. The document had also accumulated correctness gaps (log block IDs listed as 1-based when they're serialized as zero-based ordinals, missing metadata partitions, incomplete Avro schemas) that were noticed while grounding the spec against the current code.

This change refreshes the spec end-to-end against the release-1.2.0 tag and makes it the default tech spec on the site.

Scope

  • Doc header now says table version 9, "Reflects release: Hudi 1.2.0", and calls out that v9 readers cover the v8 subset but v8 readers can't in general read v9-only features.
  • Old table-version-5 spec (Aug 2022) is preserved at /learn/tech-specs-0-point-x.

Correctness fixes

  • Log block type IDs (Command, Delete, Corrupted, Avro, HFile, Parquet, CDC) renumbered to zero-based (0-6) to match the enum ordinal that actually gets serialized on the wire. Same for header metadata IDs.
  • Added missing header metadata entry BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS (ordinal 8).
  • Storage-layout tree fixed: col_stats -> column_stats, added partition_stats, bloom_filters, secondary_index_, expr_index_, .temp/.
  • Base file naming reconciled (requested_time consistent between template and prose).
  • Command block description says ROLLBACK_BLOCK (matches HoodieCommandBlockTypeEnum) instead of the invented ROLLBACK_PREVIOUS_BLOCK.
  • Typos: "Bboom filter", "fileIdLowBits, fileIdLowBits", missing "if" in writer expectations, etc.
  • Writer Expectations step "5. Writer checks" was dangling; now describes the timeline refresh inside the lock.

Content completeness

  • HoodieMetadataColumnStats table now enumerates all fields shipped in HoodieMetadata.avsc: minValue, maxValue, valueCount, nullCount, totalSize, totalUncompressedSize, isDeleted, isTightBound, valueType (HoodieValueTypeInfo).
  • HoodieRecordIndexInfo table now includes the position field and a paragraph on the partitioned-RLI file-id encoding.
  • Bloom filter section calls out SIMPLE vs DYNAMIC.
  • Secondary index section describes V1 (through v8) and V2 (default on v9) encodings without dropping V1.
  • Table Properties expanded from 11 to 26 rows to reflect the actual hoodie.properties surface: create.schema, base.file.format, log.file.format, keygenerator.class, cdc.enabled + supplemental logging mode, partial.update.mode, metadata.partitions.inflight, timeline.path, timeline.history.path, archivelog.folder, timeline.timezone, initial.version, slash-separated date partitioning, and a note on atomic writes + checksum.
  • Reader Expectations now describes rollback command blocks, BLOCK_IDENTIFIER dedup, COMPACTED_BLOCK_TIMES skipping, and positional merges anchored by BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS.
  • Metadata section enumerates all seven metadata partitions and adds a Partition Statistics subsection.
  • Async indexer section replaces the undefined "t" with the actual baseInstantTime from HoodieIndexPartitionInfo.

New content

  • Data Files -> Types: short table covering the Hudi-specific logical types added in table version 9 - VARIANT (record + logicalType variant), VECTOR (fixed + logicalType vector), BLOB (record + logicalType blob).
  • Base Files section explicitly lists Parquet, ORC, HFile, and Lance (Spark COW/read only; column stats and partition stats not populated for Lance base files).
  • Change Log now has a "version 9 - current" section split into what landed in Hudi 1.1.0 vs Hudi 1.2.0, plus updated version 8 highlights.

Site changes for the promotion

  • website/learn/tech-specs.md is now the new (v9) spec; the old Aug 2022 spec moves to website/learn/tech-specs-0-point-x.md.
  • Two self-references inside the promoted file were updated from /learn/tech-specs (which used to point at the old spec) to /learn/tech-specs-0-point-x. No other prose changes.
  • sidebarsLearn.js: entries relabeled "Tech Specs (current)" and "Tech Specs (pre-1.0)".
  • docusaurus.config.js redirects: /tech-specs-1point0 and /learn/tech-specs-1point0 both -> /learn/tech-specs so existing external links (including versioned_docs and release notes on older sites) continue to resolve. Footer link updated. Chat modal disclaimer URL updated.
  • Live-tree markdown references (docs/hudi_stack.md, blog/2025-04-02-secondary-index.md, releases/release-1.0*.md) rewritten from tech-specs-1point0 -> tech-specs.
  • versioned_docs/ left untouched by design - snapshots are frozen and the redirect covers them.

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

…omote to default

The Hudi 1.0 tech spec at /learn/tech-specs-1point0 hadn't been touched
since Dec 2024 and was scoped only to table version 8. Since then, Hudi
1.1.0 (Nov 2025) and Hudi 1.2.0 (May 2026) both shipped on table
version 9, adding new column types, a new base file format, and a
new secondary-index encoding without a further version bump. The
document had also accumulated correctness gaps (log block IDs listed
as 1-based when they're serialized as zero-based ordinals, missing
metadata partitions, incomplete Avro schemas) that were noticed while
grounding the spec against the current code.

This change refreshes the spec end-to-end against the release-1.2.0
tag and makes it the default tech spec on the site.

Scope

* Doc header now says table version 9, "Reflects release: Hudi 1.2.0",
  and calls out that v9 readers cover the v8 subset but v8 readers
  can't in general read v9-only features.
* Old table-version-5 spec (Aug 2022) is preserved at
  /learn/tech-specs-0-point-x.

Correctness fixes

* Log block type IDs (Command, Delete, Corrupted, Avro, HFile, Parquet,
  CDC) renumbered to zero-based (0-6) to match the enum ordinal that
  actually gets serialized on the wire. Same for header metadata IDs.
* Added missing header metadata entry
  BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS (ordinal 8).
* Storage-layout tree fixed: col_stats -> column_stats, added
  partition_stats, bloom_filters, secondary_index_<name>,
  expr_index_<name>, .temp/.
* Base file naming reconciled (requested_time consistent between
  template and prose).
* Command block description says ROLLBACK_BLOCK (matches
  HoodieCommandBlockTypeEnum) instead of the invented
  ROLLBACK_PREVIOUS_BLOCK.
* Typos: "Bboom filter", "fileIdLowBits, fileIdLowBits", missing "if"
  in writer expectations, etc.
* Writer Expectations step "5. Writer checks" was dangling; now
  describes the timeline refresh inside the lock.

Content completeness

* HoodieMetadataColumnStats table now enumerates all fields shipped
  in HoodieMetadata.avsc: minValue, maxValue, valueCount, nullCount,
  totalSize, totalUncompressedSize, isDeleted, isTightBound,
  valueType (HoodieValueTypeInfo).
* HoodieRecordIndexInfo table now includes the position field and a
  paragraph on the partitioned-RLI file-id encoding.
* Bloom filter section calls out SIMPLE vs DYNAMIC.
* Secondary index section describes V1 (through v8) and V2 (default
  on v9) encodings without dropping V1.
* Table Properties expanded from 11 to 26 rows to reflect the actual
  hoodie.properties surface: create.schema, base.file.format,
  log.file.format, keygenerator.class, cdc.enabled + supplemental
  logging mode, partial.update.mode, metadata.partitions.inflight,
  timeline.path, timeline.history.path, archivelog.folder,
  timeline.timezone, initial.version, slash-separated date
  partitioning, and a note on atomic writes + checksum.
* Reader Expectations now describes rollback command blocks,
  BLOCK_IDENTIFIER dedup, COMPACTED_BLOCK_TIMES skipping, and
  positional merges anchored by
  BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS.
* Metadata section enumerates all seven metadata partitions and adds
  a Partition Statistics subsection.
* Async indexer section replaces the undefined "t" with the actual
  baseInstantTime from HoodieIndexPartitionInfo.

New content

* Data Files -> Types: short table covering the Hudi-specific logical
  types added in table version 9 - VARIANT (record + logicalType
  variant), VECTOR (fixed + logicalType vector), BLOB (record +
  logicalType blob).
* Base Files section explicitly lists Parquet, ORC, HFile, and Lance
  (Spark COW/read only; column stats and partition stats not
  populated for Lance base files).
* Change Log now has a "version 9 - current" section split into what
  landed in Hudi 1.1.0 vs Hudi 1.2.0, plus updated version 8 highlights.

Site changes for the promotion

* website/learn/tech-specs.md is now the new (v9) spec; the old
  Aug 2022 spec moves to website/learn/tech-specs-0-point-x.md.
* Two self-references inside the promoted file were updated from
  /learn/tech-specs (which used to point at the old spec) to
  /learn/tech-specs-0-point-x. No other prose changes.
* sidebarsLearn.js: entries relabeled "Tech Specs (current)" and
  "Tech Specs (pre-1.0)".
* docusaurus.config.js redirects: /tech-specs-1point0 and
  /learn/tech-specs-1point0 both -> /learn/tech-specs so existing
  external links (including versioned_docs and release notes on
  older sites) continue to resolve. Footer link updated. Chat modal
  disclaimer URL updated.
* Live-tree markdown references (docs/hudi_stack.md,
  blog/2025-04-02-secondary-index.md, releases/release-1.0*.md)
  rewritten from tech-specs-1point0 -> tech-specs.
* versioned_docs/ left untouched by design - snapshots are frozen and
  the redirect covers them.
@vinothchandar vinothchandar requested a review from bhasudha July 4, 2026 03:07
@vinothchandar vinothchandar merged commit d88dac2 into apache:asf-site Jul 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant