docs: bring tech-specs current to Hudi 1.2.0 (table version 9) and promote to default#19179
Merged
Merged
Conversation
…omote to default The Hudi 1.0 tech spec at /learn/tech-specs-1point0 hadn't been touched since Dec 2024 and was scoped only to table version 8. Since then, Hudi 1.1.0 (Nov 2025) and Hudi 1.2.0 (May 2026) both shipped on table version 9, adding new column types, a new base file format, and a new secondary-index encoding without a further version bump. The document had also accumulated correctness gaps (log block IDs listed as 1-based when they're serialized as zero-based ordinals, missing metadata partitions, incomplete Avro schemas) that were noticed while grounding the spec against the current code. This change refreshes the spec end-to-end against the release-1.2.0 tag and makes it the default tech spec on the site. Scope * Doc header now says table version 9, "Reflects release: Hudi 1.2.0", and calls out that v9 readers cover the v8 subset but v8 readers can't in general read v9-only features. * Old table-version-5 spec (Aug 2022) is preserved at /learn/tech-specs-0-point-x. Correctness fixes * Log block type IDs (Command, Delete, Corrupted, Avro, HFile, Parquet, CDC) renumbered to zero-based (0-6) to match the enum ordinal that actually gets serialized on the wire. Same for header metadata IDs. * Added missing header metadata entry BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS (ordinal 8). * Storage-layout tree fixed: col_stats -> column_stats, added partition_stats, bloom_filters, secondary_index_<name>, expr_index_<name>, .temp/. * Base file naming reconciled (requested_time consistent between template and prose). * Command block description says ROLLBACK_BLOCK (matches HoodieCommandBlockTypeEnum) instead of the invented ROLLBACK_PREVIOUS_BLOCK. * Typos: "Bboom filter", "fileIdLowBits, fileIdLowBits", missing "if" in writer expectations, etc. * Writer Expectations step "5. Writer checks" was dangling; now describes the timeline refresh inside the lock. Content completeness * HoodieMetadataColumnStats table now enumerates all fields shipped in HoodieMetadata.avsc: minValue, maxValue, valueCount, nullCount, totalSize, totalUncompressedSize, isDeleted, isTightBound, valueType (HoodieValueTypeInfo). * HoodieRecordIndexInfo table now includes the position field and a paragraph on the partitioned-RLI file-id encoding. * Bloom filter section calls out SIMPLE vs DYNAMIC. * Secondary index section describes V1 (through v8) and V2 (default on v9) encodings without dropping V1. * Table Properties expanded from 11 to 26 rows to reflect the actual hoodie.properties surface: create.schema, base.file.format, log.file.format, keygenerator.class, cdc.enabled + supplemental logging mode, partial.update.mode, metadata.partitions.inflight, timeline.path, timeline.history.path, archivelog.folder, timeline.timezone, initial.version, slash-separated date partitioning, and a note on atomic writes + checksum. * Reader Expectations now describes rollback command blocks, BLOCK_IDENTIFIER dedup, COMPACTED_BLOCK_TIMES skipping, and positional merges anchored by BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS. * Metadata section enumerates all seven metadata partitions and adds a Partition Statistics subsection. * Async indexer section replaces the undefined "t" with the actual baseInstantTime from HoodieIndexPartitionInfo. New content * Data Files -> Types: short table covering the Hudi-specific logical types added in table version 9 - VARIANT (record + logicalType variant), VECTOR (fixed + logicalType vector), BLOB (record + logicalType blob). * Base Files section explicitly lists Parquet, ORC, HFile, and Lance (Spark COW/read only; column stats and partition stats not populated for Lance base files). * Change Log now has a "version 9 - current" section split into what landed in Hudi 1.1.0 vs Hudi 1.2.0, plus updated version 8 highlights. Site changes for the promotion * website/learn/tech-specs.md is now the new (v9) spec; the old Aug 2022 spec moves to website/learn/tech-specs-0-point-x.md. * Two self-references inside the promoted file were updated from /learn/tech-specs (which used to point at the old spec) to /learn/tech-specs-0-point-x. No other prose changes. * sidebarsLearn.js: entries relabeled "Tech Specs (current)" and "Tech Specs (pre-1.0)". * docusaurus.config.js redirects: /tech-specs-1point0 and /learn/tech-specs-1point0 both -> /learn/tech-specs so existing external links (including versioned_docs and release notes on older sites) continue to resolve. Footer link updated. Chat modal disclaimer URL updated. * Live-tree markdown references (docs/hudi_stack.md, blog/2025-04-02-secondary-index.md, releases/release-1.0*.md) rewritten from tech-specs-1point0 -> tech-specs. * versioned_docs/ left untouched by design - snapshots are frozen and the redirect covers them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Hudi 1.0 tech spec at /learn/tech-specs-1point0 hadn't been touched since Dec 2024 and was scoped only to table version 8. Since then, Hudi 1.1.0 (Nov 2025) and Hudi 1.2.0 (May 2026) both shipped on table version 9, adding new column types, a new base file format, and a new secondary-index encoding without a further version bump. The document had also accumulated correctness gaps (log block IDs listed as 1-based when they're serialized as zero-based ordinals, missing metadata partitions, incomplete Avro schemas) that were noticed while grounding the spec against the current code.
This change refreshes the spec end-to-end against the release-1.2.0 tag and makes it the default tech spec on the site.
Scope
Correctness fixes
Content completeness
New content
Site changes for the promotion
Describe the issue this Pull Request addresses
Summary and Changelog
Impact
Risk Level
Documentation Update
Contributor's checklist