Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Row ID specification to protocol #1610

Closed
wants to merge 8 commits into from
Closed

Add Row ID specification to protocol #1610

wants to merge 8 commits into from

Conversation

johanl-db
Copy link
Collaborator

Description

This change updates the Delta protocol to add the specification for the Row ID feature.
Row IDs are integers that are used to uniquely identify rows within a table.

See Row ID design document for additional context:

scottsand-db
scottsand-db approved these changes Mar 15, 2023
View reviewed changes
Copy link
Collaborator

@scottsand-db scottsand-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

johanl-db
johanl-db commented Mar 24, 2023
View reviewed changes
PROTOCOL.md Outdated
or with any other `add` action in the same commit.
An `add` action contains default generated Row IDs ranging from `add.baseRowId` (inclusive) to `add.baseRowId + add.stats.numRecords` (exclusive) for non-empty files. Empty files contain no default generated Row IDs.
- `add` actions for new physical files must have a value in the `baseRowId` field that is larger than the `highWaterMark` value in the last committed `rowIdHighWaterMark` action.
If the table doesn't have any committed `rowIdHighWaterMark` action, the `highWaterMark` value is -1.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If the table doesn't have any committed `rowIdHighWaterMark` action, the `highWaterMark` value is -1.
If the table version doesn't have any committed `rowIdHighWaterMark` action, the `highWaterMark` value is -1.

@felipepessoto
Copy link
Contributor

@johanl-db the design doc is not public available

@johanl-db
Copy link
Collaborator Author

@johanl-db the design doc is not public available

Sorry about that, you should now be able to access the document.

felipepessoto
felipepessoto reviewed Apr 13, 2023
View reviewed changes
PROTOCOL.md Outdated Show resolved Hide resolved
felipepessoto
felipepessoto reviewed Apr 14, 2023
View reviewed changes
PROTOCOL.md Outdated Show resolved Hide resolved
felipepessoto
felipepessoto reviewed Apr 14, 2023
View reviewed changes
PROTOCOL.md Outdated Show resolved Hide resolved
felipepessoto
felipepessoto reviewed Apr 15, 2023
View reviewed changes
PROTOCOL.md Show resolved Hide resolved
PROTOCOL.md Show resolved Hide resolved
@felipepessoto
Copy link
Contributor

@johanl-db is the plan to send the implementation PR soon?
Identity column spec has been created more than an year ago and we don't have a implementation. I'm afraid the same could happen to Row Id.

Without reference implementation other libraries are more likely to deviate from spec and create compatibility issues.

@johanl-db
Copy link
Collaborator Author

@johanl-db is the plan to send the implementation PR soon? Identity column spec has been created more than an year ago and we don't have a implementation. I'm afraid the same could happen to Row Id.

Without reference implementation other libraries are more likely to deviate from spec and create compatibility issues.

Yes, I'm actively working on it and I plan to open the first PRs implementing writing default row ids in the next few days.

Support for reading and writing materialized row ids is also in progress though it requires changes in Spark. I started shipping these (e.g. apache/spark#40545) but I'll need to wait for Delta to pick up the changes in a later Spark release before moving forward which may take some time. When that happens, I'll open PRs to support reading and writing materialized row ids.

@johanl-db johanl-db mentioned this pull request Apr 20, 2023
Closed
allisonport-db pushed a commit that referenced this pull request Apr 24, 2023
@johanl-db @allisonport-db
Introduce row ID write table feature and table property
8272ee9 
This change adds knobs to enable Row IDs when creating a new Delta table, as defined in the Row ID specification (#1610):
 - Write table feature `rowIds`: Require writers to support row IDs. Used to enable row IDs on newly created tables.
 - Table property: `rowIds.enabled`: Indicate whether all rows have an assigned row ID.
 - SQL conf `rowIds.allowForDevOnly`: restrict the use of Row IDs to testing for now.

Adding test suite RowIdSuite:
 - Test enabling Row IDs on a new table succeeds
 - Test enabling Row IDs on an existing table fails.

Closes #1702

GitOrigin-RevId: 961ff72f1ae7abf1f08d53052062ce20669d4aad
@johanl-db johanl-db mentioned this pull request Jul 11, 2023
Closed
allisonport-db pushed a commit that referenced this pull request Jul 21, 2023
@johanl-db @allisonport-db
Assign materialized Row ID and Row commit version column names
355804c 
## Description
This change is part of implementing row tracking as specified in #1610 and 7272b04.
It covers assigning a column name for the materialized Row ID and Row commit version columns by setting them in the table metadata when creating or cloning a table.

- Add test suite `rowtracking.MaterializedColumnSuite` to cover assigning materialized column names in various table creation and clone scenarios.

Closes #1896

GitOrigin-RevId: 3b963ed7e08524f24d60744160490e41a0aab3e8
This was referenced Apr 8, 2024
Closed
Closed
Merged
scottsand-db pushed a commit that referenced this pull request Apr 9, 2024
@longvu-db
[Spark] Enable Row Tracking outside of testing (#2866)
5fdba36 
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Delta Protocol for Row IDs was introduced in this PR:
#1610

Support for writing fresh row IDs / row commit versions was introduced
in the following PRs:
- #1723
- #1781
- #1896

**This is sufficient to enable row tracking on a table and write to a
table that has row tracking enabled** but not to actually read row IDs /
row commit versions back, which is also being added in Delta at the
moment ([read
BaseRowId](283ac02).
[read
defaultRowCommitVersion](#2795),
[read RowId](#2856)...)

Using row tracking is currently only allowed in testing, this change
allows enabling row tracking outside of testing so that the upcoming
Delta 3.2 release includes support for writing to tables with row
tracking enabled, making Delta writers future-proof.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Tests have already been added in previous changes, this only flips the
switch to let users enabled Row Tracking outside of tests.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
Users are now able to enable Row Tracking when creating a delta table:
```
CREATE TABLE tbl(a int) USING DELTA TBLPROPERTIES ('delta.enableRowTracking' = 'true')
```
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
andreaschat-db pushed a commit to andreaschat-db/delta that referenced this pull request Apr 16, 2024
@longvu-db @andreaschat-db
[Spark] Enable Row Tracking outside of testing (delta-io#2866)
42a5a3f 
<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Delta Protocol for Row IDs was introduced in this PR:
delta-io#1610

Support for writing fresh row IDs / row commit versions was introduced
in the following PRs:
- delta-io#1723
- delta-io#1781
- delta-io#1896

**This is sufficient to enable row tracking on a table and write to a
table that has row tracking enabled** but not to actually read row IDs /
row commit versions back, which is also being added in Delta at the
moment ([read
BaseRowId](delta-io@283ac02).
[read
defaultRowCommitVersion](delta-io#2795),
[read RowId](delta-io#2856)...)

Using row tracking is currently only allowed in testing, this change
allows enabling row tracking outside of testing so that the upcoming
Delta 3.2 release includes support for writing to tables with row
tracking enabled, making Delta writers future-proof.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
Tests have already been added in previous changes, this only flips the
switch to let users enabled Row Tracking outside of tests.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
Users are now able to enable Row Tracking when creating a delta table:
```
CREATE TABLE tbl(a int) USING DELTA TBLPROPERTIES ('delta.enableRowTracking' = 'true')
```
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@felipepessoto felipepessoto felipepessoto approved these changes

@scottsand-db scottsand-db scottsand-db approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@johanl-db @felipepessoto @scottsand-db