Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support setting table and column comments in Delta Lake #12971

Merged
merged 2 commits into from Jun 28, 2022

Conversation

findinpath
Copy link
Contributor

Description

Delta Logs corresponding to comment on table and column statements via spark-sql:

COMMENT ON TABLE testdelta IS 'this is my delta table';

ALTER TABLE testdelta ALTER COLUMN a COMMENT 'this is my a column';
[root@hadoop-master /]# hdfs dfs -cat /user/hive/warehouse/testdelta/_delta_log/00000000000000000001.json
{"metaData":{"id":"e4e5196b-311c-4883-9e93-b97b7711f1a1","description":"this is my delta table","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"a\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"b\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1656043296180}}
{"commitInfo":{"timestamp":1656043342692,"operation":"SET TBLPROPERTIES","operationParameters":{"properties":"{\"comment\":\"this is my delta table\"}"},"readVersion":0,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{},"engineInfo":"Apache-Spark/3.2.1 Delta-Lake/1.2.1","txnId":"2ae38bea-9f58-4628-be9e-b4a36fee8e96"}}
[root@hadoop-master /]# hdfs dfs -cat /user/hive/warehouse/testdelta/_delta_log/00000000000000000002.json
{"metaData":{"id":"e4e5196b-311c-4883-9e93-b97b7711f1a1","description":"this is my delta table","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"a\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{\"comment\":\"this is my a column\"}},{\"name\":\"b\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":[],"configuration":{},"createdTime":1656043296180}}
{"commitInfo":{"timestamp":1656043505250,"operation":"CHANGE COLUMN","operationParameters":{"column":"{\"name\":\"a\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{\"comment\":\"this is my a column\"}}"},"readVersion":1,"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{},"engineInfo":"Apache-Spark/3.2.1 Delta-Lake/1.2.1","txnId":"6fe8a96b-8ca6-450f-a26a-a08506db9a57"}}

Is this change a fix, improvement, new feature, refactoring, or other?

New feature.

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Delta Lake connector.

How would you describe this change to a non-technical end user or system administrator?

Add the ability to perform statements like

COMMENT ON TABLE deltatable IS 'My table description';
COMMENT ON COLUMN deltatable.deltacolumn IS 'My column description';

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Delta Lake
* Add support for `COMMENT` statement on table and column

@cla-bot cla-bot bot added the cla-signed label Jun 24, 2022
@findinpath findinpath force-pushed the comment-delta-lake branch 3 times, most recently from 7522629 to 9525b08 Compare June 24, 2022 06:34
@findinpath findinpath requested review from findepi, ebyhr and alexjo2144 and removed request for findepi June 24, 2022 06:45
@findinpath findinpath marked this pull request as ready for review June 24, 2022 06:45
@findepi findepi changed the title Add support for COMMENT statements on Delta Lake Support setting table and column comments in Delta Lake Jun 24, 2022
@findepi findepi merged commit 4f04191 into trinodb:master Jun 28, 2022
@findepi findepi mentioned this pull request Jun 28, 2022
@github-actions github-actions bot added this to the 388 milestone Jun 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

None yet

3 participants