Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docs: Update Partitions table in Flink/Spark doc #8021

Merged
merged 3 commits into from Jul 13, 2023
Merged

Docs: Update Partitions table in Flink/Spark doc #8021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db1b613
Reflect latest partitions api in Flink/Spark doc
hsiang-c Jul 9, 2023
4bc6c65
Address reviewer's comments
hsiang-c Jul 10, 2023
d3635f2
Fix grammer
hsiang-c Jul 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/flink-queries.md
View file
Open in desktop
Expand Up @@ -428,15 +428,15 @@ To show a table's current partitions:
SELECT * FROM prod.db.table$partitions;
```

| partition | record_count | file_count | spec_id |
| -------------- | ------------ | ---------- | ------- |
| {20211001, 11} | 1 | 1 | 0 |
| {20211002, 11} | 1 | 1 | 0 |
| {20211001, 10} | 1 | 1 | 0 |
| {20211002, 10} | 1 | 1 | 0 |
| partition | spec_id | record_count | file_count | total_data_file_size_in_bytes | position_delete_record_count | position_delete_file_count | equality_delete_record_count | equality_delete_file_count | last_updated_at(μs) | last_updated_snapshot_id |
| -------------- |---------|---------------|------------|--------------------------|------------------------------|----------------------------|------------------------------|----------------------------|---------------------|--------------------------|
| {20211001, 11} | 0 | 1 | 1 | 100 | 2 | 1 | 0 | 0 | 1633086034192000 | 9205185327307503337 |
| {20211002, 11} | 0 | 4 | 3 | 500 | 1 | 1 | 0 | 0 | 1633172537358000 | 867027598972211003 |
| {20211001, 10} | 0 | 7 | 4 | 700 | 0 | 0 | 0 | 0 | 1633082598716000 | 3280122546965981531 |
| {20211002, 10} | 0 | 3 | 2 | 400 | 0 | 0 | 1 | 1 | 1633169159489000 | 6941468797545315876 |

Note:
For unpartitioned tables, the partitions table will contain only the record_count and file_count columns.
For unpartitioned tables, the partitions table will not contain the partition and spec_id fields.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, i just noticed original word is columns but we updated to use fields here. do you think fields are better?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm.. dont have a strong opinion, I guess they are both equivalent, or do you see a difference?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah you are right, looking at spec and they are used interchangeably

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dramaticlly Thanks for your feedback:)


### All Metadata Tables

Expand Down
14 changes: 7 additions & 7 deletions docs/spark-queries.md
View file
Open in desktop
Expand Up @@ -311,15 +311,15 @@ To show a table's current partitions:
SELECT * FROM prod.db.table.partitions;
```

| partition | record_count | file_count | spec_id |
| -- | -- | -- | -- |
| {20211001, 11}| 1| 1| 0|
| {20211002, 11}| 1| 1| 0|
| {20211001, 10}| 1| 1| 0|
| {20211002, 10}| 1| 1| 0|
| partition | spec_id | record_count | file_count | total_data_file_size_in_bytes | position_delete_record_count | position_delete_file_count | equality_delete_record_count | equality_delete_file_count | last_updated_at(μs) | last_updated_snapshot_id |
| -------------- |---------|---------------|------------|--------------------------|------------------------------|----------------------------|------------------------------|----------------------------|---------------------|--------------------------|
| {20211001, 11} | 0 | 1 | 1 | 100 | 2 | 1 | 0 | 0 | 1633086034192000 | 9205185327307503337 |
| {20211002, 11} | 0 | 4 | 3 | 500 | 1 | 1 | 0 | 0 | 1633172537358000 | 867027598972211003 |
| {20211001, 10} | 0 | 7 | 4 | 700 | 0 | 0 | 0 | 0 | 1633082598716000 | 3280122546965981531 |
| {20211002, 10} | 0 | 3 | 2 | 400 | 0 | 0 | 1 | 1 | 1633169159489000 | 6941468797545315876 |

Note:
1. For unpartitioned tables, the partitions table will contain only the record_count and file_count columns.
1. For unpartitioned tables, the partitions table will not contain the partition and spec_id fields.

2. The partitions metadata table shows partitions with data files or delete files in the current snapshot. However, delete files are not applied, and so in some cases partitions may be shown even though all their data rows are marked deleted by delete files.

Expand Down