Docs: add flink and iceberg type compatibility by wuwenchi · Pull Request #4865 · apache/iceberg

wuwenchi · 2022-05-25T11:19:17Z

Add flink and iceberg type compatibility.

Can you help review it? thanks !
@rdblue @openinx @kbendick

kbendick

Hi a @wuwenchi! Thanks for providing this compatibility matrix.

There’s also an iceberg-docs repo, github.com/apache/iceberg-docs. You might need to make a PR there as well. cc @samredai for visibility.

For types, I’m mostly sure this is correct but need to verify a few things. Also, as a follow up, I’d like to update the language in the “not supported” section to differentiate between what is not supported by Iceberg but possible in Flink and what is simply not presently possible in Flink. Possibly somebody will be inspired to help out over there or here!

kbendick · 2022-05-26T05:49:43Z

docs/flink/flink-getting-started.md

+
+### Flink type to Iceberg type
+
+This type conversion table describes how Spark types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Spark.


Nit: this mentions Spark several times. Assuming a copy past issue?

=.= copy past issue... fix it.

kbendick · 2022-05-26T05:50:35Z

docs/flink/flink-getting-started.md


+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,


Question: for what combinations can Iceberg not do the conversion automatically? From the perspective of a new user, this might leave them with more questions than answers.

kbendick · 2022-05-26T05:58:10Z

docs/flink/flink-getting-started.md


+## Type compatibility
+
+Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,


Additionally, I would say they support “different” types. There are logic types (eg UUID), physical types (eg String or VARCHAR), and then what’s ultimately stored within the respective file types and how.

I’d focus on covering the first two here (logical and physical). And making notes of any peculiarities that are different within the matrix - for example, I believe that timestamp precision and what is and us not possible to store might be worth calling more attention to in the notes section of the matrix.

For this sentence, I’d say something more along the lines of “Iceberg’s type system is mapped to Flink’s type system. This type conversion can be done by Iceberg automatically, though the following cases need to be considered. See the notes section below for compatibility concerns and how to overcome them”.

Then I’d make a small section listing the combinations that don’t work or general discussion of the mounts of concern. As we’re migrating the docs somewhat to a new structure, so I think that would really help (even if you can’t list all cases etc)

Just a suggestion / jumping off point.

Good suggestion! I modified the description and added related notes, so that we can see some details more intuitively.

wuwenchi · 2022-05-26T11:28:38Z

There’s also an iceberg-docs repo, github.com/apache/iceberg-docs.

After this PR is completed, I will provide a PR to iceberg-docs.

samredai · 2022-05-27T15:56:03Z

There’s also an iceberg-docs repo, github.com/apache/iceberg-docs. You might need to make a PR there as well.

Since this is in the versioned part of the docs site, there's no need to open a PR against the iceberg-docs repo. I do think we need a bonafide "getting started"/"quickstart" flink guide that gives a docker environment and let's the user fully run a demo environment--that would go in the iceberg-docs repo.

On a separate note, I'm noticing there are a number of "Types Compatibility" subsections that are cropping up for various engines. I'm wondering if we should try and consolidate them to a single "Types Compatibility" page, or maybe append it to the Configurations page in PR #4801 (although it's not really 'configuration').

wuwenchi · 2022-05-28T02:57:33Z

@samredai

I do think we need a bonafide "getting started"/"quickstart" flink guide that gives a docker environment and let's the user fully run a demo environment--that would go in the iceberg-docs repo.

Good idea, look forward to this!

I'm wondering if we should try and consolidate them to a single "Types Compatibility" page, or maybe append it to the Configurations page

I prefer to add a new page to add this part.

rdblue · 2022-05-29T21:04:28Z

docs/flink/flink-getting-started.md

+| float                      | float         |                                                  |
+| double                     | double        |                                                  |
+| date                       | date          |                                                  |
+| time                       | time          | precision is fixed at 0                          |


precision is 0? I would expect that to be 6 for microseconds.

In flink, the precision of time is not supported and it default precision is 0:

in iceberg: ... case TIME: return new TimeType(); ... in flink: public static final int DEFAULT_PRECISION = 0; public TimeType() { this(DEFAULT_PRECISION); }

Of course, this is the behavior of flink, maybe we should not write this precision here.

the precision of time is not supported

Then we should not mention it here.

wuwenchi · 2022-05-30T03:08:46Z

docs/flink/flink-getting-started.md

+| symbol              |                            | Not supported |
+| logical             |                            | Not supported |
+
+### Iceberg type to Flink type


Should the type be removed here and be consistent with the above?

Yes, that would be good.

wuwenchi · 2022-05-30T09:30:06Z

Since this is in the versioned part of the docs site, there's no need to open a PR against the iceberg-docs repo.

I have another question, like #4725, the modifications in the spec should not need to submit a PR in iceberg-doc, right?

samredai · 2022-05-30T15:10:29Z

I have another question, like #4725, the modifications in the spec should not need to submit a PR in iceberg-doc, right?

Yes that's right, the spec is copied over to iceberg-docs during the release process.

rdblue · 2022-06-09T02:04:10Z

docs/flink/flink-getting-started.md

+| time                       | time          |                                                  |
+| timestamp without timezone | timestamp     | precision is fixed at 6                          |
+| timestamp with timezone    | timestamp_ltz | precision is fixed at 6                          |
+| string                     | varchar       | length is fixed at 2<sup>31</sup>-1              |


Why not use varchar[2147483647]?

rdblue · 2022-06-09T02:04:24Z

docs/flink/flink-getting-started.md

+| timestamp without timezone | timestamp     | precision is fixed at 6                          |
+| timestamp with timezone    | timestamp_ltz | precision is fixed at 6                          |
+| string                     | varchar       | length is fixed at 2<sup>31</sup>-1              |
+| uuid                       | binary        | length is fixed at 16                            |


binary(16)?

rdblue · 2022-06-09T02:04:33Z

docs/flink/flink-getting-started.md

+| double                     | double        |                                                  |
+| date                       | date          |                                                  |
+| time                       | time          |                                                  |
+| timestamp without timezone | timestamp     | precision is fixed at 6                          |


timestamp(6)?

rdblue · 2022-06-09T02:04:49Z

docs/flink/flink-getting-started.md

+| string                     | varchar       | length is fixed at 2<sup>31</sup>-1              |
+| uuid                       | binary        | length is fixed at 16                            |
+| fixed                      | binary        |                                                  |
+| binary                     | varbinary     | width is fixed at 2<sup>31</sup>-1               |


varbinary(2147483647)?

rdblue · 2022-06-09T02:05:11Z

docs/flink/flink-getting-started.md

+| uuid                       | binary        | length is fixed at 16                            |
+| fixed                      | binary        |                                                  |
+| binary                     | varbinary     | width is fixed at 2<sup>31</sup>-1               |
+| decimal                    | decimal       | precision and scale are the same as the original |


decimal(P, S) <=> decimal(P, S)?

done, looks more clear

rdblue · 2022-06-29T18:12:39Z

@wuwenchi, can you rebase?

samredai · 2022-06-29T20:59:00Z

Just FYI that there are no longer any subdirectories, so this markdown file is located at docs/flink-getting-started.md

1. Delete time's notes 2. Delete the type to be consistent with the above

fix by review

wuwenchi · 2022-06-30T02:04:59Z

@samredai ok, thanks! @rdblue done

rdblue · 2022-07-03T20:21:38Z

Thanks, @wuwenchi!

Co-authored-by: 吴文池 <wuwenchi@deepexi.com>

github-actions bot added the docs label May 25, 2022

kbendick reviewed May 26, 2022

View reviewed changes

rdblue reviewed May 29, 2022

View reviewed changes

wuwenchi commented May 30, 2022

View reviewed changes

rdblue reviewed Jun 9, 2022

View reviewed changes

wuwenchi mentioned this pull request Jun 16, 2022

Docs: missing bigint in available types #5054

Closed

rdblue closed this Jun 29, 2022

rdblue reopened this Jun 29, 2022

吴文池 and others added 5 commits June 30, 2022 09:50

Docs: add flink iceberg datatype

efecb6c

Docs: add flink iceberg datatype

79638e3

Update flink-getting-started.md

4257d5a

Update flink-getting-started.md

6c8244c

1. Delete time's notes 2. Delete the type to be consistent with the above

Update flink-getting-started.md

556efc4

fix by review

wuwenchi force-pushed the deepnova_doc_add_flink_iceberg_type branch from 39f7241 to 556efc4 Compare June 30, 2022 01:58

rdblue approved these changes Jul 3, 2022

View reviewed changes

rdblue merged commit 81dec35 into apache:master Jul 3, 2022

dungdm93 pushed a commit to dungdm93/iceberg that referenced this pull request Jul 4, 2022

Docs: Add Flink and Iceberg type compatibility tables (apache#4865)

d11f386

Co-authored-by: 吴文池 <wuwenchi@deepexi.com>

wuwenchi deleted the deepnova_doc_add_flink_iceberg_type branch July 8, 2022 01:48

namrathamyske pushed a commit to namrathamyske/iceberg that referenced this pull request Jul 10, 2022

Docs: Add Flink and Iceberg type compatibility tables (apache#4865)

bb19394

Co-authored-by: 吴文池 <wuwenchi@deepexi.com>

namrathamyske pushed a commit to namrathamyske/iceberg that referenced this pull request Jul 10, 2022

Docs: Add Flink and Iceberg type compatibility tables (apache#4865)

945e9c6

Co-authored-by: 吴文池 <wuwenchi@deepexi.com>


		### Flink type to Iceberg type

		This type conversion table describes how Spark types are converted to the Iceberg types. The conversion applies on both creating Iceberg table and writing to Iceberg table via Spark.


		## Type compatibility

		Flink and Iceberg support different set of types. Iceberg does the type conversion automatically, but not for all combinations,

Conversation

wuwenchi commented May 25, 2022

Uh oh!

kbendick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wuwenchi commented May 26, 2022

Uh oh!

samredai commented May 27, 2022

Uh oh!

wuwenchi commented May 28, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wuwenchi commented May 30, 2022

Uh oh!

samredai commented May 30, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wuwenchi Jun 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue commented Jun 29, 2022

Uh oh!

samredai commented Jun 29, 2022

Uh oh!

wuwenchi commented Jun 30, 2022

Uh oh!

rdblue commented Jul 3, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wuwenchi Jun 10, 2022 •

edited

Loading