Skip to content

Conversation

@jerryjzhang
Copy link
Contributor

What is the purpose of the change

Since 1.6 the recommended way of creating source/sink table is using connector/format/schema/ descriptors. However, when declaring map types in the schema descriptor, the following exception would be thrown:

org.apache.flink.table.api.TableException: A string representation for map types is not supported yet.

This pull request is to add string representation support for map types.

Brief change log

  • Add new Keyword and PackratParser for map types in TypeStringUtils
  • Add normalization logic for map types in TypeStringUtils.writeTypeInfo()
  • Update relevant unit tests in TypeStringUtilsTest and TableDescriptorTest

Verifying this change

This change can be verified covered by updated unit tests, such as TypeStringUtilsTest and TableDescriptorTest.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@jerryjzhang jerryjzhang changed the title [FLINK-10170 [table] Support string representation for map types in d… [FLINK-10170 [table] Support string representation for map types in descriptor-based Table API Aug 19, 2018
@jerryjzhang jerryjzhang changed the title [FLINK-10170 [table] Support string representation for map types in descriptor-based Table API [FLINK-10170] [table] Support string representation for map types in descriptor-based Table API Aug 19, 2018
@Jiayi-Liao
Copy link
Contributor

@tragicjun +1, this is good, but can you implement the same thing on other types like array? I think they should be in one PR.

Copy link
Contributor

@twalthr twalthr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @tragicjun. The changes look good to me. One thing that we could improve is making the current implementation more consistent. So far we are using ROW(...) (with parenthesis), but after looking into other implementations such as Hive, we should have used Row<> as well. We should update this in a backwards compatible way. Either in this PR or in a follow-up issue. What do you think?

Can you also update the documentation?

@jerryjzhang
Copy link
Contributor Author

@twalthr In the latest commit, I've replaced bracket () with <> for all complex types and updated the documentation. To ensure backward compatibility, bracket () is still supported by the parsers. Please let me know if anything else is missing.

@jerryjzhang jerryjzhang force-pushed the flink-10170 branch 4 times, most recently from 64ebfcb to d54672c Compare August 20, 2018 16:24
@jerryjzhang
Copy link
Contributor Author

@twalthr could you review this again? Anything else I should do before the merge, thanks.

@jerryjzhang jerryjzhang force-pushed the flink-10170 branch 2 times, most recently from 2d08599 to e3f4bad Compare August 29, 2018 17:41
@jerryjzhang
Copy link
Contributor Author

@buptljy In the latest commit, the array types are supported as well, please check it out.

@jerryjzhang jerryjzhang changed the title [FLINK-10170] [table] Support string representation for map types in descriptor-based Table API [FLINK-10170] [table] Support string representation for map and array types in descriptor-based Table API Aug 29, 2018
@Jiayi-Liao
Copy link
Contributor

lgtm.

@jerryjzhang
Copy link
Contributor Author

Ready to be merged, could you take a look @fhueske @twalthr ?

@twalthr
Copy link
Contributor

twalthr commented Sep 11, 2018

Thank you @tragicjun. The changes look good to me. I will take care of merging this...

@asfgit asfgit closed this in 1ae5983 Sep 11, 2018
asfgit pushed a commit that referenced this pull request Sep 11, 2018
…PI types

Since 1.6 the recommended way of creating source/sink tables is using
connector/format/schema descriptors. This commit adds string-based
representation for all types supported by the Table & SQL API.

We use a syntax similar to Hive and other SQL projects.

This closes #6578.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants