Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-34123][core][type] Introduce built-in serialization support for map and lists #24634

Merged
merged 2 commits into from
May 27, 2024

Conversation

X-czh
Copy link
Contributor

@X-czh X-czh commented Apr 8, 2024

What is the purpose of the change

Introduce built-in serialization support for map and lists, which falls back to Kyro previously.

Brief change log

Introduce built-in serialization support for map and lists when extracting types. The serializers and de-serializers have already been defined previously.

Set is another common Java collection type, but there's no built-in serializer for it yet (MultiSetTypeInformation utilizes MapSerializer underneath, but it could be more efficient for common Set), we can add support for it in a following PR.

Verifying this change

This change added tests that validates maps and lists are serialized using the built-in serializer instead of Kyro.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Apr 8, 2024

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@X-czh
Copy link
Contributor Author

X-czh commented Apr 8, 2024

@reswqa Hi, could you help take a look when you have time?

Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@X-czh Sorry for the delay. LGTM except the documentation part, could you please address that and rebase on latest master branch?

@X-czh
Copy link
Contributor Author

X-czh commented May 23, 2024

@reswqa Thanks for the review. I've rebased on latest master branch, and it has been highlighted in the doc content as follows:

Currently, only Map, List and its super interface Collection are supported.

Would you mind taking a look again when you have time?

@reswqa
Copy link
Member

reswqa commented May 23, 2024

Thanks @X-czh, I think we only need two commits. One for the implementation and one for documentation.

@X-czh
Copy link
Contributor Author

X-czh commented May 23, 2024

Thanks @X-czh, I think we only need two commits. One for the implementation and one for documentation.

Thanks for the notice, @reswqa. I've rearranged the commits to have only two commits: one for impl, one for doc.

Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will merge this after CI passed.

@reswqa reswqa merged commit f860631 into apache:master May 27, 2024
@X-czh X-czh deleted the FLINK-34123 branch May 27, 2024 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants