Skip to content

[Doc] add scala user doc#1028

Merged
chaokunyang merged 2 commits intoapache:mainfrom
chaokunyang:add_scala_doc
Oct 28, 2023
Merged

[Doc] add scala user doc#1028
chaokunyang merged 2 commits intoapache:mainfrom
chaokunyang:add_scala_doc

Conversation

@chaokunyang
Copy link
Collaborator

What do these changes do?

add scala user doc

Related issue number

Closes #1024

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@chaokunyang
Copy link
Collaborator Author

Hi @pjfanning , I just add a simple scala doc. It would be great if you can take a look at it.

@pjfanning
Copy link
Member

I did a quick test with Scala List and it looks like a lot of internal Scala classes are needed under the hood and they all need to be explicitly registered (unless you disable the need for registration).

I can come back to this but it could take a while. I suspect that the list of Scala library classes that need to be registered is going to be very large.

To make matters worse, Scala 2.13 rewrote a lot of Scala code relating to the collections - so the internal classes will be different from Scala 2.12. Any code to register a curated list of Scala library classes will need to try to register classes but move on if some of them fail.

This could be a fair amount of work.

@chaokunyang
Copy link
Collaborator Author

That would be an issue. Registered should be stable across scala versions.

Do we need to support registration check for some clas pattern such as scala.collection?

@pjfanning
Copy link
Member

It might be useful to support automated registration of every class in the scala package and subpackages - not just scala.collection. You might be able to exclude packages like scala.reflect. Ideally, the creation of the serializers should be lazy - only created when needed but the idea is that scala package classes would be allowed without explicit registration. I would support the idea that a registerScalaClasses call would be needed to enable this - but that this call would not to need to run through a large list of explicit class names.

@chaokunyang
Copy link
Collaborator Author

chaokunyang commented Oct 25, 2023

Auto registration is a good idea. But The registered class will be written by an auto grown Id. the order can't be ensured across scala version if we support auto register unless we add a new inferface like register but write class as name still.

@chaokunyang
Copy link
Collaborator Author

Hi @pjfanning , do you think we should merge this pr now, or hold it until we have a more scala-native serialization support?

There are some fury scala users already: #1029

@pjfanning
Copy link
Member

It's fine to merge as is.

@chaokunyang chaokunyang merged commit b515ee1 into apache:main Oct 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Java] serializing Scala Enumerations is complicated

2 participants