Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions docs/guide/scala_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,25 @@ order: 4
-- fury_frontmatter -->

# Scala serialization
Fury supports all scala object serialization by keeping compatible with JDK serialization API: `writeObject/readObject/writeReplace/readResolve/readObjectNoData/Externalizable`:
Fury supports all scala object serialization:
- `case` class serialization supported
- `pojo/bean` class serialization supported
- `object` singleton serialization supported
- `collection` serialization supported
- other types such as `tuple/either` and basic types are all supported too.

## Fury creation
When using fury for scala serialization, you should create fury at least with following options:
```scala
val fury = Fury.builder().requireClassRegistration(false).withRefTracking(true).build()
val fury = Fury.builder()
.withScalaOptimizationEnabled(true)
.requireClassRegistration(false)
.withRefTracking(true)
.build()
```
Otherwise if you serialize some scala types such as `object/Enumeration`, you will need to register some scala internal types, such as:
Otherwise if you serialize some scala types such as `collection/Enumeration`, you will need to register some scala internal types:
```scala
fury.register(classOf[ModuleSerializationProxy])
fury.register(Class.forName("scala.collection.generic.DefaultSerializationProxy"))
fury.register(Class.forName("scala.Enumeration.Val"))
```
And circular references are common in scala, `Reference tracking` should be enabled by `FuryBuilder#withRefTracking(true)`. If you don't enable reference tracking, [StackOverflowError](https://github.com/alipay/fury/issues/1032) may happen for some scala versions when serializing scala Enumeration.
Expand Down Expand Up @@ -53,9 +58,11 @@ println(fury.deserialize(fury.serialize(map)))
```

# Performance
Scala collections and generics doesn't follow java collection framework, and is not fully integrated with Fury JIT,
the execution will invoke JDK serialization API with fury `ObjectStream` implementation for scala collections,
the performance won't be as good as fury collections serialization for java. In future we may provide jit support for scala collections to
get better performance, see https://github.com/alipay/fury/issues/682.
Scala `pojo/bean/case/object` are supported by fury jit well, the performance is as good as fury java.

Scala collections and generics doesn't follow java collection framework, and is not fully integrated with Fury JIT. The performance won't be as good as fury collections serialization for java.

`case` object are supported by fury jit well, the performance is as good as fury java.
The execution for scala collections will invoke Java serialization API `writeObject/readObject/writeReplace/readResolve/readObjectNoData/Externalizable` with fury `ObjectStream` implementation. Although `io.fury.serializer.ObjectStreamSerializer` is much faster than JDK `ObjectOutputStream/ObjectInputStream`, but it still doesn't know how use scala collection generics.

In future we may provide jit support for scala collections to
get better performance, see https://github.com/alipay/fury/issues/682.
7 changes: 7 additions & 0 deletions java/fury-core/src/main/java/io/fury/config/Config.java
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ public class Config implements Serializable {
private final boolean shareMetaContext;
private final boolean asyncCompilationEnabled;
private final boolean deserializeUnexistedClass;
private final boolean scalaOptimizationEnabled;
private transient int configHash;

public Config(FuryBuilder builder) {
Expand Down Expand Up @@ -79,6 +80,7 @@ public Config(FuryBuilder builder) {
Preconditions.checkArgument(shareMetaContext || compatibleMode == CompatibleMode.COMPATIBLE);
}
asyncCompilationEnabled = builder.asyncCompilationEnabled;
scalaOptimizationEnabled = builder.scalaOptimizationEnabled;
}

public Language getLanguage() {
Expand Down Expand Up @@ -190,6 +192,11 @@ public boolean isAsyncCompilationEnabled() {
return asyncCompilationEnabled;
}

/** Whether enable scala-specific serialization optimization. */
public boolean isScalaOptimizationEnabled() {
return scalaOptimizationEnabled;
}

public int getConfigHash() {
if (configHash == 0) {
// TODO use a custom encoding to ensure different config hash different hash.
Expand Down
13 changes: 10 additions & 3 deletions java/fury-core/src/main/java/io/fury/config/FuryBuilder.java
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,10 @@ public final class FuryBuilder {
boolean requireClassRegistration = true;
boolean shareMetaContext = false;
boolean codeGenEnabled = true;
public boolean deserializeUnexistedClass = false;
public boolean asyncCompilationEnabled = false;
public boolean registerGuavaTypes = true;
boolean deserializeUnexistedClass = false;
boolean asyncCompilationEnabled = false;
boolean registerGuavaTypes = true;
boolean scalaOptimizationEnabled = false;

public FuryBuilder() {}

Expand Down Expand Up @@ -247,6 +248,12 @@ public FuryBuilder withAsyncCompilation(boolean asyncCompilation) {
return this;
}

/** Whether enable scala-specific serialization optimization. */
public FuryBuilder withScalaOptimizationEnabled(boolean enableScalaOptimization) {
this.scalaOptimizationEnabled = enableScalaOptimization;
return this;
}

private void finish() {
if (classLoader == null) {
classLoader = Thread.currentThread().getContextClassLoader();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@
import io.fury.serializer.UnexistedClassSerializers.UnexistedMetaSharedClass;
import io.fury.serializer.UnexistedClassSerializers.UnexistedSkipClass;
import io.fury.serializer.UnmodifiableSerializers;
import io.fury.serializer.scala.SingletonObjectSerializer;
import io.fury.type.ClassDef;
import io.fury.type.Descriptor;
import io.fury.type.GenericType;
Expand Down Expand Up @@ -849,6 +850,10 @@ public Class<? extends Serializer> getSerializerClass(Class<?> cls, boolean code
String.format("Class %s doesn't support serialization.", cls));
}
}
if (fury.getConfig().isScalaOptimizationEnabled()
&& ReflectionUtils.isScalaSingletonObject(cls)) {
return SingletonObjectSerializer.class;
}
if (Collection.class.isAssignableFrom(cls)) {
// Serializer of common collection such as ArrayList/LinkedList should be registered
// already.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/*
* Copyright 2023 The Fury Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.fury.serializer.scala;

import io.fury.Fury;
import io.fury.memory.MemoryBuffer;
import io.fury.serializer.Serializer;
import io.fury.util.Platform;
import java.lang.reflect.Field;

/**
* Serializer for <a href="https://docs.scala-lang.org/tour/singleton-objects.html">scala
* singleton</a>.
*
* @author chaokunyang
*/
// TODO(chaokunyang) add scala tests.
@SuppressWarnings("rawtypes")
public class SingletonObjectSerializer extends Serializer {
private final long offset;

public SingletonObjectSerializer(Fury fury, Class type) {
super(fury, type);
try {
Field field = type.getDeclaredField("MODULE$");
offset = Platform.UNSAFE.staticFieldOffset(field);
} catch (NoSuchFieldException e) {
throw new RuntimeException(type + " doesn't have `MODULE$` field", e);
}
}

@Override
public void write(MemoryBuffer buffer, Object value) {}

@Override
public Object read(MemoryBuffer buffer) {
return Platform.getObject(type, offset);
}
}
10 changes: 10 additions & 0 deletions java/fury-core/src/main/java/io/fury/util/ReflectionUtils.java
Original file line number Diff line number Diff line change
Expand Up @@ -609,4 +609,14 @@ public static boolean isDynamicGeneratedCLass(Class<?> cls) {
// TODO(chaokunyang) add cglib check
return Functions.isLambda(cls) || isJdkProxy(cls);
}

/** Returns true if a class is a scala `object` singleton. */
public static boolean isScalaSingletonObject(Class<?> cls) {
try {
cls.getDeclaredField("MODULE$");
return true;
} catch (NoSuchFieldException e) {
return false;
}
}
}
9 changes: 5 additions & 4 deletions scala/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Fury Scala
Fury supports all scala object serialization:
- case class serialization supported
- object singleton serialization supported
- collection serialization supported
- other types such as tuple/either and basic types are all supported too.
- `case` class serialization supported
- `pojo/bean` class serialization supported
- `object` singleton serialization supported
- `collection` serialization supported
- other types such as `tuple/either` and basic types are all supported too.

For user document, see [scala_guide](../docs/guide/scala_guide.md).
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ class CollectionSerializerTest extends AnyWordSpec with Matchers {
val fury: Fury = Fury.builder()
.withLanguage(Language.JAVA)
.withRefTracking(true)
.withScalaOptimizationEnabled(true)
.requireClassRegistration(false).build()

"fury scala collection support" should {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Copyright 2023 The Fury Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package io.fury.serializer

import io.fury.Fury
import io.fury.config.Language
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpec

object singleton {}

case class Pair(f1: Any, f2: Any)

class SingleObjectSerializerTest extends AnyWordSpec with Matchers {
"fury scala object support" should {
"serialize/deserialize" in {
val fury = Fury.builder()
.withLanguage(Language.JAVA)
.withRefTracking(true)
.withScalaOptimizationEnabled(true)
.requireClassRegistration(false).build()
fury.deserialize(fury.serialize(singleton)) shouldBe singleton
fury.deserialize(fury.serialize(Pair(singleton, singleton))) shouldEqual Pair(singleton, singleton)
}
}
}