Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compact documentation update #1447

Merged
merged 3 commits into from Dec 16, 2022
Merged

Compact documentation update #1447

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
903560b
Compact documentation update (#1445)
srknzl Dec 16, 2022
0d4e2b0
Add supported types
srknzl Dec 16, 2022
d4ad872
Add >
srknzl Dec 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
204 changes: 123 additions & 81 deletions DOCUMENTATION.md
View file
Open in desktop
Expand Up @@ -20,10 +20,13 @@
* [3.1 Client Properties](#31-client-properties)
* [4. Serialization](#4-serialization)
* [4.1. Compact Serialization](#41-compact-serialization)
* [4.1.1. Schema Evolution](#411-schema-evolution)
* [4.1.2. Generic Record Representation](#412-generic-record-representation)
* [4.1.3. Compact Configuration and Usage](#413-compact-configuration-and-usage)
* [4.1.4. Limitations](#414-limitations)
* [4.1.1. Configuration](#411-configuration)
* [4.1.2. Implementing CompactSerializer](#412-implementing-compactserializer)
* [4.1.3. Supported Types](#413-supported-types)
* [4.1.4. Schema Evolution](#414-schema-evolution)
* [4.1.5. Generic Record Representation](#415-generic-record-representation)
* [4.1.6. SQL Support](#416-sql-support)
* [4.1.7. Limitations](#417-limitations)
* [4.2. IdentifiedDataSerializable Serialization](#42-identifieddataserializable-serialization)
* [4.3. Portable Serialization](#43-portable-serialization)
* [4.3.1. Versioning for Portable Serialization](#431-versioning-for-portable-serialization)
Expand Down Expand Up @@ -719,27 +722,27 @@ Or, if you want to use your own serialization method, you can use [Custom Serial

## 4.1. Compact Serialization

> **NOTES: Supported in client version 5.1+ and server version 5.0+. However, it is recommended to use 5.1+ client with a 5.1+
server.
> **NOTES: Compact serialization is promoted to the stable status in the 5.2 release. The older versions released under
the BETA status are not compatible with stable the 5.2 server version.**

As an enhancement to existing serialization methods, Hazelcast offers the compact serialization, with the
following main features.

* Separates the schema from the data and stores it per type, not per object which results in less memory and bandwidth usage
compared to other formats.
* Does not require changing the source code of the class in any way.
* Does not require a class to implement an interface or change the source code of the class in any way.
* Supports schema evolution which permits adding or removing fields, or changing the types of fields.
* Platform and language independent.
* Supports partial deserialization of fields, without deserializing the whole objects during queries or indexing.
* Supports partial deserialization of fields during queries or indexing.

Hazelcast achieves these features by having a well-known schema of objects and replicating them across the cluster which enables
Hazelcast achieves these features by having well-known schemas of objects and replicating them across the cluster which enables
members and clients to fetch schemas they don’t have in their local registries. Each serialized object carries just a schema
identifier and relies on the schema distribution service or configuration to match identifiers with the actual schema. Once the
schemas are fetched, they are cached locally on the members and clients so that the next operations that use the schema do not
incur extra costs.

Schemas help Hazelcast to identify the locations of the fields on the serialized binary data. With this information, Hazelcast
can deserialize individual fields of the data, without reading the whole binary. This results in a better query and indexing
can deserialize individual fields of the data, without deserializing the whole binary. This results in a better query and indexing
performance.

Schemas can evolve freely by adding or removing fields. Even the types of the fields can be changed. Multiple versions of the
Expand All @@ -749,13 +752,94 @@ is especially useful in rolling upgrade scenarios.
The Compact serialization does not require any changes in the user classes as it doesn’t need a class to implement a particular
interface. Serializers might be implemented and registered separately from the classes.

Zero config option is not supported in Node.js client due to technical limitations. You need to register compact serializers
in the compact serialization config.
The underlying format of the compact serialized objects is platform and language independent.

The underlying format of the compact serialized objects is platform and language independent. All clients will have this feature
in the future.
### 4.1.1. Configuration

### 4.1.1. Schema Evolution
The only thing you need to configure is to register compact serializers. You can do that via
`serialization.compact.serializers` configuration option.

You have to supply a unique type name for the class in the serializer.

Choosing a type name will associate that name with the schema and will make the polyglot use cases, where there are multiple
clients from different languages, possible. Serializers in different languages can work on the same data, provided that their
read and write methods are compatible, and they have the same type name.


> **NOTES: If you evolve your class in the later versions of your application, by adding or removing fields, you should continue
using the same type name for that class.**

The following is an example of compact configuration:

```js
const client = await Client.newHazelcastClient({
serialization: {
compact: {
serializers: [new EmployeeSerializer()]
}
}
});
```

### 4.1.2. Implementing CompactSerializer

Compact serialization can be used by implementing a `CompactSerializer` for a class and registering it in the configuration.

For example, assume that you have the following `Employee` class.

```js
class Employee {
constructor(age, id) {
this.age = age;
this.id = id;
}
}
```

Then, a Compact serializer can be implemented as below.


```js
class EmployeeSerializer {
getTypeName() {
return 'Employee';
}

getClass() {
return Employee;
}

read(reader) {
const age = reader.readInt32('age');
const id = reader.readInt64('id');
return new Employee(age, id);
}

write(writer, value) {
writer.writeInt32('age', value.age);
writer.writeInt64('id', value.id);
}
}
```

The last step is to register the serializer in the member or client configuration, as shown in the
[Configuration](#411-configuration) section.

Upon serialization, a schema will be created from the serializer, and a unique schema identifier will be assigned to it
automatically.

After the configuration registration, Hazelcast will serialize instances of the `Employee` class using the `EmployeeSerializer`.

### 4.1.3. Supported Types

Compact serialization supports the types in this list
[in the reference manual](https://docs.hazelcast.com/hazelcast/latest/serialization/compact-serialization#supported-types)
as first class types. Any other type can be implemented on top of these, by using these types as building blocks.

> **NOTE: Compact serialization supports circularly-dependent types, provided that the cycle ends at some point on runtime by some
> null value.**

### 4.1.4. Schema Evolution

Compact serialization permits schemas and classes to evolve by adding or removing fields, or by changing the types of fields.
More than one version of a class may live in the same cluster and different clients or members might use different versions
Expand Down Expand Up @@ -789,12 +873,12 @@ class Employee {
constructor(id, name, age) {
this.id = id; // int32
this.name = name; // string
this.age = age; // int64
this.age = age; // int64, Newly added field
}
}
```

When faced with binary data serialized by the new writer, old readers will be able to read the following fields.
Then, when faced with binary data serialized by the new writer, old readers will be able to read the following fields.

```js
// CompactSerializer's read method
Expand All @@ -807,26 +891,31 @@ read(reader) {
}
```

When faced with binary data serialized by the old writer, new readers will be able to read the following fields.
Also, you can use the `getFieldKind` method to handle the case where such field does not exist.
Then, when faced with binary data serialized by the old writer, new readers will be able to read the following fields.
Also, Node.js client provides convenient APIs to check the existence of fields in the data when there is no such field.

```js
// CompactSerializer's read method
read(reader) {
const id = reader.readInt64("id");
const name = reader.readString("name");
// Read the "age" if it exists or set it to the default value, 0.
// Read the "age" if it exists or use the default value 0.
// reader.readInt32("age") would throw if the "age" field
// does not exist in data.
const age = reader.getFieldKind("age") === FieldKind.INT32 ? reader.readInt32("age") : 0;
return new Employee(id, name, age);
}
```

Note that, when an old reader reads a data written by an old writer, or a new reader reads a data written by a new writer,
they will be able to read all fields.
Note that, when an old reader reads data written by an old writer, or a new reader reads a data written by a new writer, they
will be able to read all fields written.

One thing to be careful while evolving the class is to not have any conditional code in the `write` method.
That method must write all the fields available in the current version of the class to the writer, with appropriate field names
and types. Node.js client uses the `write` method of the serializer to extract a schema out of the object, hence any conditional
code that may or may not run depending on the object in that method might result in an undefined behavior.

### 4.1.2. Generic Record Representation
### 4.1.5. Generic Record Representation

Compact serialized objects can also be represented by a GenericRecord. A GenericRecord is the representation of some object
when the client does not have the serializer/configuration to construct it. For example, if you read a compact object and
Expand Down Expand Up @@ -879,68 +968,21 @@ const anotherRecord = GenericRecords.compact('employee', fields, {
});
```

### 4.1.3. Compact Configuration and Usage
For more information, you can check
[the related page](https://docs.hazelcast.com/hazelcast/latest/clusters/accessing-domain-objects) in Hazelcast reference
documentation.

The only thing you need to configure is to register compact serializers. You can do that via
`serialization.compact.serializers` configuration option. The following is an example of compact configuration:

```js
'use strict';
const { Client } = require('hazelcast-client');
const Long = require('long');

class Employee {
constructor(age, id) {
this.age = age;
this.id = id;
}
}

class EmployeeSerializer {
getTypeName() {
return 'Employee';
}

getClass() {
return Employee;
}

read(reader) {
const age = reader.readInt32('age');
const id = reader.readInt64('id');
return new Employee(age, id);
}

write(writer, value) {
writer.writeInt32('age', value.age);
writer.writeInt64('id', value.id);
}
}

async function main() {
const client = await Client.newHazelcastClient({
serialization: {
compact: {
serializers: [new EmployeeSerializer()]
}
}
});
const map = await client.getMap('mapName');
await map.put(20, new Employee(23, Long.fromNumber(1)));

const employee = await map.get(20);
console.log(employee);

await client.shutdown();
}
```
### 4.1.6. SQL Support

Here, `employee` is an instance of Employee class.
Compact serialized objects can be used in SQL statements, provided that a mapping is created, similar to other serialization
formats. See [Compact Object mappings section](https://docs.hazelcast.com/hazelcast/latest/sql/mapping-to-maps#compact-objects)
in Hazelcast reference manual to learn more.

### 4.1.4. Limitations
### 4.1.7. Limitations

* Compact serialization does not work with lazy deserialization if the schema needs to be fetched. This is due to technical
limitations. In the future, lazy deserialization **may** be removed to make compact work with such APIs.
APIs with lazy deserialization, e.g `ReadOnlyLazyList` may throw `HazelcastSerializationError` in case a compact object is read
and its schema is not known by the client. This is due to a technical limitation. If the schema is fetched by the client
already, it won't throw. To fetch the schema, you can use any API that reads an object with the same schema.

## 4.2. IdentifiedDataSerializable Serialization

Expand Down