diff --git a/.agents/docs-and-formatting.md b/.agents/docs-and-formatting.md index 1d1d6da40c..e4d2d60dda 100644 --- a/.agents/docs-and-formatting.md +++ b/.agents/docs-and-formatting.md @@ -37,7 +37,7 @@ Load this file when changing documentation, public APIs, protocol specs, benchma - Keep user-facing configuration text action focused. Do not put cache ownership, cold/hot path, publish ordering, exact-local schema bypass, or TypeDef/TypeMeta routing invariants there; put implementation invariants in `docs/specification/xlang_implementation_guide.md`, - security classifications in `docs/object-serialization/deserialization-security-model.md`, and agent-only workflow rules in + security classifications in `docs/security/deserialization.md`, and agent-only workflow rules in `.agents/**`. - Documentation examples should use normal explicit imports, avoid unused imports, and keep common Fory types unqualified where that is the idiom. - Generated Markdown under `docs/benchmarks/**` should satisfy markdownlint blank-line rules at generation time: no repeated blank lines and no extra blank line after final content. diff --git a/.github/sync.yml b/.github/sync.yml index 05c5290b32..ce497fe804 100644 --- a/.github/sync.yml +++ b/.github/sync.yml @@ -18,6 +18,9 @@ apache/fory-site@main: - source: docs/index.md dest: docs/index.md + - source: docs/images/ + dest: docs/images/ + deleteOrphaned: true - source: docs/introduction/ dest: docs/introduction/ deleteOrphaned: true @@ -45,9 +48,6 @@ apache/fory-site@main: - source: docs/development/ dest: docs/development/ deleteOrphaned: true - - source: docs/images/ - dest: docs/images/ - deleteOrphaned: true - source: docs/specification/ dest: docs/specification/ deleteOrphaned: true diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index db809611bb..5585589a19 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2011,6 +2011,8 @@ jobs: uses: actions/setup-python@v5 with: python-version: 3.8 + - name: Test release documentation updater + run: python -m unittest ci.test_release - name: Set up .NET 8 uses: actions/setup-dotnet@v4 with: diff --git a/AGENTS.md b/AGENTS.md index 6f6dc1a67f..1d458373f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,11 +9,11 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th - `.agents/docs-and-formatting.md`: documentation, specification, and markdown rules. - `.agents/ci-and-pr.md`: code review workflow, CI triage, PR expectations, and commit conventions. - `.agents/testing/integration-tests.md`: `integration_tests/` prerequisites, regeneration rules, and commands. -- `docs/object-serialization/security.md`: user-facing security guidance for binary object - serialization. +- `docs/security/index.md`: contributor-facing security model index. This directory is internal + documentation and is intentionally excluded from the website sync. +- `docs/security/threat-model.md`: project trust boundaries and downstream responsibilities. +- `docs/security/deserialization.md`: implementation boundaries for untrusted deserialization. - `docs/json/security.md`: user-facing security guidance for Fory JSON. -- `docs/object-serialization/deserialization-security-model.md`: implementation boundaries for - untrusted deserialization classification. - `.agents/languages/java.md` - `.agents/languages/csharp.md` - `.agents/languages/cpp.md` @@ -34,7 +34,7 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th - Respect ownership. Keep logic, state, and helpers in their natural owner, and do not move serializer-local, context-local, runtime-type-local, or protocol-local problems into global utilities. - Check the spec before implementation. For wire behavior and xlang mapping, use the specs as the source of truth and never copy one runtime's bug into another runtime just to make tests pass. - Do not make assumptions about runtime behavior, ownership, registration, metadata construction, protocol semantics, or test coverage. Read the current code, owning docs/specs, and relevant tests before making a design judgment or implementation decision. If the evidence is incomplete, inspect more or state the uncertainty explicitly instead of filling gaps from memory or analogy with another runtime. -- For untrusted deserialization, read `docs/object-serialization/deserialization-security-model.md` before changing allocation, stream filling, skip, reference, metadata, or policy validation behavior. Variable-length deserialization must not allocate or reserve backing/output capacity from attacker-declared lengths or counts before the byte owner has proven proportional readable bytes with `checkReadableBytes` or the runtime equivalent. Root graph memory reservation is accounting only and may happen before that byte check, but it must not replace the byte check. +- For untrusted deserialization, read `docs/security/deserialization.md` before changing allocation, stream filling, skip, reference, metadata, or policy validation behavior. Variable-length deserialization must not allocate or reserve backing/output capacity from attacker-declared lengths or counts before the byte owner has proven proportional readable bytes with `checkReadableBytes` or the runtime equivalent. Root graph memory reservation is accounting only and may happen before that byte check, but it must not replace the byte check. - Malformed input must surface as a controlled root-operation error and still run root cleanup, but the exact exception type, error code, message, detection layer, and detection point are not contracts unless a public API or @@ -331,7 +331,8 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th ## Security -User-facing security guidance lives only under Object Serialization and Fory JSON. Read -`docs/object-serialization/security.md` or `docs/json/security.md` for the selected product. Before -reporting or changing allocation, stream filling, skip, reference, metadata, or policy validation -behavior, read `docs/object-serialization/deserialization-security-model.md`. +User-facing object-serialization security guidance lives in each runtime directory, such as +`docs/object-serialization/java/security.md`; Fory JSON guidance lives in `docs/json/security.md`. +Contributor-facing security models remain under `docs/security/` and are excluded from the website. +Before reporting or changing allocation, stream filling, skip, reference, metadata, or policy +validation behavior, read `docs/security/deserialization.md`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1cd0c63552..3b2d53c4dd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -42,7 +42,7 @@ Key points: ## Testing -For environmental requirements, please check [DEVELOPMENT.md](./docs/development/building.md). +For environmental requirements, see the [Development guide](./docs/development/index.md). ### Python @@ -229,7 +229,7 @@ To use Jetbrains IDEA IDE for Java Development, you need to configure the projec And due to the usage of `sun.misc.Unsafe` API, which is not visible in Java 11+, you need to configure java compiler with `--releaese` option disabled.
-
+
## Website @@ -244,4 +244,4 @@ If you want write a blog, or update other contents about the website, please sub ## Development -For more information, please refer to [Development Guide](./docs/development/building.md). +For more information, see the [Development guide](./docs/development/index.md). diff --git a/README.md b/README.md index dc5d5cd4a4..4a5e7bcb86 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@
- Apache Fory logo
+ Apache Fory logo
[![Build Status](https://img.shields.io/github/actions/workflow/status/apache/fory/ci.yml?branch=main&style=for-the-badge&label=GITHUB%20ACTIONS&logo=github)](https://github.com/apache/fory/actions/workflows/ci.yml) @@ -305,19 +305,19 @@ serializer setup. **Development From Source** -See [docs/development/building.md](docs/development/building.md). +See [docs/development/index.md](docs/development/index.md). Snapshots for Java, Scala, and Kotlin are available from `https://repository.apache.org/snapshots/` with the matching `-SNAPSHOT` version. ## Choose a Serialization Format -| Format | Use it when | Start here | -| ------------- | ------------------------------------------------------------- | ------------------------------------------------------- | -| Xlang binary | Data crosses language boundaries | [Xlang guide](docs/object-serialization/xlang/index.md) | -| Native binary | Producer and consumer are in the same runtime family | [Native guide](docs/object-serialization/native.md) | -| Row format | You need random field access or analytics-style partial reads | [Row Format](docs/row-format/index.md) | -| Fory JSON | Java applications need high-performance standard JSON | [Fory JSON](java/fory-json/README.md) | +| Format | Use it when | Start here | +| ---------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- | +| Xlang binary (default) | Data crosses language boundaries | [Cross-language interoperability](docs/object-serialization/xlang.md) | +| Native binary | Producer and consumer are in the same runtime family | [Native guide](docs/object-serialization/native.md) | +| Row format | You need random field access or analytics-style partial reads | [Row Format](docs/row-format/index.md) | +| Fory JSON | Java applications need high-performance standard JSON | [Fory JSON](java/fory-json/README.md) | For Java, Scala, Kotlin, Python, C++, Go, and Rust, use native mode for same-language traffic. It avoids xlang's cross-language type mapping and @@ -616,7 +616,7 @@ fun main() { ``` For shared/circular references, polymorphism, numeric IDs versus names, and -type-mapping rules, see the [cross-language guide](docs/object-serialization/xlang) and +type-mapping rules, see the [cross-language guide](docs/object-serialization/xlang.md) and [type mapping specification](docs/specification/xlang_type_mapping.md). ## Native Serialization @@ -909,11 +909,11 @@ security, and platform guides. | Dart | [docs/object-serialization/dart](docs/object-serialization/dart) | [View](https://fory.apache.org/docs/object-serialization/dart/) | | Scala | [docs/object-serialization/scala](docs/object-serialization/scala) | [View](https://fory.apache.org/docs/object-serialization/scala/) | | Kotlin | [docs/object-serialization/kotlin](docs/object-serialization/kotlin) | [View](https://fory.apache.org/docs/object-serialization/kotlin/) | -| Cross-language xlang | [docs/object-serialization/xlang](docs/object-serialization/xlang) | [View](https://fory.apache.org/docs/object-serialization/xlang/) | +| Cross-language xlang | [docs/object-serialization/xlang.md](docs/object-serialization/xlang.md) | [View](https://fory.apache.org/docs/object-serialization/xlang) | | Schema IDL/compiler | [docs/compiler](docs/compiler) | [View](https://fory.apache.org/docs/compiler/) | | GraalVM native image | [docs/object-serialization/java/graalvm.md](docs/object-serialization/java/graalvm.md) | [View](https://fory.apache.org/docs/object-serialization/java/graalvm) | | Android | [docs/object-serialization/java/android.md](docs/object-serialization/java/android.md) | [View](https://fory.apache.org/docs/object-serialization/java/android) | -| Development | [docs/development/building.md](docs/development/building.md) | [View](docs/development/building.md) | +| Development | [docs/development/index.md](docs/development/index.md) | [View](docs/development/index.md) | **Specifications** @@ -934,7 +934,7 @@ security, and platform guides. ## Contributing Read [CONTRIBUTING.md](CONTRIBUTING.md) and -[docs/development/building.md](docs/development/building.md) before sending pull requests. Bug +[docs/development/index.md](docs/development/index.md) before sending pull requests. Bug reports, docs fixes, tests, benchmarks, and implementation improvements are welcome. ## License diff --git a/SECURITY.md b/SECURITY.md index d83379eae6..f90a9fafc2 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -24,10 +24,11 @@ GitHub issues or pull requests for security reports. ## Security Models -User-facing guidance is product-specific: +User-facing guidance is capability- and runtime-specific: -- [Object Serialization Security](docs/object-serialization/security.md) +- [Object Serialization runtime guides](docs/object-serialization/index.md) (each runtime section + ends with its own Security page) - [Fory JSON Security](docs/json/security.md) For detailed implementation classification rules for untrusted deserialization, see the -[Deserialization Security Model](docs/object-serialization/deserialization-security-model.md). +[Deserialization Security Model](docs/security/deserialization.md). diff --git a/ci/release.py b/ci/release.py index 1bc7ef12c9..258836eeb4 100644 --- a/ci/release.py +++ b/ci/release.py @@ -1237,6 +1237,19 @@ def _update_release_doc_lines(lines, release_version): def _update_release_doc_line(line, release_version): + scoped_patterns = ( + r"(\bpyfory(?:\[[^\]]+\])?==)" + VERSION_PATTERN, + r"(\bgithub\.com/apache/fory/go/fory@v)" + VERSION_PATTERN, + r"(@apache-fory/(?:core|hps)@)" + VERSION_PATTERN, + ) + scoped_update = line + has_scoped_dependency = False + for pattern in scoped_patterns: + if re.search(pattern, scoped_update): + has_scoped_dependency = True + scoped_update = re.sub(pattern, r"\g<1>" + release_version, scoped_update) + if has_scoped_dependency: + return scoped_update if not _is_release_doc_line(line): return line if "crates.io-v" in line: diff --git a/ci/test_release.py b/ci/test_release.py new file mode 100644 index 0000000000..f06acf3bbb --- /dev/null +++ b/ci/test_release.py @@ -0,0 +1,68 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import unittest + +if __package__: + from . import release +else: + import release + + +class ReleaseDocVersionTest(unittest.TestCase): + def test_updates_release_doc_dependencies(self): + lines = [ + "python -m pip install pyfory==1.5.0\n", + 'python -m pip install "pyfory[format]==1.5.0"\n', + "go get github.com/apache/fory/go/fory@v1.5.0\n", + "npm install @apache-fory/core@1.5.0 @apache-fory/hps@1.5.0\n", + "python -m pip install pyfory==1.5.0 grpcio==1.73.0 # Fory 1.4.0\n", + "go get github.com/apache/fory/go/fory@v1.5.0 # Fory 1.4.0\n", + "npm install @apache-fory/core@1.5.0 grpc@1.73.0 @apache-fory/hps@1.5.0\n", + 'fory = "1.5.0"\n', + " fory: 1.5.0\n", + ' .package(url: "https://github.com/apache/fory.git", exact: "1.5.0")\n', + 'implementation("org.apache.fory:fory-core:1.5.0")\n', + 'libraryDependencies += "org.apache.fory" %% "fory-scala" % "1.5.0"\n', + 'implementation("io.grpc:grpc-api:1.73.0")\n', + "The wire format was introduced in version 1.5.0.\n", + ] + expected = [ + "python -m pip install pyfory==1.6.0\n", + 'python -m pip install "pyfory[format]==1.6.0"\n', + "go get github.com/apache/fory/go/fory@v1.6.0\n", + "npm install @apache-fory/core@1.6.0 @apache-fory/hps@1.6.0\n", + "python -m pip install pyfory==1.6.0 grpcio==1.73.0 # Fory 1.4.0\n", + "go get github.com/apache/fory/go/fory@v1.6.0 # Fory 1.4.0\n", + "npm install @apache-fory/core@1.6.0 grpc@1.73.0 @apache-fory/hps@1.6.0\n", + 'fory = "1.6.0"\n', + " fory: 1.6.0\n", + ' .package(url: "https://github.com/apache/fory.git", exact: "1.6.0")\n', + 'implementation("org.apache.fory:fory-core:1.6.0")\n', + 'libraryDependencies += "org.apache.fory" %% "fory-scala" % "1.6.0"\n', + 'implementation("io.grpc:grpc-api:1.73.0")\n', + "The wire format was introduced in version 1.5.0.\n", + ] + + updated = release._update_release_doc_lines(lines, "1.6.0") + + self.assertEqual(expected, updated) + self.assertEqual(expected, release._update_release_doc_lines(updated, "1.6.0")) + + +if __name__ == "__main__": + unittest.main() diff --git a/ci/test_validate_fory_site_sync.py b/ci/test_validate_fory_site_sync.py new file mode 100644 index 0000000000..d7856f4c3c --- /dev/null +++ b/ci/test_validate_fory_site_sync.py @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import pathlib +import tempfile +import unittest + +if __package__: + from . import validate_fory_site_sync +else: + import validate_fory_site_sync + + +class ForySiteSyncTest(unittest.TestCase): + def test_rejects_forbidden_doc_roots(self): + for source, dest in ( + ("docs/security/", "docs/security/"), + ("docs/development/", "docs/security/"), + ): + content = f"""apache/fory-site@main: + - source: {source} + dest: {dest} +""" + with ( + self.subTest(source=source, dest=dest), + tempfile.TemporaryDirectory() as directory, + ): + sync_file = pathlib.Path(directory) / "sync.yml" + sync_file.write_text(content, encoding="utf-8") + with self.assertRaisesRegex(RuntimeError, "must not be synced"): + validate_fory_site_sync.parse_sync_mappings(sync_file) + + +if __name__ == "__main__": + unittest.main() diff --git a/ci/validate_fory_site_sync.py b/ci/validate_fory_site_sync.py index cce69c0411..83794e2f8b 100644 --- a/ci/validate_fory_site_sync.py +++ b/ci/validate_fory_site_sync.py @@ -26,6 +26,16 @@ from typing import List, Tuple TARGET_REPO = "apache/fory-site@main" +FORBIDDEN_SYNC_ROOTS = ( + pathlib.PurePosixPath("docs/security"), +) + + +def is_forbidden_sync_path(path: str) -> bool: + candidate = pathlib.PurePosixPath(path.rstrip("/")) + return any( + candidate == root or root in candidate.parents for root in FORBIDDEN_SYNC_ROOTS + ) def parse_sync_mappings(sync_file: pathlib.Path) -> List[Tuple[str, str]]: @@ -55,6 +65,10 @@ def parse_sync_mappings(sync_file: pathlib.Path) -> List[Tuple[str, str]]: dest_match = re.match(r"^dest:\s*(.+)$", stripped) if dest_match and source: dest = dest_match.group(1).strip().strip("'\"") + if is_forbidden_sync_path(source) or is_forbidden_sync_path(dest): + raise RuntimeError( + f"path must not be synced to fory-site: {source} -> {dest}" + ) mappings.append((source, dest)) source = None diff --git a/csharp/README.md b/csharp/README.md index 1deff49129..690a428bd4 100644 --- a/csharp/README.md +++ b/csharp/README.md @@ -127,7 +127,7 @@ public sealed class User : Entity An inaccessible base member is serialized only when it has `[ForyField]`. Unmodifiable third-party bases use an external `BaseOnly` declaration. See the -[class inheritance guide](https://fory.apache.org/docs/object-serialization/csharp/core-api/#class-inheritance). +[class inheritance guide](https://fory.apache.org/docs/object-serialization/csharp/basic-serialization/#class-inheritance). ### 3. Shared and Circular References diff --git a/docs/benchmarks/index.md b/docs/benchmarks/index.md index 46af48783d..f9a50cc987 100644 --- a/docs/benchmarks/index.md +++ b/docs/benchmarks/index.md @@ -1,5 +1,5 @@ --- -title: Benchmarks +title: Results sidebar_position: 1 id: index license: | diff --git a/docs/benchmarks/object-serialization/xlang/swift/README.md b/docs/benchmarks/object-serialization/xlang/swift/README.md index f3322f902b..91d4568221 100644 --- a/docs/benchmarks/object-serialization/xlang/swift/README.md +++ b/docs/benchmarks/object-serialization/xlang/swift/README.md @@ -2,11 +2,11 @@ This benchmark compares serialization and deserialization throughput for Apache Fory, Protocol Buffers, and JSON in Swift. -## Benchmark Products +## Benchmark Executables The ordinary/xlang cases are built by `swift-benchmark`. External-type and carrier comparisons are built by the separate `swift-external-benchmark` -product, so building the ordinary product does not compile those models or +executable, so building the ordinary executable does not compile those models or serializer specializations. ## Throughput Plot diff --git a/docs/compiler/generated-code/index.md b/docs/compiler/generated-code/index.md index bf76e92ae3..ac9908fa7b 100644 --- a/docs/compiler/generated-code/index.md +++ b/docs/compiler/generated-code/index.md @@ -1,5 +1,5 @@ --- -title: Generated Code +title: Overview sidebar_position: 1 id: index license: | diff --git a/docs/development/building.md b/docs/development/building.md deleted file mode 100644 index 4e6bb13846..0000000000 --- a/docs/development/building.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: Development -sidebar_position: 20 -id: building -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -## How to build Apache Fory™ - -Clone the source tree from https://github.com/apache/fory. - -### Build Apache Fory™ Java - -```bash -cd java -mvn -T16 package -``` - -#### Environment Requirements - -- JDK 17+ -- Maven 3.6.3+ - -### Build Apache Fory™ Python - -```bash -cd python -pip install -v -e . - -# Optional: build Cython extension (replace X.Y with your Python version) -bazel build //:cp_fory_so --@rules_python//python/config_settings:python_version=X.Y -``` - -#### Environment Requirements - -- CPython 3.8+ -- Bazel 8+ (required when building Cython extensions) - -### Build Apache Fory™ C++ - -```bash -cd cpp -bazel build //cpp/... -``` - -#### Environment Requirements - -- C++17 compiler -- Bazel 8+ - -### Build Apache Fory™ Go - -```bash -cd go/fory -go test -v ./... -``` - -Run Go xlang tests from Java test module: - -```bash -cd java -mvn -T16 install -DskipTests -cd fory-core -FORY_GO_JAVA_CI=1 ENABLE_FORY_DEBUG_OUTPUT=1 mvn test -Dtest=org.apache.fory.xlang.GoXlangTest -``` - -#### Environment Requirements - -- Go 1.24+ - -### Build Apache Fory™ Rust - -```bash -cd rust -cargo build -cargo test --features tests - -# Debug a specific test -RUST_BACKTRACE=1 FORY_PANIC_ON_ERROR=1 ENABLE_FORY_DEBUG_OUTPUT=1 \ - cargo test --test mod $dir$::$test_file::$test_method -- --nocapture -``` - -#### Environment Requirements - -- Rust toolchain via rustup -- `cargo-expand` (optional, for macro expansion debugging) - -### Build Apache Fory™ JavaScript - -```bash -cd javascript -npm install - -npm run build -node ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit -``` - -#### Environment Requirements - -- Node.js (LTS) -- npm - -### Lint Markdown Docs - -```bash -cd docs -npx prettier --write "**/*.md" -``` - -#### Environment Requirements - -- Node.js (LTS) -- npm - -## Contributing - -For contribution details, see [How to contribute to Apache Fory™](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). -For AI-assisted contributions, follow the -[AI Contribution Policy](https://github.com/apache/fory/blob/main/AI_POLICY.md), including the -required self-review, two-reviewer AI review loop, disclosure, and verification evidence for -substantial AI assistance. diff --git a/docs/development/cpp-debugging.md b/docs/development/cpp-debugging.md index 360eef894b..761a972e42 100644 --- a/docs/development/cpp-debugging.md +++ b/docs/development/cpp-debugging.md @@ -18,9 +18,9 @@ license: | limitations under the License. --- -import JumpGeneratedFile from "@site/docs/images/jmp_generate_file.png"; -import VscodeDebugFory from "@site/docs/images/vscode_debug_fory.jpg"; -import VscodeSelectDebugRun from "@site/docs/images/vscode_select_debug_run.png"; +import JumpGeneratedFile from "@site/docs/development/jmp_generate_file.png"; +import VscodeDebugFory from "@site/docs/development/vscode_debug_fory.jpg"; +import VscodeSelectDebugRun from "@site/docs/development/vscode_select_debug_run.png"; ## Debugging C++ diff --git a/docs/development/index.md b/docs/development/index.md index 7044b921de..1584659bb0 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -1,5 +1,5 @@ --- -title: Development +title: Build and Test sidebar_position: 1 id: index license: | @@ -22,11 +22,121 @@ license: | Contributor documentation covers repository setup, builds, tests, debugging, and release-oriented workflows. It is separate from the user guides for released artifacts. -## Start here +Clone the source tree from https://github.com/apache/fory. -- [Build and test the repository](building.md) -- [Debug the C++ implementation](cpp-debugging.md) -- [Contributing guide](https://github.com/apache/fory/blob/main/CONTRIBUTING.md) +## Java + +```bash +cd java +mvn -T16 package +``` + +Requirements: + +- JDK 17+ +- Maven 3.6.3+ + +## Python + +```bash +cd python +pip install -v -e . + +# Optional: build Cython extension (replace X.Y with your Python version) +bazel build //:cp_fory_so --@rules_python//python/config_settings:python_version=X.Y +``` + +Requirements: + +- CPython 3.8+ +- Bazel 8+ (required when building Cython extensions) + +## C++ + +```bash +cd cpp +bazel build //cpp/... +``` + +Requirements: + +- C++17 compiler +- Bazel 8+ + +## Go + +```bash +cd go/fory +go test -v ./... +``` + +Run Go xlang tests from the Java test module: + +```bash +cd java +mvn -T16 install -DskipTests +cd fory-core +FORY_GO_JAVA_CI=1 ENABLE_FORY_DEBUG_OUTPUT=1 mvn test -Dtest=org.apache.fory.xlang.GoXlangTest +``` + +Requirement: Go 1.24+. + +## Rust + +```bash +cd rust +cargo build +cargo test --features tests + +# Debug a specific test +RUST_BACKTRACE=1 FORY_PANIC_ON_ERROR=1 ENABLE_FORY_DEBUG_OUTPUT=1 \ + cargo test --test mod $dir$::$test_file::$test_method -- --nocapture +``` + +Requirements: + +- Rust toolchain via rustup +- `cargo-expand` (optional, for macro expansion debugging) + +## JavaScript + +```bash +cd javascript +npm install + +npm run build +node ./node_modules/.bin/jest --ci --reporters=default --reporters=jest-junit +``` + +Requirements: + +- Node.js (LTS) +- npm + +## Markdown + +```bash +cd docs +npx prettier --write "**/*.md" +``` + +Requirements: + +- Node.js (LTS) +- npm + +## Debugging + +See [Debugging C++](cpp-debugging.md) for the repository's VS Code, Bazel, LLDB, and GDB setup. Runtime-specific contributor instructions remain in each runtime's source tree and contributor README when they are not part of the shared repository build. + +## Contributing + +For contribution details, see +[How to contribute to Apache Fory™](https://github.com/apache/fory/blob/main/CONTRIBUTING.md). +For AI-assisted contributions, follow the +[AI Contribution Policy](https://github.com/apache/fory/blob/main/AI_POLICY.md), including the +required self-review, two-reviewer AI review loop, disclosure, and verification evidence for +substantial AI assistance. diff --git a/docs/images/jmp_generate_file.png b/docs/development/jmp_generate_file.png similarity index 100% rename from docs/images/jmp_generate_file.png rename to docs/development/jmp_generate_file.png diff --git a/docs/images/vscode_debug_fory.jpg b/docs/development/vscode_debug_fory.jpg similarity index 100% rename from docs/images/vscode_debug_fory.jpg rename to docs/development/vscode_debug_fory.jpg diff --git a/docs/images/vscode_select_debug_run.png b/docs/development/vscode_select_debug_run.png similarity index 100% rename from docs/images/vscode_select_debug_run.png rename to docs/development/vscode_select_debug_run.png diff --git a/docs/grpc/architecture.md b/docs/grpc/architecture.md deleted file mode 100644 index ffe4c30162..0000000000 --- a/docs/grpc/architecture.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Architecture -sidebar_position: 2 -id: architecture -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Generated service companions use normal gRPC servers, channels, method descriptors, deadlines, -status codes, interceptors, and streaming APIs. Fory-generated marshallers encode and decode the -generated request and response models. - -## Ownership boundary - -Fory can generate service companions for application-provided gRPC runtimes. -Those companions provide Fory serialization for request and response objects; -the application and gRPC stack still own listeners, channels, credentials, -authentication, authorization, deadlines, retries, and transport lifecycle. - -Fory packages do not add a gRPC implementation as a hard dependency. The application selects and -configures the runtime's gRPC libraries. - -## Generated service surface - -The compiler emits runtime-idiomatic service bases, clients or stubs, method metadata, and Fory -marshallers. Model generation is documented under -[Generated Code](../compiler/generated-code/index.md); the runtime pages document server and client -integration. diff --git a/docs/grpc/index.md b/docs/grpc/index.md index 5e019a79ff..a0c4712e0a 100644 --- a/docs/grpc/index.md +++ b/docs/grpc/index.md @@ -1,5 +1,5 @@ --- -title: Fory gRPC +title: Overview sidebar_position: 1 id: index license: | @@ -54,8 +54,52 @@ generation with `--grpc-python-mode=sync`. JavaScript uses `@grpc/grpc-js` for Node.js; browser clients are generated separately with `--grpc-web` and use `grpc-web`. +## Architecture + +Generated service companions use normal gRPC servers, channels, method descriptors, deadlines, +status codes, interceptors, and streaming APIs. Fory-generated marshallers encode and decode the +generated request and response models. + +### Ownership Boundary + +Fory can generate service companions for application-provided gRPC runtimes. Those companions +provide Fory serialization for request and response objects; the application and gRPC stack still +own listeners, channels, credentials, authentication, authorization, deadlines, retries, and +transport lifecycle. + +Fory packages do not add a gRPC implementation as a hard dependency. The application selects and +configures the runtime's gRPC libraries. + +### Generated Service Surface + +The compiler emits runtime-idiomatic service bases, clients or stubs, method metadata, and Fory +marshallers. Model generation is documented under +[Generated Code](../compiler/generated-code/index.md); the runtime pages document server and client +integration. + +## Interoperability + +Fory gRPC peers interoperate only when they use the same generated service contract, matching Fory +type identities, and compatible generated model schemas. + +### Protocol Boundary + +The transport is gRPC, but the message bytes are Fory payloads. Generic protobuf clients and server +reflection tools cannot decode those payloads as protobuf messages. Generate every peer through a +supported Fory compiler frontend. + +### Verification + +Test at least one unary call and every streaming shape used by the service. A protobuf +`UNIMPLEMENTED` or decode failure usually means the peer used an ordinary protobuf stub or a +different generated service contract. + ## Runtime Guides +Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Dart, Scala, and Kotlin have documented gRPC +companions. Use the [support matrix](../introduction/support-matrix.md) and the selected runtime page +for current dependencies and streaming support. + | Runtime | Guide | | --------------------- | ------------------------------------- | | Java | [Java](java.md) | @@ -68,6 +112,3 @@ Node.js; browser clients are generated separately with `--grpc-web` and use | Dart | [Dart](dart.md) | | Scala | [Scala](scala.md) | | Kotlin | [Kotlin](kotlin.md) | - -See [Architecture](architecture.md) for payload and transport ownership and -[Interoperability](interoperability.md) for peer compatibility. diff --git a/docs/grpc/interoperability.md b/docs/grpc/interoperability.md deleted file mode 100644 index 1de36c4885..0000000000 --- a/docs/grpc/interoperability.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -title: Interoperability -sidebar_position: 3 -id: interoperability -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Fory gRPC peers interoperate only when they use the same generated service contract, matching Fory -type identities, and compatible generated model schemas. - -## Protocol boundary - -The transport is gRPC, but the message bytes are Fory payloads. Generic protobuf clients and server -reflection tools cannot decode those payloads as protobuf messages. Generate every peer through a -supported Fory compiler frontend. - -## Supported generated companions - -Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Dart, Scala, and Kotlin have documented gRPC -companions. Use the [support matrix](../introduction/support-matrix.md) and the selected runtime page -for current dependencies and streaming support. - -## Verification - -Test at least one unary call and every streaming shape used by the service. A protobuf -`UNIMPLEMENTED` or decode failure usually means the peer used an ordinary protobuf stub or a -different generated service contract. diff --git a/docs/images/logo/fory-horizontal.png b/docs/images/fory-logo.png similarity index 100% rename from docs/images/logo/fory-horizontal.png rename to docs/images/fory-logo.png diff --git a/docs/images/idea_jdk11.png b/docs/images/idea-jdk11.png similarity index 100% rename from docs/images/idea_jdk11.png rename to docs/images/idea-jdk11.png diff --git a/docs/images/logo/fory-horizontal-black.png b/docs/images/logo/fory-horizontal-black.png deleted file mode 100644 index 2fff2442c4..0000000000 Binary files a/docs/images/logo/fory-horizontal-black.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal-black1.png b/docs/images/logo/fory-horizontal-black1.png deleted file mode 100644 index d2495dc8d6..0000000000 Binary files a/docs/images/logo/fory-horizontal-black1.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal-white.png b/docs/images/logo/fory-horizontal-white.png deleted file mode 100644 index 759074c222..0000000000 Binary files a/docs/images/logo/fory-horizontal-white.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal-white1.png b/docs/images/logo/fory-horizontal-white1.png deleted file mode 100644 index a310816cf9..0000000000 Binary files a/docs/images/logo/fory-horizontal-white1.png and /dev/null differ diff --git a/docs/images/logo/fory-horizontal1.png b/docs/images/logo/fory-horizontal1.png deleted file mode 100644 index eb7039f122..0000000000 Binary files a/docs/images/logo/fory-horizontal1.png and /dev/null differ diff --git a/docs/images/logo/fory-icon-black.png b/docs/images/logo/fory-icon-black.png deleted file mode 100644 index 5056720de4..0000000000 Binary files a/docs/images/logo/fory-icon-black.png and /dev/null differ diff --git a/docs/images/logo/fory-icon-white.png b/docs/images/logo/fory-icon-white.png deleted file mode 100644 index 6f75304c23..0000000000 Binary files a/docs/images/logo/fory-icon-white.png and /dev/null differ diff --git a/docs/images/logo/fory-icon.png b/docs/images/logo/fory-icon.png deleted file mode 100644 index fb3c9749f3..0000000000 Binary files a/docs/images/logo/fory-icon.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-black.png b/docs/images/logo/fory-vertical-black.png deleted file mode 100644 index 4a6cba6723..0000000000 Binary files a/docs/images/logo/fory-vertical-black.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-black1.png b/docs/images/logo/fory-vertical-black1.png deleted file mode 100644 index 2bef2eda27..0000000000 Binary files a/docs/images/logo/fory-vertical-black1.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-white.png b/docs/images/logo/fory-vertical-white.png deleted file mode 100644 index d2daf413e8..0000000000 Binary files a/docs/images/logo/fory-vertical-white.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical-white1.png b/docs/images/logo/fory-vertical-white1.png deleted file mode 100644 index c56e4a66bf..0000000000 Binary files a/docs/images/logo/fory-vertical-white1.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical.png b/docs/images/logo/fory-vertical.png deleted file mode 100644 index 8962eba810..0000000000 Binary files a/docs/images/logo/fory-vertical.png and /dev/null differ diff --git a/docs/images/logo/fory-vertical1.png b/docs/images/logo/fory-vertical1.png deleted file mode 100644 index 1fa11e55c4..0000000000 Binary files a/docs/images/logo/fory-vertical1.png and /dev/null differ diff --git a/docs/introduction/choose-a-format.md b/docs/introduction/choose-a-format.md index 0d7225c6a4..240590d1c5 100644 --- a/docs/introduction/choose-a-format.md +++ b/docs/introduction/choose-a-format.md @@ -19,12 +19,16 @@ license: | limitations under the License. --- -| Format | Use it when | Start here | -| ------------- | ------------------------------------------------------------- | -------------------------------------------------------------- | -| Xlang binary | Data crosses language boundaries | [Cross-language guide](../object-serialization/xlang/index.md) | -| Native binary | Producer and consumer are in the same language | Language guide | -| Row format | You need random field access or analytics-style partial reads | [Row format spec](../row-format/index.md) | -| Fory JSON | Java applications need high-performance standard JSON | [Fory JSON guide](../json/index.md) | +| Format or mode | Data model | Use it when | Interoperability | Start here | +| -------------- | ------------------------------------ | ------------------------------------------------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------- | +| Xlang binary | Portable object graphs | Data crosses language boundaries | Shared wire format across supported runtimes | [Cross-language guide](../object-serialization/xlang.md) | +| Native binary | Runtime-native object graphs | Producer and consumer use the same language | Same runtime family only | [Object Serialization](../object-serialization/index.md) | +| Row Format | Random-access binary rows | You need random field access or analytics-style partial reads | Standard Row is shared by Java, Python, C++, and Rust; Compact is Java-only | [Row Format guide](../row-format/index.md) | +| Fory JSON | Standard JSON mapped to Java objects | Java applications need high-performance standard JSON | Standard JSON text | [Fory JSON guide](../json/index.md) | + +Xlang and native are sibling modes of Object Serialization. Use them when the receiver needs to +reconstruct an object graph. Row Format and Fory JSON are separate formats, not additional Object +Serialization modes. For Java, Scala, Kotlin, Python, C++, Go, and Rust, use native mode for same-language traffic. It avoids xlang's cross-language type mapping and @@ -59,3 +63,12 @@ Choose xlang or native mode when you need to reconstruct object graphs. Choose R trusted analytical data that benefits from random field access. Choose Fory JSON for standard JSON in Java applications. Use Fory IDL and the compiler when multiple teams need one schema-first contract; it generates models that use the relevant Fory capability. + +## Related capabilities + +[Fory IDL and the compiler](../compiler/index.md) generate native models for supported runtimes. +Service definitions can also generate [Fory gRPC](../grpc/index.md) code. Neither capability defines +another serialization format. + +Protocol implementers should follow the normative format documentation under +[Specifications](../specification/xlang_serialization_spec.md). diff --git a/docs/introduction/index.md b/docs/introduction/index.md index 3f0bd9eed0..2245ef3700 100644 --- a/docs/introduction/index.md +++ b/docs/introduction/index.md @@ -1,5 +1,5 @@ --- -title: Introduction +title: Overview sidebar_position: 1 id: index license: | diff --git a/docs/introduction/product-model.md b/docs/introduction/product-model.md deleted file mode 100644 index 3379dd6b52..0000000000 --- a/docs/introduction/product-model.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: Product Model -sidebar_position: 3 -id: product-model -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory exposes three serialization products and one schema toolchain. Start from the data -contract you need, then choose a runtime API. - -## Serialization products - -| Product | Data model | Interoperability boundary | -| ---------------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------- | -| Binary Object Serialization: xlang mode | Portable object graphs | Shared wire format across supported runtimes | -| Binary Object Serialization: native mode | Runtime-native object graphs | Same runtime family only | -| Row Format | Random-access binary rows | Standard Row is shared by Java, Python, C++, and Rust; Compact Row is Java-only | -| Fory JSON | Standard JSON mapped to Java objects | Java API with text interoperability | - -Xlang and native are sibling modes of Binary Object Serialization. Row Format and Fory JSON are -separate products; neither is a third object-serialization mode. - -## Schema and services - -[Fory IDL and the compiler](../compiler/index.md) generate native models for supported runtimes. -Service definitions can also generate [Fory gRPC](../grpc/index.md) companions. The compiler and -gRPC integration do not define additional serialization formats. - -## Normative formats - -Protocol implementers should use the unchanged [Specification](../specification/xlang_serialization_spec.md) -surface. User guides explain tasks and link to the exact specification instead of duplicating its -wire-level rules. diff --git a/docs/introduction/support-matrix.md b/docs/introduction/support-matrix.md index 5127bd48fb..7fee55a395 100644 --- a/docs/introduction/support-matrix.md +++ b/docs/introduction/support-matrix.md @@ -20,7 +20,7 @@ license: | --- Use this matrix to confirm the documented API surface before choosing a capability. A runtime page -does not imply support for every Fory product. +does not imply support for every Fory capability. | Capability | Documented runtimes | Interoperability | | --------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------- | @@ -33,5 +33,5 @@ does not imply support for every Fory product. | Fory gRPC | Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Dart, Scala, Kotlin | Peers must use matching generated Fory service contracts | Platform constraints such as [Android](../object-serialization/java/android.md) and -[GraalVM Native Image](../object-serialization/java/graalvm.md) are documented by the owning Java -object-serialization and Fory JSON products. +[GraalVM Native Image](../object-serialization/java/graalvm.md) are documented in the Java Object +Serialization and Fory JSON guides. diff --git a/docs/json/index.md b/docs/json/index.md index 6ab963dd7a..47a2e3683f 100644 --- a/docs/json/index.md +++ b/docs/json/index.md @@ -1,5 +1,5 @@ --- -title: Fory JSON +title: Overview sidebar_position: 1 id: index license: | @@ -63,6 +63,6 @@ native or xlang protocol when reference identity or cycles are required. ## Related Java guides For binary serialization, start with [Java Object Serialization](../object-serialization/java/index.md) -and choose [xlang](../object-serialization/java/xlang.md) or +and choose [xlang](../object-serialization/java/basic-serialization.md#cross-language-interoperability) or [native](../object-serialization/java/native.md). Binary builder options are documented separately in [Java Configuration](../object-serialization/java/configuration.md). diff --git a/docs/object-serialization/core-concepts.md b/docs/object-serialization/core-concepts.md new file mode 100644 index 0000000000..c122e6d450 --- /dev/null +++ b/docs/object-serialization/core-concepts.md @@ -0,0 +1,122 @@ +--- +title: Core Concepts +sidebar_position: 1 +id: core-concepts +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Fory object serialization turns an object graph into bytes and reconstructs that graph later. The +same concepts apply to the default [xlang mode](xlang.md) and to supported +[native modes](native.md); the selected mode determines which types and wire rules are available. + +## Object graphs + +A root value may contain scalar fields, collections, maps, nested objects, repeated references, and +cycles. Serialization walks that graph from the root. Deserialization creates a new graph from the +encoded type and field data. + +This is different from serializing a row or a JSON document. Object serialization can preserve +runtime types and object identity so the reader can reconstruct application objects rather than +only values. Use [Row Format](../row-format/index.md) for trusted analytical rows and +[Fory JSON](../json/index.md) for JSON interchange. + +## Runtime instances and registration + +A Fory instance owns its mode, schema behavior, reference settings, registered types, custom +serializers, and read limits. Configure and register the instance before its first root +serialization or deserialization operation, then reuse it. Registration is frozen after the first +root operation so the same instance always resolves a type in the same way. + +Thread-safety differs by runtime. Some runtimes provide a thread-safe wrapper or pool; others use +one instance per thread or task. Follow the selected language guide instead of sharing an ordinary +instance without checking its concurrency contract. + +## Types and type identity + +Built-in types have identities owned by Fory. Application structs, classes, enums, unions, and +extension types use a registered numeric ID or name. Type identity answers _which serializer and +model should read this value_; a field schema describes _what data that model contains_. + +A statically known field can use its declared type directly. A dynamic field also carries the +concrete runtime type needed for interfaces, abstract classes, trait objects, broad object types, +or heterogeneous values. Dynamic typing is more flexible but requires every possible concrete type +to be registered and supported by the selected mode. + +In xlang mode, peers must coordinate the same portable type identity and mapping. Native mode may +use runtime-specific identities and types. See [Xlang Serialization](xlang.md) for the portable +rules and each language's Type Registration page for its exact API. + +## Schemas and evolution + +A schema describes the fields and nested types of a structured value. Compatible mode carries +metadata that lets a reader handle supported additions, removals, reordering, and type adaptations. +Use it when readers and writers may deploy independently. + +Same-schema mode assumes both sides use the same type identity, fields, nested types, nullability, +and reference metadata. It reduces metadata and payload size, but a schema mismatch is an error. +Use it only when one release process keeps every reader and writer aligned. + +Field IDs or names should remain stable after a contract is published. Renaming or reusing an +identity can turn an intended evolution into a different field or type. + +## Nullability + +Nullability determines whether a value position may contain no value. Languages express it through +nullable references, option types, pointers, annotations, or schema metadata. A nullable field is +not the same as a field whose value happens to use a default. + +Keep nullability consistent across readers and writers. Compatible mode can handle documented +nullable and missing-field cases, but it cannot place a remote null into a local carrier that has +no valid null or missing-value representation. + +## Reference tracking + +Reference tracking preserves object identity. Enable it when a graph contains the same object more +than once or contains a cycle. Without reference tracking, repeated values may become separate +objects and cycles may recurse until the operation fails. + +Leave reference tracking disabled for value-shaped, acyclic data when identity does not matter; it +adds per-object metadata and lookup work. Some runtimes combine a global setting with field-level +metadata, so use the language-specific References or Basic Serialization page for exact behavior. + +## Polymorphism + +Polymorphism stores the concrete type of a value whose declared position is broader. The reader +must know and accept that concrete type, and the type must be representable in the selected mode. + +Host-language inheritance alone does not create a portable contract. For cross-language data, +model only alternatives that have xlang mappings on every peer. For same-runtime data, native mode +may support additional runtime-specific class, trait, or hook behavior. + +## Custom serializers + +Use a custom serializer when a type needs a representation that built-in schema inference cannot +provide. Registration connects the custom serializer to the application type. A custom serializer +must follow the selected mode's rules: xlang serializers need a portable representation, while +native serializers may use runtime-specific data and hooks. + +Prefer built-in serializers and generated models when they already describe the type. They keep +schema evolution, reference handling, and cross-language behavior easier to reason about. + +## Continue with a mode + +- [Xlang Serialization](xlang.md) is the default and is required when different language runtimes + exchange bytes. +- [Native Serialization](native.md) is for supported same-runtime use cases that need native types + or behavior. +- Choose a language section after selecting a mode to find installation, API, configuration, + registration, platform, security, and troubleshooting guidance. diff --git a/docs/object-serialization/cpp/core-api.md b/docs/object-serialization/cpp/basic-serialization.md similarity index 51% rename from docs/object-serialization/cpp/core-api.md rename to docs/object-serialization/cpp/basic-serialization.md index c9ca5cbc3d..275ad53ed4 100644 --- a/docs/object-serialization/cpp/core-api.md +++ b/docs/object-serialization/cpp/basic-serialization.md @@ -1,7 +1,7 @@ --- title: Basic Serialization -sidebar_position: 3 -id: core-api +sidebar_position: 1 +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,7 +19,7 @@ license: | limitations under the License. --- -This page covers basic object graph serialization and the core serialization APIs. +This page covers basic object graph serialization and the core serialization APIs in the default xlang mode. ## Object Graph Serialization @@ -355,6 +355,285 @@ fory.register_struct(2); - **Disable Tracking**: Use `track_ref(false)` when references aren't needed - **Compact Encoding**: Variable-length encoding for space efficiency +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + +This page explains how to use Fory xlang serialization between C++ and other languages. + +### Overview + +Apache Fory™ enables seamless data exchange between C++, Java, Python, Go, Rust, +JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Xlang mode ensures +binary compatibility across all supported languages. + +### Xlang Configuration + +C++ defaults to xlang mode. Compatible schema evolution is also the xlang default. Set the mode explicitly in xlang examples: + +```cpp +#include "fory/serialization/fory.h" + +using namespace fory::serialization; + +auto fory = Fory::builder().xlang(true).build(); +``` + +### Xlang Example + +#### C++ Producer + +```cpp +#include "fory/serialization/fory.h" +#include + +using namespace fory::serialization; + +struct Message { + std::string topic; + int64_t timestamp; + std::map headers; + std::vector payload; + + bool operator==(const Message &other) const { + return topic == other.topic && timestamp == other.timestamp && + headers == other.headers && payload == other.payload; + } +}; +FORY_STRUCT(Message, topic, timestamp, headers, payload); + +int main() { + auto fory = Fory::builder().xlang(true).build(); + fory.register_struct(100); + + Message msg{ + "events.user", + 1699999999000, + {{"content-type", "application/json"}}, + {'h', 'e', 'l', 'l', 'o'} + }; + + auto result = fory.serialize(msg); + if (result.ok()) { + auto bytes = std::move(result).value(); + // write to file, send over network, etc. + std::ofstream file("message.bin", std::ios::binary); + file.write(reinterpret_cast(bytes.data()), bytes.size()); + } + return 0; +} +``` + +#### Java Consumer + +```java +import org.apache.fory.Fory; + +public class Message { + public String topic; + public long timestamp; + public Map headers; + public byte[] payload; +} + +public class Consumer { + public static void main(String[] args) throws Exception { + Fory fory = Fory.builder() + .withXlang(true) + .build(); + fory.register(Message.class, 100); // Same ID as C++ + + byte[] bytes = Files.readAllBytes(Path.of("message.bin")); + Message msg = (Message) fory.deserialize(bytes); + + System.out.println("Topic: " + msg.topic); + System.out.println("Timestamp: " + msg.timestamp); + } +} +``` + +#### Python Consumer + +```python +import pyfory + +class Message: + topic: str + timestamp: int + headers: dict[str, str] + payload: bytes + +fory = pyfory.Fory(xlang=True) +fory.register(Message, type_id=100) # Same ID as C++ + +with open("message.bin", "rb") as f: + data = f.read() + +msg = fory.deserialize(data) +print(f"Topic: {msg.topic}") +print(f"Timestamp: {msg.timestamp}") +``` + +### Type Mapping + +#### Primitive Types + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------------ | ---------- | ----------------- | ------------------- | ---------- | +| `bool` | `boolean` | `bool` | `bool` | `bool` | +| `int8_t` | `byte` | `int` | `int8` | `i8` | +| `int16_t` | `short` | `int` | `int16` | `i16` | +| `int32_t` | `int` | `int` | `int32` | `i32` | +| `int64_t` | `long` | `int` | `int64` | `i64` | +| `float` | `float` | `float` | `float32` | `f32` | +| `double` | `double` | `float` | `float64` | `f64` | +| `fory::float16_t` | `Float16` | `pyfory.Float16` | `float16.Float16` | `Float16` | +| `fory::bfloat16_t` | `BFloat16` | `pyfory.BFloat16` | `bfloat16.BFloat16` | `BFloat16` | + +#### String Types + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------- | --------- | ----------- | -------- | --------- | +| `std::string` | `String` | `str` | `string` | `String` | + +#### Collection Types + +| C++ Type | Java Type | Python Type | Go Type | Rust Type | +| ------------------------------------------- | -------------- | --------------- | --------------------- | --------------- | +| `std::vector` | `List` | `list` | `[]T` | `Vec` | +| `std::vector` | `Float16List` | `Float16Array` | `[]float16.Float16` | `Vec` | +| `std::vector` | `BFloat16List` | `BFloat16Array` | `[]bfloat16.BFloat16` | `Vec` | +| `std::set` | `Set` | `set` | `map[T]struct{}` | `HashSet` | +| `std::map` / `std::unordered_map` | `Map` | `dict` | `map[K]V` | `HashMap` | + +#### Lists and Dense Arrays + +`std::vector` maps to Fory `list` by default in handwritten C++ structs. +Use the field metadata DSL's array node when the schema is dense `array`. + +| Fory schema | C++ metadata sketch | +| ----------------- | ---------------------------------------- | +| `list` | `fory::F(id).list(fory::T::int32())` | +| `array` | `fory::F(id).array(fory::T::bool_())` | +| `array` | `fory::F(id).array(fory::T::int8())` | +| `array` | `fory::F(id).array(fory::T::int16())` | +| `array` | `fory::F(id).array(fory::T::int32())` | +| `array` | `fory::F(id).array(fory::T::int64())` | +| `array` | `fory::F(id).array(fory::T::uint8())` | +| `array` | `fory::F(id).array(fory::T::uint16())` | +| `array` | `fory::F(id).array(fory::T::uint32())` | +| `array` | `fory::F(id).array(fory::T::uint64())` | +| `array` | `fory::F(id).array(fory::T::float16())` | +| `array` | `fory::F(id).array(fory::T::bfloat16())` | +| `array` | `fory::F(id).array(fory::T::float32())` | +| `array` | `fory::F(id).array(fory::T::float64())` | + +#### Temporal Types + +| C++ Type | Java Type | Python Type | Go Type | +| ----------------- | ----------- | --------------- | --------------- | +| `fory::Timestamp` | `Instant` | `datetime` | `time.Time` | +| `fory::Duration` | `Duration` | `timedelta` | `time.Duration` | +| `fory::Date` | `LocalDate` | `datetime.date` | `time.Time` | + +### Field Order Requirements + +**Critical:** Fields are sorted by snake_case field name. The converted names must match across languages. + +#### C++ + +```cpp +struct Person { + std::string name; // Field 0 + int32_t age; // Field 1 + std::string email; // Field 2 +}; +FORY_STRUCT(Person, name, age, email); // Order matters! +``` + +#### Java + +```java +public class Person { + public String name; // Field 0 + public int age; // Field 1 + public String email; // Field 2 +} +``` + +#### Python + +```python +class Person: + name: str # Field 0 + age: int # Field 1 + email: str # Field 2 +``` + +### Type ID Consistency + +All languages must use the same type IDs: + +```cpp +// C++ +fory.register_struct(100); +fory.register_struct
(101); +fory.register_struct(102); +``` + +```java +// Java +fory.register(Person.class, 100); +fory.register(Address.class, 101); +fory.register(Order.class, 102); +``` + +```python +# Python +fory.register(Person, type_id=100) +fory.register(Address, type_id=101) +fory.register(Order, type_id=102) +``` + +### Compatible Mode + +Xlang mode already uses compatible schema evolution by default. Keep that default for schemas that +may evolve independently: + +```cpp +auto fory = Fory::builder().xlang(true).build(); +``` + +Compatible mode allows: + +- Adding new fields (with defaults) +- Removing unused fields +- Reordering fields + +### Interoperability Troubleshooting + +#### Type Mismatch Errors + +``` +Error: Type mismatch: expected 100, got 101 +``` + +**Solution:** Ensure type IDs match across all languages. + +#### Encoding Errors + +``` +Error: Invalid UTF-8 sequence +``` + +**Solution:** Ensure strings are valid UTF-8 in all languages. + +### Related Guides + +- [Configuration](configuration.md) - Builder options +- [Type Registration](type-registration.md) - Registering types +- [Supported Types](supported-types.md) - Type compatibility + ## Related Topics - [Configuration](configuration.md) - Builder options diff --git a/docs/object-serialization/cpp/configuration.md b/docs/object-serialization/cpp/configuration.md index 389636732f..78c77939d2 100644 --- a/docs/object-serialization/cpp/configuration.md +++ b/docs/object-serialization/cpp/configuration.md @@ -252,20 +252,10 @@ auto fory = Fory::builder().build_thread_safe(); // Returns ThreadSafeFory ## Security -Security-related configuration: - -- Register all structs and polymorphic implementations before deserializing untrusted payloads. -- Use `check_struct_version(true)` with `compatible(false)` for intentional same-schema payloads. -- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a - positive value for a trusted workload that needs a different collection/map/struct gate. -- Keep `max_dyn_depth(...)` as low as your model permits to reject unexpectedly deep polymorphic - graphs. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer concrete fields over broad polymorphic fields for untrusted input. +See [C++ Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) - Using configured Fory -- [Xlang Serialization](xlang.md) - xlang mode details +- [Basic Serialization](basic-serialization.md) - Using configured Fory +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode details - [Type Registration](type-registration.md) - Registering types diff --git a/docs/object-serialization/cpp/custom-serializers.md b/docs/object-serialization/cpp/custom-serializers.md index bc7106e79e..089e10df08 100644 --- a/docs/object-serialization/cpp/custom-serializers.md +++ b/docs/object-serialization/cpp/custom-serializers.md @@ -366,6 +366,6 @@ static MyType read_data(ReadContext &ctx) { ## Related Topics - [Type Registration](type-registration.md) - Registering serializers -- [Basic Serialization](core-api.md) - Using FORY_STRUCT macro +- [Basic Serialization](basic-serialization.md) - Using FORY_STRUCT macro - [Schema Evolution](schema-evolution.md) - Compatible mode -- [Xlang Serialization](xlang.md) - Cross-language serialization +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language serialization diff --git a/docs/object-serialization/cpp/index.md b/docs/object-serialization/cpp/index.md index c69bcd5215..a18ee4f393 100644 --- a/docs/object-serialization/cpp/index.md +++ b/docs/object-serialization/cpp/index.md @@ -222,7 +222,7 @@ Use xlang mode for cross-language payloads and schemas shared with other Fory im Use native mode for C++-only traffic. Native mode is selected with `.xlang(false)` and keeps C++ object serialization in C++-native form. It is optimized for C++ types and avoids portable xlang type-mapping constraints when the payload never leaves C++. Compatible mode is enabled by default. Set `.compatible(false)` only when every reader and writer uses the same C++ schema and you want faster serialization and smaller size. -See [Xlang Serialization](xlang.md) for C++ xlang registration and interoperability rules, and [Native Serialization](native.md) for C++-only payloads. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for C++ xlang registration and interoperability rules, and [Native Serialization](native.md) for C++-only payloads. ## Thread Safety @@ -264,8 +264,7 @@ std::thread t2([&]() { ## Next Steps - [Configuration](configuration.md) - Builder options and modes -- [Basic Serialization](core-api.md) - Object graph serialization -- [Xlang Serialization](xlang.md) - xlang mode and interoperability +- [Basic Serialization](basic-serialization.md) - Default xlang object graphs and interoperability - [Native Serialization](native.md) - C++-only serialization - [Schema Metadata](schema-metadata.md) - Field-level metadata (nullable, ref tracking) - [Schema Evolution](schema-evolution.md) - Compatible mode and schema changes @@ -274,3 +273,6 @@ std::thread t2([&]() { - [Custom Serializers](custom-serializers.md) - Extend serialization behavior - [Row Format](../../row-format/cpp.md) - Zero-copy row-based format - [gRPC Support](../../grpc/cpp.md) - Fory payloads over gRPC C++ + +Before decoding bytes from outside the application trust boundary, read +[C++ Security](security.md). diff --git a/docs/object-serialization/cpp/native.md b/docs/object-serialization/cpp/native.md index a93c9ed06d..70df2a91ba 100644 --- a/docs/object-serialization/cpp/native.md +++ b/docs/object-serialization/cpp/native.md @@ -23,7 +23,7 @@ C++ native serialization is the C++-only wire mode selected with `.xlang(false)` writer and reader is C++ and the payload should follow C++ type behavior instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default C++ mode, when +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default C++ mode, when bytes must be read by Java, Python, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-C++ Fory implementation. @@ -206,9 +206,9 @@ Enable `.track_ref(true)` and verify the graph uses supported pointer patterns. ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language C++ payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language C++ payloads - [Configuration](configuration.md) - Builder options -- [Basic Serialization](core-api.md) - Object graph serialization +- [Basic Serialization](basic-serialization.md) - Object graph serialization - [Supported Types](supported-types.md) - C++ type support - [Polymorphic Serialization](polymorphism.md) - Polymorphic object models - [Schema Evolution](schema-evolution.md) - Compatible mode diff --git a/docs/object-serialization/cpp/polymorphism.md b/docs/object-serialization/cpp/polymorphism.md index 3f609ad280..383ba7c36d 100644 --- a/docs/object-serialization/cpp/polymorphism.md +++ b/docs/object-serialization/cpp/polymorphism.md @@ -487,4 +487,4 @@ if (!decoded_result.ok()) { - [Schema Metadata](schema-metadata.md) - Field-level metadata and options - [Supported Types](supported-types.md) - Smart pointers and collections - [Configuration](configuration.md) - `max_dyn_depth` and other settings -- [Basic Serialization](core-api.md) - Core serialization concepts +- [Basic Serialization](basic-serialization.md) - Core serialization concepts diff --git a/docs/object-serialization/cpp/schema-evolution.md b/docs/object-serialization/cpp/schema-evolution.md index 8ca540479d..4b7ab928f8 100644 --- a/docs/object-serialization/cpp/schema-evolution.md +++ b/docs/object-serialization/cpp/schema-evolution.md @@ -421,4 +421,4 @@ Both instances can exchange data even with different schema versions. - [Configuration](configuration.md) - Enabling compatible mode - [Type Registration](type-registration.md) - Type ID management -- [Xlang Serialization](xlang.md) - Cross-language considerations +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language considerations diff --git a/docs/object-serialization/cpp/security.md b/docs/object-serialization/cpp/security.md new file mode 100644 index 0000000000..fe86e37d2d --- /dev/null +++ b/docs/object-serialization/cpp/security.md @@ -0,0 +1,57 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a C++ reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register all structs and polymorphic implementations before deserializing untrusted payloads. +- Use `check_struct_version(true)` with `compatible(false)` for intentional same-schema payloads. +- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a + positive value for a trusted workload that needs a different collection/map/struct gate. +- Keep `max_dyn_depth(...)` as low as your model permits to reject unexpectedly deep polymorphic + graphs. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer concrete fields over broad polymorphic fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/cpp/supported-types.md b/docs/object-serialization/cpp/supported-types.md index 11f21a9e33..4c95f5307e 100644 --- a/docs/object-serialization/cpp/supported-types.md +++ b/docs/object-serialization/cpp/supported-types.md @@ -295,6 +295,6 @@ Currently not supported: ## Related Topics -- [Basic Serialization](core-api.md) - Using these types +- [Basic Serialization](basic-serialization.md) - Using these types - [Type Registration](type-registration.md) - Registering types -- [Xlang Serialization](xlang.md) - Cross-language compatibility +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language compatibility diff --git a/docs/object-serialization/cpp/type-registration.md b/docs/object-serialization/cpp/type-registration.md index 6e2a3e5189..62f525ef94 100644 --- a/docs/object-serialization/cpp/type-registration.md +++ b/docs/object-serialization/cpp/type-registration.md @@ -249,6 +249,6 @@ if (!result.ok()) { ## Related Topics -- [Basic Serialization](core-api.md) - Using registered types -- [Xlang Serialization](xlang.md) - Cross-language considerations +- [Basic Serialization](basic-serialization.md) - Using registered types +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language considerations - [Supported Types](supported-types.md) - All supported types diff --git a/docs/object-serialization/cpp/xlang.md b/docs/object-serialization/cpp/xlang.md deleted file mode 100644 index 726f63bda5..0000000000 --- a/docs/object-serialization/cpp/xlang.md +++ /dev/null @@ -1,295 +0,0 @@ ---- -title: Xlang Serialization -sidebar_position: 1 -id: xlang -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page explains how to use Fory xlang serialization between C++ and other languages. - -## Overview - -Apache Fory™ enables seamless data exchange between C++, Java, Python, Go, Rust, -JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Xlang mode ensures -binary compatibility across all supported languages. - -## Create an Xlang Fory Instance - -C++ defaults to xlang mode. Compatible schema evolution is also the xlang default. Set the mode explicitly in xlang examples: - -```cpp -#include "fory/serialization/fory.h" - -using namespace fory::serialization; - -auto fory = Fory::builder().xlang(true).build(); -``` - -## Xlang Example - -### C++ Producer - -```cpp -#include "fory/serialization/fory.h" -#include - -using namespace fory::serialization; - -struct Message { - std::string topic; - int64_t timestamp; - std::map headers; - std::vector payload; - - bool operator==(const Message &other) const { - return topic == other.topic && timestamp == other.timestamp && - headers == other.headers && payload == other.payload; - } -}; -FORY_STRUCT(Message, topic, timestamp, headers, payload); - -int main() { - auto fory = Fory::builder().xlang(true).build(); - fory.register_struct(100); - - Message msg{ - "events.user", - 1699999999000, - {{"content-type", "application/json"}}, - {'h', 'e', 'l', 'l', 'o'} - }; - - auto result = fory.serialize(msg); - if (result.ok()) { - auto bytes = std::move(result).value(); - // write to file, send over network, etc. - std::ofstream file("message.bin", std::ios::binary); - file.write(reinterpret_cast(bytes.data()), bytes.size()); - } - return 0; -} -``` - -### Java Consumer - -```java -import org.apache.fory.Fory; - -public class Message { - public String topic; - public long timestamp; - public Map headers; - public byte[] payload; -} - -public class Consumer { - public static void main(String[] args) throws Exception { - Fory fory = Fory.builder() - .withXlang(true) - .build(); - fory.register(Message.class, 100); // Same ID as C++ - - byte[] bytes = Files.readAllBytes(Path.of("message.bin")); - Message msg = (Message) fory.deserialize(bytes); - - System.out.println("Topic: " + msg.topic); - System.out.println("Timestamp: " + msg.timestamp); - } -} -``` - -### Python Consumer - -```python -import pyfory - -class Message: - topic: str - timestamp: int - headers: dict[str, str] - payload: bytes - -fory = pyfory.Fory(xlang=True) -fory.register(Message, type_id=100) # Same ID as C++ - -with open("message.bin", "rb") as f: - data = f.read() - -msg = fory.deserialize(data) -print(f"Topic: {msg.topic}") -print(f"Timestamp: {msg.timestamp}") -``` - -## Type Mapping - -### Primitive Types - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------------ | ---------- | ----------------- | ------------------- | ---------- | -| `bool` | `boolean` | `bool` | `bool` | `bool` | -| `int8_t` | `byte` | `int` | `int8` | `i8` | -| `int16_t` | `short` | `int` | `int16` | `i16` | -| `int32_t` | `int` | `int` | `int32` | `i32` | -| `int64_t` | `long` | `int` | `int64` | `i64` | -| `float` | `float` | `float` | `float32` | `f32` | -| `double` | `double` | `float` | `float64` | `f64` | -| `fory::float16_t` | `Float16` | `pyfory.Float16` | `float16.Float16` | `Float16` | -| `fory::bfloat16_t` | `BFloat16` | `pyfory.BFloat16` | `bfloat16.BFloat16` | `BFloat16` | - -### String Types - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------- | --------- | ----------- | -------- | --------- | -| `std::string` | `String` | `str` | `string` | `String` | - -### Collection Types - -| C++ Type | Java Type | Python Type | Go Type | Rust Type | -| ------------------------------------------- | -------------- | --------------- | --------------------- | --------------- | -| `std::vector` | `List` | `list` | `[]T` | `Vec` | -| `std::vector` | `Float16List` | `Float16Array` | `[]float16.Float16` | `Vec` | -| `std::vector` | `BFloat16List` | `BFloat16Array` | `[]bfloat16.BFloat16` | `Vec` | -| `std::set` | `Set` | `set` | `map[T]struct{}` | `HashSet` | -| `std::map` / `std::unordered_map` | `Map` | `dict` | `map[K]V` | `HashMap` | - -### Lists and Dense Arrays - -`std::vector` maps to Fory `list` by default in handwritten C++ structs. -Use the field metadata DSL's array node when the schema is dense `array`. - -| Fory schema | C++ metadata sketch | -| ----------------- | ---------------------------------------- | -| `list` | `fory::F(id).list(fory::T::int32())` | -| `array` | `fory::F(id).array(fory::T::bool_())` | -| `array` | `fory::F(id).array(fory::T::int8())` | -| `array` | `fory::F(id).array(fory::T::int16())` | -| `array` | `fory::F(id).array(fory::T::int32())` | -| `array` | `fory::F(id).array(fory::T::int64())` | -| `array` | `fory::F(id).array(fory::T::uint8())` | -| `array` | `fory::F(id).array(fory::T::uint16())` | -| `array` | `fory::F(id).array(fory::T::uint32())` | -| `array` | `fory::F(id).array(fory::T::uint64())` | -| `array` | `fory::F(id).array(fory::T::float16())` | -| `array` | `fory::F(id).array(fory::T::bfloat16())` | -| `array` | `fory::F(id).array(fory::T::float32())` | -| `array` | `fory::F(id).array(fory::T::float64())` | - -### Temporal Types - -| C++ Type | Java Type | Python Type | Go Type | -| ----------------- | ----------- | --------------- | --------------- | -| `fory::Timestamp` | `Instant` | `datetime` | `time.Time` | -| `fory::Duration` | `Duration` | `timedelta` | `time.Duration` | -| `fory::Date` | `LocalDate` | `datetime.date` | `time.Time` | - -## Field Order Requirements - -**Critical:** Fields are sorted by snake_case field name. The converted names must match across languages. - -### C++ - -```cpp -struct Person { - std::string name; // Field 0 - int32_t age; // Field 1 - std::string email; // Field 2 -}; -FORY_STRUCT(Person, name, age, email); // Order matters! -``` - -### Java - -```java -public class Person { - public String name; // Field 0 - public int age; // Field 1 - public String email; // Field 2 -} -``` - -### Python - -```python -class Person: - name: str # Field 0 - age: int # Field 1 - email: str # Field 2 -``` - -## Type ID Consistency - -All languages must use the same type IDs: - -```cpp -// C++ -fory.register_struct(100); -fory.register_struct
(101); -fory.register_struct(102); -``` - -```java -// Java -fory.register(Person.class, 100); -fory.register(Address.class, 101); -fory.register(Order.class, 102); -``` - -```python -# Python -fory.register(Person, type_id=100) -fory.register(Address, type_id=101) -fory.register(Order, type_id=102) -``` - -## Compatible Mode - -Xlang mode already uses compatible schema evolution by default. Keep that default for schemas that -may evolve independently: - -```cpp -auto fory = Fory::builder().xlang(true).build(); -``` - -Compatible mode allows: - -- Adding new fields (with defaults) -- Removing unused fields -- Reordering fields - -## Troubleshooting - -### Type Mismatch Errors - -``` -Error: Type mismatch: expected 100, got 101 -``` - -**Solution:** Ensure type IDs match across all languages. - -### Encoding Errors - -``` -Error: Invalid UTF-8 sequence -``` - -**Solution:** Ensure strings are valid UTF-8 in all languages. - -## Related Topics - -- [Configuration](configuration.md) - Builder options -- [Type Registration](type-registration.md) - Registering types -- [Supported Types](supported-types.md) - Type compatibility diff --git a/docs/object-serialization/csharp/core-api.md b/docs/object-serialization/csharp/basic-serialization.md similarity index 55% rename from docs/object-serialization/csharp/core-api.md rename to docs/object-serialization/csharp/basic-serialization.md index 43020f8130..7c77560a7d 100644 --- a/docs/object-serialization/csharp/core-api.md +++ b/docs/object-serialization/csharp/basic-serialization.md @@ -1,7 +1,7 @@ --- title: Basic Serialization -sidebar_position: 3 -id: core-api +sidebar_position: 1 +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,7 +19,7 @@ license: | limitations under the License. --- -This page covers typed serialization APIs in Apache Fory™ C#. +This page covers typed serialization APIs in the default xlang mode for Apache Fory™ C#. ## Object Graph Serialization @@ -159,6 +159,125 @@ fory.Serialize(dynamicWriter, value); - Register user types handled by `[ForyStruct]`, `[ForyEnum]`, `[ForyUnion]`, external structural serializers, or custom serializers explicitly. +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + +Apache Fory™ C# supports xlang serialization with other Fory implementations. + +### Xlang Configuration + +C# always writes and reads the xlang frame header. There is no mode switch, so interoperability code +only needs to configure the remaining settings such as compatibility mode and reference +tracking. + +```csharp +Fory fory = Fory.Builder() + .Build(); +``` + +### Register with Stable IDs + +```csharp +[ForyStruct] +public sealed class Person +{ + public string Name { get; set; } = string.Empty; + public int Age { get; set; } +} + +Fory fory = Fory.Builder() + .Build(); + +fory.Register(100); +``` + +Use the same ID mapping on all languages. + +Third-party classes, structs, and enums can use +[external-type serialization](external-types.md). Register the target type, +not its local serializer declaration, with the same ID or name used by the +other language peers. + +### Register by Name + +```csharp +fory.Register("com.example.Person"); +``` + +### Xlang Example + +#### C# (Serializer) + +```csharp +Person person = new() { Name = "Alice", Age = 30 }; +byte[] payload = fory.Serialize(person); +``` + +#### Java (Deserializer) + +```java +Fory fory = Fory.builder() + .withXlang(true) + .withRefTracking(true) + .build(); + +fory.register(Person.class, 100); +Person value = (Person) fory.deserialize(payloadFromCSharp); +``` + +#### Python (Deserializer) + +```python +import pyfory + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register_type(Person, type_id=100) +value = fory.deserialize(payload_from_csharp) +``` + +### Type Mapping Reference + +See [cross-language interoperability guide](../xlang.md) for complete mapping. + +For reduced-precision numeric payloads, use `Half` / `Half[]` or `List` for xlang `float16`, and `BFloat16` / `BFloat16[]` or `List` for xlang `bfloat16`. + +### Lists and Dense Arrays + +C# `List` maps to Fory `list`. Use the schema marker +`Apache.Fory.Schema.Types.Array` when a field is dense `array`. + +| Fory schema | C# schema marker sketch | +| ----------------- | ----------------------- | +| `list` | `S.List` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | +| `array` | `S.Array` | + +### Interoperability Best Practices + +1. Keep type IDs stable and documented. +2. Keep compatible mode enabled for rolling upgrades. +3. Register all user types on both read/write peers. +4. Validate integration with real payload round trips. + +### Related Guides + +- [Type Registration](type-registration.md) +- [External Types](external-types.md) +- [Schema Evolution](schema-evolution.md) +- [Supported Types](supported-types.md) + ## Related Topics - [Type Registration](type-registration.md) diff --git a/docs/object-serialization/csharp/configuration.md b/docs/object-serialization/csharp/configuration.md index 2d45d409d9..5bac8df215 100644 --- a/docs/object-serialization/csharp/configuration.md +++ b/docs/object-serialization/csharp/configuration.md @@ -200,19 +200,10 @@ ThreadSafeFory fory = Fory.Builder() ## Security -Security-related configuration: - -- Register only the expected types before deserializing untrusted payloads. -- Use `CheckStructVersion(true)` with `Compatible(false)` for intentional same-schema payloads. -- Set `MaxDepth(...)` to reject unexpectedly deep dynamic object graphs. -- Set `MaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input bytes. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer generated or registered concrete models over broad dynamic fields for untrusted input. +See [C# Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/csharp/external-types.md b/docs/object-serialization/csharp/external-types.md index b1802f07be..0b1e18fd1a 100644 --- a/docs/object-serialization/csharp/external-types.md +++ b/docs/object-serialization/csharp/external-types.md @@ -220,7 +220,7 @@ long as each concrete annotated child has a legal parameterless construction path. First-party bases use direct `[ForyStruct]` annotations instead. See -[Class Inheritance](core-api.md#class-inheritance). +[Class Inheritance](basic-serialization.md#class-inheritance). ## Declaration and Target Requirements diff --git a/docs/object-serialization/csharp/index.md b/docs/object-serialization/csharp/index.md index 66e3d2b5ff..0b98a6df9f 100644 --- a/docs/object-serialization/csharp/index.md +++ b/docs/object-serialization/csharp/index.md @@ -86,24 +86,26 @@ User decoded = fory.Deserialize(payload); ## Documentation -| Topic | Description | -| ------------------------------------------- | --------------------------------------------- | -| [Configuration](configuration.md) | Builder options and mode settings | -| [Basic Serialization](core-api.md) | Typed and dynamic serialization APIs | -| [Xlang Serialization](xlang.md) | Interoperability guidance | -| [Schema Metadata](schema-metadata.md) | `[ForyField]` ids and schema type descriptors | -| [Type Registration](type-registration.md) | Registering user types and custom serializers | -| [External Types](external-types.md) | Serializers for third-party types | -| [Custom Serializers](custom-serializers.md) | Implementing `Serializer` | -| [References](references.md) | Shared/circular reference handling | -| [Schema Evolution](schema-evolution.md) | Compatible mode behavior | -| [Supported Types](supported-types.md) | Built-in and generated type support | -| [Thread Safety](thread-safety.md) | `Fory` vs `ThreadSafeFory` usage | -| [gRPC Support](../../grpc/csharp.md) | Generated Fory-backed gRPC service companions | -| [Troubleshooting](troubleshooting.md) | Common errors and debugging steps | +| Topic | Description | +| --------------------------------------------- | --------------------------------------------- | +| [Configuration](configuration.md) | Builder options and mode settings | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs and interoperability | +| [Schema Metadata](schema-metadata.md) | `[ForyField]` ids and schema type descriptors | +| [Type Registration](type-registration.md) | Registering user types and custom serializers | +| [External Types](external-types.md) | Serializers for third-party types | +| [Custom Serializers](custom-serializers.md) | Implementing `Serializer` | +| [References](references.md) | Shared/circular reference handling | +| [Schema Evolution](schema-evolution.md) | Compatible mode behavior | +| [Supported Types](supported-types.md) | Built-in and generated type support | +| [Thread Safety](thread-safety.md) | `Fory` vs `ThreadSafeFory` usage | +| [gRPC Support](../../grpc/csharp.md) | Generated Fory-backed gRPC service companions | +| [Troubleshooting](troubleshooting.md) | Common errors and debugging steps | ## Related Resources - [Xlang serialization specification](../../specification/xlang_serialization_spec.md) -- [Xlang guide](../xlang/index.md) +- [Cross-language guide](../xlang.md) - [C# source directory](https://github.com/apache/fory/tree/main/csharp) + +Before decoding bytes from outside the application trust boundary, read +[C# Security](security.md). diff --git a/docs/object-serialization/csharp/references.md b/docs/object-serialization/csharp/references.md index 00d044ba07..723503136b 100644 --- a/docs/object-serialization/csharp/references.md +++ b/docs/object-serialization/csharp/references.md @@ -76,6 +76,6 @@ supported; Fory rejects unresolved refs instead of returning a partial union. ## Related Topics - [Configuration](configuration.md) -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [External Types](external-types.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/csharp/security.md b/docs/object-serialization/csharp/security.md new file mode 100644 index 0000000000..06416a73e0 --- /dev/null +++ b/docs/object-serialization/csharp/security.md @@ -0,0 +1,56 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a C# reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected types before deserializing untrusted payloads. +- Use `CheckStructVersion(true)` with `Compatible(false)` for intentional same-schema payloads. +- Set `MaxDepth(...)` to reject unexpectedly deep dynamic object graphs. +- Set `MaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input bytes. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer generated or registered concrete models over broad dynamic fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/csharp/supported-types.md b/docs/object-serialization/csharp/supported-types.md index 7f4a060456..bf4f0ad08d 100644 --- a/docs/object-serialization/csharp/supported-types.md +++ b/docs/object-serialization/csharp/supported-types.md @@ -107,11 +107,11 @@ Dynamic object payloads via `Serialize` / `Deserialize` suppor ## Notes - User-defined types should be registered explicitly. -- For cross-language usage, follow the [xlang guide](../xlang/index.md). +- For cross-language usage, follow the [cross-language interoperability guide](../xlang.md). ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [External Types](external-types.md) - [Type Registration](type-registration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/csharp/type-registration.md b/docs/object-serialization/csharp/type-registration.md index 2d64ee2c49..0c6b629310 100644 --- a/docs/object-serialization/csharp/type-registration.md +++ b/docs/object-serialization/csharp/type-registration.md @@ -94,7 +94,7 @@ fory.Register(101); ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [External Types](external-types.md) - [Custom Serializers](custom-serializers.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/csharp/xlang.md b/docs/object-serialization/csharp/xlang.md deleted file mode 100644 index fe17a459c5..0000000000 --- a/docs/object-serialization/csharp/xlang.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -title: Xlang Serialization -sidebar_position: 1 -id: xlang -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory™ C# supports xlang serialization with other Fory implementations. - -## Xlang Fory Instance - -C# always writes and reads the xlang frame header. There is no mode switch, so interoperability code -only needs to configure the remaining settings such as compatibility mode and reference -tracking. - -```csharp -Fory fory = Fory.Builder() - .Build(); -``` - -## Register with Stable IDs - -```csharp -[ForyStruct] -public sealed class Person -{ - public string Name { get; set; } = string.Empty; - public int Age { get; set; } -} - -Fory fory = Fory.Builder() - .Build(); - -fory.Register(100); -``` - -Use the same ID mapping on all languages. - -Third-party classes, structs, and enums can use -[external-type serialization](external-types.md). Register the target type, -not its local serializer declaration, with the same ID or name used by the -other language peers. - -## Register by Name - -```csharp -fory.Register("com.example.Person"); -``` - -## Xlang Example - -### C# (Serializer) - -```csharp -Person person = new() { Name = "Alice", Age = 30 }; -byte[] payload = fory.Serialize(person); -``` - -### Java (Deserializer) - -```java -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); - -fory.register(Person.class, 100); -Person value = (Person) fory.deserialize(payloadFromCSharp); -``` - -### Python (Deserializer) - -```python -import pyfory - -fory = pyfory.Fory(xlang=True, ref=True) -fory.register_type(Person, type_id=100) -value = fory.deserialize(payload_from_csharp) -``` - -## Type Mapping Reference - -See [xlang guide](../xlang/index.md) for complete mapping. - -For reduced-precision numeric payloads, use `Half` / `Half[]` or `List` for xlang `float16`, and `BFloat16` / `BFloat16[]` or `List` for xlang `bfloat16`. - -## Lists and Dense Arrays - -C# `List` maps to Fory `list`. Use the schema marker -`Apache.Fory.Schema.Types.Array` when a field is dense `array`. - -| Fory schema | C# schema marker sketch | -| ----------------- | ----------------------- | -| `list` | `S.List` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | -| `array` | `S.Array` | - -## Best Practices - -1. Keep type IDs stable and documented. -2. Keep compatible mode enabled for rolling upgrades. -3. Register all user types on both read/write peers. -4. Validate integration with real payload round trips. - -## Related Topics - -- [Type Registration](type-registration.md) -- [External Types](external-types.md) -- [Schema Evolution](schema-evolution.md) -- [Supported Types](supported-types.md) diff --git a/docs/object-serialization/dart/xlang.md b/docs/object-serialization/dart/basic-serialization.md similarity index 68% rename from docs/object-serialization/dart/xlang.md rename to docs/object-serialization/dart/basic-serialization.md index e65c9210f9..0b28adb02b 100644 --- a/docs/object-serialization/dart/xlang.md +++ b/docs/object-serialization/dart/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,9 +19,132 @@ license: | limitations under the License. --- +This page shows how to serialize and deserialize values in the default xlang mode for Apache Fory™ Dart. + +## Create a `Fory` Instance + +Create one instance and reuse it — creating a new `Fory` for every call wastes resources. + +```dart +import 'package:fory/fory.dart'; + +final fory = Fory(); +``` + +## Serialize and Deserialize Annotated Types + +```dart +import 'package:fory/fory.dart'; + +part 'person.fory.dart'; + +@ForyStruct() +class Person { + Person(); + + String name = ''; + + @ForyField(type: Int32Type()) + int age = 0; +} + +void main() { + final fory = Fory(); + PersonForyModule.register( + fory, + Person, + name: 'example.Person', + ); + + final person = Person() + ..name = 'Ada' + ..age = 36; + + final bytes = fory.serialize(person); + final roundTrip = fory.deserialize(bytes); + print(roundTrip.name); +} +``` + +`deserialize` returns the decoded value cast to `T`. If the payload describes a different type than `T`, it throws. + +## Null Values + +Serializing `null` is supported directly: + +```dart +final fory = Fory(); +final bytes = fory.serialize(null); +final value = fory.deserialize(bytes); +``` + +## Serialize Collections and Dynamic Payloads + +You can serialize collection values directly: + +```dart +final fory = Fory(); +final bytes = fory.serialize([ + 'hello', + 42, + true, +]); +final value = fory.deserialize>(bytes); +``` + +For heterogeneous collections, deserialize to `Object?`, `List`, or `Map`. + +## Reference Tracking + +By default, Fory does not track object identity — if the same object appears twice in a list, it is serialized twice. Enable reference tracking when your data contains shared references or circular structures. + +For a top-level collection: + +```dart +final fory = Fory(); +final shared = String.fromCharCodes('shared'.codeUnits); +final bytes = fory.serialize([shared, shared], trackRef: true); +final roundTrip = fory.deserialize>(bytes); +print(identical(roundTrip[0], roundTrip[1])); // true +``` + +For fields inside a generated struct, use `@ForyField(ref: true)` on that field instead. + +## Reusing a Buffer + +If you want to avoid allocating a new `Uint8List` on every call, use `serializeTo` and `deserializeFrom` with an explicit `Buffer`: + +```dart +final fory = Fory(); +final buffer = Buffer(); + +fory.serializeTo('Ada', buffer); +final value = fory.deserializeFrom(buffer); +``` + +This is an optimization. For most applications the default `serialize`/`deserialize` pair is fine. + +## Register Your Types Before Serializing + +Before you can serialize a custom class or enum, register it with `Fory`. The generated code makes this easy: + +```dart +PersonForyModule.register( + fory, + Person, + id: 100, +); +``` + +If you skip registration, deserialization fails with `Type ... is not registered`. See [Type Registration](type-registration.md) and [Code Generation](code-generation.md). + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Apache Fory™ Dart serializes to the same binary format as the Java, Go, C#, Python, Rust, and Swift Fory implementations. You can write a message in Dart and read it in Java — or any other direction — without any conversion layer. -## Setup +### Xlang Configuration Create a `Fory` instance as normal. There is no separate xlang option to enable in Dart: @@ -31,11 +154,11 @@ final fory = Fory(); // xlang payloads with compatible schema evolution The key requirement is that both sides register the same type using the same identity. -## Registration Identity +### Registration Identity The most important rule: **use the same type identity on every side**. You have two options: -### Numeric ID +#### Numeric ID Simpler for small, tightly-coordinated teams: @@ -44,7 +167,7 @@ Simpler for small, tightly-coordinated teams: ModelsForyModule.register(fory, Person, id: 100); ``` -### Namespace + Type Name +#### Namespace + Type Name Better when multiple teams define types independently: @@ -59,7 +182,7 @@ ModelsForyModule.register( Do not mix the two strategies for the same type across implementations. -## External Types +### External Types For a struct class owned by another Dart package, define an [external structural serializer](external-types.md) and register the target @@ -87,9 +210,9 @@ define the Dart-side xlang schema. An external declaration may explicitly list an accessible inherited target property, but Fory does not automatically scan the external target hierarchy. -## Dart to Java Example +### Dart to Java Example -### Dart +#### Dart ```dart import 'package:fory/fory.dart'; @@ -113,7 +236,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### Java +#### Java ```java Fory fory = Fory.builder() @@ -124,9 +247,9 @@ fory.register(Person.class, 100); Person value = (Person) fory.deserialize(bytesFromDart); ``` -## Dart to C# Example +### Dart to C# Example -### Dart +#### Dart ```dart final fory = Fory(); @@ -136,7 +259,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### CSharp +#### CSharp ```csharp [ForyStruct] @@ -153,9 +276,9 @@ fory.Register(100); Person person = fory.Deserialize(payloadFromDart); ``` -## Dart to Go Example +### Dart to Go Example -### Dart +#### Dart ```dart final fory = Fory(); @@ -165,7 +288,7 @@ final bytes = fory.serialize(Person() ..age = 30); ``` -### Go +#### Go ```go type Person struct { @@ -180,7 +303,7 @@ var person Person _ = f.Deserialize(bytesFromDart, &person) ``` -## Field Matching Rules +### Field Matching Rules Fory matches fields by name or by stable field ID. For robust cross-language interop: @@ -204,7 +327,7 @@ metadata use the same reference behavior as fields declared directly on the child. Inheritance does not change xlang reference framing or add parent-level reference state. -## Type Mapping Notes for Dart +### Type Mapping Notes for Dart Because Dart `int` is not itself a promise about the exact xlang wire width, prefer explicit field metadata when exact cross-language interpretation matters: @@ -217,7 +340,7 @@ Because Dart `int` is not itself a promise about the exact xlang wire width, pre - `Float16List` and `Bfloat16List` for 16-bit floating-point array payloads - `Timestamp`, `LocalDate`, and `Duration` for explicit temporal semantics -### Lists and Dense Arrays +#### Lists and Dense Arrays `List` always represents Fory `list` unless a field has explicit array metadata. Use `array` only for dense one-dimensional bool or numeric data. @@ -241,7 +364,7 @@ metadata. Use `array` only for dense one-dimensional bool or numeric data. See [Supported Types](supported-types.md) and [xlang type mapping](../../specification/xlang_type_mapping.md). -## Validation +### Validation Before relying on a cross-language contract in production, test a payload end-to-end through every implementation you support. @@ -253,10 +376,16 @@ dart analyze dart test ``` -## Related Topics +### Related Guides - [Struct Inheritance](inheritance.md) - [Type Registration](type-registration.md) - [External-Type Serialization](external-types.md) - [Schema Evolution](schema-evolution.md) -- [Xlang guide](../xlang/index.md) +- [Cross-language guide](../xlang.md) + +## Related Topics + +- [Configuration](configuration.md) +- [Type Registration](type-registration.md) +- [Schema Metadata](schema-metadata.md) diff --git a/docs/object-serialization/dart/configuration.md b/docs/object-serialization/dart/configuration.md index cbacbb5f65..7094e9b4a9 100644 --- a/docs/object-serialization/dart/configuration.md +++ b/docs/object-serialization/dart/configuration.md @@ -162,19 +162,10 @@ When Fory is used to communicate between services written in different languages ## Security -Security-related configuration: - -- Register only the expected generated models before deserializing untrusted payloads. -- Use `checkStructVersion: true` with `compatible: false` for intentional same-schema payloads. -- Set `maxDepth` to reject unexpectedly deep payload shapes. -- Keep `maxGraphMemoryBytes` at the default for most inputs, or set an explicit positive byte gate - for known trusted collection/map/struct-heavy payloads. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer generated schemas and explicit field metadata over broad dynamic fields for untrusted input. +See [Dart Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/dart/core-api.md b/docs/object-serialization/dart/core-api.md deleted file mode 100644 index 08d586971a..0000000000 --- a/docs/object-serialization/dart/core-api.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 2 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page shows how to serialize and deserialize values with Apache Fory™ Dart. - -## Create a `Fory` Instance - -Create one instance and reuse it — creating a new `Fory` for every call wastes resources. - -```dart -import 'package:fory/fory.dart'; - -final fory = Fory(); -``` - -## Serialize and Deserialize Annotated Types - -```dart -import 'package:fory/fory.dart'; - -part 'person.fory.dart'; - -@ForyStruct() -class Person { - Person(); - - String name = ''; - - @ForyField(type: Int32Type()) - int age = 0; -} - -void main() { - final fory = Fory(); - PersonForyModule.register( - fory, - Person, - name: 'example.Person', - ); - - final person = Person() - ..name = 'Ada' - ..age = 36; - - final bytes = fory.serialize(person); - final roundTrip = fory.deserialize(bytes); - print(roundTrip.name); -} -``` - -`deserialize` returns the decoded value cast to `T`. If the payload describes a different type than `T`, it throws. - -## Null Values - -Serializing `null` is supported directly: - -```dart -final fory = Fory(); -final bytes = fory.serialize(null); -final value = fory.deserialize(bytes); -``` - -## Serialize Collections and Dynamic Payloads - -You can serialize collection values directly: - -```dart -final fory = Fory(); -final bytes = fory.serialize([ - 'hello', - 42, - true, -]); -final value = fory.deserialize>(bytes); -``` - -For heterogeneous collections, deserialize to `Object?`, `List`, or `Map`. - -## Reference Tracking - -By default, Fory does not track object identity — if the same object appears twice in a list, it is serialized twice. Enable reference tracking when your data contains shared references or circular structures. - -For a top-level collection: - -```dart -final fory = Fory(); -final shared = String.fromCharCodes('shared'.codeUnits); -final bytes = fory.serialize([shared, shared], trackRef: true); -final roundTrip = fory.deserialize>(bytes); -print(identical(roundTrip[0], roundTrip[1])); // true -``` - -For fields inside a generated struct, use `@ForyField(ref: true)` on that field instead. - -## Reusing a Buffer - -If you want to avoid allocating a new `Uint8List` on every call, use `serializeTo` and `deserializeFrom` with an explicit `Buffer`: - -```dart -final fory = Fory(); -final buffer = Buffer(); - -fory.serializeTo('Ada', buffer); -final value = fory.deserializeFrom(buffer); -``` - -This is an optimization. For most applications the default `serialize`/`deserialize` pair is fine. - -## Register Your Types Before Serializing - -Before you can serialize a custom class or enum, register it with `Fory`. The generated code makes this easy: - -```dart -PersonForyModule.register( - fory, - Person, - id: 100, -); -``` - -If you skip registration, deserialization fails with `Type ... is not registered`. See [Type Registration](type-registration.md) and [Code Generation](code-generation.md). - -## Related Topics - -- [Configuration](configuration.md) -- [Type Registration](type-registration.md) -- [Schema Metadata](schema-metadata.md) diff --git a/docs/object-serialization/dart/custom-serializers.md b/docs/object-serialization/dart/custom-serializers.md index 7284d14851..55ae78a846 100644 --- a/docs/object-serialization/dart/custom-serializers.md +++ b/docs/object-serialization/dart/custom-serializers.md @@ -140,5 +140,5 @@ Skipping this step causes back-references to that object to resolve to `null`. - [Type Registration](type-registration.md) - [External-Type Serialization](external-types.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/dart/index.md b/docs/object-serialization/dart/index.md index abe3b3eae1..e43822d760 100644 --- a/docs/object-serialization/dart/index.md +++ b/docs/object-serialization/dart/index.md @@ -137,26 +137,28 @@ constructors, mixins, and field inclusion options. ## Documentation -| Topic | Description | -| ------------------------------------------------ | --------------------------------------------------------------- | -| [Configuration](configuration.md) | Fory options, compatible mode, and safety limits | -| [Basic Serialization](core-api.md) | `serialize`, `deserialize`, generated registration, root graphs | -| [Code Generation](code-generation.md) | `@ForyStruct`, build runner, and generated modules | -| [Struct Inheritance](inheritance.md) | Superclasses, mixins, private fields, and constructors | -| [External-Type Serialization](external-types.md) | Generated serializers for classes owned by another package | -| [Xlang Serialization](xlang.md) | Interoperability rules and field alignment | -| [Schema Metadata](schema-metadata.md) | `@ForyField`, field IDs, nullability, references, polymorphism | -| [Type Registration](type-registration.md) | ID-based vs name-based registration and registration rules | -| [Custom Serializers](custom-serializers.md) | Custom `Serializer` implementations and unions | -| [Supported Types](supported-types.md) | Built-in xlang values, wrappers, collections, and structs | -| [Schema Evolution](schema-evolution.md) | Compatible structs and evolving schemas | -| [Web Platform Support](web-platform-support.md) | Dart VM/AOT, Flutter, and web support, limits, and validation | -| [gRPC Support](../../grpc/dart.md) | Generated Fory-backed gRPC service companions | -| [Troubleshooting](troubleshooting.md) | Common errors, diagnostics, and validation steps | +| Topic | Description | +| ------------------------------------------------ | -------------------------------------------------------------- | +| [Configuration](configuration.md) | Fory options, compatible mode, and safety limits | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs, registration, and interoperability | +| [Code Generation](code-generation.md) | `@ForyStruct`, build runner, and generated modules | +| [Struct Inheritance](inheritance.md) | Superclasses, mixins, private fields, and constructors | +| [External-Type Serialization](external-types.md) | Generated serializers for classes owned by another package | +| [Schema Metadata](schema-metadata.md) | `@ForyField`, field IDs, nullability, references, polymorphism | +| [Type Registration](type-registration.md) | ID-based vs name-based registration and registration rules | +| [Custom Serializers](custom-serializers.md) | Custom `Serializer` implementations and unions | +| [Supported Types](supported-types.md) | Built-in xlang values, wrappers, collections, and structs | +| [Schema Evolution](schema-evolution.md) | Compatible structs and evolving schemas | +| [Web Platform Support](web-platform-support.md) | Dart VM/AOT, Flutter, and web support, limits, and validation | +| [gRPC Support](../../grpc/dart.md) | Generated Fory-backed gRPC service companions | +| [Troubleshooting](troubleshooting.md) | Common errors, diagnostics, and validation steps | ## Related Resources - [Xlang serialization specification](../../specification/xlang_serialization_spec.md) - [Xlang implementation guide](../../specification/xlang_implementation_guide.md) -- [Xlang guide](../xlang/index.md) +- [Cross-language guide](../xlang.md) - [Dart implementation source directory](https://github.com/apache/fory/tree/main/dart) + +Before decoding bytes from outside the application trust boundary, read +[Dart Security](security.md). diff --git a/docs/object-serialization/dart/schema-evolution.md b/docs/object-serialization/dart/schema-evolution.md index 45f5c328e1..6f290413a1 100644 --- a/docs/object-serialization/dart/schema-evolution.md +++ b/docs/object-serialization/dart/schema-evolution.md @@ -132,4 +132,4 @@ runtime reference protocol or add a compatibility reader. - [Configuration](configuration.md) - [External-Type Serialization](external-types.md) - [Schema Metadata](schema-metadata.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/dart/schema-metadata.md b/docs/object-serialization/dart/schema-metadata.md index bd2edffe7d..3073cf6bb8 100644 --- a/docs/object-serialization/dart/schema-metadata.md +++ b/docs/object-serialization/dart/schema-metadata.md @@ -169,4 +169,4 @@ When the same model is defined in multiple languages: - [Code Generation](code-generation.md) - [External-Type Serialization](external-types.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/dart/security.md b/docs/object-serialization/dart/security.md new file mode 100644 index 0000000000..4367148291 --- /dev/null +++ b/docs/object-serialization/dart/security.md @@ -0,0 +1,56 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Dart reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected generated models before deserializing untrusted payloads. +- Use `checkStructVersion: true` with `compatible: false` for intentional same-schema payloads. +- Set `maxDepth` to reject unexpectedly deep payload shapes. +- Keep `maxGraphMemoryBytes` at the default for most inputs, or set an explicit positive byte gate + for known trusted collection/map/struct-heavy payloads. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer generated schemas and explicit field metadata over broad dynamic fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/dart/supported-types.md b/docs/object-serialization/dart/supported-types.md index 2136bf78de..52c17b9b7c 100644 --- a/docs/object-serialization/dart/supported-types.md +++ b/docs/object-serialization/dart/supported-types.md @@ -173,5 +173,5 @@ width is one of the most common cross-language bugs. - [Struct Inheritance](inheritance.md) - [Schema Metadata](schema-metadata.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Schema Evolution](schema-evolution.md) diff --git a/docs/object-serialization/dart/troubleshooting.md b/docs/object-serialization/dart/troubleshooting.md index 0a73e13874..038ee4c9d6 100644 --- a/docs/object-serialization/dart/troubleshooting.md +++ b/docs/object-serialization/dart/troubleshooting.md @@ -217,7 +217,7 @@ separate protobuf service endpoint for generic protobuf clients. ## Related Topics - [Struct Inheritance](inheritance.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Code Generation](code-generation.md) - [Custom Serializers](custom-serializers.md) - [Web Platform Support](web-platform-support.md) diff --git a/docs/object-serialization/dart/type-registration.md b/docs/object-serialization/dart/type-registration.md index ff57fdbd1c..2364c81bd6 100644 --- a/docs/object-serialization/dart/type-registration.md +++ b/docs/object-serialization/dart/type-registration.md @@ -119,12 +119,12 @@ See [Custom Serializers](custom-serializers.md) for how to implement a serialize ## Xlang Requirements -The same numeric ID or name must be used in every peer that reads or writes the type. See [Xlang Serialization](xlang.md) for examples. +The same numeric ID or name must be used in every peer that reads or writes the type. See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for examples. ## Related Topics - [Struct Inheritance](inheritance.md) - [Code Generation](code-generation.md) - [External-Type Serialization](external-types.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Custom Serializers](custom-serializers.md) diff --git a/docs/object-serialization/go/xlang.md b/docs/object-serialization/go/basic-serialization.md similarity index 54% rename from docs/object-serialization/go/xlang.md rename to docs/object-serialization/go/basic-serialization.md index d2f9472696..e5b19b3fc5 100644 --- a/docs/object-serialization/go/xlang.md +++ b/docs/object-serialization/go/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,11 +19,397 @@ license: | limitations under the License. --- +This guide covers the core serialization APIs in the default xlang mode for Fory Go. + +## Creating a Fory Instance + +Create a Fory instance and register your types before serialization: + +```go +import "github.com/apache/fory/go/fory" + +f := fory.New(fory.WithXlang(true)) + +// Register struct with a type ID +f.RegisterStruct(User{}, 1) +f.RegisterStruct(Order{}, 2) + +// Or register with a name (more flexible, less prone to ID conflicts, but higher serialization cost) +f.RegisterStructByName(User{}, "example.User") + +// Register enum types +f.RegisterEnum(Color(0), 3) +``` + +`fory.New()` uses xlang mode with compatible schema evolution. The example sets +`fory.WithXlang(true)` explicitly so the mode choice is visible. For Go-only +payloads that need native mode, configure `fory.WithXlang(false)` explicitly in +the native-mode examples. + +**Important**: The Fory instance should be reused across serialization calls. Creating a new instance involves allocating internal buffers, type caches, and resolvers, which is expensive. The default Fory instance is not thread-safe; for concurrent usage, use the thread-safe wrapper (see [Thread Safety](thread-safety.md)). + +See [Type Registration](type-registration.md) for more details. + +## Core API + +### Serialize and Deserialize + +The primary API for serialization: + +```go +// Serialize any value +data, err := f.Serialize(value) +if err != nil { + // Handle error +} + +// Deserialize into target +var result MyType +err = f.Deserialize(data, &result) +if err != nil { + // Handle error +} +``` + +### Marshal and Unmarshal + +Aliases for `Serialize` and `Deserialize` (familiar to Go developers): + +```go +data, err := f.Marshal(value) +err = f.Unmarshal(data, &result) +``` + +## Serializing Primitives + +```go +// Integers +data, _ := f.Serialize(int64(42)) +var i int64 +f.Deserialize(data, &i) // i = 42 + +// Floats +data, _ = f.Serialize(float64(3.14)) +var fl float64 +f.Deserialize(data, &fl) // fl = 3.14 + +// Strings +data, _ = f.Serialize("hello") +var s string +f.Deserialize(data, &s) // s = "hello" + +// Booleans +data, _ = f.Serialize(true) +var b bool +f.Deserialize(data, &b) // b = true +``` + +## Serializing Collections + +### Slices + +```go +// String slice +strs := []string{"a", "b", "c"} +data, _ := f.Serialize(strs) + +var result []string +f.Deserialize(data, &result) +// result = ["a", "b", "c"] + +// Integer slice +nums := []int64{1, 2, 3} +data, _ = f.Serialize(nums) + +var intResult []int64 +f.Deserialize(data, &intResult) +// intResult = [1, 2, 3] +``` + +### Maps + +```go +// String to string map +m := map[string]string{"key": "value"} +data, _ := f.Serialize(m) + +var result map[string]string +f.Deserialize(data, &result) +// result = {"key": "value"} + +// String to int map +m2 := map[string]int64{"count": 42} +data, _ = f.Serialize(m2) + +var result2 map[string]int64 +f.Deserialize(data, &result2) +// result2 = {"count": 42} +``` + +## Serializing Structs + +### Basic Struct Serialization + +Only **exported fields** (starting with uppercase) are serialized: + +```go +type User struct { + ID int64 // Serialized + Name string // Serialized + password string // NOT serialized (unexported) +} + +f.RegisterStruct(User{}, 1) + +user := &User{ID: 1, Name: "Alice", password: "secret"} +data, _ := f.Serialize(user) + +var result User +f.Deserialize(data, &result) +// result.ID = 1, result.Name = "Alice", result.password = "" +``` + +### Nested Structs + +```go +type Address struct { + City string + Country string +} + +type Person struct { + Name string + Address Address +} + +f.RegisterStruct(Address{}, 1) +f.RegisterStruct(Person{}, 2) + +person := &Person{ + Name: "Alice", + Address: Address{City: "NYC", Country: "USA"}, +} + +data, _ := f.Serialize(person) + +var result Person +f.Deserialize(data, &result) +// result.Address.City = "NYC" +``` + +### Pointer Fields + +```go +type Node struct { + Value int32 + Child *Node +} + +// Use WithTrackRef for pointer fields +f := fory.New(fory.WithXlang(true), fory.WithTrackRef(true)) +f.RegisterStruct(Node{}, 1) + +root := &Node{ + Value: 1, + Child: &Node{Value: 2, Child: nil}, +} + +data, _ := f.Serialize(root) + +var result Node +f.Deserialize(data, &result) +// result.Child.Value = 2 +``` + +## Streaming API + +For scenarios where you want to control the buffer: + +### SerializeTo + +Serialize to an existing buffer: + +```go +buf := fory.NewByteBuffer(nil) + +// Serialize multiple values to same buffer +f.SerializeTo(buf, value1) +f.SerializeTo(buf, value2) + +// Get all serialized data +data := buf.GetByteSlice(0, buf.WriterIndex()) +``` + +### DeserializeFrom + +Deserialize from an existing buffer: + +```go +buf := fory.NewByteBuffer(data) + +var result1, result2 MyType +f.DeserializeFrom(buf, &result1) +f.DeserializeFrom(buf, &result2) +``` + +## Generic API (Type-Safe) + +Fory Go provides generic functions for type-safe serialization: + +```go +import "github.com/apache/fory/go/fory" + +type User struct { + ID int64 + Name string +} + +// Type-safe serialization +user := &User{ID: 1, Name: "Alice"} +data, err := fory.Serialize(f, user) + +// Type-safe deserialization +var result User +err = fory.Deserialize(f, data, &result) +``` + +The generic API: + +- Infers type at compile time +- Provides better type safety +- May offer performance benefits + +## Error Handling + +Always check errors from serialization operations: + +```go +data, err := f.Serialize(value) +if err != nil { + switch e := err.(type) { + case fory.Error: + fmt.Printf("Fory error: %s (kind: %d)\n", e.Error(), e.Kind()) + default: + fmt.Printf("Unknown error: %v\n", err) + } + return +} + +err = f.Deserialize(data, &result) +if err != nil { + // Handle deserialization error +} +``` + +Common error kinds: + +- `ErrKindBufferOutOfBound`: Read/write beyond buffer bounds +- `ErrKindTypeMismatch`: Type ID mismatch during deserialization +- `ErrKindUnknownType`: Unknown type encountered +- `ErrKindMaxDepthExceeded`: Recursion depth limit exceeded +- `ErrKindHashMismatch`: Struct hash mismatch (schema changed) + +See [Troubleshooting](troubleshooting.md) for error resolution. + +## Nil Handling + +### Nil Pointers + +```go +var ptr *User = nil +data, _ := f.Serialize(ptr) + +var result *User +f.Deserialize(data, &result) +// result = nil +``` + +### Empty Collections + +```go +// Nil slice +var slice []string = nil +data, _ := f.Serialize(slice) + +var result []string +f.Deserialize(data, &result) +// result = nil + +// Empty slice (different from nil) +empty := []string{} +data, _ = f.Serialize(empty) + +f.Deserialize(data, &result) +// result = [] (empty, not nil) +``` + +## Complete Example + +```go +package main + +import ( + "fmt" + "github.com/apache/fory/go/fory" +) + +type Order struct { + ID int64 + Customer string + Items []Item + Total float64 +} + +type Item struct { + Name string + Quantity int32 + Price float64 +} + +func main() { + f := fory.New(fory.WithXlang(true)) + f.RegisterStruct(Order{}, 1) + f.RegisterStruct(Item{}, 2) + + order := &Order{ + ID: 12345, + Customer: "Alice", + Items: []Item{ + {Name: "Widget", Quantity: 2, Price: 9.99}, + {Name: "Gadget", Quantity: 1, Price: 24.99}, + }, + Total: 44.97, + } + + // Serialize + data, err := f.Serialize(order) + if err != nil { + panic(err) + } + fmt.Printf("Serialized %d bytes\n", len(data)) + + // Deserialize + var result Order + if err := f.Deserialize(data, &result); err != nil { + panic(err) + } + + fmt.Printf("Order ID: %d\n", result.ID) + fmt.Printf("Customer: %s\n", result.Customer) + fmt.Printf("Items: %d\n", len(result.Items)) + fmt.Printf("Total: %.2f\n", result.Total) +} +``` + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Fory Go enables seamless data exchange with Java, Python, C++, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. This guide covers xlang compatibility and type mapping. -## Create an Xlang Fory Instance +### Xlang Configuration Go defaults to xlang mode with compatible schema evolution. Set the mode explicitly in xlang examples: @@ -31,11 +417,11 @@ Go defaults to xlang mode with compatible schema evolution. Set the mode explici f := fory.New(fory.WithXlang(true)) ``` -## Type Registration for Xlang +### Type Registration for Xlang Use consistent type IDs across all languages: -### Go +#### Go ```go type User struct { @@ -48,7 +434,7 @@ f.RegisterStruct(User{}, 1) data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) ``` -### Java +#### Java ```java public class User { @@ -60,7 +446,7 @@ fory.register(User.class, 1); User user = fory.deserialize(data, User.class); ``` -### Python +#### Python ```python from dataclasses import dataclass @@ -76,11 +462,11 @@ fory.register(User, type_id=1) user = fory.deserialize(data) ``` -## Type Mapping +### Type Mapping See [Type Mapping Specification](../../specification/xlang_type_mapping.md) for detailed type mappings across all languages. -## Field Ordering +### Field Ordering Cross-language serialization requires consistent field ordering. Fory sorts fields by their snake_case names alphabetically. @@ -106,9 +492,9 @@ type Example struct { } ``` -## Examples +### Examples -### Go to Java +#### Go to Java **Go (Serializer)**: @@ -149,7 +535,7 @@ fory.register(Order.class, 1); Order order = fory.deserialize(data, Order.class); ``` -### Python to Go +#### Python to Go **Python (Serializer)**: @@ -187,11 +573,11 @@ f.Deserialize(data, &msg) fmt.Println(msg.Content) // "Hello from Python" ``` -### Nested Structures +#### Nested Structures Cross-language nested structures require all types to be registered: -## Lists and Dense Arrays +### Lists and Dense Arrays Go slices are ordinary `list` carriers unless a field tag explicitly requests the dense `array` schema. Use `array` only for one-dimensional bool or @@ -251,9 +637,9 @@ fory.register(Address.class, 1); fory.register(Company.class, 2); ``` -## Common Issues +### Common Issues -### Field Name Mismatch +#### Field Name Mismatch Go uses PascalCase, other languages may use camelCase or snake_case. Fields are matched by their snake_case conversion: @@ -269,7 +655,7 @@ public class User { } ``` -### Type Interpretation +#### Type Interpretation Go unsigned types map to Java signed types with the same bit pattern: @@ -279,7 +665,7 @@ var value uint64 = 18446744073709551615 // Max uint64 Java's `long` holds the same bits but interprets as -1. Use `Long.toUnsignedString()` in Java if unsigned interpretation is needed. -### Nil vs Null +#### Nil vs Null Go nil slices/maps serialize differently based on configuration: @@ -290,7 +676,7 @@ var slice []string = nil Ensure other languages handle null appropriately. -## Best Practices +### Interoperability Best Practices 1. **Use consistent type IDs**: Same numeric ID for the same type across all languages 2. **Register all types**: Including nested struct types @@ -298,7 +684,7 @@ Ensure other languages handle null appropriately. 4. **Test cross-language**: Run integration tests early and often 5. **Handle type differences**: Be aware of signed/unsigned interpretation differences -## Related Topics +### Related Guides - [Type Registration](type-registration.md) - [Supported Types](supported-types.md) @@ -306,7 +692,7 @@ Ensure other languages handle null appropriately. - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Specification](../../specification/xlang_type_mapping.md) -## Built-in values +### Built-in values ```go package main @@ -344,7 +730,7 @@ func main() { } ``` -## Custom values +### Custom values ```go package main @@ -407,7 +793,7 @@ func main() { } ``` -## Shared and circular references +### Shared and circular references ```go package main @@ -440,3 +826,10 @@ func main() { fmt.Println(newValue) } ``` + +## Related Topics + +- [Configuration](configuration.md) +- [Type Registration](type-registration.md) +- [Supported Types](supported-types.md) +- [References](references.md) diff --git a/docs/object-serialization/go/configuration.md b/docs/object-serialization/go/configuration.md index 1e94af8942..e3de7416ed 100644 --- a/docs/object-serialization/go/configuration.md +++ b/docs/object-serialization/go/configuration.md @@ -417,17 +417,11 @@ for req := range requests { ## Security -Security-related configuration: - -- Register only the expected structs before deserializing untrusted data. -- Use `WithMaxDepth(...)` to reject unexpectedly deep payloads. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer concrete struct fields over broad `any` or interface-typed fields for untrusted input. +See [Go Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [References](references.md) - [Schema Evolution](schema-evolution.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/go/core-api.md b/docs/object-serialization/go/core-api.md deleted file mode 100644 index 0a92aa70cf..0000000000 --- a/docs/object-serialization/go/core-api.md +++ /dev/null @@ -1,409 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This guide covers the core serialization APIs in Fory Go. - -## Creating a Fory Instance - -Create a Fory instance and register your types before serialization: - -```go -import "github.com/apache/fory/go/fory" - -f := fory.New(fory.WithXlang(true)) - -// Register struct with a type ID -f.RegisterStruct(User{}, 1) -f.RegisterStruct(Order{}, 2) - -// Or register with a name (more flexible, less prone to ID conflicts, but higher serialization cost) -f.RegisterStructByName(User{}, "example.User") - -// Register enum types -f.RegisterEnum(Color(0), 3) -``` - -`fory.New()` uses xlang mode with compatible schema evolution. The example sets -`fory.WithXlang(true)` explicitly so the mode choice is visible. For Go-only -payloads that need native mode, configure `fory.WithXlang(false)` explicitly in -the native-mode examples. - -**Important**: The Fory instance should be reused across serialization calls. Creating a new instance involves allocating internal buffers, type caches, and resolvers, which is expensive. The default Fory instance is not thread-safe; for concurrent usage, use the thread-safe wrapper (see [Thread Safety](thread-safety.md)). - -See [Type Registration](type-registration.md) for more details. - -## Core API - -### Serialize and Deserialize - -The primary API for serialization: - -```go -// Serialize any value -data, err := f.Serialize(value) -if err != nil { - // Handle error -} - -// Deserialize into target -var result MyType -err = f.Deserialize(data, &result) -if err != nil { - // Handle error -} -``` - -### Marshal and Unmarshal - -Aliases for `Serialize` and `Deserialize` (familiar to Go developers): - -```go -data, err := f.Marshal(value) -err = f.Unmarshal(data, &result) -``` - -## Serializing Primitives - -```go -// Integers -data, _ := f.Serialize(int64(42)) -var i int64 -f.Deserialize(data, &i) // i = 42 - -// Floats -data, _ = f.Serialize(float64(3.14)) -var fl float64 -f.Deserialize(data, &fl) // fl = 3.14 - -// Strings -data, _ = f.Serialize("hello") -var s string -f.Deserialize(data, &s) // s = "hello" - -// Booleans -data, _ = f.Serialize(true) -var b bool -f.Deserialize(data, &b) // b = true -``` - -## Serializing Collections - -### Slices - -```go -// String slice -strs := []string{"a", "b", "c"} -data, _ := f.Serialize(strs) - -var result []string -f.Deserialize(data, &result) -// result = ["a", "b", "c"] - -// Integer slice -nums := []int64{1, 2, 3} -data, _ = f.Serialize(nums) - -var intResult []int64 -f.Deserialize(data, &intResult) -// intResult = [1, 2, 3] -``` - -### Maps - -```go -// String to string map -m := map[string]string{"key": "value"} -data, _ := f.Serialize(m) - -var result map[string]string -f.Deserialize(data, &result) -// result = {"key": "value"} - -// String to int map -m2 := map[string]int64{"count": 42} -data, _ = f.Serialize(m2) - -var result2 map[string]int64 -f.Deserialize(data, &result2) -// result2 = {"count": 42} -``` - -## Serializing Structs - -### Basic Struct Serialization - -Only **exported fields** (starting with uppercase) are serialized: - -```go -type User struct { - ID int64 // Serialized - Name string // Serialized - password string // NOT serialized (unexported) -} - -f.RegisterStruct(User{}, 1) - -user := &User{ID: 1, Name: "Alice", password: "secret"} -data, _ := f.Serialize(user) - -var result User -f.Deserialize(data, &result) -// result.ID = 1, result.Name = "Alice", result.password = "" -``` - -### Nested Structs - -```go -type Address struct { - City string - Country string -} - -type Person struct { - Name string - Address Address -} - -f.RegisterStruct(Address{}, 1) -f.RegisterStruct(Person{}, 2) - -person := &Person{ - Name: "Alice", - Address: Address{City: "NYC", Country: "USA"}, -} - -data, _ := f.Serialize(person) - -var result Person -f.Deserialize(data, &result) -// result.Address.City = "NYC" -``` - -### Pointer Fields - -```go -type Node struct { - Value int32 - Child *Node -} - -// Use WithTrackRef for pointer fields -f := fory.New(fory.WithXlang(true), fory.WithTrackRef(true)) -f.RegisterStruct(Node{}, 1) - -root := &Node{ - Value: 1, - Child: &Node{Value: 2, Child: nil}, -} - -data, _ := f.Serialize(root) - -var result Node -f.Deserialize(data, &result) -// result.Child.Value = 2 -``` - -## Streaming API - -For scenarios where you want to control the buffer: - -### SerializeTo - -Serialize to an existing buffer: - -```go -buf := fory.NewByteBuffer(nil) - -// Serialize multiple values to same buffer -f.SerializeTo(buf, value1) -f.SerializeTo(buf, value2) - -// Get all serialized data -data := buf.GetByteSlice(0, buf.WriterIndex()) -``` - -### DeserializeFrom - -Deserialize from an existing buffer: - -```go -buf := fory.NewByteBuffer(data) - -var result1, result2 MyType -f.DeserializeFrom(buf, &result1) -f.DeserializeFrom(buf, &result2) -``` - -## Generic API (Type-Safe) - -Fory Go provides generic functions for type-safe serialization: - -```go -import "github.com/apache/fory/go/fory" - -type User struct { - ID int64 - Name string -} - -// Type-safe serialization -user := &User{ID: 1, Name: "Alice"} -data, err := fory.Serialize(f, user) - -// Type-safe deserialization -var result User -err = fory.Deserialize(f, data, &result) -``` - -The generic API: - -- Infers type at compile time -- Provides better type safety -- May offer performance benefits - -## Error Handling - -Always check errors from serialization operations: - -```go -data, err := f.Serialize(value) -if err != nil { - switch e := err.(type) { - case fory.Error: - fmt.Printf("Fory error: %s (kind: %d)\n", e.Error(), e.Kind()) - default: - fmt.Printf("Unknown error: %v\n", err) - } - return -} - -err = f.Deserialize(data, &result) -if err != nil { - // Handle deserialization error -} -``` - -Common error kinds: - -- `ErrKindBufferOutOfBound`: Read/write beyond buffer bounds -- `ErrKindTypeMismatch`: Type ID mismatch during deserialization -- `ErrKindUnknownType`: Unknown type encountered -- `ErrKindMaxDepthExceeded`: Recursion depth limit exceeded -- `ErrKindHashMismatch`: Struct hash mismatch (schema changed) - -See [Troubleshooting](troubleshooting.md) for error resolution. - -## Nil Handling - -### Nil Pointers - -```go -var ptr *User = nil -data, _ := f.Serialize(ptr) - -var result *User -f.Deserialize(data, &result) -// result = nil -``` - -### Empty Collections - -```go -// Nil slice -var slice []string = nil -data, _ := f.Serialize(slice) - -var result []string -f.Deserialize(data, &result) -// result = nil - -// Empty slice (different from nil) -empty := []string{} -data, _ = f.Serialize(empty) - -f.Deserialize(data, &result) -// result = [] (empty, not nil) -``` - -## Complete Example - -```go -package main - -import ( - "fmt" - "github.com/apache/fory/go/fory" -) - -type Order struct { - ID int64 - Customer string - Items []Item - Total float64 -} - -type Item struct { - Name string - Quantity int32 - Price float64 -} - -func main() { - f := fory.New(fory.WithXlang(true)) - f.RegisterStruct(Order{}, 1) - f.RegisterStruct(Item{}, 2) - - order := &Order{ - ID: 12345, - Customer: "Alice", - Items: []Item{ - {Name: "Widget", Quantity: 2, Price: 9.99}, - {Name: "Gadget", Quantity: 1, Price: 24.99}, - }, - Total: 44.97, - } - - // Serialize - data, err := f.Serialize(order) - if err != nil { - panic(err) - } - fmt.Printf("Serialized %d bytes\n", len(data)) - - // Deserialize - var result Order - if err := f.Deserialize(data, &result); err != nil { - panic(err) - } - - fmt.Printf("Order ID: %d\n", result.ID) - fmt.Printf("Customer: %s\n", result.Customer) - fmt.Printf("Items: %d\n", len(result.Items)) - fmt.Printf("Total: %.2f\n", result.Total) -} -``` - -## Related Topics - -- [Configuration](configuration.md) -- [Type Registration](type-registration.md) -- [Supported Types](supported-types.md) -- [References](references.md) diff --git a/docs/object-serialization/go/custom-serializers.md b/docs/object-serialization/go/custom-serializers.md index 5ab02224d2..3f809fd32b 100644 --- a/docs/object-serialization/go/custom-serializers.md +++ b/docs/object-serialization/go/custom-serializers.md @@ -282,4 +282,4 @@ func TestMySerializer(t *testing.T) { - [Type Registration](type-registration.md) - [Supported Types](supported-types.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/go/index.md b/docs/object-serialization/go/index.md index b053d1e0b7..a4e11d7e18 100644 --- a/docs/object-serialization/go/index.md +++ b/docs/object-serialization/go/index.md @@ -91,7 +91,7 @@ Use xlang mode for cross-language payloads and schemas shared with other Fory im Use native mode for Go-only traffic. Native mode is selected with `fory.WithXlang(false)` and keeps Go object serialization in Go-native form. It is optimized for Go structs, pointers, interfaces, and Go-specific type behavior that does not need a portable xlang mapping. Compatible mode is enabled by default. Set `fory.WithCompatible(false)` only when every reader and writer uses the same Go struct schema and you want faster serialization and smaller size. -See [Xlang Serialization](xlang.md) for Go xlang registration and interoperability rules, and [Native Serialization](native.md) for Go-only payloads. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Go xlang registration and interoperability rules, and [Native Serialization](native.md) for Go-only payloads. ## Configuration @@ -133,28 +133,30 @@ data, _ := f.Serialize(&User{ID: 1, Name: "Alice"}) // 'data' can be deserialized by Java, Python, etc. ``` -See [Xlang Serialization](xlang.md) for type mapping and compatibility details. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for type mapping and compatibility details. ## Documentation -| Topic | Description | -| ------------------------------------------- | -------------------------------------- | -| [Basic Serialization](core-api.md) | Core APIs and usage patterns | -| [Xlang Serialization](xlang.md) | Multi-language serialization | -| [Native Serialization](native.md) | Go-only serialization | -| [Configuration](configuration.md) | Options and settings | -| [Schema Metadata](schema-metadata.md) | Field-level configuration | -| [Type Registration](type-registration.md) | Registering types for serialization | -| [Supported Types](supported-types.md) | Complete type support reference | -| [References](references.md) | Circular references and shared objects | -| [Schema Evolution](schema-evolution.md) | Forward/backward compatibility | -| [Custom Serializers](custom-serializers.md) | Extend serialization behavior | -| [Thread Safety](thread-safety.md) | Concurrent usage patterns | -| [gRPC Support](../../grpc/go.md) | Fory payloads over grpc-go | -| [Troubleshooting](troubleshooting.md) | Common issues and solutions | +| Topic | Description | +| --------------------------------------------- | --------------------------------------- | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs and interoperability | +| [Native Serialization](native.md) | Go-only serialization | +| [Configuration](configuration.md) | Options and settings | +| [Schema Metadata](schema-metadata.md) | Field-level configuration | +| [Type Registration](type-registration.md) | Registering types for serialization | +| [Supported Types](supported-types.md) | Complete type support reference | +| [References](references.md) | Circular references and shared objects | +| [Schema Evolution](schema-evolution.md) | Forward/backward compatibility | +| [Custom Serializers](custom-serializers.md) | Extend serialization behavior | +| [Thread Safety](thread-safety.md) | Concurrent usage patterns | +| [gRPC Support](../../grpc/go.md) | Fory payloads over grpc-go | +| [Troubleshooting](troubleshooting.md) | Common issues and solutions | ## Related Resources - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Xlang Type Mapping](../../specification/xlang_type_mapping.md) - [GitHub Repository](https://github.com/apache/fory) + +Before decoding bytes from outside the application trust boundary, read +[Go Security](security.md). diff --git a/docs/object-serialization/go/native.md b/docs/object-serialization/go/native.md index ac1b831f72..cf72cb866c 100644 --- a/docs/object-serialization/go/native.md +++ b/docs/object-serialization/go/native.md @@ -23,7 +23,7 @@ Go native serialization is the Go-only wire mode selected with `fory.WithXlang(f when every writer and reader is a Go service and the payload should follow Go's type system instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default Go mode, when bytes must be read by +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Go mode, when bytes must be read by Java, Python, C++, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-Go Fory implementation. @@ -210,7 +210,7 @@ The default `Fory` instance reuses its buffer. Copy the byte slice or use `threa ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language Go payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Go payloads - [Configuration](configuration.md) - Go options - [Type Registration](type-registration.md) - Struct and enum registration - [References](references.md) - Shared and circular references diff --git a/docs/object-serialization/go/references.md b/docs/object-serialization/go/references.md index 92470b5193..0469e33eb8 100644 --- a/docs/object-serialization/go/references.md +++ b/docs/object-serialization/go/references.md @@ -353,4 +353,4 @@ func main() { - [Configuration](configuration.md) - [Struct Tags](schema-metadata.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/go/schema-evolution.md b/docs/object-serialization/go/schema-evolution.md index b46aecf209..6a8afb47ba 100644 --- a/docs/object-serialization/go/schema-evolution.md +++ b/docs/object-serialization/go/schema-evolution.md @@ -389,5 +389,5 @@ func main() { ## Related Topics - [Configuration](configuration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/go/schema-metadata.md b/docs/object-serialization/go/schema-metadata.md index 02ac8f5d09..878661dabb 100644 --- a/docs/object-serialization/go/schema-metadata.md +++ b/docs/object-serialization/go/schema-metadata.md @@ -389,5 +389,5 @@ type Session struct { ## Related Topics - [References](references.md) -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) diff --git a/docs/object-serialization/go/security.md b/docs/object-serialization/go/security.md new file mode 100644 index 0000000000..916643e3e3 --- /dev/null +++ b/docs/object-serialization/go/security.md @@ -0,0 +1,53 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Go reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected structs before deserializing untrusted data. +- Use `WithMaxDepth(...)` to reject unexpectedly deep payloads. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer concrete struct fields over broad `any` or interface-typed fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/go/supported-types.md b/docs/object-serialization/go/supported-types.md index e21e0c48de..c0098d6649 100644 --- a/docs/object-serialization/go/supported-types.md +++ b/docs/object-serialization/go/supported-types.md @@ -354,7 +354,7 @@ data, _ := f.Serialize(status) | `time.Time` | Instant | datetime | - | - | | `time.Duration` | Duration | timedelta | - | - | -See [Xlang Serialization](xlang.md) for detailed mapping. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for detailed mapping. ## Unsupported Types @@ -370,5 +370,5 @@ Attempting to serialize these types will result in an error. ## Related Topics - [Type Registration](type-registration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [References](references.md) diff --git a/docs/object-serialization/go/thread-safety.md b/docs/object-serialization/go/thread-safety.md index 7d78e3bb9b..89b5367667 100644 --- a/docs/object-serialization/go/thread-safety.md +++ b/docs/object-serialization/go/thread-safety.md @@ -343,5 +343,5 @@ go func() { ## Related Topics - [Configuration](configuration.md) -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/go/troubleshooting.md b/docs/object-serialization/go/troubleshooting.md index e9a04edcd4..9c51ba6be0 100644 --- a/docs/object-serialization/go/troubleshooting.md +++ b/docs/object-serialization/go/troubleshooting.md @@ -438,6 +438,6 @@ If you encounter issues not covered here: ## Related Topics - [Configuration](configuration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Schema Evolution](schema-evolution.md) - [Thread Safety](thread-safety.md) diff --git a/docs/object-serialization/go/type-registration.md b/docs/object-serialization/go/type-registration.md index 18abc8ad63..9367affdd3 100644 --- a/docs/object-serialization/go/type-registration.md +++ b/docs/object-serialization/go/type-registration.md @@ -258,7 +258,7 @@ Two types registered with the same ID will conflict. ## Related Topics -- [Basic Serialization](core-api.md) -- [Xlang Serialization](xlang.md) +- [Basic Serialization](basic-serialization.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Supported Types](supported-types.md) - [Troubleshooting](troubleshooting.md) diff --git a/docs/object-serialization/index.md b/docs/object-serialization/index.md index 996a5080dd..02eeae901e 100644 --- a/docs/object-serialization/index.md +++ b/docs/object-serialization/index.md @@ -1,5 +1,5 @@ --- -title: Object Serialization +title: Introduction sidebar_position: 0 id: index license: | @@ -24,27 +24,43 @@ collections, polymorphic values, and optional shared references. ## Choose a mode -| Mode | Use it when | Start here | -| ------ | ---------------------------------------------------- | ---------------------------- | -| Xlang | Bytes cross runtime boundaries | [Xlang mode](xlang/index.md) | -| Native | Every writer and reader uses the same runtime family | [Native mode](native.md) | +| Mode | Use it when | Start here | +| --------------- | ---------------------------------------------------- | --------------------------------- | +| Xlang (default) | Bytes cross runtime boundaries | [Xlang Serialization](xlang.md) | +| Native | Every writer and reader uses the same runtime family | [Native serialization](native.md) | Xlang and native are the only object-serialization modes. Row Format is a random-access analytical representation, and Fory JSON is a Java JSON codec; use the [format chooser](../introduction/choose-a-format.md) when object reconstruction is not your goal. +## Read the concepts and modes + +Read these pages before choosing a runtime API: + +1. [Core Concepts](core-concepts.md) explains object graphs, types, schemas, references, + polymorphism, and schema evolution across both modes. +2. [Xlang Serialization](xlang.md) explains the portable format used by default and the rules that + different language runtimes must share. +3. [Native Serialization](native.md) explains when a same-runtime native format is appropriate. + ## Browse by runtime Choose a runtime to find its installation route, lifecycle, exact APIs, configuration, type registration, schema behavior, extensions, platforms, and troubleshooting: -[Browse supported runtimes](runtimes.md) - -## Security - -Before decoding externally supplied bytes, read [Object Serialization Security](security.md). It -covers accepted-type policy, registration, resource limits, transport responsibilities, and -negative verification for both modes. +| Runtime | Modes | Documentation | +| --------------------- | -------------------- | ------------------------------------------------------ | +| Java | xlang and native | [Java runtime](./java/index.md) | +| Python | xlang and native | [Python runtime](./python/index.md) | +| C++ | xlang and native | [C++ runtime](./cpp/index.md) | +| Go | xlang and native | [Go runtime](./go/index.md) | +| Rust | xlang and native | [Rust runtime](./rust/index.md) | +| JavaScript/TypeScript | xlang | [JavaScript/TypeScript runtime](./javascript/index.md) | +| C# | xlang | [C# runtime](./csharp/index.md) | +| Swift | xlang | [Swift runtime](./swift/index.md) | +| Dart | xlang | [Dart runtime](./dart/index.md) | +| Scala | xlang and JVM native | [Scala runtime](./scala/index.md) | +| Kotlin | xlang and JVM native | [Kotlin runtime](./kotlin/index.md) | ## Specifications diff --git a/docs/object-serialization/java/advanced-features.md b/docs/object-serialization/java/advanced-features.md index 1f58d604ff..ea7cce50bc 100644 --- a/docs/object-serialization/java/advanced-features.md +++ b/docs/object-serialization/java/advanced-features.md @@ -157,4 +157,4 @@ static { - [Configuration](configuration.md) - All ForyBuilder options - [Native Serialization](native.md) - Java-only serialization, JDK hooks, and zero-copy buffers - [Object Copy](object-copy.md) - Deep copy functionality -- [Xlang Serialization](xlang.md) - Java xlang interoperability +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Java xlang interoperability diff --git a/docs/object-serialization/java/xlang.md b/docs/object-serialization/java/basic-serialization.md similarity index 68% rename from docs/object-serialization/java/xlang.md rename to docs/object-serialization/java/basic-serialization.md index c18b13d538..70fb3ea7dc 100644 --- a/docs/object-serialization/java/xlang.md +++ b/docs/object-serialization/java/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,12 +19,120 @@ license: | limitations under the License. --- +This page covers the Java xlang quickstart. Xlang mode is the default Java wire format and is the +right first choice for cross-language payloads. + +## Create a Fory Instance + +For a single-threaded xlang Fory instance, set the mode explicitly: + +```java +import org.apache.fory.Fory; + +Fory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .build(); +``` + +For a thread-safe Fory instance, build `ThreadSafeFory` from the same builder: + +```java +import org.apache.fory.ThreadSafeFory; + +ThreadSafeFory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .buildThreadSafeFory(); +``` + +Default Java xlang mode also defaults to compatible schema mode, so independently deployed services +can add and remove fields when their schema metadata remains compatible. Use +`withCompatible(false)` only when every reader and writer always uses the same schema and you want +faster serialization and smaller size. Use the `compatible=false` opt-out only after verifying that every language uses the same xlang schema, or when native types are generated from Fory schema IDL. + +## Register Custom Types + +Register application classes with the same type identity on every peer. Numeric IDs are compact and +fast, while name registration is easier to coordinate across independently owned services. + +```java +import org.apache.fory.annotation.ForyField; + +public class User { + @ForyField(id = 0) + public String name; + + @ForyField(id = 1) + public int age; +} + +Fory fory = Fory.builder() + .withXlang(true) + .requireClassRegistration(true) + .build(); + +fory.register(User.class, "example", "User"); +``` + +Use field IDs for long-lived schemas so field identity is stable even if Java field names change. +See [Schema Metadata](schema-metadata.md) for Java annotations, nullability, reference tracking, and +enum metadata. + +## Serialize And Deserialize + +```java +User user = new User(); +user.name = "Alice"; +user.age = 30; + +byte[] bytes = fory.serialize(user); +User decoded = fory.deserialize(bytes, User.class); +``` + +When xlang bytes cross languages, every peer must register the same type identity and compatible +field metadata. The shared rules live in [Cross-Language Interoperability](../xlang.md), and the Java-specific +interoperability requirements are covered below. + +## Use Native Serialization For Java-Only Traffic + +For same-language Java/JVM traffic, native mode is usually the better fit: + +```java +Fory fory = Fory.builder() + .withXlang(false) + .build(); +``` + +Native mode supports the broad Java object serialization surface, including JDK serialization hooks, +object copy, and native-mode zero-copy buffers. See [Native Serialization](native.md). + +## Common Options + +- `withRefTracking(true)` preserves shared references and circular references. +- `requireClassRegistration(true)` keeps the default registered-type policy. +- Compatible mode is enabled by default for native-mode and xlang payloads. Use + `withCompatible(false)` only when every reader and writer uses the same schema and you want faster + serialization and smaller size. For xlang payloads, use the `compatible=false` opt-out only after verifying that every language uses the same schema, or when native types are generated from Fory schema IDL. +- `withAsyncCompilation(true)` enables asynchronous serializer compilation where supported. + +## Best Practices + +1. **Reuse Fory instances**: Creating Fory is expensive, always reuse instances +2. **Use appropriate thread safety**: Choose between single-thread and thread-safe based on your needs +3. **Register classes**: Keep type identity stable across every xlang peer +4. **Configure reference tracking**: Enable it only when the object graph needs identity or cycles + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Apache Fory™ xlang serialization is the Java wire mode for payloads that must be read by Python, Rust, Go, JavaScript/TypeScript, C++, C#, Swift, Dart, Scala, Kotlin, or another non-Java Fory implementation. Java defaults to xlang mode with compatible schema evolution, but examples set the mode explicitly so the payload contract is visible in code. -## Create an Xlang Fory Instance +### Xlang Configuration Use one long-lived `Fory` or `ThreadSafeFory` instance per configuration. Creating a Fory instance is expensive because Fory caches type metadata and generated serializers. @@ -45,12 +153,12 @@ identity or cycles. Disable it for value-shaped schemas. Use [Native Serialization](native.md) instead when every writer and reader is Java and the payload should preserve Java-specific object behavior. -## Register Types +### Register Types Types must be registered with consistent IDs or names across all languages. Fory supports two registration methods. -### Register by ID (Recommended for Performance) +#### Register by ID (Recommended for Performance) ```java public record Person(String name, int age) {} @@ -67,7 +175,7 @@ Benefits: faster serialization and smaller binary size. Trade-off: every service must coordinate IDs so the same logical type uses the same number. -### Register by Name (Recommended for Flexibility) +#### Register by Name (Recommended for Flexibility) ```java public record Person(String name, int age) {} @@ -88,9 +196,9 @@ Trade-off: the payload includes string identity, so it is larger than ID-based r The Java API also supports a single string type name, such as `fory.register(Person.class, "example.Person")`. Use the same logical identity on every peer. -## Java To Python Example +### Java To Python Example -### Java (Serializer) +#### Java (Serializer) ```java import org.apache.fory.Fory; @@ -116,7 +224,7 @@ public class Example { } ``` -### Python (Deserializer) +#### Python (Deserializer) ```python import pyfory @@ -137,7 +245,7 @@ with open("person.bin", "rb") as input_file: print(f"{person.name}, {person.age}") # Output: Bob, 25 ``` -## Handling Circular and Shared References +### Handling Circular and Shared References Xlang mode supports circular and shared references when reference tracking is enabled: @@ -166,7 +274,7 @@ byte[] bytes = fory.serialize(node1); // Python/Rust/Go can correctly deserialize this with circular references preserved ``` -## Type Mapping Considerations +### Type Mapping Considerations Not all Java types have equivalents in other languages. When using xlang mode: @@ -181,7 +289,7 @@ Not all Java types have equivalents in other languages. When using xlang mode: - See [Type Mapping Guide](../../specification/xlang_type_mapping.md) for the complete compatibility matrix. -### Lists and Dense Arrays +#### Lists and Dense Arrays Java primitive arrays are dense `array` carriers, except plain `byte[]`, which defaults to `bytes`. General Java collections and Fory primitive-list @@ -212,7 +320,7 @@ private @UInt32Type int[] ids; private @BFloat16Type short[] values; ``` -### Compatible Types +#### Compatible Types ```java public record UserData( @@ -223,7 +331,7 @@ public record UserData( ) {} ``` -### Problematic Types +#### Problematic Types ```java public record UserData( @@ -233,7 +341,7 @@ public record UserData( ) {} ``` -## Performance Considerations +### Performance Considerations Xlang mode has additional overhead compared to Java native mode: @@ -246,45 +354,45 @@ Xlang mode has additional overhead compared to Java native mode: - **Disable reference tracking** if you don't need circular references (`withRefTracking(false)`) - **Use native mode** (`withXlang(false)`) when only Java serialization is needed -## Best Practices +### Interoperability Best Practices 1. Use explicit type IDs or namespace/type names for every user type. 2. Keep compatible mode for independently deployed services. 3. Test payloads through every peer before relying on a schema in production. 4. Use native serialization for Java-only traffic that needs Java-specific object behavior. -## Troubleshooting +### Interoperability Troubleshooting -### "Type not registered" errors +#### "Type not registered" errors - Verify type is registered with same ID/name on both sides - Check if type name has typos or case differences -### "Type mismatch" errors +#### "Type mismatch" errors - Ensure field types are compatible across languages - Review [Type Mapping Guide](../../specification/xlang_type_mapping.md) -### Data corruption or unexpected values +#### Data corruption or unexpected values - Verify both sides use xlang payloads - Ensure both sides have compatible Fory versions -## See Also +### Specifications and References - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Reference](../../specification/xlang_type_mapping.md) -- [Python Xlang Serialization Guide](../python/xlang.md) -- [Rust Xlang Serialization Guide](../rust/xlang.md) +- [Python Interoperability Guide](../python/basic-serialization.md#cross-language-interoperability) +- [Rust Interoperability Guide](../rust/basic-serialization.md#cross-language-interoperability) -## Related Topics +### Related Guides - [Schema Evolution](schema-evolution.md) - Compatible mode - [Type Registration](type-registration.md) - Registration methods - [Native Serialization](native.md) - Java-only serialization features - [Row Format](../../row-format/java.md) - Cross-language row format -## Built-in values +### Built-in values ```java import org.apache.fory.*; @@ -310,7 +418,7 @@ public class Example1 { } ``` -## Custom values +### Custom values ```java import org.apache.fory.*; @@ -369,3 +477,10 @@ public class Example2 { } } ``` + +## Related Topics + +- [Configuration](configuration.md) - All ForyBuilder options +- [Native Serialization](native.md) - Java-only serialization features +- [Schema Metadata](schema-metadata.md) - Field IDs, nullability, reference tracking, and enum IDs +- [Troubleshooting](troubleshooting.md) - Common API usage issues diff --git a/docs/object-serialization/java/configuration.md b/docs/object-serialization/java/configuration.md index 0f1f44ae81..6463c69b74 100644 --- a/docs/object-serialization/java/configuration.md +++ b/docs/object-serialization/java/configuration.md @@ -86,43 +86,7 @@ For xlang payloads, call `withCompatible(false)` only after verifying that every ## Security -Keep class registration enabled for production and any untrusted payload source: - -```java -Fory fory = Fory.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .build(); -``` - -Security-related options: - -- `requireClassRegistration(true)` restricts deserialization to registered classes. -- `withMaxDepth(...)` rejects unexpectedly deep object graphs. -- `withMaxGraphMemoryBytes(...)` sets an approximate gate for materialized graph memory during one - root deserialization. The estimate mainly covers collections, maps, arrays, structs, and objects; - Fory core primitive arrays and primitive lists count their primitive storage from the decoded - length. It skips leaf values such as strings, primitive scalars, and dedicated binary values that - do not use a primitive-array serializer. Actual process memory can be higher than this limit. Leaf - values remain protected by byte-availability checks: if the unread input does not contain enough - bytes, Fory will not read or create that leaf value. The default is a fixed `128 MiB`; set a - positive byte limit when trusted workloads need a larger or smaller gate. -- `withMaxUnbackedContainerItems(...)` limits count-driven collection and map work whose repeated - read bodies do not consume proportional input. The default is `8192`; zero is a strict limit. -- `withMaxTypeFields(...)` and `withMaxTypeMetaBytes(...)` bound the field count - and encoded body size of one received remote metadata body. -- `withMaxSchemaVersionsPerType(...)` and - `withMaxAverageSchemaVersionsPerType(...)` bound accepted remote metadata versions without - changing registration, dynamic loading, or schema-evolution semantics. -- `withDeserializeUnknownClass(false)` avoids materializing unknown classes from metadata. -- `checkJdkClassSerializable(true)` keeps the JDK serializability check for `java.*` classes. -- Class registration warnings can be useful during security audits; use - `suppressClassRegistrationWarnings(false)` when you need to surface unexpected types. - -Use `requireClassRegistration(false)` only for trusted payloads, and pair it with a `TypeChecker` -allow list when dynamic class loading is required. +See [Java Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics diff --git a/docs/object-serialization/java/core-api.md b/docs/object-serialization/java/core-api.md deleted file mode 100644 index a8f10bb452..0000000000 --- a/docs/object-serialization/java/core-api.md +++ /dev/null @@ -1,132 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers the Java xlang quickstart. Xlang mode is the default Java wire format and is the -right first choice for cross-language payloads. - -## Create a Fory Instance - -For a single-threaded xlang Fory instance, set the mode explicitly: - -```java -import org.apache.fory.Fory; - -Fory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .build(); -``` - -For a thread-safe Fory instance, build `ThreadSafeFory` from the same builder: - -```java -import org.apache.fory.ThreadSafeFory; - -ThreadSafeFory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .buildThreadSafeFory(); -``` - -Default Java xlang mode also defaults to compatible schema mode, so independently deployed services -can add and remove fields when their schema metadata remains compatible. Use -`withCompatible(false)` only when every reader and writer always uses the same schema and you want -faster serialization and smaller size. Use the `compatible=false` opt-out only after verifying that every language uses the same xlang schema, or when native types are generated from Fory schema IDL. - -## Register Custom Types - -Register application classes with the same type identity on every peer. Numeric IDs are compact and -fast, while name registration is easier to coordinate across independently owned services. - -```java -import org.apache.fory.annotation.ForyField; - -public class User { - @ForyField(id = 0) - public String name; - - @ForyField(id = 1) - public int age; -} - -Fory fory = Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .build(); - -fory.register(User.class, "example", "User"); -``` - -Use field IDs for long-lived schemas so field identity is stable even if Java field names change. -See [Schema Metadata](schema-metadata.md) for Java annotations, nullability, reference tracking, and -enum metadata. - -## Serialize And Deserialize - -```java -User user = new User(); -user.name = "Alice"; -user.age = 30; - -byte[] bytes = fory.serialize(user); -User decoded = fory.deserialize(bytes, User.class); -``` - -When xlang bytes cross languages, every peer must register the same type identity and compatible -field metadata. The shared rules live in [Xlang](../xlang/index.md), while Java-specific API calls -are in [Xlang Serialization](xlang.md). - -## Use Native Serialization For Java-Only Traffic - -For same-language Java/JVM traffic, native mode is usually the better fit: - -```java -Fory fory = Fory.builder() - .withXlang(false) - .build(); -``` - -Native mode supports the broad Java object serialization surface, including JDK serialization hooks, -object copy, and native-mode zero-copy buffers. See [Native Serialization](native.md). - -## Common Options - -- `withRefTracking(true)` preserves shared references and circular references. -- `requireClassRegistration(true)` keeps the default registered-type policy. -- Compatible mode is enabled by default for native-mode and xlang payloads. Use - `withCompatible(false)` only when every reader and writer uses the same schema and you want faster - serialization and smaller size. For xlang payloads, use the `compatible=false` opt-out only after verifying that every language uses the same schema, or when native types are generated from Fory schema IDL. -- `withAsyncCompilation(true)` enables asynchronous serializer compilation where supported. - -## Best Practices - -1. **Reuse Fory instances**: Creating Fory is expensive, always reuse instances -2. **Use appropriate thread safety**: Choose between single-thread and thread-safe based on your needs -3. **Register classes**: Keep type identity stable across every xlang peer -4. **Configure reference tracking**: Enable it only when the object graph needs identity or cycles - -## Related Topics - -- [Configuration](configuration.md) - All ForyBuilder options -- [Native Serialization](native.md) - Java-only serialization features -- [Schema Metadata](schema-metadata.md) - Field IDs, nullability, reference tracking, and enum IDs -- [Xlang Serialization](xlang.md) - Java xlang interoperability -- [Troubleshooting](troubleshooting.md) - Common API usage issues diff --git a/docs/object-serialization/java/index.md b/docs/object-serialization/java/index.md index 88a69b87ef..2d5add91f8 100644 --- a/docs/object-serialization/java/index.md +++ b/docs/object-serialization/java/index.md @@ -22,7 +22,7 @@ license: | Apache Fory Java provides high-performance binary object serialization. Use xlang mode for payloads shared with other supported runtimes and native mode for Java/JVM-only object graphs. -This runtime guide is scoped to Binary Object Serialization. For other Java products, use +This runtime guide is scoped to Binary Object Serialization. For other Java capabilities, use [Row Format](../../row-format/java.md), [Fory JSON](../../json/index.md), [Fory IDL and compiler](../../compiler/index.md), or [Fory gRPC](../../grpc/java.md). @@ -169,7 +169,7 @@ Use xlang mode for cross-language payloads and schemas shared with non-Java impl Use native mode for Java-only traffic. Native mode is selected with `.withXlang(false)` and owns Java-specific object behavior such as JDK serialization hooks, `Externalizable`, dynamic object graphs, object copy, and Java native-mode zero-copy buffers. It is optimized for the JVM type system and supports a broader Java object surface than xlang mode. Compatible mode is enabled by default. Set `.withCompatible(false)` only when every reader and writer uses the same class schema and you want faster serialization and smaller size. If you are replacing JDK serialization, Kryo, FST, Hessian, or Java-only Protocol Buffers payloads, start with native mode. -See [Native Serialization](native.md) for Java-only serialization details and [Xlang Serialization](xlang.md) for Java xlang registration and interoperability rules. +See [Native Serialization](native.md) for Java-only serialization details and [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Java xlang registration and interoperability rules. ### Thread Safety @@ -257,10 +257,10 @@ model-discovery workflows differ from Fory Core. | Group | Pages | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Serialization modes | [Xlang](xlang.md), [Native](native.md) | -| Common | [Core API](core-api.md), [Configuration](configuration.md), [Type Registration](type-registration.md), [Schema Evolution](schema-evolution.md), [Schema Metadata](schema-metadata.md), [Custom Serializers](custom-serializers.md) | +| Serialization modes | [Basic Serialization](basic-serialization.md), [Native Serialization](native.md) | +| Common | [Configuration](configuration.md), [Type Registration](type-registration.md), [Schema Evolution](schema-evolution.md), [Schema Metadata](schema-metadata.md), [Custom Serializers](custom-serializers.md) | | Java-specific features | [Advanced Features](advanced-features.md), [Compression](compression.md), [Object Copy](object-copy.md), [JDK Custom Serialization](jdk-serialization.md), [Static Generated Serializers](static-generated-serializers.md), [Virtual Threads](virtual-threads.md) | | Platform and operate | [Android](android.md), [GraalVM Native Image](graalvm.md), [Troubleshooting](troubleshooting.md) | Before decoding externally supplied binary payloads, read -[Object Serialization Security](../security.md). +[Java Security](security.md). diff --git a/docs/object-serialization/java/native.md b/docs/object-serialization/java/native.md index 6832213e73..6737b2f53d 100644 --- a/docs/object-serialization/java/native.md +++ b/docs/object-serialization/java/native.md @@ -28,7 +28,7 @@ payloads. Native serialization in this page means Fory's `xlang=false` wire mode. It is separate from GraalVM native image support, which is covered in [GraalVM Native Image](graalvm.md). -Use [Xlang Serialization](xlang.md), the default Java mode, when bytes must be read by +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Java mode, when bytes must be read by non-Java Fory implementations. ## When To Use Native Serialization @@ -326,8 +326,8 @@ Use `JavaSerializer.serializedByJDK(...)` only at the mixed-format boundary, the ## Related Topics -- [Basic Serialization](core-api.md) - Xlang-first Java quickstart -- [Xlang Serialization](xlang.md) - Cross-language Java payloads +- [Basic Serialization](basic-serialization.md) - Xlang-first Java quickstart +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Java payloads - [Configuration](configuration.md) - Java builder options - [Schema Evolution](schema-evolution.md) - compatible mode and same-schema optimization - [Type Registration](type-registration.md) - Registration and security diff --git a/docs/object-serialization/java/object-copy.md b/docs/object-serialization/java/object-copy.md index 002b79b334..b0c02b23ec 100644 --- a/docs/object-serialization/java/object-copy.md +++ b/docs/object-serialization/java/object-copy.md @@ -357,7 +357,7 @@ Fix it by either: ## Related Topics -- [Basic Serialization](core-api.md) - Fory instance creation and core APIs +- [Basic Serialization](basic-serialization.md) - Fory instance creation and core APIs - [Configuration](configuration.md) - Builder options including `withRefCopy` - [Custom Serializers](custom-serializers.md) - Serializer design and registration - [Virtual Threads](virtual-threads.md) - Thread-safe Fory guidance diff --git a/docs/object-serialization/java/schema-evolution.md b/docs/object-serialization/java/schema-evolution.md index 1e71d7e97e..9a2796306e 100644 --- a/docs/object-serialization/java/schema-evolution.md +++ b/docs/object-serialization/java/schema-evolution.md @@ -266,5 +266,5 @@ public class SameSchemaMessage { ## Related Topics - [Configuration](configuration.md) - All ForyBuilder options -- [Xlang Serialization](xlang.md) - xlang mode +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode - [Troubleshooting](troubleshooting.md) - Common schema issues diff --git a/docs/object-serialization/java/schema-metadata.md b/docs/object-serialization/java/schema-metadata.md index 0e631edf9b..981e47184a 100644 --- a/docs/object-serialization/java/schema-metadata.md +++ b/docs/object-serialization/java/schema-metadata.md @@ -710,7 +710,7 @@ public class User { ## Related Topics -- [Basic Serialization](core-api.md) - Getting started with Fory serialization +- [Basic Serialization](basic-serialization.md) - Getting started with Fory serialization - [Configuration](configuration.md) - `ForyBuilder` options - [Schema Evolution](schema-evolution.md) - Compatible mode and schema evolution -- [Xlang Serialization](xlang.md) - Interoperability with Python, Rust, C++, Go +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Interoperability with Python, Rust, C++, Go diff --git a/docs/object-serialization/java/security.md b/docs/object-serialization/java/security.md new file mode 100644 index 0000000000..780d39b2d2 --- /dev/null +++ b/docs/object-serialization/java/security.md @@ -0,0 +1,83 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Java reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Keep class registration enabled for production and any untrusted payload source: + +```java +Fory fory = Fory.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .build(); +``` + +Security-related options: + +- `requireClassRegistration(true)` restricts deserialization to registered classes. +- `withMaxDepth(...)` rejects unexpectedly deep object graphs. +- `withMaxGraphMemoryBytes(...)` sets an approximate gate for materialized graph memory during one + root deserialization. The estimate mainly covers collections, maps, arrays, structs, and objects; + Fory core primitive arrays and primitive lists count their primitive storage from the decoded + length. It skips leaf values such as strings, primitive scalars, and dedicated binary values that + do not use a primitive-array serializer. Actual process memory can be higher than this limit. Leaf + values remain protected by byte-availability checks: if the unread input does not contain enough + bytes, Fory will not read or create that leaf value. The default is a fixed `128 MiB`; set a + positive byte limit when trusted workloads need a larger or smaller gate. +- `withMaxUnbackedContainerItems(...)` limits count-driven collection and map work whose repeated + read bodies do not consume proportional input. The default is `8192`; zero is a strict limit. +- `withMaxTypeFields(...)` and `withMaxTypeMetaBytes(...)` bound the field count + and encoded body size of one received remote metadata body. +- `withMaxSchemaVersionsPerType(...)` and + `withMaxAverageSchemaVersionsPerType(...)` bound accepted remote metadata versions without + changing registration, dynamic loading, or schema-evolution semantics. +- `withDeserializeUnknownClass(false)` avoids materializing unknown classes from metadata. +- `checkJdkClassSerializable(true)` keeps the JDK serializability check for `java.*` classes. +- Class registration warnings can be useful during security audits; use + `suppressClassRegistrationWarnings(false)` when you need to surface unexpected types. + +Use `requireClassRegistration(false)` only for trusted payloads, and pair it with a `TypeChecker` +allow list when dynamic class loading is required. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/javascript/xlang.md b/docs/object-serialization/javascript/basic-serialization.md similarity index 54% rename from docs/object-serialization/javascript/xlang.md rename to docs/object-serialization/javascript/basic-serialization.md index 24452ee559..ce14adb1ce 100644 --- a/docs/object-serialization/javascript/xlang.md +++ b/docs/object-serialization/javascript/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,6 +19,228 @@ license: | limitations under the License. --- +This guide covers the core serialization APIs in the default xlang mode for Apache Fory JavaScript. + +## Create a `Fory` Instance + +```ts +import Fory from "@apache-fory/core"; + +const fory = new Fory(); +``` + +Create one instance, register your schemas, and reuse it. Fory caches the generated serializers after the first `register` call, so recreating it on every request wastes that work. + +## Define a Schema with `Type.struct` + +The most common path is to define a schema and register it. + +```ts +import Fory, { Type } from "@apache-fory/core"; + +const accountType = Type.struct( + { typeName: "example.account" }, + { + id: Type.int64(), + owner: Type.string(), + active: Type.bool(), + nickname: Type.string().setNullable(true), + }, +); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(accountType); +``` + +## Serialize and Deserialize + +```ts +const bytes = serialize({ + id: 42n, + owner: "Alice", + active: true, + nickname: null, +}); + +const value = deserialize(bytes); +console.log(value); +// { id: 42n, owner: 'Alice', active: true, nickname: null } +``` + +The returned `bytes` value is a `Uint8Array`/platform buffer and can be sent over the network or written to storage. + +## Root-Level Dynamic Serialization + +`Fory` can also serialize dynamic root values without first binding a schema-specific serializer. + +```ts +const fory = new Fory(); + +const bytes = fory.serialize( + new Map([ + ["name", "Alice"], + ["age", 30], + ]), +); + +const value = fory.deserialize(bytes); +``` + +This is convenient for dynamic payloads, but explicit schemas are usually better for stable interfaces and cross-language contracts. + +## Primitive Values + +```ts +const fory = new Fory(); + +fory.deserialize(fory.serialize(true)); +// true + +fory.deserialize(fory.serialize("hello")); +// 'hello' + +fory.deserialize(fory.serialize(123)); +// 123 + +fory.deserialize(fory.serialize(123n)); +// 123n + +fory.deserialize(fory.serialize(new Date("2021-10-20T09:13:00Z"))); +// Date +``` + +### Number and `bigint` + +JavaScript `number` is a 64-bit float, which cannot exactly represent all 64-bit integers. For cross-language contracts or anywhere exact integer sizes matter, use explicit field types in your schema: + +- `Type.int32()` — 32-bit integer; use JavaScript `number` +- `Type.int64()` — 64-bit integer; use JavaScript `bigint` +- `Type.float32()` / `Type.float64()` — floating-point + +Dynamic root serialization (calling `fory.serialize(someNumber)` without a schema) will infer a type, but the inferred type is not guaranteed by the API. Use a schema for any stable contract. + +## Arrays, Maps, and Sets + +```ts +const inventoryType = Type.struct("example.inventory", { + tags: Type.list(Type.string()), + counts: Type.map(Type.string(), Type.int32()), + labels: Type.set(Type.string()), +}); + +const fory = new Fory({ ref: true }); +const { serialize, deserialize } = fory.register(inventoryType); + +const bytes = serialize({ + tags: ["hot", "new"], + counts: new Map([ + ["apple", 3], + ["pear", 8], + ]), + labels: new Set(["featured", "seasonal"]), +}); + +const value = deserialize(bytes); +``` + +## Nested Structs + +```ts +const addressType = Type.struct("example.address", { + city: Type.string(), + country: Type.string(), +}); + +const userType = Type.struct("example.user", { + name: Type.string(), + address: Type.struct("example.address", { + city: Type.string(), + country: Type.string(), + }), +}); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(userType); + +const bytes = serialize({ + name: "Alice", + address: { city: "Hangzhou", country: "CN" }, +}); + +const user = deserialize(bytes); +``` + +If a nested value can be missing, mark it nullable: + +```ts +const wrapperType = Type.struct("example.wrapper", { + child: Type.struct("example.child", { + name: Type.string(), + }).setNullable(true), +}); +``` + +## Decorator-Based Registration + +TypeScript decorators are also supported. + +```ts +import Fory, { Type } from "@apache-fory/core"; + +@Type.struct("example.user") +class User { + @Type.int64() + id!: bigint; + + @Type.string() + name!: string; +} + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(User); + +const user = new User(); +user.id = 1n; +user.name = "Alice"; + +const copy = deserialize(serialize(user)); +console.log(copy instanceof User); // true +``` + +## Nullability + +Field nullability is explicit in schema-based structs. + +```ts +const nullableType = Type.struct("example.optional_user", { + name: Type.string(), + email: Type.string().setNullable(true), +}); +``` + +If a field is not marked nullable and you try to write `null`, serialization throws. + +## Debugging Generated Code + +You can inspect generated serializer code with `hooks.afterCodeGenerated`. + +```ts +const fory = new Fory({ + hooks: { + afterCodeGenerated(code) { + console.log(code); + return code; + }, + }, +}); +``` + +This is useful when debugging schema behavior, field ordering, or generated fast paths. + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Fory JavaScript serializes to the same binary format as the Java, Python, C++, Go, Rust, C#, Swift, Dart, Scala, and Kotlin Fory implementations. You can write a message in JavaScript and read it in Java, or any other direction, without a @@ -29,7 +251,7 @@ Things to keep in mind: - Fory JavaScript reads and writes cross-language payloads only; it does not support any native-mode format. - JavaScript does not support out-of-band mode. -## Requirements for a Successful Round Trip +### Requirements for a Successful Round Trip For a message to survive a round trip between JavaScript and another language: @@ -39,7 +261,7 @@ For a message to survive a round trip between JavaScript and another language: 4. Compatible schema evolution on both sides. JavaScript enables it by default. 5. **Same reference tracking config** if your data has shared or circular references. -## Step-by-Step: JavaScript to Another Peer +### Step-by-Step: JavaScript to Another Peer 1. Define the JavaScript schema with the same type name or numeric ID used by the peer. 2. Register the schema in both peers. @@ -75,14 +297,14 @@ On the other side, register the same `example.message` type (same name or same n - [Go guide](../go/index.md) - [Rust guide](../rust/index.md) -## Field Naming +### Field Naming Fory matches fields by name. When models are defined in multiple languages, keep field names consistent — or at minimum use a naming scheme that maps unambiguously across languages (e.g. `snake_case` everywhere). With the default compatible schema evolution, field order differences are tolerated, but the names themselves must still match. -## Numeric Types +### Numeric Types JavaScript `number` is a 64-bit float, which does not map cleanly to every integer type in other languages. Use explicit schema types: @@ -90,7 +312,7 @@ JavaScript `number` is a 64-bit float, which does not map cleanly to every integ - `Type.int64()` with `bigint` values for 64-bit integers (Java `long`, Go `int64`) - `Type.float32()` or `Type.float64()` for floating-point values -## Lists and Dense Arrays +### Lists and Dense Arrays Use `Type.list(T)` for ordinary JavaScript `Array` values and Fory `list` schema. Dense bool/numeric vectors use the explicit array builders @@ -113,13 +335,13 @@ listed below. | `array` | `Type.float32Array()` | | `array` | `Type.float64Array()` | -## Date and Time +### Date and Time - `Type.timestamp()` — a point in time; round-trips as a JavaScript `Date` - `Type.date()` — a date without time; deserializes as `Date` - `Type.duration()` — exposed as a numeric millisecond value in JavaScript -## Polymorphic Fields +### Polymorphic Fields `Type.any()` lets a field hold different concrete types, but it is harder to keep in sync across languages. Prefer explicit field schemas whenever possible. @@ -132,7 +354,7 @@ const wrapperType = Type.struct( ); ``` -## Enums +### Enums Enum member **order** must match across languages. Fory encodes enums by ordinal position, not by value. @@ -144,17 +366,17 @@ fory.register(Type.enum({ typeId: 210 }, Color)); Use the same type ID or type name in every peer. -## Safety Limits +### Safety Limits The `maxDepth` option bounds nested payloads. It does not change the binary format; it only controls what the local `Fory` instance accepts. -## Related Topics +### Related Guides - [Supported Types](supported-types.md) - [Schema Evolution](schema-evolution.md) - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) -## Built-in values +### Built-in values ```javascript import Fory from "@apache-fory/core"; @@ -165,7 +387,7 @@ const result = fory.deserialize(input); console.log(result); ``` -## Custom values +### Custom values ```javascript import Fory, { Type } from "@apache-fory/core"; @@ -184,7 +406,7 @@ const result = deserialize(input); console.log(result); ``` -## Shared and circular references +### Shared and circular references ```javascript import Fory, { Type } from "@apache-fory/core"; @@ -204,3 +426,9 @@ const input = serialize(data); const result = deserialize(input); console.log(result.bar.foo === result.foo); ``` + +## Related Topics + +- [Type Registration](type-registration.md) +- [Supported Types](supported-types.md) +- [References](references.md) diff --git a/docs/object-serialization/javascript/configuration.md b/docs/object-serialization/javascript/configuration.md index f7d76d30e1..c73194882e 100644 --- a/docs/object-serialization/javascript/configuration.md +++ b/docs/object-serialization/javascript/configuration.md @@ -140,24 +140,11 @@ Leave this unset unless you run on Node.js 20+ and have benchmarked your workloa ## Security -Security-related configuration: - -- Register only the expected schemas before deserializing untrusted payloads. -- Set `maxDepth` for the maximum nesting depth your service accepts. -- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep `maxTypeFields` and `maxTypeMetaBytes` at their defaults unless the data - is not malicious and a trusted peer sends larger remote metadata. -- Keep `maxSchemaVersionsPerType` and - `maxAverageSchemaVersionsPerType` at their defaults unless the data is not - malicious and a trusted peer sends many remote schema versions. -- Prefer explicit `Type.struct(...)` schemas over `Type.any()` for untrusted input. -- Pass `hps` only from the official package version you deploy with Fory. +See [JavaScript/TypeScript Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Metadata](schema-metadata.md) - [Schema Evolution](schema-evolution.md) - [References](references.md) diff --git a/docs/object-serialization/javascript/core-api.md b/docs/object-serialization/javascript/core-api.md deleted file mode 100644 index 6b4650e8f2..0000000000 --- a/docs/object-serialization/javascript/core-api.md +++ /dev/null @@ -1,244 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This guide covers the core serialization APIs in Apache Fory JavaScript. - -## Create a `Fory` Instance - -```ts -import Fory from "@apache-fory/core"; - -const fory = new Fory(); -``` - -Create one instance, register your schemas, and reuse it. Fory caches the generated serializers after the first `register` call, so recreating it on every request wastes that work. - -## Define a Schema with `Type.struct` - -The most common path is to define a schema and register it. - -```ts -import Fory, { Type } from "@apache-fory/core"; - -const accountType = Type.struct( - { typeName: "example.account" }, - { - id: Type.int64(), - owner: Type.string(), - active: Type.bool(), - nickname: Type.string().setNullable(true), - }, -); - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(accountType); -``` - -## Serialize and Deserialize - -```ts -const bytes = serialize({ - id: 42n, - owner: "Alice", - active: true, - nickname: null, -}); - -const value = deserialize(bytes); -console.log(value); -// { id: 42n, owner: 'Alice', active: true, nickname: null } -``` - -The returned `bytes` value is a `Uint8Array`/platform buffer and can be sent over the network or written to storage. - -## Root-Level Dynamic Serialization - -`Fory` can also serialize dynamic root values without first binding a schema-specific serializer. - -```ts -const fory = new Fory(); - -const bytes = fory.serialize( - new Map([ - ["name", "Alice"], - ["age", 30], - ]), -); - -const value = fory.deserialize(bytes); -``` - -This is convenient for dynamic payloads, but explicit schemas are usually better for stable interfaces and cross-language contracts. - -## Primitive Values - -```ts -const fory = new Fory(); - -fory.deserialize(fory.serialize(true)); -// true - -fory.deserialize(fory.serialize("hello")); -// 'hello' - -fory.deserialize(fory.serialize(123)); -// 123 - -fory.deserialize(fory.serialize(123n)); -// 123n - -fory.deserialize(fory.serialize(new Date("2021-10-20T09:13:00Z"))); -// Date -``` - -### Number and `bigint` - -JavaScript `number` is a 64-bit float, which cannot exactly represent all 64-bit integers. For cross-language contracts or anywhere exact integer sizes matter, use explicit field types in your schema: - -- `Type.int32()` — 32-bit integer; use JavaScript `number` -- `Type.int64()` — 64-bit integer; use JavaScript `bigint` -- `Type.float32()` / `Type.float64()` — floating-point - -Dynamic root serialization (calling `fory.serialize(someNumber)` without a schema) will infer a type, but the inferred type is not guaranteed by the API. Use a schema for any stable contract. - -## Arrays, Maps, and Sets - -```ts -const inventoryType = Type.struct("example.inventory", { - tags: Type.list(Type.string()), - counts: Type.map(Type.string(), Type.int32()), - labels: Type.set(Type.string()), -}); - -const fory = new Fory({ ref: true }); -const { serialize, deserialize } = fory.register(inventoryType); - -const bytes = serialize({ - tags: ["hot", "new"], - counts: new Map([ - ["apple", 3], - ["pear", 8], - ]), - labels: new Set(["featured", "seasonal"]), -}); - -const value = deserialize(bytes); -``` - -## Nested Structs - -```ts -const addressType = Type.struct("example.address", { - city: Type.string(), - country: Type.string(), -}); - -const userType = Type.struct("example.user", { - name: Type.string(), - address: Type.struct("example.address", { - city: Type.string(), - country: Type.string(), - }), -}); - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(userType); - -const bytes = serialize({ - name: "Alice", - address: { city: "Hangzhou", country: "CN" }, -}); - -const user = deserialize(bytes); -``` - -If a nested value can be missing, mark it nullable: - -```ts -const wrapperType = Type.struct("example.wrapper", { - child: Type.struct("example.child", { - name: Type.string(), - }).setNullable(true), -}); -``` - -## Decorator-Based Registration - -TypeScript decorators are also supported. - -```ts -import Fory, { Type } from "@apache-fory/core"; - -@Type.struct("example.user") -class User { - @Type.int64() - id!: bigint; - - @Type.string() - name!: string; -} - -const fory = new Fory(); -const { serialize, deserialize } = fory.register(User); - -const user = new User(); -user.id = 1n; -user.name = "Alice"; - -const copy = deserialize(serialize(user)); -console.log(copy instanceof User); // true -``` - -## Nullability - -Field nullability is explicit in schema-based structs. - -```ts -const nullableType = Type.struct("example.optional_user", { - name: Type.string(), - email: Type.string().setNullable(true), -}); -``` - -If a field is not marked nullable and you try to write `null`, serialization throws. - -## Debugging Generated Code - -You can inspect generated serializer code with `hooks.afterCodeGenerated`. - -```ts -const fory = new Fory({ - hooks: { - afterCodeGenerated(code) { - console.log(code); - return code; - }, - }, -}); -``` - -This is useful when debugging schema behavior, field ordering, or generated fast paths. - -## Related Topics - -- [Type Registration](type-registration.md) -- [Supported Types](supported-types.md) -- [References](references.md) diff --git a/docs/object-serialization/javascript/index.md b/docs/object-serialization/javascript/index.md index 994a350ccb..31a158002c 100644 --- a/docs/object-serialization/javascript/index.md +++ b/docs/object-serialization/javascript/index.md @@ -156,21 +156,23 @@ options; see [Configuration](configuration.md). ## Documentation -| Topic | Description | -| -------------------------------------------- | ------------------------------------------------------- | -| [Basic Serialization](core-api.md) | Core APIs and everyday usage | -| [Configuration](configuration.md) | Fory options, compatible mode, limits, and HPS | -| [Type Registration](type-registration.md) | Numeric IDs, names, decorators, and schema registration | -| [Schema Metadata](schema-metadata.md) | Type builders, field options, and decorators | -| [Supported Types](supported-types.md) | Primitive, collection, time, enum, and struct mappings | -| [References](references.md) | Shared references and circular object graphs | -| [Schema Evolution](schema-evolution.md) | Compatible mode and evolving structs | -| [Xlang Serialization](xlang.md) | Interop guidance and mapping rules | -| [Fory IDL Compiler](../../compiler/index.md) | Generate TypeScript models from `.fdl` schemas | -| [gRPC Support](../../grpc/javascript.md) | Node.js gRPC and browser gRPC-Web generated clients | -| [Troubleshooting](troubleshooting.md) | Common issues, limits, and debugging tips | +| Topic | Description | +| --------------------------------------------- | ------------------------------------------------------- | +| [Basic Serialization](basic-serialization.md) | Default xlang APIs, usage, and interoperability | +| [Configuration](configuration.md) | Fory options, compatible mode, limits, and HPS | +| [Type Registration](type-registration.md) | Numeric IDs, names, decorators, and schema registration | +| [Schema Metadata](schema-metadata.md) | Type builders, field options, and decorators | +| [Supported Types](supported-types.md) | Primitive, collection, time, enum, and struct mappings | +| [References](references.md) | Shared references and circular object graphs | +| [Schema Evolution](schema-evolution.md) | Compatible mode and evolving structs | +| [Fory IDL Compiler](../../compiler/index.md) | Generate TypeScript models from `.fdl` schemas | +| [gRPC Support](../../grpc/javascript.md) | Node.js gRPC and browser gRPC-Web generated clients | +| [Troubleshooting](troubleshooting.md) | Common issues, limits, and debugging tips | ## Related Resources - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Xlang Type Mapping](../../specification/xlang_type_mapping.md) + +Before decoding bytes from outside the application trust boundary, read +[JavaScript/TypeScript Security](security.md). diff --git a/docs/object-serialization/javascript/references.md b/docs/object-serialization/javascript/references.md index 38b496843f..5f0f118bdb 100644 --- a/docs/object-serialization/javascript/references.md +++ b/docs/object-serialization/javascript/references.md @@ -106,6 +106,6 @@ Reference tracking is part of the Fory binary protocol and works across language ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/schema-evolution.md b/docs/object-serialization/javascript/schema-evolution.md index 22d77da5d4..f91d96ef5e 100644 --- a/docs/object-serialization/javascript/schema-evolution.md +++ b/docs/object-serialization/javascript/schema-evolution.md @@ -108,9 +108,9 @@ reads expecting compatible metadata, deserialization will fail. ## Xlang Requirement -Compatible mode only protects you from schema differences in the _fields_ of a type. You still need the same type identity (same numeric ID or same `typeName`) on every side. See [Xlang Serialization](xlang.md). +Compatible mode only protects you from schema differences in the _fields_ of a type. You still need the same type identity (same numeric ID or same `typeName`) on every side. See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). ## Related Topics - [Type Registration](type-registration.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/security.md b/docs/object-serialization/javascript/security.md new file mode 100644 index 0000000000..6a01ac5b73 --- /dev/null +++ b/docs/object-serialization/javascript/security.md @@ -0,0 +1,60 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a JavaScript/TypeScript reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected schemas before deserializing untrusted payloads. +- Set `maxDepth` for the maximum nesting depth your service accepts. +- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep `maxTypeFields` and `maxTypeMetaBytes` at their defaults unless the data + is not malicious and a trusted peer sends larger remote metadata. +- Keep `maxSchemaVersionsPerType` and + `maxAverageSchemaVersionsPerType` at their defaults unless the data is not + malicious and a trusted peer sends many remote schema versions. +- Prefer explicit `Type.struct(...)` schemas over `Type.any()` for untrusted input. +- Pass `hps` only from the official package version you deploy with Fory. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/javascript/supported-types.md b/docs/object-serialization/javascript/supported-types.md index 37cfda8b4e..57ca8eb011 100644 --- a/docs/object-serialization/javascript/supported-types.md +++ b/docs/object-serialization/javascript/supported-types.md @@ -172,6 +172,6 @@ For types that need completely custom encoding, use `Type.ext(...)` and pass a c ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [References](references.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/troubleshooting.md b/docs/object-serialization/javascript/troubleshooting.md index c1aa890a1b..b215dd37b7 100644 --- a/docs/object-serialization/javascript/troubleshooting.md +++ b/docs/object-serialization/javascript/troubleshooting.md @@ -85,6 +85,6 @@ const fory = new Fory({ ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [References](references.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/javascript/type-registration.md b/docs/object-serialization/javascript/type-registration.md index b109c2f822..a0c471c075 100644 --- a/docs/object-serialization/javascript/type-registration.md +++ b/docs/object-serialization/javascript/type-registration.md @@ -154,11 +154,11 @@ Use **names** when: ## Xlang -For a message to round-trip between JavaScript and another language, both sides must use the same identity for a given type: same numeric ID, or same `typeName`. Use `.` inside `typeName` to add a namespace prefix. See [Xlang Serialization](xlang.md). +For a message to round-trip between JavaScript and another language, both sides must use the same identity for a given type: same numeric ID, or same `typeName`. Use `.` inside `typeName` to add a namespace prefix. See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). ## Related Topics -- [Basic Serialization](core-api.md) +- [Basic Serialization](basic-serialization.md) - [Schema Metadata](schema-metadata.md) - [Schema Evolution](schema-evolution.md) -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) diff --git a/docs/object-serialization/kotlin/xlang.md b/docs/object-serialization/kotlin/basic-serialization.md similarity index 82% rename from docs/object-serialization/kotlin/xlang.md rename to docs/object-serialization/kotlin/basic-serialization.md index 1f5b86e3bf..0fdbfb3c04 100644 --- a/docs/object-serialization/kotlin/xlang.md +++ b/docs/object-serialization/kotlin/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Kotlin Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,6 +19,12 @@ license: | limitations under the License. --- +Xlang is the default serialization mode for Fory Kotlin. This page covers the basic serialization API and interoperability rules for that default mode. + +## Cross-Language Interoperability + +The following sections cover model generation, registration, and cross-language round trips in the default xlang mode. + Kotlin xlang serialization uses the JVM Fory implementation through `ForyKotlin`. Use it when Kotlin payloads must be read by another supported Fory runtime. Register portable model types with the same identity and field schema on every peer. @@ -27,7 +33,7 @@ Kotlin data classes, enums, and sealed-class models use the Kotlin integration a serializers where applicable. Exact portable carrier mappings remain defined by the [xlang type mapping](../../specification/xlang_type_mapping.md). -## Create an xlang instance +### Create a Fory Instance ```kotlin import org.apache.fory.kotlin.ForyKotlin @@ -37,7 +43,7 @@ val fory = ForyKotlin.builder() .build() ``` -## First round trip +### First round trip ```kotlin import org.apache.fory.ThreadSafeFory diff --git a/docs/object-serialization/kotlin/configuration.md b/docs/object-serialization/kotlin/configuration.md index 175b2a4d4e..f58b2d83a9 100644 --- a/docs/object-serialization/kotlin/configuration.md +++ b/docs/object-serialization/kotlin/configuration.md @@ -148,31 +148,4 @@ For xlang payloads, call `withCompatible(false)` only after verifying that every ## Security -Kotlin uses the Java configuration surface. Keep class registration enabled for production -and any untrusted payload source: - -```kotlin -val fory = ForyKotlin.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .withMaxTypeFields(512) - .withMaxTypeMetaBytes(4096) - .build() -``` - -Security-related configuration: - -- Keep `requireClassRegistration(true)` and register application classes or generated modules. -- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. -- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a - larger root allowance. Zero rejects every unbacked item. -- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits - at their defaults unless the data is not malicious and a trusted peer sends larger metadata or - many schema versions. -- Follow [Java Configuration](../java/configuration.md#security) for allow-listing and unknown-class - controls. +See [Kotlin Security](security.md) for trust boundaries, safe reader configuration, and verification. diff --git a/docs/object-serialization/kotlin/index.md b/docs/object-serialization/kotlin/index.md index 76abeddabc..5b5910fff2 100644 --- a/docs/object-serialization/kotlin/index.md +++ b/docs/object-serialization/kotlin/index.md @@ -117,7 +117,7 @@ See [Configuration](configuration.md) for Kotlin builder setup and [Java Native Fory Kotlin is built on top of Fory Java. Most configuration options, features, and concepts from Fory Java apply directly to Kotlin. Refer to the Java documentation for: - [Configuration](../java/configuration.md) - All ForyBuilder options -- [Basic Serialization](../java/core-api.md) - Serialization patterns and APIs +- [Basic Serialization](../java/basic-serialization.md) - Serialization patterns and APIs - [Type Registration](../java/type-registration.md) - Class registration and security - [Schema Evolution](../java/schema-evolution.md) - Forward/backward compatibility - [Custom Serializers](../java/custom-serializers.md) - Implement custom serializers @@ -126,7 +126,7 @@ Fory Kotlin is built on top of Fory Java. Most configuration options, features, ## Kotlin-Specific Documentation -- [Xlang Serialization](xlang.md) - Kotlin APIs for direct cross-language object serialization +- [Basic Serialization](basic-serialization.md) - Models, APIs, and interoperability in the default xlang mode - [Configuration](configuration.md) - Kotlin-specific Fory setup requirements - [Native Serialization](native.md) - Serializing Kotlin types in JVM-native mode - [Schema Metadata](schema-metadata.md) - Kotlin annotations, nullability, references, and integer metadata @@ -134,3 +134,6 @@ Fory Kotlin is built on top of Fory Java. Most configuration options, features, - [Static Generated Serializers](static-generated-serializers.md) - KSP xlang/schema serializer generation - [Kotlin gRPC Support](../../grpc/kotlin.md) - Coroutine stubs and service bases for Fory IDL services - [Android Support](../java/android.md) - Android setup, R8 behavior, and release-build validation + +Before decoding bytes from outside the application trust boundary, read +[Kotlin Security](security.md). diff --git a/docs/object-serialization/kotlin/native.md b/docs/object-serialization/kotlin/native.md index 5b16278525..99a88b9670 100644 --- a/docs/object-serialization/kotlin/native.md +++ b/docs/object-serialization/kotlin/native.md @@ -20,7 +20,7 @@ license: | --- This page covers serialization of Kotlin-specific JVM types in native mode. For -cross-language Kotlin models, use [Kotlin Xlang Serialization](xlang.md). +cross-language Kotlin models, use [Kotlin Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). When compatible mode is enabled, Kotlin readers use the JVM compatible-read rules for selected scalar field type changes. A matched field can read between `Boolean`, `String`, numeric scalars, diff --git a/docs/object-serialization/kotlin/security.md b/docs/object-serialization/kotlin/security.md new file mode 100644 index 0000000000..56bbafd556 --- /dev/null +++ b/docs/object-serialization/kotlin/security.md @@ -0,0 +1,74 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Kotlin reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Kotlin uses the Java configuration surface. Keep class registration enabled for production +and any untrusted payload source: + +```kotlin +val fory = ForyKotlin.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .withMaxTypeFields(512) + .withMaxTypeMetaBytes(4096) + .build() +``` + +Security-related configuration: + +- Keep `requireClassRegistration(true)` and register application classes or generated modules. +- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. +- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a + larger root allowance. Zero rejects every unbacked item. +- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits + at their defaults unless the data is not malicious and a trusted peer sends larger metadata or + many schema versions. +- Follow [Java Security](../java/security.md) for allow-listing and unknown-class + controls. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the Kotlin option reference and +[Java Type Registration](../java/type-registration.md) for the underlying registration API. diff --git a/docs/object-serialization/native.md b/docs/object-serialization/native.md index e85a2a05d4..d5bf9a87dc 100644 --- a/docs/object-serialization/native.md +++ b/docs/object-serialization/native.md @@ -1,6 +1,6 @@ --- title: Native Serialization -sidebar_position: 2 +sidebar_position: 3 id: native license: | Licensed to the Apache Software Foundation (ASF) under one or more @@ -26,7 +26,7 @@ system. It is not one shared cross-language protocol. Use native mode for same-runtime traffic that needs language-specific object shapes, migration from a host serializer, or a smaller/faster format without xlang type-mapping constraints. Use -[xlang mode](xlang/index.md) whenever a different runtime must read the bytes. +[xlang mode](xlang.md) whenever a different runtime must read the bytes. ## Supported runtime families diff --git a/docs/object-serialization/python/xlang.md b/docs/object-serialization/python/basic-serialization.md similarity index 76% rename from docs/object-serialization/python/xlang.md rename to docs/object-serialization/python/basic-serialization.md index 3f9b53d19c..489bc77929 100644 --- a/docs/object-serialization/python/xlang.md +++ b/docs/object-serialization/python/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,12 +19,101 @@ license: | limitations under the License. --- +This page covers the Python xlang quickstart. `pyfory.Fory()` defaults to xlang mode with +compatible schema evolution; examples set `xlang=True` explicitly so the mode choice is visible. + +## Basic Object Serialization + +Serialize and deserialize Python objects with a simple API: + +```python +import pyfory + +fory = pyfory.Fory(xlang=True) + +# Serialize xlang-compatible values +data = fory.dumps({"name": "Alice", "age": 30, "scores": [95, 87, 92]}) + +# Deserialize back to Python object +obj = fory.loads(data) +print(obj) # {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]} +``` + +**Note**: `dumps()`/`loads()` are aliases for `serialize()`/`deserialize()`. Both APIs are identical, use whichever feels more intuitive. + +## Custom Class Serialization + +Use dataclasses and type annotations for stable xlang payloads: + +```python +import pyfory +from dataclasses import dataclass +from typing import List, Dict + +@dataclass +class Person: + name: str + age: pyfory.Int32 + scores: List[pyfory.Int32] + metadata: Dict[str, str] + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register(Person, name="example.Person") +person = Person("Bob", 25, [88, 92, 85], {"team": "engineering"}) +data = fory.serialize(person) +result = fory.deserialize(data) +print(result) # Person(name='Bob', age=25, ...) +``` + +## Reference Tracking & Circular References + +Handle repeated references safely when the payload uses xlang-compatible types: + +```python +import pyfory + +f = pyfory.Fory(xlang=True, ref=True) + +shared = ["shared"] +value = [shared, shared] + +data = f.serialize(value) +result = f.deserialize(data) +assert result[0] is result[1] +``` + +For arbitrary Python object graphs, local classes, functions, and methods, use +[Native Serialization](native.md). + +## Performance Tips + +1. **Disable `ref=True` if not needed**: Reference tracking has overhead +2. **Use type_id instead of name**: Integer IDs are faster than string names +3. **Reuse Fory instances**: Create once, use many times +4. **Enable Cython**: Make sure `ENABLE_FORY_CYTHON_SERIALIZATION=1` + +```python +# Good: Reuse instance +fory = pyfory.Fory(xlang=True) +for obj in objects: + data = fory.dumps(obj) + +# Bad: Create new instance each time +for obj in objects: + fory = pyfory.Fory(xlang=True) # Wasteful! + data = fory.dumps(obj) +``` + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + `pyfory` supports xlang object graph serialization, allowing you to serialize data in Python and deserialize it in Java, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another supported language. -## Create an Xlang Fory Instance +### Xlang Configuration Python defaults to xlang mode with compatible schema evolution. Set the mode explicitly in xlang examples: @@ -33,9 +122,9 @@ import pyfory fory = pyfory.Fory(xlang=True, ref=False, strict=True) ``` -## Xlang Example +### Xlang Example -### Python (Serializer) +#### Python (Serializer) ```python import pyfory @@ -56,7 +145,7 @@ binary_data = f.serialize(person) # binary_data can now be sent to Java, Go, etc. ``` -### Java (Deserializer) +#### Java (Deserializer) ```java import org.apache.fory.*; @@ -75,7 +164,7 @@ fory.register(Person.class, "example.Person"); Person person = (Person) fory.deserialize(binaryData); ``` -### Rust (Deserializer) +#### Rust (Deserializer) ```rust use fory::Fory; @@ -93,7 +182,7 @@ fory.register_by_name::("example.Person"); let person: Person = fory.deserialize(&binary_data)?; ``` -## Type Annotations for Xlang +### Type Annotations for Xlang Use pyfory type annotations for explicit xlang type mapping: @@ -119,7 +208,7 @@ Nested collection annotations are part of the field schema. Compatible-mode reads consume bytes with the remote schema metadata, then assign only when the decoded value safely satisfies the local schema. -## Reduced-Precision Types +### Reduced-Precision Types `pyfory.Float16` and `pyfory.BFloat16` are reserved annotation markers for xlang reduced-precision fields. They are not value wrapper classes; scalar values deserialize as native @@ -130,7 +219,7 @@ numeric values with `pyfory.Float16Array.from_values([...])` or `pyfory.BFloat16Array.from_values([...])`. Use `from_buffer(...)` and `to_buffer()` only when you already need packed little-endian `uint16` storage and want the raw-buffer fast path. -## Type Mapping +### Type Mapping | Python marker/carrier | Java | Rust | Go | | ---------------------- | -------------- | --------------- | --------------------- | @@ -147,7 +236,7 @@ already need packed little-endian `uint16` storage and want the raw-buffer fast | `list` | `List` | `Vec` | `[]T` | | `dict` | `Map` | `HashMap` | `map[K]V` | -### Lists and Dense Arrays +#### Lists and Dense Arrays Python `List[T]` maps to Fory `list`. Use `pyfory.Array[T]`, `pyfory.NDArray[T]`, or `pyfory.PyArray[T]` only when the schema is the dense @@ -197,24 +286,24 @@ carrier. Use `pyfory.format.from_arrow_schema(...)` and `pyfory.format.to_arrow_schema(...)` to convert between PyArrow schemas and Fory row-format schemas. -## Differences from Python Native Mode +### Differences from Python Native Mode The binary protocol and API are similar to `pyfory`'s Python native mode, but Python native mode can serialize any Python object—including global functions, local functions, lambdas, local classes, and types with custom serialization using `__getstate__/__reduce__/__reduce_ex__`, which are **not allowed** in xlang mode. -## See Also +### Specifications and References - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Reference](../../specification/xlang_type_mapping.md) -- [Java Xlang Serialization Guide](../java/xlang.md) -- [Rust Xlang Serialization Guide](../rust/xlang.md) +- [Java Interoperability Guide](../java/basic-serialization.md#cross-language-interoperability) +- [Rust Interoperability Guide](../rust/basic-serialization.md#cross-language-interoperability) -## Related Topics +### Related Guides - [Configuration](configuration.md) - xlang mode settings - [Schema Evolution](schema-evolution.md) - Compatible mode - [Type Registration](type-registration.md) - Registration patterns -## Read the Java file example +### Read the Java file example ```python import pyfory @@ -236,7 +325,7 @@ print(f"Name: {person.name}, Age: {person.age}") # Output: Name: Alice, Age: 30 ``` -## Built-in values +### Built-in values ```python import pyfory @@ -255,7 +344,7 @@ new_map = fory.deserialize(data) print(new_map) ``` -## Custom values +### Custom values ```python from dataclasses import dataclass @@ -312,7 +401,7 @@ if __name__ == "__main__": print(f.deserialize(data)) ``` -## Shared and circular references +### Shared and circular references ```python from typing import Dict @@ -332,3 +421,10 @@ data = fory.serialize(obj) # bytes can be deserialized by other languages print(fory.deserialize(data)) ``` + +## Related Topics + +- [Configuration](configuration.md) - Fory parameters +- [Type Registration](type-registration.md) - Registration patterns +- [Native Serialization](native.md) - Functions and lambdas +- [Out-of-Band Serialization](out-of-band.md) - Buffer callback APIs diff --git a/docs/object-serialization/python/configuration.md b/docs/object-serialization/python/configuration.md index 57d1cca945..04652322e7 100644 --- a/docs/object-serialization/python/configuration.md +++ b/docs/object-serialization/python/configuration.md @@ -180,126 +180,10 @@ Use `strict=False` only for trusted data, preferably with a `policy=` deserializ ## Security -Treat native-mode bytes from untrusted sources the same way you would treat untrusted pickle bytes. -Native mode can reconstruct Python objects, import modules, invoke reduction hooks, and rebuild -dynamic classes or functions when `strict=False`. - -### Production Configuration - -Keep `strict=True` for production payloads unless the whole data source is trusted and a -`DeserializationPolicy` owns the remaining trust decisions: - -```python -import pyfory - -fory = pyfory.Fory( - xlang=True, - ref=False, - strict=True, - max_depth=50, - max_type_fields=512, - max_type_meta_bytes=4096, - max_schema_versions_per_type=10, - max_average_schema_versions_per_type=3, - max_graph_memory_bytes=128 * 1024 * 1024, -) - -fory.register(UserModel, name="example.User") -fory.register(OrderModel, name="example.Order") -``` - -Use dynamic native-mode deserialization (`strict=False`) only for trusted Python-only payloads: - -```python -import pyfory - -fory = pyfory.Fory( - xlang=False, - ref=True, - strict=False, - max_depth=100, -) -``` - -Received remote metadata is also limited: - -- `max_type_fields` limits the number of fields accepted in one received struct metadata body. -- `max_type_meta_bytes` limits the encoded body bytes accepted for one received TypeDef body. -- `max_schema_versions_per_type` limits accepted remote metadata versions for one logical type. -- `max_average_schema_versions_per_type` limits the average across accepted remote types. -- `max_graph_memory_bytes` sets an approximate gate for materialized graph memory during one root - deserialization. The estimate mainly covers lists, tuples, sets, dicts, object arrays, structs, - and Python objects. It skips leaf values such as strings, binary data, primitive scalars, and - dense primitive arrays, so actual process memory can be higher than this value. Leaf values remain - protected by byte-availability checks: if the unread input does not contain enough bytes, Fory - will not read or create that leaf value. The default is a fixed `128 MiB` for all root input - forms. Set a positive byte value for trusted payloads that legitimately need a larger or smaller - gate. -- `max_unbacked_container_items` limits collection elements and map entries whose repeated read - bodies do not consume proportional input during one root deserialization. The default is `8192`; - zero is a strict limit. - -These limits do not change `strict`, `policy`, dynamic loading, unknown-class handling, or -schema-evolution semantics. - -### DeserializationPolicy - -When `strict=False` is necessary, use `DeserializationPolicy` to restrict the dynamic types and -hooks accepted during deserialization: - -```python -import pyfory -from pyfory import DeserializationPolicy - -dangerous_modules = {"subprocess", "os", "__builtin__"} - -class SafeDeserializationPolicy(DeserializationPolicy): - def validate_class(self, cls, is_local, **kwargs): - if cls.__module__ in dangerous_modules: - raise ValueError(f"Blocked dangerous class: {cls.__module__}.{cls.__name__}") - - def intercept_reduce_call(self, callable_obj, args, **kwargs): - if getattr(callable_obj, "__name__", "") == "Popen": - raise ValueError("Blocked attempt to invoke subprocess.Popen") - return None - - def intercept_setstate(self, obj, state, **kwargs): - if isinstance(state, dict) and "password" in state: - state["password"] = "***REDACTED***" - return None - -policy = SafeDeserializationPolicy() -fory = pyfory.Fory(xlang=False, ref=True, strict=False, policy=policy) -``` - -Available policy hooks include: - -Reference validation hooks reject by raising exceptions and otherwise leave deserialized references -unchanged. - -| Hook | Description | -| -------------------------------------------- | --------------------------------------------------- | -| `validate_class(cls, is_local)` | Validate or block class types | -| `validate_module(module_name, is_local)` | Validate or block module imports | -| `validate_function(func, is_local)` | Validate or block function references | -| `validate_method(method, is_local)` | Validate or block method references | -| `intercept_reduce_call(callable_obj, args)` | Intercept `__reduce__` invocations | -| `inspect_reduced_object(obj)` | Inspect or replace objects created via `__reduce__` | -| `intercept_setstate(obj, state)` | Sanitize state before `__setstate__` | -| `authorize_instantiation(cls, args, kwargs)` | Control class instantiation | - -### Security Checklist - -- Keep `strict=True` for untrusted data. -- Register all expected application types before deserialization. -- Use `DeserializationPolicy` when `strict=False` is necessary. -- Keep `max_depth` low enough to reject unexpectedly deep payloads. -- Keep `max_graph_memory_bytes` at the fixed `128 MiB` default for most inputs, or set a positive - explicit gate for trusted workloads with different legitimate collection/map/struct sizes. -- Do not treat xlang/native mode choice as a security control. +See [Python Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) - Using configured Fory +- [Basic Serialization](basic-serialization.md) - Using configured Fory - [Type Registration](type-registration.md) - Registration patterns - [Native Serialization](native.md) - Python-only object serialization diff --git a/docs/object-serialization/python/core-api.md b/docs/object-serialization/python/core-api.md deleted file mode 100644 index a14ab689d5..0000000000 --- a/docs/object-serialization/python/core-api.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers the Python xlang quickstart. `pyfory.Fory()` defaults to xlang mode with -compatible schema evolution; examples set `xlang=True` explicitly so the mode choice is visible. - -## Basic Object Serialization - -Serialize and deserialize Python objects with a simple API: - -```python -import pyfory - -fory = pyfory.Fory(xlang=True) - -# Serialize xlang-compatible values -data = fory.dumps({"name": "Alice", "age": 30, "scores": [95, 87, 92]}) - -# Deserialize back to Python object -obj = fory.loads(data) -print(obj) # {'name': 'Alice', 'age': 30, 'scores': [95, 87, 92]} -``` - -**Note**: `dumps()`/`loads()` are aliases for `serialize()`/`deserialize()`. Both APIs are identical, use whichever feels more intuitive. - -## Custom Class Serialization - -Use dataclasses and type annotations for stable xlang payloads: - -```python -import pyfory -from dataclasses import dataclass -from typing import List, Dict - -@dataclass -class Person: - name: str - age: pyfory.Int32 - scores: List[pyfory.Int32] - metadata: Dict[str, str] - -fory = pyfory.Fory(xlang=True, ref=True) -fory.register(Person, name="example.Person") -person = Person("Bob", 25, [88, 92, 85], {"team": "engineering"}) -data = fory.serialize(person) -result = fory.deserialize(data) -print(result) # Person(name='Bob', age=25, ...) -``` - -## Reference Tracking & Circular References - -Handle repeated references safely when the payload uses xlang-compatible types: - -```python -import pyfory - -f = pyfory.Fory(xlang=True, ref=True) - -shared = ["shared"] -value = [shared, shared] - -data = f.serialize(value) -result = f.deserialize(data) -assert result[0] is result[1] -``` - -For arbitrary Python object graphs, local classes, functions, and methods, use -[Native Serialization](native.md). - -## Performance Tips - -1. **Disable `ref=True` if not needed**: Reference tracking has overhead -2. **Use type_id instead of name**: Integer IDs are faster than string names -3. **Reuse Fory instances**: Create once, use many times -4. **Enable Cython**: Make sure `ENABLE_FORY_CYTHON_SERIALIZATION=1` - -```python -# Good: Reuse instance -fory = pyfory.Fory(xlang=True) -for obj in objects: - data = fory.dumps(obj) - -# Bad: Create new instance each time -for obj in objects: - fory = pyfory.Fory(xlang=True) # Wasteful! - data = fory.dumps(obj) -``` - -## Related Topics - -- [Configuration](configuration.md) - Fory parameters -- [Type Registration](type-registration.md) - Registration patterns -- [Native Serialization](native.md) - Functions and lambdas -- [Out-of-Band Serialization](out-of-band.md) - Buffer callback APIs diff --git a/docs/object-serialization/python/custom-serializers.md b/docs/object-serialization/python/custom-serializers.md index e934c3454f..da907d4302 100644 --- a/docs/object-serialization/python/custom-serializers.md +++ b/docs/object-serialization/python/custom-serializers.md @@ -135,4 +135,4 @@ fory.register(MyClass, name="com.example.MyClass", serializer=MySerializer(fory. - [Type Registration](type-registration.md) - Registration patterns - [Configuration](configuration.md) - Fory parameters -- [Xlang Serialization](xlang.md) - type registration and schema rules for xlang +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - type registration and schema rules for xlang diff --git a/docs/object-serialization/python/index.md b/docs/object-serialization/python/index.md index b356e36b94..cf05c7510a 100644 --- a/docs/object-serialization/python/index.md +++ b/docs/object-serialization/python/index.md @@ -147,12 +147,11 @@ Use xlang mode for cross-language payloads and dataclass schemas shared with oth Use native mode for Python-only traffic. Native mode is selected with `xlang=False` and owns pickle/cloudpickle-style behavior such as functions, lambdas, classes, methods, `__reduce__`, `__getstate__`, and out-of-band pickle protocol 5 buffers. It is optimized for Python's type system and supports a broader Python object surface than xlang mode, so use it when replacing pickle or cloudpickle. Compatible mode is enabled by default. Set `compatible=False` only when every reader and writer uses the same Python class schema and you want faster serialization and smaller size. -See [Native Serialization](native.md) for Python-only serialization details and [Xlang Serialization](xlang.md) for Python xlang registration and interoperability rules. +See [Native Serialization](native.md) for Python-only serialization details and [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Python xlang registration and interoperability rules. ## Next Steps -- [Basic Serialization](core-api.md) - Basic usage patterns -- [Xlang Serialization](xlang.md) - xlang mode +- [Basic Serialization](basic-serialization.md) - Default xlang APIs and interoperability - [Native Serialization](native.md) - Python-only serialization - [Configuration](configuration.md) - Fory parameters, modes, and security - [Type Registration](type-registration.md) - User-defined type registration @@ -167,3 +166,6 @@ See [Native Serialization](native.md) for Python-only serialization details and - **PyPI**: https://pypi.org/project/pyfory/ - **Issues**: https://github.com/apache/fory/issues - **Slack**: https://join.slack.com/t/fory-project/shared_invite/zt-36g0qouzm-kcQSvV_dtfbtBKHRwT5gsw + +Before decoding bytes from outside the application trust boundary, read +[Python Security](security.md). diff --git a/docs/object-serialization/python/native.md b/docs/object-serialization/python/native.md index fbab2d008e..e77e24bbea 100644 --- a/docs/object-serialization/python/native.md +++ b/docs/object-serialization/python/native.md @@ -23,7 +23,7 @@ Python native serialization is the Python-only wire mode selected with `xlang=Fa every writer and reader is Python and the payload should follow Python's object model instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default Python mode, when bytes must be read +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Python mode, when bytes must be read by Java, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-Python Fory implementation. @@ -186,7 +186,7 @@ Keep the payload in native mode. Xlang mode does not execute Python `__reduce__` ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language Python payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Python payloads - [Configuration](configuration.md) - Python `Fory` options - [Out-of-Band Serialization](out-of-band.md) - Zero-copy buffer support - [Configuration](configuration.md#security) - Deserialization policies diff --git a/docs/object-serialization/python/numpy-integration.md b/docs/object-serialization/python/numpy-integration.md index 4884f6e2d5..8188696ace 100644 --- a/docs/object-serialization/python/numpy-integration.md +++ b/docs/object-serialization/python/numpy-integration.md @@ -100,4 +100,4 @@ assert np.array_equal(array, result) ## Related Topics - [Out-of-Band Serialization](out-of-band.md) - Zero-copy buffers -- [Basic Serialization](core-api.md) - Standard usage +- [Basic Serialization](basic-serialization.md) - Standard usage diff --git a/docs/object-serialization/python/out-of-band.md b/docs/object-serialization/python/out-of-band.md index 8a3ff0399f..8db222fecb 100644 --- a/docs/object-serialization/python/out-of-band.md +++ b/docs/object-serialization/python/out-of-band.md @@ -175,5 +175,5 @@ for buffer_obj in buffer_objects: ## Related Topics - [NumPy Integration](numpy-integration.md) - NumPy array serialization -- [Basic Serialization](core-api.md) - Standard serialization +- [Basic Serialization](basic-serialization.md) - Standard serialization - [Configuration](configuration.md) - Fory parameters diff --git a/docs/object-serialization/python/schema-evolution.md b/docs/object-serialization/python/schema-evolution.md index 52ba957c78..a8927c92c0 100644 --- a/docs/object-serialization/python/schema-evolution.md +++ b/docs/object-serialization/python/schema-evolution.md @@ -137,5 +137,5 @@ class SameSchemaMessage: ## Related Topics - [Configuration](configuration.md) - Compatible mode settings -- [Xlang Serialization](xlang.md) - Schema evolution across languages +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Schema evolution across languages - [Type Registration](type-registration.md) - Registration patterns diff --git a/docs/object-serialization/python/schema-metadata.md b/docs/object-serialization/python/schema-metadata.md index ad8db45551..913cafb4f8 100644 --- a/docs/object-serialization/python/schema-metadata.md +++ b/docs/object-serialization/python/schema-metadata.md @@ -524,6 +524,6 @@ class User: ## Related Topics -- [Basic Serialization](core-api.md) - Getting started with Fory serialization +- [Basic Serialization](basic-serialization.md) - Getting started with Fory serialization - [Schema Evolution](schema-evolution.md) - Compatible mode and schema evolution -- [Xlang Serialization](xlang.md) - Interoperability with Java, Rust, C++, Go +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Interoperability with Java, Rust, C++, Go diff --git a/docs/object-serialization/python/security.md b/docs/object-serialization/python/security.md new file mode 100644 index 0000000000..db9e96c980 --- /dev/null +++ b/docs/object-serialization/python/security.md @@ -0,0 +1,163 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Python reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Treat native-mode bytes from untrusted sources the same way you would treat untrusted pickle bytes. +Native mode can reconstruct Python objects, import modules, invoke reduction hooks, and rebuild +dynamic classes or functions when `strict=False`. + +### Production Configuration + +Keep `strict=True` for production payloads unless the whole data source is trusted and a +`DeserializationPolicy` owns the remaining trust decisions: + +```python +import pyfory + +fory = pyfory.Fory( + xlang=True, + ref=False, + strict=True, + max_depth=50, + max_type_fields=512, + max_type_meta_bytes=4096, + max_schema_versions_per_type=10, + max_average_schema_versions_per_type=3, + max_graph_memory_bytes=128 * 1024 * 1024, +) + +fory.register(UserModel, name="example.User") +fory.register(OrderModel, name="example.Order") +``` + +Use dynamic native-mode deserialization (`strict=False`) only for trusted Python-only payloads: + +```python +import pyfory + +fory = pyfory.Fory( + xlang=False, + ref=True, + strict=False, + max_depth=100, +) +``` + +Received remote metadata is also limited: + +- `max_type_fields` limits the number of fields accepted in one received struct metadata body. +- `max_type_meta_bytes` limits the encoded body bytes accepted for one received TypeDef body. +- `max_schema_versions_per_type` limits accepted remote metadata versions for one logical type. +- `max_average_schema_versions_per_type` limits the average across accepted remote types. +- `max_graph_memory_bytes` sets an approximate gate for materialized graph memory during one root + deserialization. The estimate mainly covers lists, tuples, sets, dicts, object arrays, structs, + and Python objects. It skips leaf values such as strings, binary data, primitive scalars, and + dense primitive arrays, so actual process memory can be higher than this value. Leaf values remain + protected by byte-availability checks: if the unread input does not contain enough bytes, Fory + will not read or create that leaf value. The default is a fixed `128 MiB` for all root input + forms. Set a positive byte value for trusted payloads that legitimately need a larger or smaller + gate. +- `max_unbacked_container_items` limits collection elements and map entries whose repeated read + bodies do not consume proportional input during one root deserialization. The default is `8192`; + zero is a strict limit. + +These limits do not change `strict`, `policy`, dynamic loading, unknown-class handling, or +schema-evolution semantics. + +### DeserializationPolicy + +When `strict=False` is necessary, use `DeserializationPolicy` to restrict the dynamic types and +hooks accepted during deserialization: + +```python +import pyfory +from pyfory import DeserializationPolicy + +dangerous_modules = {"subprocess", "os", "__builtin__"} + +class SafeDeserializationPolicy(DeserializationPolicy): + def validate_class(self, cls, is_local, **kwargs): + if cls.__module__ in dangerous_modules: + raise ValueError(f"Blocked dangerous class: {cls.__module__}.{cls.__name__}") + + def intercept_reduce_call(self, callable_obj, args, **kwargs): + if getattr(callable_obj, "__name__", "") == "Popen": + raise ValueError("Blocked attempt to invoke subprocess.Popen") + return None + + def intercept_setstate(self, obj, state, **kwargs): + if isinstance(state, dict) and "password" in state: + state["password"] = "***REDACTED***" + return None + +policy = SafeDeserializationPolicy() +fory = pyfory.Fory(xlang=False, ref=True, strict=False, policy=policy) +``` + +Available policy hooks include: + +Reference validation hooks reject by raising exceptions and otherwise leave deserialized references +unchanged. + +| Hook | Description | +| -------------------------------------------- | --------------------------------------------------- | +| `validate_class(cls, is_local)` | Validate or block class types | +| `validate_module(module_name, is_local)` | Validate or block module imports | +| `validate_function(func, is_local)` | Validate or block function references | +| `validate_method(method, is_local)` | Validate or block method references | +| `intercept_reduce_call(callable_obj, args)` | Intercept `__reduce__` invocations | +| `inspect_reduced_object(obj)` | Inspect or replace objects created via `__reduce__` | +| `intercept_setstate(obj, state)` | Sanitize state before `__setstate__` | +| `authorize_instantiation(cls, args, kwargs)` | Control class instantiation | + +### Security Checklist + +- Keep `strict=True` for untrusted data. +- Register all expected application types before deserialization. +- Use `DeserializationPolicy` when `strict=False` is necessary. +- Keep `max_depth` low enough to reject unexpectedly deep payloads. +- Keep `max_graph_memory_bytes` at the fixed `128 MiB` default for most inputs, or set a positive + explicit gate for trusted workloads with different legitimate collection/map/struct sizes. +- Do not treat xlang/native mode choice as a security control. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/runtimes.md b/docs/object-serialization/runtimes.md deleted file mode 100644 index c820c929d9..0000000000 --- a/docs/object-serialization/runtimes.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: Object Serialization Runtimes -sidebar_position: 3 -id: runtimes -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Choose the runtime whose API your application uses. Runtime documentation is scoped to -binary object serialization; Row Format, Fory JSON, compiler, and gRPC pages remain with -their product owners. - -| Runtime | Modes | Documentation | -| --------------------- | -------------------- | ------------------------------------------------------ | -| Java | xlang and native | [Java runtime](./java/index.md) | -| Python | xlang and native | [Python runtime](./python/index.md) | -| C++ | xlang and native | [C++ runtime](./cpp/index.md) | -| Go | xlang and native | [Go runtime](./go/index.md) | -| Rust | xlang and native | [Rust runtime](./rust/index.md) | -| JavaScript/TypeScript | xlang | [JavaScript/TypeScript runtime](./javascript/index.md) | -| C# | xlang | [C# runtime](./csharp/index.md) | -| Swift | xlang | [Swift runtime](./swift/index.md) | -| Dart | xlang | [Dart runtime](./dart/index.md) | -| Scala | xlang and JVM native | [Scala runtime](./scala/index.md) | -| Kotlin | xlang and JVM native | [Kotlin runtime](./kotlin/index.md) | diff --git a/docs/object-serialization/rust/xlang.md b/docs/object-serialization/rust/basic-serialization.md similarity index 53% rename from docs/object-serialization/rust/xlang.md rename to docs/object-serialization/rust/basic-serialization.md index c3b573a639..b8fcf9e873 100644 --- a/docs/object-serialization/rust/xlang.md +++ b/docs/object-serialization/rust/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,10 +19,204 @@ license: | limitations under the License. --- +This page covers basic object graph serialization and supported types in the default xlang mode for Fory Rust. + +## Object Graph Serialization + +Apache Fory™ provides automatic serialization of complex object graphs, preserving the structure and relationships between objects. The `#[derive(ForyStruct)]` macro generates efficient serialization code at compile time, eliminating reflection overhead. + +**Key capabilities:** + +- Nested struct serialization with arbitrary depth +- Collection types (Vec, HashMap, HashSet, BTreeMap) +- Optional fields with `Option` +- Automatic handling of primitive types and strings +- Efficient binary encoding with variable-length integers + +```rust +use fory::{Fory, Error}; +use fory::ForyStruct; +use std::collections::HashMap; + +#[derive(ForyStruct, Debug, PartialEq)] +struct Person { + name: String, + age: i32, + address: Address, + hobbies: Vec, + metadata: HashMap, +} + +#[derive(ForyStruct, Debug, PartialEq)] +struct Address { + street: String, + city: String, + country: String, +} + +let mut fory = Fory::builder().xlang(true).build(); +fory.register_by_name::
("example.Address").unwrap(); +fory.register_by_name::("example.Person").unwrap(); + +let person = Person { + name: "John Doe".to_string(), + age: 30, + address: Address { + street: "123 Main St".to_string(), + city: "New York".to_string(), + country: "USA".to_string(), + }, + hobbies: vec!["reading".to_string(), "coding".to_string()], + metadata: HashMap::from([ + ("role".to_string(), "developer".to_string()), + ]), +}; + +let bytes = fory.serialize(&person).unwrap(); +let decoded: Person = fory.deserialize(&bytes)?; +assert_eq!(person, decoded); +``` + +## Supported Types + +### Primitive Types + +| Rust Type | Description | +| ------------------------- | --------------------------- | +| `bool` | Boolean | +| `i8`, `i16`, `i32`, `i64` | Signed integers | +| `f32`, `f64` | Floating point | +| `BFloat16` | 16-bit brain floating point | +| `String` | UTF-8 string | + +### Collections + +| Rust Type | Description | +| ---------------- | ------------------ | +| `Vec` | Dynamic array | +| `VecDeque` | Double-ended queue | +| `LinkedList` | Doubly-linked list | +| `HashMap` | Hash map | +| `BTreeMap` | Ordered map | +| `HashSet` | Hash set | +| `BTreeSet` | Ordered set | +| `BinaryHeap` | Binary heap | +| `Option` | Optional value | + +`Vec` is the dense carrier when the schema is `array`. + +### Smart Pointers + +| Rust Type | Description | +| ------------ | ---------------------------------------------------- | +| `Box` | Heap allocation | +| `Rc` | Reference counting (shared refs tracked) | +| `Arc` | Thread-safe reference counting (shared refs tracked) | +| `RcWeak` | Weak reference to `Rc` (breaks circular refs) | +| `ArcWeak` | Weak reference to `Arc` (breaks circular refs) | +| `RefCell` | Interior mutability (runtime borrow checking) | +| `Mutex` | Thread-safe interior mutability | + +### Date and Time + +| Rust Type | Description | +| ----------- | ------------------------------------------------------- | +| `Date` | Date without timezone, stored as epoch days | +| `Timestamp` | Point in time, stored as epoch seconds and nanos | +| `Duration` | Signed duration, stored as seconds and normalized nanos | + +The built-in carriers expose dependency-free constructors, accessors, conversions, and checked +arithmetic: + +```rust +use fory::{Date, Duration, Timestamp}; + +let date = Date::from_epoch_days(19_782); +assert_eq!(date.checked_add_days(1)?.epoch_days(), 19_783); + +let timestamp = Timestamp::from_epoch_millis(-1); +assert_eq!(timestamp.to_epoch_millis()?, -1); + +let duration = Duration::from_parts(1, 1_500_000_000)?; +assert_eq!(duration.to_millis()?, 2_500); +let later = timestamp.checked_add_duration(duration)?; +``` + +`chrono::NaiveDate`, `chrono::NaiveDateTime`, and `chrono::Duration` are supported when the Rust +`chrono` feature is enabled: + +```toml +[dependencies] +fory = { version = "1.5.0", features = ["chrono"] } +``` + +### Custom Types + +Use `#[derive(ForyStruct)]` for object graph serialization. The separate +[Rust Row Format guide](../../row-format/rust.md) documents `#[derive(ForyRow)]` and its supported +type set. + +## Serialization APIs + +```rust +use fory::{Fory, Reader}; + +let mut fory = Fory::builder().xlang(true).build(); +fory.register::(1)?; + +let obj = MyStruct { /* ... */ }; + +// Basic serialize/deserialize +let bytes = fory.serialize(&obj)?; +let decoded: MyStruct = fory.deserialize(&bytes)?; + +// Serialize to existing buffer +let mut buf: Vec = vec![]; +fory.serialize_to(&mut buf, &obj)?; + +// Deserialize from reader +let mut reader = Reader::new(&buf); +let decoded: MyStruct = fory.deserialize_from(&mut reader)?; +``` + +When the Rust value type uses an external structural serializer or custom +serializer, select it explicitly at the root: + +```rust +let bytes = fory.serialize_with::(&user)?; +let decoded: third_party::User = + fory.deserialize_with::(&bytes)?; +``` + +Carrier serializers compose the same selection for a root container: + +```rust +use fory::VecSerializer; + +let bytes = + fory.serialize_with::>(&users)?; +let decoded: Vec = + fory.deserialize_with::>(&bytes)?; +``` + +See [External-Type Serialization](external-types.md) for field annotations, +all supported carriers, and registration. + +## Performance Tips + +- **Buffer Pre-allocation**: Minimizes memory allocations during serialization +- **Compact Encoding**: Variable-length encoding for space efficiency +- **Little-Endian**: Optimized for modern CPU architectures +- **Reference Deduplication**: Shared objects serialized only once + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Apache Fory™ supports seamless data exchange across Java, Python, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. -## Create an Xlang Fory Instance +### Xlang Configuration Rust defaults to xlang mode with compatible schema evolution. Set the mode explicitly in xlang examples: @@ -39,9 +233,9 @@ fory.register::(100)?; // fory.register_by_name::("com.example.MyStruct")?; ``` -## Type Registration for Xlang +### Type Registration for Xlang -### Register by ID +#### Register by ID For fast, compact serialization with consistent IDs across languages: @@ -51,7 +245,7 @@ let mut fory = Fory::builder().xlang(true).build(); fory.register::(100)?; // Same ID in Java, Python, etc. ``` -### Register by Name +#### Register by Name For more flexible type naming: @@ -59,9 +253,9 @@ For more flexible type naming: fory.register_by_name::("com.example.User")?; ``` -## Xlang Example +### Xlang Example -### Rust (Serializer) +#### Rust (Serializer) ```rust use fory::Fory; @@ -86,7 +280,7 @@ let bytes = fory.serialize(&person)?; // bytes can be deserialized by Java, Python, etc. ``` -### Third-Party Rust Types +#### Third-Party Rust Types An external structural serializer gives a third-party Rust type the same xlang schema as an equivalent local derive: @@ -120,7 +314,7 @@ multiple tuple or named fields is supported with `xlang(false)`, but its serializer registration is rejected in xlang mode. See [External-Type Serialization](external-types.md). -### Dynamic Rust Carriers +#### Dynamic Rust Carriers `Box`, `Rc`, `Arc`, and application `dyn Trait` carriers can be used in xlang mode when every selected concrete @@ -128,7 +322,7 @@ target has an xlang-compatible structural or EXT identity. Fory writes the concrete registered target identity; the Rust trait or erased-carrier identity does not appear on the wire. -### Java (Deserializer) +#### Java (Deserializer) ```java import org.apache.fory.*; @@ -149,7 +343,7 @@ fory.register(Person.class, 100); // Same ID as Rust Person person = (Person) fory.deserialize(bytesFromRust); ``` -### Python (Deserializer) +#### Python (Deserializer) ```python import pyfory @@ -166,11 +360,11 @@ fory.register_type(Person, type_id=100) # Same ID as Rust person = fory.deserialize(bytes_from_rust) ``` -## Type Mapping +### Type Mapping See [xlang_type_mapping.md](../../specification/xlang_type_mapping.md) for complete type mapping across languages. -### Common Type Mappings +#### Common Type Mappings | Rust | Java | Python | | --------------- | -------------- | --------------- | @@ -189,7 +383,7 @@ See [xlang_type_mapping.md](../../specification/xlang_type_mapping.md) for compl | `HashMap` | `Map` | `Dict[K,V]` | | `Option` | nullable `T` | `Optional[T]` | -### Lists and Dense Arrays +#### Lists and Dense Arrays Rust `Vec` maps to Fory `list` by default for manual structs. Use an explicit array field attribute when the schema is dense `array`. @@ -211,28 +405,28 @@ explicit array field attribute when the schema is dense `array`. | `array` | `#[fory(array)] Vec` | | `array` | `#[fory(array)] Vec` | -## Best Practices +### Interoperability Best Practices 1. **Use consistent type IDs** across all languages 2. **Keep compatible mode** for schema evolution 3. **Register all types** before serialization 4. **Test cross-language** compatibility during development -## See Also +### Specifications and References - [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - [Type Mapping Reference](../../specification/xlang_type_mapping.md) -- [Java Xlang Serialization Guide](../java/xlang.md) -- [Python Xlang Serialization Guide](../python/xlang.md) +- [Java Interoperability Guide](../java/basic-serialization.md#cross-language-interoperability) +- [Python Interoperability Guide](../python/basic-serialization.md#cross-language-interoperability) -## Related Topics +### Related Guides - [Configuration](configuration.md) - xlang mode configuration - [Schema Evolution](schema-evolution.md) - Compatible mode - [Type Registration](type-registration.md) - Registration methods - [External-Type Serialization](external-types.md) - Third-party values in xlang mode -## Built-in values +### Built-in values ```rust use fory::Fory; @@ -245,7 +439,7 @@ fn run() { } ``` -## Custom values +### Custom values ```rust use chrono::{NaiveDate, NaiveDateTime}; @@ -309,6 +503,14 @@ fn complex_struct() { } ``` -## Shared and circular references +### Shared and circular references Circular references cannot be implemented in Rust due to ownership restrictions. + +## Related Topics + +- [Type Registration](type-registration.md) - Registering types +- [References](references.md) - Shared and circular references +- [Custom Serializers](custom-serializers.md) - Custom serialization +- [External-Type Serialization](external-types.md) - Third-party values and carrier roots +- [Row Format](../../row-format/rust.md) - Standard Row Format and zero-copy borrowed views diff --git a/docs/object-serialization/rust/configuration.md b/docs/object-serialization/rust/configuration.md index 8edeb023d0..f5fb04eefc 100644 --- a/docs/object-serialization/rust/configuration.md +++ b/docs/object-serialization/rust/configuration.md @@ -201,21 +201,10 @@ Use `.compatible(false)` only when the schema used to deserialize every payload ## Security -Security-related configuration: - -- Register application structs and trait-object implementations before deserializing untrusted - payloads. -- Use `max_dyn_depth(...)` to reject unexpectedly deep dynamic object graphs. -- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a - positive byte gate for trusted workloads with different legitimate collection/map/struct sizes. -- Keep `max_unbacked_container_items(...)` at `8192` unless trusted compact codecs require a - larger root allowance. Zero rejects every unbacked item. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. -- Prefer concrete typed fields over `dyn Any` or broad trait-object fields for untrusted input. +See [Rust Security](security.md) for trust boundaries, safe reader configuration, and verification. ## Related Topics -- [Basic Serialization](core-api.md) - Using configured Fory +- [Basic Serialization](basic-serialization.md) - Using configured Fory - [Schema Evolution](schema-evolution.md) - Compatible mode details -- [Xlang Serialization](xlang.md) - xlang mode +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode diff --git a/docs/object-serialization/rust/core-api.md b/docs/object-serialization/rust/core-api.md deleted file mode 100644 index 5bce4c68e4..0000000000 --- a/docs/object-serialization/rust/core-api.md +++ /dev/null @@ -1,218 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers basic object graph serialization and supported types. - -## Object Graph Serialization - -Apache Fory™ provides automatic serialization of complex object graphs, preserving the structure and relationships between objects. The `#[derive(ForyStruct)]` macro generates efficient serialization code at compile time, eliminating reflection overhead. - -**Key capabilities:** - -- Nested struct serialization with arbitrary depth -- Collection types (Vec, HashMap, HashSet, BTreeMap) -- Optional fields with `Option` -- Automatic handling of primitive types and strings -- Efficient binary encoding with variable-length integers - -```rust -use fory::{Fory, Error}; -use fory::ForyStruct; -use std::collections::HashMap; - -#[derive(ForyStruct, Debug, PartialEq)] -struct Person { - name: String, - age: i32, - address: Address, - hobbies: Vec, - metadata: HashMap, -} - -#[derive(ForyStruct, Debug, PartialEq)] -struct Address { - street: String, - city: String, - country: String, -} - -let mut fory = Fory::builder().xlang(true).build(); -fory.register_by_name::
("example.Address").unwrap(); -fory.register_by_name::("example.Person").unwrap(); - -let person = Person { - name: "John Doe".to_string(), - age: 30, - address: Address { - street: "123 Main St".to_string(), - city: "New York".to_string(), - country: "USA".to_string(), - }, - hobbies: vec!["reading".to_string(), "coding".to_string()], - metadata: HashMap::from([ - ("role".to_string(), "developer".to_string()), - ]), -}; - -let bytes = fory.serialize(&person).unwrap(); -let decoded: Person = fory.deserialize(&bytes)?; -assert_eq!(person, decoded); -``` - -## Supported Types - -### Primitive Types - -| Rust Type | Description | -| ------------------------- | --------------------------- | -| `bool` | Boolean | -| `i8`, `i16`, `i32`, `i64` | Signed integers | -| `f32`, `f64` | Floating point | -| `BFloat16` | 16-bit brain floating point | -| `String` | UTF-8 string | - -### Collections - -| Rust Type | Description | -| ---------------- | ------------------ | -| `Vec` | Dynamic array | -| `VecDeque` | Double-ended queue | -| `LinkedList` | Doubly-linked list | -| `HashMap` | Hash map | -| `BTreeMap` | Ordered map | -| `HashSet` | Hash set | -| `BTreeSet` | Ordered set | -| `BinaryHeap` | Binary heap | -| `Option` | Optional value | - -`Vec` is the dense carrier when the schema is `array`. - -### Smart Pointers - -| Rust Type | Description | -| ------------ | ---------------------------------------------------- | -| `Box` | Heap allocation | -| `Rc` | Reference counting (shared refs tracked) | -| `Arc` | Thread-safe reference counting (shared refs tracked) | -| `RcWeak` | Weak reference to `Rc` (breaks circular refs) | -| `ArcWeak` | Weak reference to `Arc` (breaks circular refs) | -| `RefCell` | Interior mutability (runtime borrow checking) | -| `Mutex` | Thread-safe interior mutability | - -### Date and Time - -| Rust Type | Description | -| ----------- | ------------------------------------------------------- | -| `Date` | Date without timezone, stored as epoch days | -| `Timestamp` | Point in time, stored as epoch seconds and nanos | -| `Duration` | Signed duration, stored as seconds and normalized nanos | - -The built-in carriers expose dependency-free constructors, accessors, conversions, and checked -arithmetic: - -```rust -use fory::{Date, Duration, Timestamp}; - -let date = Date::from_epoch_days(19_782); -assert_eq!(date.checked_add_days(1)?.epoch_days(), 19_783); - -let timestamp = Timestamp::from_epoch_millis(-1); -assert_eq!(timestamp.to_epoch_millis()?, -1); - -let duration = Duration::from_parts(1, 1_500_000_000)?; -assert_eq!(duration.to_millis()?, 2_500); -let later = timestamp.checked_add_duration(duration)?; -``` - -`chrono::NaiveDate`, `chrono::NaiveDateTime`, and `chrono::Duration` are supported when the Rust -`chrono` feature is enabled: - -```toml -[dependencies] -fory = { version = "1.5.0", features = ["chrono"] } -``` - -### Custom Types - -Use `#[derive(ForyStruct)]` for object graph serialization. The separate -[Rust Row Format guide](../../row-format/rust.md) documents `#[derive(ForyRow)]` and its supported -type set. - -## Serialization APIs - -```rust -use fory::{Fory, Reader}; - -let mut fory = Fory::builder().xlang(true).build(); -fory.register::(1)?; - -let obj = MyStruct { /* ... */ }; - -// Basic serialize/deserialize -let bytes = fory.serialize(&obj)?; -let decoded: MyStruct = fory.deserialize(&bytes)?; - -// Serialize to existing buffer -let mut buf: Vec = vec![]; -fory.serialize_to(&mut buf, &obj)?; - -// Deserialize from reader -let mut reader = Reader::new(&buf); -let decoded: MyStruct = fory.deserialize_from(&mut reader)?; -``` - -When the Rust value type uses an external structural serializer or custom -serializer, select it explicitly at the root: - -```rust -let bytes = fory.serialize_with::(&user)?; -let decoded: third_party::User = - fory.deserialize_with::(&bytes)?; -``` - -Carrier serializers compose the same selection for a root container: - -```rust -use fory::VecSerializer; - -let bytes = - fory.serialize_with::>(&users)?; -let decoded: Vec = - fory.deserialize_with::>(&bytes)?; -``` - -See [External-Type Serialization](external-types.md) for field annotations, -all supported carriers, and registration. - -## Performance Tips - -- **Buffer Pre-allocation**: Minimizes memory allocations during serialization -- **Compact Encoding**: Variable-length encoding for space efficiency -- **Little-Endian**: Optimized for modern CPU architectures -- **Reference Deduplication**: Shared objects serialized only once - -## Related Topics - -- [Type Registration](type-registration.md) - Registering types -- [References](references.md) - Shared and circular references -- [Custom Serializers](custom-serializers.md) - Custom serialization -- [External-Type Serialization](external-types.md) - Third-party values and carrier roots -- [Row Format](../../row-format/rust.md) - Standard Row Format and zero-copy borrowed views diff --git a/docs/object-serialization/rust/index.md b/docs/object-serialization/rust/index.md index 00a29191c8..6f1156060e 100644 --- a/docs/object-serialization/rust/index.md +++ b/docs/object-serialization/rust/index.md @@ -101,7 +101,7 @@ Use xlang mode for cross-language payloads and schemas shared with other Fory im Use native mode for Rust-only traffic. Native mode is selected with `.xlang(false)` and keeps Rust object serialization in Rust-native form. It supports native-only concrete targets and data-enum shapes that have no xlang representation. Dynamic `Any`, application trait, and shared-reference carriers can also be used in xlang mode when every selected concrete target is xlang-compatible. Compatible mode is enabled by default. Set `.compatible(false)` only when every reader and writer uses the same Rust schema and you want faster serialization and smaller size. -See [Xlang Serialization](xlang.md) for Rust xlang registration and interoperability rules, and [Native Serialization](native.md) for Rust-only payloads. +See [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) for Rust xlang registration and interoperability rules, and [Native Serialization](native.md) for Rust-only payloads. ## Thread Safety @@ -158,8 +158,7 @@ fn main() -> Result<(), Error> { ## Next Steps - [Configuration](configuration.md) - Fory builder options and modes -- [Basic Serialization](core-api.md) - Object graph serialization -- [Xlang Serialization](xlang.md) - xlang mode +- [Basic Serialization](basic-serialization.md) - Default xlang object graphs and interoperability - [Native Serialization](native.md) - Rust-only serialization - [References](references.md) - Shared and circular references - [Polymorphism](polymorphism.md) - Trait object serialization @@ -168,3 +167,6 @@ fn main() -> Result<(), Error> { plus carrier composition - [Row Format](../../row-format/rust.md) - Standard Row Format with borrowed views - [gRPC Support](../../grpc/rust.md) - Fory payloads over tonic + +Before decoding bytes from outside the application trust boundary, read +[Rust Security](security.md). diff --git a/docs/object-serialization/rust/native.md b/docs/object-serialization/rust/native.md index feaaf0cf33..084d6ab118 100644 --- a/docs/object-serialization/rust/native.md +++ b/docs/object-serialization/rust/native.md @@ -23,7 +23,7 @@ Rust native serialization is the Rust-only wire mode selected with `.xlang(false every writer and reader is Rust and the payload should preserve Rust object-graph behavior instead of the portable xlang type system. -Use [Xlang Serialization](xlang.md), the default Rust mode, when bytes must be read +Use [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability), the default Rust mode, when bytes must be read by Java, Python, C++, Go, JavaScript/TypeScript, C#, Swift, Dart, Scala, Kotlin, or another non-Rust Fory implementation. @@ -115,7 +115,7 @@ Native serialization owns the Rust-specific object surface: is appropriate. - Date and time carriers, including optional `chrono` support. -Use [Basic Serialization](core-api.md), [References](references.md), and +Use [Basic Serialization](basic-serialization.md), [References](references.md), and [Trait Object Serialization](polymorphism.md) for focused examples. ## Shared And Circular References @@ -264,9 +264,9 @@ Native serialization defaults to compatible mode. Keep that default when schemas ## Related Topics -- [Xlang Serialization](xlang.md) - Cross-language Rust payloads +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Cross-language Rust payloads - [Configuration](configuration.md) - Builder options -- [Basic Serialization](core-api.md) - Object graph serialization +- [Basic Serialization](basic-serialization.md) - Object graph serialization - [Shared & Circular References](references.md) - `Rc`, `Arc`, and weak pointers - [Trait Object Serialization](polymorphism.md) - Trait objects and dynamic dispatch - [Schema Evolution](schema-evolution.md) - Compatible mode diff --git a/docs/object-serialization/rust/references.md b/docs/object-serialization/rust/references.md index 7bc0e584c3..4cbbc01f79 100644 --- a/docs/object-serialization/rust/references.md +++ b/docs/object-serialization/rust/references.md @@ -208,6 +208,6 @@ for child in &decoded.lock().unwrap().children { ## Related Topics -- [Basic Serialization](core-api.md) - Supported types +- [Basic Serialization](basic-serialization.md) - Supported types - [Polymorphism](polymorphism.md) - Trait objects with Rc/Arc - [Configuration](configuration.md) - Reference tracking options diff --git a/docs/object-serialization/rust/schema-evolution.md b/docs/object-serialization/rust/schema-evolution.md index 29e3624be6..81505ac2bf 100644 --- a/docs/object-serialization/rust/schema-evolution.md +++ b/docs/object-serialization/rust/schema-evolution.md @@ -278,5 +278,5 @@ assert_eq!(data, decoded); - [Configuration](configuration.md) - Compatible mode settings - [Polymorphism](polymorphism.md) - Trait objects with schema evolution -- [Xlang Serialization](xlang.md) - Schema evolution across languages +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Schema evolution across languages - [External-Type Serialization](external-types.md) - Compatible schemas for third-party values diff --git a/docs/object-serialization/rust/schema-metadata.md b/docs/object-serialization/rust/schema-metadata.md index ebf226e195..249f9a906a 100644 --- a/docs/object-serialization/rust/schema-metadata.md +++ b/docs/object-serialization/rust/schema-metadata.md @@ -497,7 +497,7 @@ struct User { ## Related Topics -- [Basic Serialization](core-api.md) - Getting started with Fory serialization +- [Basic Serialization](basic-serialization.md) - Getting started with Fory serialization - [Schema Evolution](schema-evolution.md) - Compatible mode and schema evolution -- [Xlang Serialization](xlang.md) - Interoperability with Java, C++, Go, Python +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - Interoperability with Java, C++, Go, Python - [External-Type Serialization](external-types.md) - Selecting serializers for third-party fields diff --git a/docs/object-serialization/rust/security.md b/docs/object-serialization/rust/security.md new file mode 100644 index 0000000000..7a17d95fec --- /dev/null +++ b/docs/object-serialization/rust/security.md @@ -0,0 +1,58 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Rust reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register application structs and trait-object implementations before deserializing untrusted + payloads. +- Use `max_dyn_depth(...)` to reject unexpectedly deep dynamic object graphs. +- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a + positive byte gate for trusted workloads with different legitimate collection/map/struct sizes. +- Keep `max_unbacked_container_items(...)` at `8192` unless trusted compact codecs require a + larger root allowance. Zero rejects every unbacked item. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. +- Prefer concrete typed fields over `dyn Any` or broad trait-object fields for untrusted input. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/rust/type-registration.md b/docs/object-serialization/rust/type-registration.md index 3efe057ae3..162eb4a954 100644 --- a/docs/object-serialization/rust/type-registration.md +++ b/docs/object-serialization/rust/type-registration.md @@ -136,6 +136,6 @@ let handles: Vec<_> = (0..4) ## Related Topics - [Configuration](configuration.md) - Fory builder options -- [Xlang Serialization](xlang.md) - xlang mode registration +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - xlang mode registration - [Custom Serializers](custom-serializers.md) - Custom serialization - [External-Type Serialization](external-types.md) - Third-party targets and carrier roots diff --git a/docs/object-serialization/scala/xlang.md b/docs/object-serialization/scala/basic-serialization.md similarity index 93% rename from docs/object-serialization/scala/xlang.md rename to docs/object-serialization/scala/basic-serialization.md index e49a86a4b9..c1768d45e5 100644 --- a/docs/object-serialization/scala/xlang.md +++ b/docs/object-serialization/scala/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Scala Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,11 +19,17 @@ license: | limitations under the License. --- +Xlang is the default serialization mode for Fory Scala. This page covers the basic serialization API and interoperability rules for that default mode. + +## Cross-Language Interoperability + +The following sections cover model generation, registration, and cross-language round trips in the default xlang mode. + The Fory schema IDL Scala target generates Scala 3 source for xlang payloads. The Fory Scala artifact remains cross-built for Scala 2.13 and Scala 3; only the schema IDL output and quoted macro derivation require Scala 3. -## Setup +### Setup Generated Scala code uses the public macro API in `org.apache.fory.scala` and the shared JVM annotations in `org.apache.fory.annotation`. Macro internals live @@ -55,7 +61,7 @@ serializers or Scala-specific registration state in Java core. Enums and unions are registered with their serializers directly because their derived serializers own case dispatch. -## Generated Messages +### Generated Messages Acyclic messages generate case classes: @@ -107,7 +113,7 @@ supported mutable collection interfaces such as `scala.collection.Seq` and `scala.collection.Map`, but concrete mutable collection classes are outside the schema IDL surface unless explicitly generated. -## Generated Enums +### Generated Enums IDL enums generate Scala 3 enums only. The compiler does not emit Java enum files. @@ -127,7 +133,7 @@ enum Status { Generated registration uses `ScalaSerializers.registerEnum(...)` so the stable Fory enum IDs from case-level `@ForyEnumId` metadata are used in xlang mode. -## Generated Unions +### Generated Unions IDL unions generate Scala 3 ADT enums with macro-derived serializers: @@ -167,7 +173,7 @@ locally. The macro writes the existing xlang union envelope directly. It does not allocate temporary Java `Union` carriers. -## Manual Scala 3 Derivation +### Manual Scala 3 Derivation Manual Scala 3 models can derive the same serializer typeclass: @@ -192,7 +198,7 @@ constructor-owned value that participates in the cycle, such as a Scala enum case or case class, the serializer fails with a clear error because no copied identity can be published until construction has completed. -## First round trip +### First round trip ```scala import org.apache.fory.Fory diff --git a/docs/object-serialization/scala/configuration.md b/docs/object-serialization/scala/configuration.md index 18aee91be5..2524eaeb0e 100644 --- a/docs/object-serialization/scala/configuration.md +++ b/docs/object-serialization/scala/configuration.md @@ -192,31 +192,4 @@ For xlang payloads, call `withCompatible(false)` only after verifying that every ## Security -Scala uses the Java configuration surface. Keep class registration enabled for production -and any untrusted payload source: - -```scala -val fory = ForyScala.builder() - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .withMaxTypeFields(512) - .withMaxTypeMetaBytes(4096) - .build() -``` - -Security-related configuration: - -- Keep `requireClassRegistration(true)` and register application classes or generated modules. -- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. -- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a - larger root allowance. Zero rejects every unbacked item. -- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits - at their defaults unless the data is not malicious and a trusted peer sends larger metadata or - many schema versions. -- Follow [Java Configuration](../java/configuration.md#security) for allow-listing and unknown-class - controls. +See [Scala Security](security.md) for trust boundaries, safe reader configuration, and verification. diff --git a/docs/object-serialization/scala/index.md b/docs/object-serialization/scala/index.md index 0d60f59af6..de1d3efec9 100644 --- a/docs/object-serialization/scala/index.md +++ b/docs/object-serialization/scala/index.md @@ -105,7 +105,7 @@ See [Configuration](configuration.md) for Scala builder setup and [Java Native S Fory Scala is built on top of Fory Java. Most configuration options, features, and concepts from Fory Java apply directly to Scala. Refer to the Java documentation for: - [Configuration](../java/configuration.md) - All ForyBuilder options -- [Basic Serialization](../java/core-api.md) - Serialization patterns and APIs +- [Basic Serialization](../java/basic-serialization.md) - Serialization patterns and APIs - [Type Registration](../java/type-registration.md) - Class registration and security - [Schema Evolution](../java/schema-evolution.md) - Forward/backward compatibility - [Custom Serializers](../java/custom-serializers.md) - Implement custom serializers @@ -118,5 +118,8 @@ Fory Scala is built on top of Fory Java. Most configuration options, features, a - [Native Serialization](native.md) - Serializing Scala types in JVM-native mode - [Schema Metadata](schema-metadata.md) - Scala annotations, references, enum IDs, and union metadata - [Default Values](default-values.md) - Scala class default values support -- [Xlang Serialization](xlang.md) - Scala 3 generated models and macro-derived xlang serializers +- [Basic Serialization](basic-serialization.md) - Models, APIs, and interoperability in the default xlang mode - [gRPC Support](../../grpc/scala.md) - Scala 3 generated gRPC service companions + +Before decoding bytes from outside the application trust boundary, read +[Scala Security](security.md). diff --git a/docs/object-serialization/scala/native.md b/docs/object-serialization/scala/native.md index e3992c8773..20fd963b97 100644 --- a/docs/object-serialization/scala/native.md +++ b/docs/object-serialization/scala/native.md @@ -21,7 +21,7 @@ license: | This page covers serialization of Scala-specific JVM types in native mode. For cross-language Scala models, use the xlang path described in -[Xlang Serialization](xlang.md). +[Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability). When compatible mode is enabled, Scala readers use the JVM compatible-read rules for selected scalar field type changes. A matched field can read between `Boolean`, `String`, numeric scalars, diff --git a/docs/object-serialization/scala/schema-metadata.md b/docs/object-serialization/scala/schema-metadata.md index 1d04dce8ab..d630690824 100644 --- a/docs/object-serialization/scala/schema-metadata.md +++ b/docs/object-serialization/scala/schema-metadata.md @@ -119,6 +119,6 @@ reflection is not the source of truth for generated Scala metadata. ## Related Topics -- [Xlang Serialization](xlang.md) +- [Cross-Language Interoperability](basic-serialization.md#cross-language-interoperability) - [Configuration](configuration.md) - [Default Values](default-values.md) diff --git a/docs/object-serialization/scala/security.md b/docs/object-serialization/scala/security.md new file mode 100644 index 0000000000..f5f40c61c7 --- /dev/null +++ b/docs/object-serialization/scala/security.md @@ -0,0 +1,74 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Scala reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Scala uses the Java configuration surface. Keep class registration enabled for production +and any untrusted payload source: + +```scala +val fory = ForyScala.builder() + .requireClassRegistration(true) + .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .withMaxUnbackedContainerItems(8192) + .withMaxTypeFields(512) + .withMaxTypeMetaBytes(4096) + .build() +``` + +Security-related configuration: + +- Keep `requireClassRegistration(true)` and register application classes or generated modules. +- Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. +- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep `withMaxUnbackedContainerItems(...)` at `8192` unless trusted compact codecs require a + larger root allowance. Zero rejects every unbacked item. +- Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits + at their defaults unless the data is not malicious and a trusted peer sends larger metadata or + many schema versions. +- Follow [Java Security](../java/security.md) for allow-listing and unknown-class + controls. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the Scala option reference and +[Java Type Registration](../java/type-registration.md) for the underlying registration API. diff --git a/docs/object-serialization/security.md b/docs/object-serialization/security.md deleted file mode 100644 index 28789dc466..0000000000 --- a/docs/object-serialization/security.md +++ /dev/null @@ -1,210 +0,0 @@ ---- -title: Object Serialization Security -sidebar_position: 4 -id: security -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This guide defines the trust boundary and safe operating model for Fory binary object serialization in xlang and native mode. Contributor-facing classification rules live in the [deserialization security model](deserialization-security-model.md). - -Fory is an in-process serialization library. Applications link Fory into their -own process, configure serializers and type policies, and call Fory APIs to -serialize application-owned objects or deserialize encoded Fory data. Fory does -not provide a standalone network service, daemon, authentication system, or -transport protocol. - -## Trust Boundaries - -Fory's primary security boundary is encoded bytes or streams passed to -deserialization APIs from untrusted or partially trusted sources. The embedding -application owns where those bytes come from and which Fory configuration, -registered types, schemas, and policies are used to read them. - -The adversary model for untrusted deserialization is a sender that can craft -encoded bytes or stream behavior presented to a Fory read API. It does not assume -the sender can change the embedding application's Fory configuration, registered -type set, `TypeChecker` or equivalent allow-list policy, schema definitions, -classloader, or other active policy objects unless the application itself exposes -those controls. - -Fory security boundaries include: - -- Runtime safety, including avoiding crashes, panics, undefined behavior, and - out-of-bounds memory access. -- Resource ownership, including memory, CPU progress, stream buffers, native - allocations, callbacks, and retained read-side state. -- Explicit Fory policy checks, such as class, type, function, method, - registration, or deserialization policies that restrict what may be - materialized. -- Cleanup boundaries, where state created during a failed root operation must - not leak into later operations. - -Runtime serializer code generation and JIT compilation are not paths for -executing encoded input. They operate on types and schemas after the active -registration check, `TypeChecker`, schema check, or policy check has accepted the -type surface. When class registration is disabled, `TypeChecker` or an -equivalent allow-list policy is the relevant gate. Generated serializer code is -derived from checked type descriptors rather than from attacker-controlled byte -contents. - -The [deserialization security model](deserialization-security-model.md) defines how to -classify these boundaries for untrusted deserialization paths. - -## Non-Goals - -Fory does not provide: - -- Encoded-data authenticity, integrity, confidentiality, signing, MACs, or - encryption. -- Transport security or protection for bytes while they are stored or moved - outside Fory. -- Application-level authorization or validation for the business meaning of a - successfully deserialized value. -- A sandbox for user-registered classes, functions, constructors, setters, - finalizers, or other application-owned logic. - -Applications that receive Fory data from untrusted sources should authenticate -or integrity-check those bytes before passing them to Fory when authenticity or -tamper resistance matters. - -## Downstream Responsibilities - -Applications are responsible for: - -- Choosing whether a byte source is trusted enough for the configured - deserialization mode. -- Keeping class or type registration enabled for untrusted data unless another - explicit Fory policy owns the accepted type surface. -- Registering only types and serializers that are safe for the application's - trust boundary. -- Configuring depth and resource limits for the largest data shape the - application intends to accept. -- Treating cross-language peers and schemas as part of the application's trust - relationship. - -Disabling registration or using dynamic deserialization on trusted data is a -configuration choice. For untrusted data, bypassing an explicit Fory policy, -crashing, leaking resources, retaining attacker-controlled state, or allocating -disproportionately remains security-relevant as described in the -[deserialization security model](deserialization-security-model.md). - -## Resource Limits - -### Depth Limits - -Set the runtime's depth limit to the deepest graph the application deliberately -accepts. Some runtimes apply this limit to every nested value; others apply a -separate dynamic-object depth limit. Use the selected runtime's configuration -page for its exact scope and default. A depth limit prevents excessively nested -input from turning into unbounded recursion, but it is not a byte or memory -quota. - -### Graph Memory Limit - -`maxGraphMemoryBytes`, or the runtime-equivalent option, is an approximate gate -for graph owners materialized by one root deserialization operation. The fixed -default is 128 MiB, and explicit values must be positive. Each root operation -starts with the full configured budget, including after a failed read. - -The budget covers runtime-owned collections, maps, arrays, structs, and objects -according to each implementation's storage model. It is not exact heap -accounting, an input-size limit, or a replacement for readable-byte checks. -Actual process memory may be higher. Keep external body or file-size limits at -the boundary that receives the bytes. - -### Remote Schema Metadata Limits - -Compatible mode may receive remote metadata (`TypeDef` or `TypeMeta`) for types that are not already -known by the reader. Fory limits how many distinct remote metadata versions can be accepted, and -also limits the size of each received metadata body: - -- `maxSchemaVersionsPerType`: maximum accepted remote metadata versions for one logical type. The - default is `10`. -- `maxAverageSchemaVersionsPerType`: average accepted remote metadata versions across all accepted - remote types. The default is `3`; the effective global floor is `8192` metadata entries. -- `maxTypeFields`: maximum fields declared by one received struct metadata body. The default is - `512`. -- `maxTypeMetaBytes`: maximum encoded metadata body bytes for one received TypeDef or TypeMeta body, - excluding the 8-byte header and any extended-size varint. The default is `4096`. - -These limits are resource protections. They do not change wire format, registration requirements, -dynamic type loading, unknown-type handling, or schema-evolution compatibility. - -Raise these values only when a known peer deliberately sends larger metadata or -many schema versions. - -### Count-Driven Container Work Limit - -Every runtime limits collection elements and map entries whose repeated read -bodies do not consume proportional input. The default root allowance is `8192`. -Zero is a strict limit, and negative values are rejected. Raise the limit only -for trusted payloads that intentionally use compact zero-byte element codecs or -empty Struct bodies. This is a reader resource limit and does not change the -wire format or writer behavior. - -## Configure a Runtime - -Keep registration enabled for untrusted input, choose the wire mode explicitly, -and set limits to values derived from the endpoint's accepted models. A minimal -Java boundary looks like this: - -```java -Fory fory = - Fory.builder() - .withXlang(true) - .requireClassRegistration(true) - .withMaxDepth(50) - .withMaxGraphMemoryBytes(128L * 1024 * 1024) - .withMaxUnbackedContainerItems(8192) - .build(); -``` - -Register only the application types that the endpoint accepts. If registration -is disabled, configure the runtime's explicit type checker or allow-list before -reading external data. - -Exact option names, defaults, and mode-specific behavior belong to the runtime -configuration guides: - -| Runtime | Configuration | -| --------------------- | ------------------------------------------------------- | -| Java | [Java configuration](java/configuration.md) | -| Python | [Python configuration](python/configuration.md) | -| C++ | [C++ configuration](cpp/configuration.md) | -| Go | [Go configuration](go/configuration.md) | -| Rust | [Rust configuration](rust/configuration.md) | -| JavaScript/TypeScript | [JavaScript configuration](javascript/configuration.md) | -| C# | [C# configuration](csharp/configuration.md) | -| Swift | [Swift configuration](swift/configuration.md) | -| Dart | [Dart configuration](dart/configuration.md) | -| Scala | [Scala configuration](scala/configuration.md) | -| Kotlin | [Kotlin configuration](kotlin/configuration.md) | - -## Verify the Boundary - -Add negative tests alongside the normal round trip. Verify that the configured -reader rejects: - -- an unregistered or disallowed application type; -- a graph deeper than the accepted model; -- a graph that exceeds the configured memory budget; -- excessive remote schema versions or metadata size in compatible xlang mode; -- excessive count-driven container work; and -- a malformed root followed by a valid root on the same reusable runtime. - -Also verify the application's external authentication, integrity, request-size, -timeout, and domain-validation controls independently of Fory. diff --git a/docs/object-serialization/swift/xlang.md b/docs/object-serialization/swift/basic-serialization.md similarity index 55% rename from docs/object-serialization/swift/xlang.md rename to docs/object-serialization/swift/basic-serialization.md index b49b2e798e..1f1eaadb3f 100644 --- a/docs/object-serialization/swift/xlang.md +++ b/docs/object-serialization/swift/basic-serialization.md @@ -1,7 +1,7 @@ --- -title: Xlang Serialization +title: Basic Serialization sidebar_position: 1 -id: xlang +id: basic-serialization license: | Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with @@ -19,17 +19,177 @@ license: | limitations under the License. --- +This page covers object graph serialization and core API usage in the default xlang mode for Fory Swift. + +## Object Graph Serialization + +Use `@ForyStruct`, `@ForyEnum`, or `@ForyUnion`, register types, then serialize and deserialize. + +```swift +import Foundation +import Fory + +@ForyStruct +struct Address: Equatable { + var street: String = "" + var zip: Int32 = 0 +} + +@ForyStruct +struct Person: Equatable { + var id: Int64 = 0 + var name: String = "" + var nickname: String? = nil + var tags: Set = [] + var scores: [Int32] = [] + var addresses: [Address] = [] + var metadata: [Int8: Int32?] = [:] +} + +let fory = Fory() +try fory.register(Address.self, id: 100) +try fory.register(Person.self, id: 101) + +let person = Person( + id: 42, + name: "Alice", + nickname: nil, + tags: ["swift", "xlang"], + scores: [10, 20, 30], + addresses: [Address(street: "Main", zip: 94107)], + metadata: [1: 100, 2: nil] +) + +let data = try fory.serialize(person) +let decoded: Person = try fory.deserialize(data) +assert(decoded == person) +``` + +## Working with Existing Buffers + +Append serialized bytes to an existing `Data` and deserialize from `ByteBuffer`. + +```swift +var output = Data() +try fory.serialize(person, to: &output) + +let inputBuffer = ByteBuffer(data: output) +let fromBuffer: Person = try fory.deserialize(from: inputBuffer) +assert(fromBuffer == person) +``` + +## Selecting a Serializer + +A type that implements `Serializer` with `Target == Self` selects itself: + +```swift +let data = try fory.serialize(person) +let decoded: Person = try fory.deserialize(data) +``` + +This implicit selection composes through generated fields and ordinary +optionals, arrays, sets, and dictionaries. It also applies when an application +intentionally gives an external type one retroactive self-target conformance. + +When a separate serializer targets the value, select it with `with`: + +```swift +try fory.register(UserSerializer.self, id: 200) + +let data = try fory.serialize( + externalUser, + with: UserSerializer.self +) +let decoded = try fory.deserialize( + data, + with: UserSerializer.self +) +``` + +The same selection works with existing buffers: + +```swift +var output = Data() +try fory.serialize( + externalUser, + with: UserSerializer.self, + to: &output +) + +let input = ByteBuffer(data: output) +let decoded = try fory.deserialize( + from: input, + with: UserSerializer.self +) +``` + +See [External-Type Serialization](external-types.md) for structural +serializers and recursive carrier roots. See +[Custom Serializers](custom-serializers.md) for serializers implemented +directly by a type, retroactive conformances, and separate custom serializers. + +## Built-in Supported Types + +### Primitive and scalar + +- `Bool` +- `Int8`, `Int16`, `Int32`, `Int64`, `Int` +- `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` +- `Float`, `Double` +- `String` +- `Data` + +### Date and time + +- `Date` +- `LocalDate` +- `Duration` + +Use `Date` for timestamp values and `LocalDate` for day-only dates. `LocalDate` +supports epoch-day and `Date` conversions through `fromEpochDay(_:)`, +`toEpochDay()`, `init(utcDate:)`, and `toUTCDate()`. + +### Collections + +- Optionals and arrays whose values directly implement `Serializer` +- Sets whose elements directly implement `Serializer` and are `Hashable` +- Dictionaries whose keys and values directly implement `Serializer`, with + `Hashable` keys + +Children that use a separate serializer compose with: + +- `OptionalSerializer` +- `ArraySerializer` +- `SetSerializer` +- `DictionarySerializer` + +### Dynamic + +- `Any` and `AnyObject` +- `AnyHashable` +- Arbitrary application protocol values +- Supported heterogeneous arrays and dictionaries + +`Any` and `AnyObject` roots use direct root APIs. Arbitrary application +protocol roots and dynamic values nested in carriers use explicit `with:` +selection. +See [Polymorphism and Dynamic Types](polymorphism.md). + +## Cross-Language Interoperability + +The default xlang format is shared by all Fory runtimes. The following sections cover its cross-language type mapping, type identity, and interoperability requirements. + Fory Swift can exchange payloads with other Fory implementations using the xlang protocol. -## Recommended Xlang Configuration +### Recommended Xlang Configuration ```swift let fory = Fory() ``` -## Register Types with Shared Identity +### Register Types with Shared Identity -### ID-based registration +#### ID-based registration ```swift @ForyStruct @@ -42,13 +202,13 @@ let fory = Fory() try fory.register(Order.self, id: 100) ``` -### Name-based registration +#### Name-based registration ```swift try fory.register(Order.self, name: "com.example.Order") ``` -## Xlang Rules +### Xlang Rules - Keep type registration mapping consistent across languages - Keep compatible mode enabled when independently evolving schemas. Swift enables it by default. @@ -58,7 +218,7 @@ try fory.register(Order.self, name: "com.example.Order") intentional retroactive self-target conformance for a type owned by another module -## Lists and Dense Arrays +### Lists and Dense Arrays Swift `Array` fields map to Fory `list` unless field metadata explicitly requests dense `array`. Use `array` only for one-dimensional bool or @@ -84,7 +244,7 @@ numeric data. An array that uses a separate element serializer still uses normal list encoding. Use `@ArrayField` only for supported dense bool or numeric arrays. -## External Targets +### External Targets External structural serializers produce the same xlang STRUCT, ENUM, or UNION schema and value bytes as an equivalent ordinary Swift model: @@ -110,7 +270,7 @@ Swift has no native serialization mode. A known `@ForyUnion` case has zero or one associated value. Use a struct payload for a union alternative with multiple logical fields. -## Swift IDL Workflow +### Swift IDL Workflow Generate Swift models directly from Fory IDL/Proto/FBS inputs: @@ -135,7 +295,7 @@ let payload = try fory.serialize(book) let decoded: Addressbook.AddressBook = try fory.deserialize(payload) ``` -### Run Swift IDL Integration Tests +#### Run Swift IDL Integration Tests ```bash cd integration_tests/idl_tests @@ -144,7 +304,7 @@ cd integration_tests/idl_tests This runs Swift roundtrip matrix tests and Java peer roundtrip checks (`IDL_PEER_LANG=swift`). -## Debugging Xlang Tests +### Debugging Xlang Tests Enable debug output when running xlang tests: @@ -152,7 +312,7 @@ Enable debug output when running xlang tests: ENABLE_FORY_DEBUG_OUTPUT=1 FORY_SWIFT_JAVA_CI=1 mvn -T16 test -Dtest=org.apache.fory.xlang.SwiftXlangTest ``` -## First round trip +### First round trip ```swift import Fory @@ -175,7 +335,7 @@ print("\(result.name) \(result.age)") For more cross-language rules and examples, see: -- [Cross-Language Serialization Guide](../xlang/index.md) +- [Cross-Language Interoperability](../xlang.md) - [Java Guide](../java/index.md) - [Python Guide](../python/index.md) - [Dart Guide](../dart/index.md) diff --git a/docs/object-serialization/swift/configuration.md b/docs/object-serialization/swift/configuration.md index cffcedab15..1d3bae58ad 100644 --- a/docs/object-serialization/swift/configuration.md +++ b/docs/object-serialization/swift/configuration.md @@ -157,13 +157,4 @@ let fory = Fory(compatible: false) ## Security -Security-related configuration: - -- Register only the expected generated models before deserializing untrusted payloads. -- Use `checkClassVersion` with `compatible: false` for intentional same-schema payloads. -- Set `maxDepth` for the largest dynamic `Any` nesting depth your service accepts. -- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, class, and - object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input - bytes. -- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a - trusted peer sends larger metadata or many schema versions. +See [Swift Security](security.md) for trust boundaries, safe reader configuration, and verification. diff --git a/docs/object-serialization/swift/core-api.md b/docs/object-serialization/swift/core-api.md deleted file mode 100644 index bdff028dde..0000000000 --- a/docs/object-serialization/swift/core-api.md +++ /dev/null @@ -1,176 +0,0 @@ ---- -title: Basic Serialization -sidebar_position: 3 -id: core-api -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers object graph serialization and core API usage in Swift. - -## Object Graph Serialization - -Use `@ForyStruct`, `@ForyEnum`, or `@ForyUnion`, register types, then serialize and deserialize. - -```swift -import Foundation -import Fory - -@ForyStruct -struct Address: Equatable { - var street: String = "" - var zip: Int32 = 0 -} - -@ForyStruct -struct Person: Equatable { - var id: Int64 = 0 - var name: String = "" - var nickname: String? = nil - var tags: Set = [] - var scores: [Int32] = [] - var addresses: [Address] = [] - var metadata: [Int8: Int32?] = [:] -} - -let fory = Fory() -try fory.register(Address.self, id: 100) -try fory.register(Person.self, id: 101) - -let person = Person( - id: 42, - name: "Alice", - nickname: nil, - tags: ["swift", "xlang"], - scores: [10, 20, 30], - addresses: [Address(street: "Main", zip: 94107)], - metadata: [1: 100, 2: nil] -) - -let data = try fory.serialize(person) -let decoded: Person = try fory.deserialize(data) -assert(decoded == person) -``` - -## Working with Existing Buffers - -Append serialized bytes to an existing `Data` and deserialize from `ByteBuffer`. - -```swift -var output = Data() -try fory.serialize(person, to: &output) - -let inputBuffer = ByteBuffer(data: output) -let fromBuffer: Person = try fory.deserialize(from: inputBuffer) -assert(fromBuffer == person) -``` - -## Selecting a Serializer - -A type that implements `Serializer` with `Target == Self` selects itself: - -```swift -let data = try fory.serialize(person) -let decoded: Person = try fory.deserialize(data) -``` - -This implicit selection composes through generated fields and ordinary -optionals, arrays, sets, and dictionaries. It also applies when an application -intentionally gives an external type one retroactive self-target conformance. - -When a separate serializer targets the value, select it with `with`: - -```swift -try fory.register(UserSerializer.self, id: 200) - -let data = try fory.serialize( - externalUser, - with: UserSerializer.self -) -let decoded = try fory.deserialize( - data, - with: UserSerializer.self -) -``` - -The same selection works with existing buffers: - -```swift -var output = Data() -try fory.serialize( - externalUser, - with: UserSerializer.self, - to: &output -) - -let input = ByteBuffer(data: output) -let decoded = try fory.deserialize( - from: input, - with: UserSerializer.self -) -``` - -See [External-Type Serialization](external-types.md) for structural -serializers and recursive carrier roots. See -[Custom Serializers](custom-serializers.md) for serializers implemented -directly by a type, retroactive conformances, and separate custom serializers. - -## Built-in Supported Types - -### Primitive and scalar - -- `Bool` -- `Int8`, `Int16`, `Int32`, `Int64`, `Int` -- `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` -- `Float`, `Double` -- `String` -- `Data` - -### Date and time - -- `Date` -- `LocalDate` -- `Duration` - -Use `Date` for timestamp values and `LocalDate` for day-only dates. `LocalDate` -supports epoch-day and `Date` conversions through `fromEpochDay(_:)`, -`toEpochDay()`, `init(utcDate:)`, and `toUTCDate()`. - -### Collections - -- Optionals and arrays whose values directly implement `Serializer` -- Sets whose elements directly implement `Serializer` and are `Hashable` -- Dictionaries whose keys and values directly implement `Serializer`, with - `Hashable` keys - -Children that use a separate serializer compose with: - -- `OptionalSerializer` -- `ArraySerializer` -- `SetSerializer` -- `DictionarySerializer` - -### Dynamic - -- `Any` and `AnyObject` -- `AnyHashable` -- Arbitrary application protocol values -- Supported heterogeneous arrays and dictionaries - -`Any` and `AnyObject` roots use direct root APIs. Arbitrary application -protocol roots and dynamic values nested in carriers use explicit `with:` -selection. -See [Polymorphism and Dynamic Types](polymorphism.md). diff --git a/docs/object-serialization/swift/index.md b/docs/object-serialization/swift/index.md index 718bea7860..135c02a5e6 100644 --- a/docs/object-serialization/swift/index.md +++ b/docs/object-serialization/swift/index.md @@ -52,8 +52,7 @@ targets: [ ## Guide Contents - [Configuration](configuration.md) -- [Basic Serialization](core-api.md) -- [Xlang Serialization](xlang.md) +- [Basic Serialization](basic-serialization.md) - [Schema Metadata](schema-metadata.md) - [Type Registration](type-registration.md) - [External-Type Serialization](external-types.md) @@ -83,3 +82,6 @@ let output: User = try fory.deserialize(data) assert(input == output) ``` + +Before decoding bytes from outside the application trust boundary, read +[Swift Security](security.md). diff --git a/docs/object-serialization/swift/security.md b/docs/object-serialization/swift/security.md new file mode 100644 index 0000000000..a03e6c58b5 --- /dev/null +++ b/docs/object-serialization/swift/security.md @@ -0,0 +1,56 @@ +--- +title: Security +sidebar_position: 99 +id: security +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Use this page when a Swift reader accepts bytes from outside the application's trust boundary. +Fory reconstructs application values; it does not authenticate the sender, protect transport +integrity, or decide whether a valid value is authorized for a business operation. + +## Application boundary + +Before deserialization: + +- Authenticate the sender and protect message integrity at the transport or storage layer. +- Enforce request or file size, timeout, and concurrency limits outside Fory. +- Register only the application types the endpoint accepts and configure the reader before its + first root operation. +- Validate the deserialized value against application authorization and domain rules before use. + +## Runtime safeguards + +Security-related configuration: + +- Register only the expected generated models before deserializing untrusted payloads. +- Use `checkClassVersion` with `compatible: false` for intentional same-schema payloads. +- Set `maxDepth` for the largest dynamic `Any` nesting depth your service accepts. +- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, class, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. +- Keep the remote schema metadata limits at their defaults unless the data is not malicious and a + trusted peer sends larger metadata or many schema versions. + +## Verification + +Add negative tests for the boundary as well as normal round trips. Verify that the configured reader +rejects unexpected application types, excessive nesting, resource-limit violations, and malformed +input. After a failed read, verify that a valid root can still be read with the reusable runtime. + +See [Configuration](configuration.md) for the complete option reference and +[Type Registration](type-registration.md) for the runtime's registration API. diff --git a/docs/object-serialization/xlang.md b/docs/object-serialization/xlang.md new file mode 100644 index 0000000000..bfd8ea7b4f --- /dev/null +++ b/docs/object-serialization/xlang.md @@ -0,0 +1,367 @@ +--- +title: Xlang Serialization +sidebar_position: 2 +id: xlang +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Xlang is Fory's default object serialization mode. It uses one portable binary format across Java, +Python, C++, Go, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin. Each runtime's +[Basic Serialization](#runtime-guides) page owns its API and model examples; this page explains the +rules that peers must share. + +Read [Core Concepts](core-concepts.md) first for the object graph, schema, reference, and +polymorphism concepts shared by xlang and native modes. + +## Overview + +Use xlang serialization when bytes cross runtime boundaries, including polyglot services, data +pipelines, and frontend/backend communication. It provides: + +- Direct serialization of native language models without requiring an IDL. +- Coordinated numeric or named identities for application types. +- Compatible schema evolution for independently deployed peers. +- Optional shared-reference and circular-reference preservation. +- Polymorphic values when every concrete type has a portable mapping. +- Out-of-band buffers for large binary and numeric data where the runtime supports them. + +Use [Native Serialization](native.md) instead when every writer and reader uses the same supported +runtime and the object graph needs language-specific behavior such as Java serialization hooks or +Python pickle-compatible objects. + +### Supported Runtimes + +| Runtime | Package or target | Modes | +| --------------------- | ------------------------------------------ | ------------ | +| Java | `org.apache.fory:fory-core` | xlang/native | +| Python | `pyfory` | xlang/native | +| C++ | Fory C++ CMake or Bazel target | xlang/native | +| Go | `github.com/apache/fory/go/fory` | xlang/native | +| Rust | `fory` crate | xlang/native | +| JavaScript/TypeScript | `@apache-fory/core` | xlang | +| C# | `Apache.Fory` | xlang | +| Swift | `Fory` Swift Package Manager target | xlang | +| Dart | `fory` package | xlang | +| Scala | `org.apache.fory:fory-scala` | xlang/native | +| Kotlin | `org.apache.fory:fory-kotlin` and Java API | xlang/native | + +### First Cross-Language Round Trip + +Register the same logical type identity and compatible fields on every peer. The following example +uses a shared type name. + +Rust producer: + +```rust +use fory::{Fory, ForyStruct}; + +#[derive(ForyStruct)] +struct Person { + name: String, + age: i32, +} + +let mut fory = Fory::builder().xlang(true).build(); +fory.register_by_name::("example.Person").unwrap(); + +let person = Person { + name: "Alice".to_string(), + age: 30, +}; +let bytes = fory.serialize(&person).unwrap(); +``` + +Python consumer: + +```python +from dataclasses import dataclass +import pyfory + +@dataclass +class Person: + name: str + age: pyfory.Int32 + +fory = pyfory.Fory(xlang=True) +fory.register_type(Person, name="example.Person") +person = fory.deserialize(bytes_from_rust) +``` + +Although xlang is the default, examples often select it explicitly so the transport contract is +visible in application code. + +### Native Models or Fory IDL + +| Approach | Use when | +| --------------------------------- | ---------------------------------------------------------------------- | +| Native language models | The contract is small and teams want to start without a compiler step | +| [Fory IDL](../compiler/index.md) | Many messages or teams need one schema and generated models | +| Native first, then migrate to IDL | A small contract is becoming a long-lived, multi-team service boundary | + +A minimal Fory IDL message looks like this: + +```protobuf +package example; + +message Person { + string name = 1; + int32 age = 2; + optional string email = 3; +} +``` + +Generate the required runtime targets with `foryc`; generated models use consistent field and type +metadata across those targets. + +## Type System and Type Identity + +### Built-in and Custom Types + +Primitive numeric values, strings, binary values, temporal values, lists, sets, maps, dense numeric +arrays, enums, structs, and unions have shared xlang schemas. Built-in values do not require user +registration. Application structs, enums, unions, and extension types require a coordinated +identity. + +The normative [type mapping](../specification/xlang_type_mapping.md) defines the exact host-language +carrier for every xlang type. Important cases include: + +- Python uses markers such as `pyfory.Int32`, `pyfory.Float16`, and `pyfory.BFloat16` when the native + Python type does not express the required width. +- Java, Dart, and other runtimes use annotations or schema metadata where one host type can represent + multiple xlang types. +- Reduced-precision `float16` and `bfloat16` values and dense arrays use runtime-specific carriers. +- `list` and dense `array` are distinct schemas. In compatible mode, a direct struct field may + adapt between a list and dense bool/numeric array when the element domain is compatible and the + actual list contains no unrepresentable null or reference-tracked element. + +Use the mapping specification instead of inferring compatibility from similar host-language names. + +### Coordinate Type Identity + +Every peer must register a custom type with either the same numeric ID or the same namespace and type +name. Numeric IDs produce smaller metadata; names are easier to coordinate across independently +owned services. Do not register one peer by ID and another by name for the same contract. + +Registration must happen before the first root serialization or deserialization operation. Keep a +small contract registry or use generated Fory IDL modules when multiple teams own the peers. + +### Static and Dynamic Fields + +A statically known field uses its declared serializer without writing a concrete runtime type. A +dynamic field carries enough type information to select the concrete registered type. Dynamic +metadata is needed for interfaces, abstract types, trait objects, and other polymorphic positions; +it is unnecessary for primitives and exact final types. + +| Runtime | Dynamic field model | +| ------- | ----------------------------------------------------------------- | +| Java | `@ForyField(dynamic = ...)` controls automatic or forced metadata | +| Python | `pyfory.field(dynamic=...)` controls object-field metadata | +| C++ | `fory::F(...).dynamic(...)` overrides automatic detection | +| Go | Interface fields express dynamic values | +| Rust | Trait-object carriers express dynamic values | + +Writing dynamic metadata costs space and type-resolution work. Disable it only when the field can +never contain another concrete type. Exact annotation and registration examples belong to each +runtime's schema metadata, type registration, and polymorphism pages. + +## Nullability and Reference Tracking + +Nullability and reference tracking solve different problems: + +| Concern | Purpose | +| ------------------ | ------------------------------------------------------------- | +| Nullability | Allows a field or value position to contain no value | +| Reference tracking | Preserves repeated object identity and supports object cycles | + +The wire framing is defined by the +[xlang serialization specification](../specification/xlang_serialization_spec.md). Applications +should configure the semantic behavior through runtime APIs rather than depend on flag values. + +### Nullability + +Xlang struct fields are non-nullable by default. Non-nullable fields are smaller and make required +data explicit. Nullable or optional carriers include Java boxed/annotated values, Python +`Optional[T]`, C++ `std::optional`, Go pointers, Rust `Option`, and Scala `Option[T]`. + +Keep corresponding fields consistently nullable across peers. In same-schema mode, changing +nullability changes the schema and is incompatible. Compatible mode supports the documented +nullable and scalar adaptations, but a remote null still cannot be materialized into a local type +that has no valid null or missing-value behavior. + +### Shared and Circular References + +Enable reference tracking when the graph contains the same object more than once or contains a +cycle. Leave it disabled for value-shaped data to avoid identity-table overhead. + +```java +import org.apache.fory.Fory; +import org.apache.fory.annotation.Ref; + +public class Node { + public String value; + @Ref public Node next; +} + +Node first = new Node(); +Node second = new Node(); +first.next = second; +second.next = first; + +Fory fory = Fory.builder() + .withXlang(true) + .withRefTracking(true) + .build(); +``` + +Global reference tracking enables the runtime mechanism; field metadata selects which positions +participate. Common field-level controls are Java and Scala `@Ref`, Go `fory:"ref"` tags, Rust +`#[fory(ref = true)]`, and C++ smart-pointer or `fory::F().ref()` metadata. Consult the runtime guide +because default tracking differs by carrier and language. + +Reference support also follows the host ownership model. For example, Rust can preserve supported +shared-reference carriers, while cycles require representable ownership and weak-reference shapes. + +## Polymorphism + +Xlang polymorphism preserves a value's concrete registered type when its declared field, collection +element, or root type is broader. Every receiving peer must: + +1. Register the same concrete type identity. +2. Provide a compatible field schema for that concrete type. +3. Mark or model the position as dynamic when the runtime cannot infer it. +4. Use a concrete type that has a portable xlang mapping. + +Host-language inheritance alone does not make a type portable. If a shape has no xlang mapping, use +the runtime's native mode for same-language traffic or define a portable model. See the runtime +polymorphism pages for interfaces, trait objects, unions, and generated-code syntax. + +## Schema Evolution + +Compatible mode is the xlang default. It carries schema metadata so independently deployed readers +can tolerate supported field additions, removals, reordering, and documented compatible type +adaptations. + +Keep compatible mode when peers may deploy separately. Select same-schema mode only when every +reader and writer uses the same: + +- Type identity and field IDs or names. +- Field types and nested generic shapes. +- Nullability and reference metadata. +- Polymorphic alternatives. + +Same-schema mode reduces metadata and payload size, but any mismatch can produce a schema hash or +type error. Generated Fory IDL models make exact coordination easier when all peers are released +together. Normative compatibility behavior lives in the +[xlang serialization specification](../specification/xlang_serialization_spec.md). + +## Zero-Copy Serialization + +Some runtimes can move large binary or numeric buffers out of the main serialized byte stream. This +avoids copying those buffers into one contiguous payload. + +The transport flow is: + +1. Serialize the object graph and collect selected buffer objects through a callback. +2. Send the main metadata bytes and the collected buffers separately. +3. Provide the buffers in the same order during deserialization. + +Java: + +```java +Collection objects = new ArrayList<>(); +byte[] metadata = fory.serialize(value, object -> !objects.add(object)); +List buffers = objects.stream() + .map(BufferObject::toBuffer) + .toList(); +Object decoded = fory.deserialize(metadata, buffers); +``` + +Python: + +```python +objects = [] +metadata = fory.serialize(value, buffer_callback=objects.append) +buffers = [obj.to_buffer() for obj in objects] +decoded = fory.deserialize(metadata, buffers=buffers) +``` + +Go exposes the equivalent callback-buffer flow through its serialization and buffer APIs. Use the +runtime documentation for the current method names and supported buffer carriers. + +Out-of-band serialization helps when buffers are large and the transport can send them without an +additional copy. For small arrays, callback and multi-buffer transport overhead may cost more than +copying. The application owns buffer ordering, lifetime, and transport framing. See +[Python Out-of-Band Serialization](python/out-of-band.md) for Python and NumPy details. + +## Troubleshooting + +| Symptom | Likely cause | Resolution | +| ---------------------------------------- | --------------------------------------------------- | -------------------------------------------------------------- | +| Type is not registered | Registration missing or performed too late | Register every custom type before the first root operation | +| Type ID or name mismatch | Peers use different identities | Use the same numeric ID or the same namespace and type name | +| Integer overflow or float precision loss | Host carriers use different numeric widths | Follow the type mapping and use explicit width metadata | +| Fields decode incorrectly | Field IDs, names, or types differ | Align field metadata or regenerate all peers from the same IDL | +| Stack overflow on a cyclic graph | Reference tracking is disabled | Enable global and field-level reference tracking | +| Shared objects become duplicates | The value position does not track references | Enable reference tracking for that carrier or field | +| Unsupported host type | The type has no portable xlang representation | Use a portable model or native mode for same-language traffic | +| Schema/hash mismatch | Same-schema peers have different schemas | Align every peer or restore compatible mode | +| Failure after an upgrade | Peers run incompatible protocol versions | Align supported Fory versions and review release notes | +| Payload rejected immediately | One peer wrote native bytes and another reads xlang | Keep all peers on xlang for a cross-language contract | + +### Diagnostic Checklist + +1. Confirm that every peer uses xlang mode and a mutually supported Fory version. +2. Compare the registered type identity, field IDs or names, numeric widths, nullability, and + reference metadata. +3. Reproduce a same-runtime round trip before testing the cross-runtime direction. +4. Test both directions for every language pair used in production. +5. Reduce the value to one type and field, then add fields back until the mismatch appears. +6. Inspect the runtime-specific troubleshooting page for generated-code, platform, or API errors. + +When diagnosing binary layout, use the specifications and runtime debug facilities. Do not treat a +hex dump or internal flag value as a stable application API. + +## Runtime Guides + +- [Java](java/basic-serialization.md#cross-language-interoperability) +- [Python](python/basic-serialization.md#cross-language-interoperability) +- [C++](cpp/basic-serialization.md#cross-language-interoperability) +- [Go](go/basic-serialization.md#cross-language-interoperability) +- [Rust](rust/basic-serialization.md#cross-language-interoperability) +- [JavaScript/TypeScript](javascript/basic-serialization.md#cross-language-interoperability) +- [C#](csharp/basic-serialization.md#cross-language-interoperability) +- [Swift](swift/basic-serialization.md#cross-language-interoperability) +- [Dart](dart/basic-serialization.md#cross-language-interoperability) +- [Scala](scala/basic-serialization.md#cross-language-interoperability) +- [Kotlin](kotlin/basic-serialization.md#cross-language-interoperability) + +## Related Documentation + +- [Xlang Serialization Format](../specification/xlang_serialization_spec.md) — normative wire format +- [Xlang Type Mapping](../specification/xlang_type_mapping.md) — exact runtime carrier mappings +- [Fory IDL and Compiler](../compiler/index.md) — schema-first models and code generation +- [Getting Started](../start/index.md) — installation and first serialization for each runtime +- [Row Format](../row-format/index.md) — random-access analytical rows for trusted data + +## Operational Best Practices + +1. Coordinate one type identity and field contract across every peer. +2. Keep compatible mode unless all readers and writers deploy the same schema together. +3. Enable reference tracking only for identity-bearing or cyclic graphs. +4. Reuse configured Fory instances instead of rebuilding them for each operation. +5. Validate every production language pair in both directions before deployment. +6. Prefer Fory IDL once a contract spans many messages, services, or independently owned teams. diff --git a/docs/object-serialization/xlang/_category_.json b/docs/object-serialization/xlang/_category_.json deleted file mode 100644 index d2418b5e5e..0000000000 --- a/docs/object-serialization/xlang/_category_.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "position": 1, - "label": "Xlang", - "collapsible": true, - "collapsed": true -} diff --git a/docs/object-serialization/xlang/index.md b/docs/object-serialization/xlang/index.md deleted file mode 100644 index 2777ce9d59..0000000000 --- a/docs/object-serialization/xlang/index.md +++ /dev/null @@ -1,192 +0,0 @@ ---- -title: Xlang Serialization Guide -sidebar_position: 0 -id: index -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Apache Fory™ xlang serialization is the default wire format for cross-language payloads. Serialize -data in one language and deserialize it in another without manual conversion. You can use direct -language model types for small contracts, or use Fory IDL and code generation when a schema-first -workflow is a better fit. - -## Features - -- **No IDL required**: Serialize objects directly with language model types. -- **Multi-language support**: Java, Python, C++, Go, Rust, - JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin interoperate through - the same xlang format. -- **Reference support**: Shared and circular references work across language boundaries when reference tracking is enabled in each peer. -- **Schema evolution**: Compatible mode is the xlang default so readers can tolerate added, removed, or reordered fields. -- **Out-of-band buffers**: Language implementations can expose zero-copy buffer paths for large binary data. -- **High performance**: Fory implementations use generated serializers, JIT serializers, or optimized code paths where available. - -## Supported Languages - -| Language | Status | Package or target | -| --------------------- | --------- | -------------------------------- | -| Java | Supported | `org.apache.fory:fory-core` | -| Python | Supported | `pyfory` | -| C++ | Supported | Bazel/CMake build | -| Go | Supported | `github.com/apache/fory/go/fory` | -| Rust | Supported | `fory` crate | -| JavaScript/TypeScript | Supported | `@apache-fory/core` | -| C# | Supported | `Apache.Fory` | -| Swift | Supported | Swift Package Manager target | -| Dart | Supported | `fory` package | -| Scala | Supported | `org.apache.fory:fory-scala` | -| Kotlin | Supported | `org.apache.fory:fory-kotlin` | - -## When to Use Xlang Mode - -Use xlang mode when: - -- Building multi-language microservices -- Creating polyglot data pipelines -- Sharing data between frontend JavaScript/TypeScript and backend services such - as Java, Python, Go, C#, Scala, or Kotlin - -Use native mode for same-language traffic in Java, Scala, Kotlin, Python, C++, -Go, or Rust: - -- All serialization/deserialization happens in the same language -- You need language-specific features such as Python pickle-style objects or Java serialization hooks -- You want native-mode payloads for same-language services - -## Quick Example - -### Java (Producer) - -```java -import org.apache.fory.*; -import org.apache.fory.config.*; - -public class Person { - public String name; - public int age; -} - -Fory fory = Fory.builder().withXlang(true).build(); -fory.register(Person.class, "example.Person"); - -Person person = new Person(); -person.name = "Alice"; -person.age = 30; -byte[] bytes = fory.serialize(person); -// Send bytes to Python, Go, Rust, etc. -``` - -### Python (Consumer) - -```python -import pyfory -from dataclasses import dataclass - -@dataclass -class Person: - name: str - age: pyfory.Int32 - -fory = pyfory.Fory(xlang=True) -fory.register_type(Person, name="example.Person") - -# Receive bytes from Java -person = fory.deserialize(bytes_from_java) -print(f"{person.name}, {person.age}") # Alice, 30 -``` - -## Fory IDL - -For schema-first projects, Fory also provides **Fory IDL** and code generation. - -- Compiler docs: [Fory IDL Overview](../../compiler/index.md) -- Best for large multi-language message contracts and long-lived schemas - -### Minimal IDL Example - -Create `person.fdl`: - -```protobuf -package example; - -message Person { - string name = 1; - int32 age = 2; - optional string email = 3; -} -``` - -Generate code: - -```bash -foryc person.fdl --lang java,python,cpp,go,rust,javascript,csharp,swift,dart,scala,kotlin --output ./generated -``` - -This generates native language types with consistent field/type mappings across all targets. - -## When to Fory IDL - -| Option | Use When | Why | -| ---------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -| Native xlang types (no IDL) | You only have a few message types and want to move quickly | Avoids the integration/setup cost of introducing and operating the compiler | -| Fory IDL (schema-first + codegen) | You have many messages across multiple languages/teams/services | Provides a single contract, stronger consistency, and easier long-term evolution | -| Hybrid (start native, move to IDL) | Project starts small but message count and cross-team dependency grows | Lets you keep early velocity, then standardize once schema complexity increases | - -## Documentation - -| Topic | Description | -| --------------------------------------------------------- | ----------------------------------------------- | -| [Getting Started](../../start/index.md) | Installation and basic setup for all languages | -| [Type Mapping](../../specification/xlang_type_mapping.md) | Xlang type mapping reference | -| [Type System](type-system.md) | Built-in types and cross-runtime type behavior | -| [Type Identity](type-identity.md) | Coordinate names and numeric IDs across peers | -| [Nullability](nullability.md) | Nullable field behavior and configuration | -| [References](references.md) | Shared and circular object references | -| [Polymorphism](polymorphism.md) | Runtime type selection and registration | -| [Schema Evolution](schema-evolution.md) | Compatible and same-schema mode choices | -| [Zero-Copy](zero-copy.md) | Out-of-band serialization for large data | -| [Row Format](../../row-format/index.md) | Cache-friendly binary format with random access | -| [Troubleshooting](troubleshooting.md) | Common issues and solutions | - -## Language-Specific Guides - -For language-specific details and API reference: - -- [Java Xlang Serialization Guide](../java/xlang.md) -- [Python Xlang Serialization Guide](../python/xlang.md) -- [C++ Xlang Serialization Guide](../cpp/xlang.md) -- [Go Xlang Serialization Guide](../go/xlang.md) -- [Rust Xlang Serialization Guide](../rust/xlang.md) -- [JavaScript/TypeScript Xlang Serialization Guide](../javascript/xlang.md) -- [C# Xlang Serialization Guide](../csharp/xlang.md) -- [Swift Xlang Serialization Guide](../swift/xlang.md) -- [Dart Xlang Serialization Guide](../dart/xlang.md) -- [Scala Xlang Serialization Guide](../scala/xlang.md) -- [Kotlin Xlang Serialization Guide](../kotlin/xlang.md) - -## Specifications - -- [Xlang Serialization Specification](../../specification/xlang_serialization_spec.md) - Binary protocol details -- [Type Mapping Specification](../../specification/xlang_type_mapping.md) - Complete type mapping reference - -## Operational best practices - -1. **Use consistent type names**: Ensure all languages use the same type name or ID -2. **Enable reference tracking**: If your data has circular or shared references -3. **Reuse Fory instances**: Creating Fory is expensive; reuse instances -4. **Use type annotations**: In Python, use markers such as `pyfory.Int32` for precise type mapping -5. **Test cross-language**: Verify serialization works across all target languages diff --git a/docs/object-serialization/xlang/nullability.md b/docs/object-serialization/xlang/nullability.md deleted file mode 100644 index e7b7fe2d72..0000000000 --- a/docs/object-serialization/xlang/nullability.md +++ /dev/null @@ -1,267 +0,0 @@ ---- -title: Field Nullability -sidebar_position: 4 -id: nullability -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page explains how Fory handles field nullability in cross-language (xlang) serialization mode. - -## Default Behavior - -In xlang mode, **fields are non-nullable by default**. This means: - -- Values must always be present (non-null) -- No null flag byte is written for the field -- Serialization is more compact - -The following types are nullable by default: - -- `Optional` (Java, C++) -- Java boxed types (`Integer`, `Long`, `Double`, etc.) -- Go pointer types (`*int32`, `*string`, etc.) -- Rust `Option` -- Python `Optional[T]` -- Scala `Option[T]` - -| Field Type | Default Nullable | Null Flag Written | -| ------------------------------------------ | ---------------- | ----------------- | -| Primitives (`int`, `bool`, `float`, etc.) | No | No | -| `String` | No | No | -| `List`, `Map`, `Set` | No | No | -| Custom structs | No | No | -| Enums | No | No | -| Java boxed types (`Integer`, `Long`, etc.) | Yes | Yes | -| Go pointer types (`*int32`, `*string`) | Yes | Yes | -| `Optional` / `Option` | Yes | Yes | - -## Wire Format - -The nullable flag controls whether a **null flag byte** is written before the field value: - -``` -Non-nullable field: [value data] -Nullable field: [null_flag] [value data if not null] -``` - -Where `null_flag` is: - -- `-1` (NULL_FLAG): Value is null -- `-2` (NOT_NULL_VALUE_FLAG): Value is present - -## Nullable vs Reference Tracking - -These are related but distinct concepts: - -| Concept | Purpose | Flag Values | -| ---------------------- | ------------------------------------ | ------------------------------------------- | -| **Nullable** | Allow null values for a field | `-1` (null), `-2` (not null) | -| **Reference Tracking** | Deduplicate shared object references | `-1` (null), `-2` (not null), `≥0` (ref ID) | - -Key differences: - -- **Nullable only**: Writes `-1` or `-2` flag, no reference deduplication -- **Reference tracking**: Extends nullable semantics with reference IDs (`≥0`) for previously seen objects -- Both use the same flag byte position—ref tracking is a superset of nullable - -When `refTracking=true`, the null flag byte doubles as a ref flag: - -``` -ref_flag = -1 → null value -ref_flag = -2 → new object (first occurrence) -ref_flag >= 0 → reference to object at index ref_flag -``` - -For detailed reference tracking behavior, see [Reference Tracking](references.md). - -## Language-Specific Examples - -### Java - -```java -public class Person { - // Non-nullable by default in xlang mode - String name; // Must not be null - int age; // Primitive, always non-nullable - List tags; // Must not be null - - // Explicitly nullable - @Nullable - String nickname; // Can be null - - // Optional wrapper - nullable by default - Optional bio; // Can be empty/null -} - -Fory fory = Fory.builder() - .withXlang(true) - .build(); -fory.register(Person.class, "example.Person"); -``` - -### Python - -```python -from dataclasses import dataclass -from typing import Optional, List -import pyfory - -@dataclass -class Person: - # Non-nullable by default - name: str # Must have a value - age: pyfory.Int32 # Primitive - tags: List[str] # Must not be None - - # Optional makes it nullable - nickname: Optional[str] = None # Can be None - bio: Optional[str] = None # Can be None - -fory = pyfory.Fory(xlang=True) -fory.register_type(Person, name="example.Person") -``` - -### Rust - -```rust -use fory::{Fory, ForyStruct}; - -#[derive(ForyStruct)] -struct Person { - // Non-nullable by default - name: String, - age: i32, - tags: Vec, - - // Option is nullable - nickname: Option, // Can be None - bio: Option, // Can be None -} -``` - -### Go - -```go -type Person struct { - // Non-nullable by default - Name string - Age int32 - Tags []string - - // Pointer types for nullable fields - Nickname *string // Can be nil - Bio *string // Can be nil -} - -fory := forygo.NewFory(forygo.WithXlang(true)) -fory.RegisterStructByName(Person{}, "example.Person") -``` - -### C++ - -```cpp -struct Person { - // Non-nullable by default - std::string name; - int32_t age; - std::vector tags; - - // std::optional for nullable - std::optional nickname; - std::optional bio; -}; -FORY_STRUCT(Person, name, age, tags, nickname, bio); -``` - -## Customizing Nullability - -### Java: @Nullable Annotation - -```java -public class Config { - @Nullable - String optionalSetting; // Explicitly nullable - - String requiredSetting; // Explicitly non-nullable (default) -} -``` - -### C++: FORY_STRUCT Field Config - -```cpp -struct Config { - std::optional optional_setting; - std::string required_setting; -}; - -FORY_STRUCT(Config, - (optional_setting, fory::F(1)), - (required_setting, fory::F(2)) -); -``` - -For nullable pointer carriers, opt in with `.nullable()`: - -```cpp -struct ConfigRef { - std::shared_ptr optional_setting; - std::shared_ptr required_setting; -}; - -FORY_STRUCT(ConfigRef, - (optional_setting, fory::F(1).nullable()), - (required_setting, fory::F(2)) -); -``` - -## Null Value Handling - -When a non-nullable field receives a null value: - -| Language | Behavior | -| -------- | ---------------------------------------------------- | -| Java | Throws `NullPointerException` or serialization error | -| Python | Raises `TypeError` or serialization error | -| Rust | Compile-time error (non-Option types can't be None) | -| Go | Zero value is used (empty string, 0, etc.) | -| C++ | Default-constructed value or undefined behavior | - -## Schema Compatibility - -The nullable flag is part of the struct schema fingerprint. When compatible mode is disabled, changing a field's nullability is a **breaking change** that will cause schema version mismatch errors. - -``` -Schema A: { name: String (non-nullable) } -Schema B: { name: String (nullable) } -// These have different fingerprints when compatible mode is disabled -``` - -In compatible mode, top-level scalar fields can still be matched when their scalar type is otherwise compatible and the nullability or optional wrapper differs. Present values are read through compatible scalar conversion and must satisfy the normal lossless conversion checks. Remote null values follow the compatible-read null/default behavior for the local field. - -## Best Practices - -1. **Use non-nullable by default**: Only make fields nullable when null is a valid semantic value -2. **Use Optional/Option wrappers**: Instead of raw types with nullable annotation -3. **Be consistent across languages**: Use the same nullability for corresponding fields -4. **Document nullable fields**: Make it clear which fields can be null in your API - -## See Also - -- [Reference Tracking](references.md) - Shared and circular reference handling -- [Xlang Overview](index.md) - Cross-language serialization workflow and runtime guides -- [Type Mapping](../../specification/xlang_type_mapping.md) - Cross-language type mapping reference -- [Xlang Specification](../../specification/xlang_serialization_spec.md) - Binary protocol details diff --git a/docs/object-serialization/xlang/polymorphism.md b/docs/object-serialization/xlang/polymorphism.md deleted file mode 100644 index 7851e17829..0000000000 --- a/docs/object-serialization/xlang/polymorphism.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Polymorphism -sidebar_position: 6 -id: polymorphism -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Xlang polymorphism preserves a value's concrete registered type when a field, element, or root is -declared through a broader type. Every peer must agree on the concrete type identity and compatible -field schema. - -## Runtime rules - -Use the selected runtime's schema metadata, registration, and polymorphism pages for exact syntax. -The [xlang serialization specification](../../specification/xlang_serialization_spec.md) defines -the shared behavior and limitations. - -Do not infer xlang support from host-language inheritance alone. A concrete subtype must have a -portable xlang mapping and coordinated registration on every peer that can receive it. diff --git a/docs/object-serialization/xlang/references.md b/docs/object-serialization/xlang/references.md deleted file mode 100644 index 6e12328c4e..0000000000 --- a/docs/object-serialization/xlang/references.md +++ /dev/null @@ -1,303 +0,0 @@ ---- -title: Reference Tracking -sidebar_position: 5 -id: references -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page explains how Fory handles reference tracking for shared and circular references in cross-language serialization. - -## Overview - -Reference tracking enables: - -- **Shared references**: Same object referenced multiple times is serialized once -- **Circular references**: Objects that reference themselves or form cycles -- **Memory efficiency**: No duplicate data for repeated objects - -## Enabling Reference Tracking - -### Java - -```java -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); -``` - -### Python - -```python -fory = pyfory.Fory(xlang=True, ref=True) -``` - -### Go - -```go -fory := forygo.NewFory( - forygo.WithXlang(true), - forygo.WithTrackRef(true), -) -``` - -### C++ - -```cpp -auto fory = fory::serialization::Fory::builder().xlang(true).track_ref(true).build(); -``` - -### Rust - -```rust -let fory = Fory::builder() - .xlang(true) - .track_ref(true).build(); -``` - -### Scala - -```scala -import org.apache.fory.scala.ForyScala - -val fory = ForyScala.builder() - .withXlang(true) - .withRefTracking(true) - .build() -``` - -## Wire Format - -When reference tracking is enabled, nullable fields write a **ref flag byte** before the value: - -``` -[ref_flag] [value data if not null/ref] -``` - -Where `ref_flag` is: - -| Value | Meaning | -| -------------------------- | ----------------------------------------------------- | -| `-1` (NULL_FLAG) | Value is null | -| `-2` (NOT_NULL_VALUE_FLAG) | Value is present, first occurrence | -| `≥0` | Reference ID pointing to previously serialized object | - -## Reference Tracking vs Nullability - -These are **independent** concepts: - -| Concept | Purpose | Controlled By | -| ---------------------- | ------------------------------------------ | ---------------------------------------- | -| **Nullability** | Whether a field can hold null values | Field type (`Optional`) or annotation | -| **Reference Tracking** | Whether duplicate objects are deduplicated | Global `refTracking` option | - -Key behavior: - -- Ref flag bytes are **only written for nullable fields** -- Non-nullable fields skip ref flags entirely, even with `refTracking=true` -- Reference deduplication only applies to objects that appear multiple times - -```java -// Reference tracking enabled, but non-nullable fields still skip ref flags -Fory fory = Fory.builder() - .withXlang(true) - .withRefTracking(true) - .build(); -``` - -## Per-Field Reference Tracking - -By default, **most fields do not track references** even when global `refTracking=true`. Only specific pointer/smart pointer types track references by default. - -### Default Behavior by Language - -| Language | Default Ref Tracking | Types That Track Refs by Default | -| -------- | -------------------- | ---------------------------------------------------------- | -| Java | No | None (use annotation to enable) | -| Python | No | None (use annotation to enable) | -| Go | No | None (use `fory:"ref"` to enable) | -| C++ | Yes | `std::shared_ptr`, `fory::serialization::SharedWeak` | -| Rust | No | `Rc`, `Arc`, `Weak` | -| Scala | No | None (use `@Ref` to enable) | - -### Customizing Per-Field Ref Tracking - -#### Java: @Ref Annotation - -```java -public class Document { - // Default: no ref tracking - String title; - - // Enable ref tracking for this field - @Ref - Author author; - - // Shared across documents, track refs to avoid duplicates - List<@Ref Tag> tags; -} -``` - -#### C++: FORY_STRUCT Field Config - -```cpp -struct Document { - std::string title; - - // shared_ptr/SharedWeak track refs by default - std::shared_ptr author; - fory::serialization::SharedWeak data; - - std::shared_ptr tag_owner; -}; -FORY_STRUCT(Document, - title, - author, - data, - (tag_owner, fory::F().ref()) -); -``` - -To disable reference tracking for C++ entirely, set -`Fory::builder().xlang(true).track_ref(false).build()` on the serializer. - -#### Rust: Field Attributes - -```rust -use fory::ForyStruct; -use std::rc::Rc; - -#[derive(ForyStruct)] -struct Document { - title: String, - - // Rc/Arc track refs by default - author: Rc, - - // Explicitly enable ref tracking - #[fory(ref = true)] - tags: Vec, -} -``` - -#### Scala: @Ref Annotation - -Scala schema IDL and Scala 3 macro derivation use the same shared JVM `@Ref` -annotation: - -```scala -import org.apache.fory.annotation.{ForyField, ForyStruct, Ref} -import org.apache.fory.scala.ForySerializer - -@ForyStruct -final class Node() derives ForySerializer { - @ForyField(id = 1) - var children: List[Node @Ref] = List.empty - - @Ref - @ForyField(id = 2) - var parent: Option[Node] = None -} -``` - -For Scala, top-level field reference tracking is owned by `@Ref` on the field or -constructor parameter. Type-use `T @Ref` is for nested element/value/payload -references, such as `List[Node @Ref]`. - -#### Go: Struct Tags - -```go -type Document struct { - Title string - - // Enable ref tracking for pointer to struct - Author *Author `fory:"ref"` - - // Enable ref tracking for slice - Tags []Tag `fory:"ref"` -} -``` - -### When to Enable Per-Field Ref Tracking - -Enable ref tracking for fields that: - -- May contain the same object instance multiple times -- Are part of circular reference chains -- Hold large objects that might be shared - -Disable (or leave default) for fields that: - -- Always contain unique values -- Are primitives or simple value types -- Don't participate in object sharing - -## Example: Shared References - -```java -public class Container { - List data; - List sameData; // Points to same list -} - -Container obj = new Container(); -obj.data = Arrays.asList("a", "b", "c"); -obj.sameData = obj.data; // Shared reference - -// With refTracking=true: data serialized once, sameData stores reference ID -// With refTracking=false: data serialized twice (duplicate) -``` - -## Example: Circular References - -```java -public class Node { - String value; - Node next; -} - -Node a = new Node("A"); -Node b = new Node("B"); -a.next = b; -b.next = a; // Circular reference - -// With refTracking=true: works correctly -// With refTracking=false: infinite recursion error -``` - -## Language Support - -| Language | Shared Refs | Circular Refs | -| ---------- | ----------- | -------------------- | -| Java | Yes | Yes | -| Python | Yes | Yes | -| Go | Yes | Yes | -| C++ | Yes | Yes | -| JavaScript | Yes | Yes | -| Rust | Yes | No (ownership rules) | - -## Performance Considerations - -- **Overhead**: Reference tracking adds a hash map lookup per object -- **When to enable**: Use when data has shared/circular references -- **When to disable**: Use for simple data structures without sharing - -## See Also - -- [Field Nullability](nullability.md) - How nullability affects serialization -- [Xlang Overview](index.md) - Cross-language serialization workflow and runtime guides -- [Xlang Specification](../../specification/xlang_serialization_spec.md) - Binary protocol details diff --git a/docs/object-serialization/xlang/schema-evolution.md b/docs/object-serialization/xlang/schema-evolution.md deleted file mode 100644 index 6e519da3c0..0000000000 --- a/docs/object-serialization/xlang/schema-evolution.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Schema Evolution -sidebar_position: 3 -id: schema-evolution -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Xlang compatible mode carries schema metadata so readers can tolerate supported field additions, -removals, and reordering. Same-schema mode reduces metadata only when every reader and writer uses -the same schema. - -## Choose compatible or same-schema mode - -Keep compatible mode for independently deployed peers. Use same-schema mode only after verifying -that every runtime uses the same field identities, nullability, reference metadata, and types. - -Runtime APIs and examples live in each runtime's `schema-evolution.md` page. Normative schema -metadata and compatibility behavior are defined by the -[xlang serialization specification](../../specification/xlang_serialization_spec.md). diff --git a/docs/object-serialization/xlang/troubleshooting.md b/docs/object-serialization/xlang/troubleshooting.md deleted file mode 100644 index e6514c1589..0000000000 --- a/docs/object-serialization/xlang/troubleshooting.md +++ /dev/null @@ -1,321 +0,0 @@ ---- -title: Troubleshooting -sidebar_position: 90 -id: troubleshooting -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -This page covers common issues and solutions when using cross-language serialization. - -## Type Registration Errors - -### "Type not registered" Error - -**Symptom:** - -``` -Error: Type 'example.Person' is not registered -``` - -**Cause:** The type was not registered before deserialization, or the type name doesn't match. - -**Solution:** - -1. Ensure the type is registered with the same name on both sides: - - ```java - // Java - fory.register(Person.class, "example.Person"); - ``` - - ```python - # Python - fory.register_type(Person, name="example.Person") - ``` - -2. Check for typos or case differences in type names - -3. Register types before any serialization/deserialization calls - -### "Type ID mismatch" Error - -**Symptom:** - -``` -Error: Expected type ID 100, got 101 -``` - -**Cause:** Different type IDs used across languages. - -**Solution:** Use consistent type IDs: - -```java -// Java -fory.register(Person.class, 100); -fory.register(Address.class, 101); -``` - -```python -# Python -fory.register_type(Person, type_id=100) -fory.register_type(Address, type_id=101) -``` - -## Type Mapping Issues - -### Integer Overflow - -**Symptom:** Values are truncated or wrapped unexpectedly. - -**Cause:** Using different integer sizes across languages. - -**Solution:** - -1. In Python, use explicit type annotations: - - ```python - @dataclass - class Data: - value: pyfory.Int32 # Not just 'int' - ``` - -2. Ensure integer ranges are compatible: - - `int8`: -128 to 127 - - `int16`: -32,768 to 32,767 - - `int32`: -2,147,483,648 to 2,147,483,647 - -### Float Precision Loss - -**Symptom:** Float values have unexpected precision. - -**Cause:** Mixing `float32` and `float64` types. - -**Solution:** - -1. Use consistent float types: - - ```python - @dataclass - class Data: - value: pyfory.Float32 # Explicit 32-bit float - ``` - -2. Be aware that Python's `float` maps to `float64` by default - -### String Encoding Errors - -**Symptom:** - -``` -Error: Invalid UTF-8 sequence -``` - -**Cause:** Non-UTF-8 encoded strings. - -**Solution:** - -1. Ensure all strings are valid UTF-8 -2. In Python, decode bytes before serialization: - - ```python - text = raw_bytes.decode('utf-8') - ``` - -## Field Order Issues - -### "Field mismatch" Error - -**Symptom:** Deserialized objects have wrong field values. - -**Cause:** Field order differs between languages. - -**Solution:** Fory sorts fields by their snake_cased names. Ensure field names are consistent: - -```java -// Java - fields will be sorted: age, email, name -public class Person { - public String name; - public int age; - public String email; -} -``` - -```python -# Python - same field order -@dataclass -class Person: - name: str - age: pyfory.Int32 - email: str -``` - -## Reference Tracking Issues - -### Stack Overflow with Circular References - -**Symptom:** - -``` -StackOverflowError or RecursionError -``` - -**Cause:** Reference tracking is disabled but data has circular references. - -**Solution:** Enable reference tracking: - -```java -// Java -Fory fory = Fory.builder() - .withRefTracking(true) - .build(); -``` - -```python -# Python -fory = pyfory.Fory(ref=True) -``` - -### Duplicate Objects - -**Symptom:** Shared objects are duplicated after deserialization. - -**Cause:** Reference tracking is disabled. - -**Solution:** Enable reference tracking if objects are shared within the graph. - -## Xlang Type Issues - -### Incompatible Types in Xlang Mode - -**Symptom:** - -``` -Error: Type 'Optional' is not supported in xlang mode -``` - -**Cause:** Using Java-specific types that don't have cross-language equivalents. - -**Solution:** Use compatible types: - -```java -// Instead of Optional -public String email; // nullable - -// Instead of BigDecimal -public double amount; - -// Instead of EnumSet -public Set statuses; -``` - -## Version Compatibility - -### Schema Hash Mismatch - -**Symptom:** Deserialization fails with an error such as `class version hash mismatch`, -`schema version mismatch`, `struct version mismatch`, or `hash mismatch`. - -**Cause:** The writer and reader have disabled compatible mode while their struct/class schemas -differ. In xlang mode this can happen even when each language made a reasonable local change, -because field names, type annotations, field IDs, nullability, and generated schema metadata must -still align exactly. - -**Solution:** - -1. Align the schemas carefully on every service and language: field names or field IDs, field order, - type annotations, nullability, and type registration IDs/names. -2. Xlang mode defaults to compatible mode in current implementations. If a peer has explicitly selected - `compatible=false`, remove that override or enable compatible mode on every peer. - Compatible mode writes extra schema metadata, so payloads are larger, but it is recommended - for xlang services that may evolve independently. -3. Set `compatible=false` only when every reader and writer always uses the same schema. For xlang payloads, do this only after verifying that every language uses that schema, or when native types are generated from Fory schema IDL. - -### Serialization Format Changed - -**Symptom:** Deserialization fails after upgrading Fory. - -**Cause:** Breaking changes in serialization format. - -**Solution:** - -1. Ensure all services use compatible Fory versions -2. Check release notes for breaking changes -3. Consider using schema evolution (compatible mode) for gradual upgrades - -## Debugging Tips - -### Enable Debug Logging - -**Java:** - -```java -// Add to JVM options --Dfory.debug=true -``` - -**Python:** - -```python -import logging -logging.getLogger('pyfory').setLevel(logging.DEBUG) -``` - -### Inspect Serialized Data - -Use hex dump to inspect the binary format: - -```python -data = fory.serialize(obj) -print(data.hex()) -``` - -### Test Round-Trip - -Always test round-trip serialization in each language: - -```java -byte[] bytes = fory.serialize(obj); -Object result = fory.deserialize(bytes); -assert obj.equals(result); -``` - -### Cross-Language Testing - -Test serialization across all target languages before deployment: - -```bash -# Serialize in Java -java -jar serializer.jar > data.bin - -# Deserialize in Python -python deserializer.py data.bin -``` - -## Common Mistakes - -1. **Not registering types**: Always register custom types before use -2. **Inconsistent type names/IDs**: Use the same names/IDs across all languages -3. **Mixing xlang and native payloads**: Keep every peer on the xlang wire format -4. **Wrong type annotations**: Use markers such as `pyfory.Int32` in Python -5. **Ignoring reference tracking**: Enable for circular/shared references - -## See Also - -- [Type Mapping](../../specification/xlang_type_mapping.md) - Cross-language type mapping reference -- [Getting Started](../../start/index.md) - Runtime setup guides -- [Java Troubleshooting](../java/troubleshooting.md) - Java-specific issues -- [Python Troubleshooting](../python/troubleshooting.md) - Python-specific issues diff --git a/docs/object-serialization/xlang/type-identity.md b/docs/object-serialization/xlang/type-identity.md deleted file mode 100644 index 844b7e4b52..0000000000 --- a/docs/object-serialization/xlang/type-identity.md +++ /dev/null @@ -1,298 +0,0 @@ ---- -title: Type Identity and Dynamic Fields -sidebar_position: 2 -id: type-identity -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Field type meta configuration controls whether type information is written during serialization for struct fields. This is essential for supporting polymorphism where the actual concrete type may differ from the declared field type. - -## Overview - -When serializing a struct field, Fory needs to determine whether to write type metadata: - -- **Static typing**: Use the declared field type's serializer directly (no type info written) -- **Dynamic typing**: Write type information to support subtypes - -## When Type Meta Is Needed - -Type metadata is required when: - -1. **Interface/abstract fields**: The declared type is abstract, so concrete type must be recorded -2. **Polymorphic fields**: The concrete type may be a subclass of the declared type -3. **Cross-language compatibility**: When the receiver needs type information to deserialize correctly - -Type metadata is NOT needed when: - -1. **Final/concrete types**: The declared type is final/sealed and cannot be subclassed -2. **Primitive types**: Type is known at compile time -3. **Performance optimization**: When you know the concrete type always matches the declared type - -## Language-Specific Configuration - -### Java - -Java requires explicit configuration because concrete classes can be subclassed unless marked `final`. - -Use the `@ForyField` annotation with the `dynamic` parameter: - -```java -import org.apache.fory.annotation.ForyField; -import org.apache.fory.annotation.ForyField.Dynamic; - -public class Container { - // AUTO (default): Interface types write type info, concrete types don't - @ForyField(id = 0) - private Shape shape; // Interface - type info written - - // FALSE: Never write type info (use declared type's serializer) - @ForyField(id = 1, dynamic = Dynamic.FALSE) - private Circle circle; // Always treated as Circle - - // TRUE: Always write type info (support subtypes) - @ForyField(id = 2, dynamic = Dynamic.TRUE) - private Shape concreteShape; // Type info written even if concrete -} -``` - -**Dynamic Options**: - -| Value | Behavior | -| ------- | ------------------------------------------------------ | -| `AUTO` | Interface/abstract types are dynamic, concrete are not | -| `FALSE` | Never write type info, use declared type's serializer | -| `TRUE` | Always write type info to support subtypes | - -**Use Cases**: - -- `AUTO`: Default behavior, suitable for most cases -- `FALSE`: Performance optimization when you know the exact type -- `TRUE`: When a concrete field may hold subclass instances - -### C++ - -C++ uses the `.dynamic(bool)` builder method inside `FORY_STRUCT`: - -```cpp -#include "fory/serialization/fory.h" - -// Abstract base class with pure virtual methods -struct Animal { - virtual ~Animal() = default; - virtual std::string speak() const = 0; -}; - -struct Zoo { - // Auto: type info written because Animal is polymorphic (std::is_polymorphic) - std::shared_ptr animal; - - // Force non-dynamic: skip type info even though Animal is polymorphic - std::shared_ptr fixed_animal; - - // Force dynamic: write type info even for non-polymorphic types - std::shared_ptr polymorphic_data; -}; -FORY_STRUCT(Zoo, - (animal, fory::F(0).nullable()), // Auto-detect polymorphism - (fixed_animal, fory::F(1).nullable().dynamic(false)), // Skip type info - (polymorphic_data, fory::F(2).dynamic(true)) // Force type info -); -``` - -**Default Behavior**: Fory auto-detects polymorphism via `std::is_polymorphic`. Types with pure virtual methods are treated as dynamic by default. - -### Go and Rust - -Go and Rust do **not** require explicit dynamic configuration because: - -- **Go**: Interface types are inherently dynamic - Fory can determine from the type whether it's an interface -- **Rust**: Trait objects (`dyn Trait`) are explicitly marked in the type system - -The type system in these languages already indicates whether a field is polymorphic: - -```go -// Go: interface types are automatically dynamic -type Container struct { - Shape Shape // Interface - type info written automatically - Circle Circle // Concrete struct - no type info needed -} -``` - -```rust -// Rust: trait objects are explicitly marked -struct Container { - shape: Box, // Trait object - type info written automatically - circle: Circle, // Concrete type - no type info needed -} -``` - -### Python - -Use `pyfory.field()` with the `dynamic` parameter: - -```python -from dataclasses import dataclass -from abc import ABC, abstractmethod -import pyfory - -class Shape(ABC): - @abstractmethod - def area(self) -> float: - pass - -@dataclass -class Circle(Shape): - radius: float = 0.0 - - def area(self) -> float: - return 3.14159 * self.radius * self.radius - -@dataclass -class Container: - # Abstract class: dynamic is always True (type info written) - shape: Shape = pyfory.field(id=0) - - # Concrete type with explicit dynamic=True (force type info) - circle: Circle = pyfory.field(id=1, dynamic=True) - - # Concrete type with explicit dynamic=False (skip type info) - fixed_circle: Circle = pyfory.field(id=2, dynamic=False) -``` - -**Default Behavior**: - -| Mode | Abstract Class | Concrete Object Types | Numeric/str/time Types | -| ----------- | -------------- | --------------------- | ---------------------- | -| Native mode | `True` | `True` | `False` | -| Xlang mode | `True` | `False` | `False` | - -- **Abstract classes**: `dynamic` is always `True` (type info must be written) -- **Native mode**: `dynamic` defaults to `True` for object types, `False` for numeric/str/time types -- **Xlang mode**: `dynamic` defaults to `False` for concrete types - -## Default Behavior - -| Language | Interface/Abstract Types | Concrete Types | -| -------- | ------------------------ | ---------------- | -| Java | Dynamic (write type) | Static (no type) | -| C++ | Dynamic (virtual) | Static | -| Go | Dynamic (interface) | Static (struct) | -| Rust | Dynamic (dyn Trait) | Static | -| Python | Dynamic (all objects) | Dynamic | - -## Performance Considerations - -Writing type metadata has overhead: - -- **Space**: Type information adds bytes to serialized output -- **Time**: Type resolution during serialization/deserialization - -Use `dynamic = FALSE` (Java) or `dynamic(false)` (C++) when: - -- You're certain the concrete type matches the declared type -- Performance is critical and polymorphism is not needed -- The field type is effectively final - -## Cross-Language Compatibility - -When serializing data for cross-language consumption: - -1. **Use consistent type registration**: Register types with the same ID across languages -2. **Prefer explicit configuration**: Use `dynamic = TRUE` when unsure about receiver's expectations -3. **Document polymorphic fields**: Make it clear which fields may contain subtypes - -## Example: Polymorphic Container - -### Java - -```java -public interface Animal { - String speak(); -} - -public class Dog implements Animal { - private String name; - - @Override - public String speak() { return "Woof!"; } -} - -public class Cat implements Animal { - private String name; - - @Override - public String speak() { return "Meow!"; } -} - -public class Zoo { - // Type info written because Animal is an interface - @ForyField(id = 0) - private Animal animal; - - // Force type info for concrete type that may hold subtypes - @ForyField(id = 1, dynamic = Dynamic.TRUE) - private Dog maybeMixedBreed; -} -``` - -### C++ - -```cpp -// Abstract base class with pure virtual methods -class Animal { -public: - virtual std::string speak() const = 0; - virtual ~Animal() = default; -}; - -class Dog : public Animal { -public: - std::string name; - std::string speak() const override { return "Woof!"; } -}; - -struct Zoo { - std::shared_ptr animal; - std::shared_ptr maybe_mixed_breed; -}; - -FORY_STRUCT(Zoo, - (animal, fory::F(0).nullable()), // Auto-detect (Animal is polymorphic) - (maybe_mixed_breed, fory::F(1).dynamic(true)) // Force dynamic for concrete type -); -``` - -## Related Topics - -- [Field Nullability](nullability.md) - Controlling null handling for fields -- [Reference Tracking](references.md) - Managing shared/circular references -- [Type Mapping](../../specification/xlang_type_mapping.md) - Cross-language type compatibility - -## Coordinate Type Identity Across Peers - -Every peer must register a custom type with the same numeric ID or the same -namespace and type name. Numeric IDs produce smaller metadata; names are -easier to coordinate across independently deployed services. Do not mix an -ID registration on one peer with a name registration on another. - -Use the selected runtime guide for the exact registration API: -[Java](../java/xlang.md), [Python](../python/xlang.md), -[C++](../cpp/xlang.md), [Go](../go/xlang.md), -[Rust](../rust/xlang.md), [JavaScript](../javascript/xlang.md), -[C#](../csharp/xlang.md), [Swift](../swift/xlang.md), -[Dart](../dart/xlang.md), [Scala](../scala/xlang.md), and -[Kotlin](../kotlin/xlang.md). diff --git a/docs/object-serialization/xlang/type-system.md b/docs/object-serialization/xlang/type-system.md deleted file mode 100644 index ac711295e8..0000000000 --- a/docs/object-serialization/xlang/type-system.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: Type System -sidebar_position: 1 -id: type-system -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -## Serialize Built-in Types - -Common types can be serialized automatically without registration: primitive numeric types, string, binary, array, list, map, and more. - -Reduced-precision floating-point values are also part of the built-in xlang type system: - -- `float16` and `array` -- `bfloat16` and `array` - -Use the language-specific carrier types documented in the type mapping reference. Python uses `pyfory.Float16` and `pyfory.BFloat16` as annotation markers only; scalar values are native Python `float`, and dense reduced-precision arrays use `pyfory.Float16Array` and `pyfory.BFloat16Array`. Go uses the `float16` and `bfloat16` packages for scalar, slice, and array carriers; JavaScript uses `number` for scalar `float16` and `bfloat16`, and dense array carriers `BoolArray`, `Float16Array`, and `BFloat16Array` for the corresponding `array` schemas. Dart uses `double` plus `Float16Type` or `Bfloat16Type` metadata for scalar fields, and `Float16List` / `Bfloat16List` for dense arrays. Java uses `@ArrayType` on supported reduced-precision carriers for `array` / `array` schema, while general object arrays stay on the `list` path; C++, Rust, and C# provide their own dedicated scalar and array carriers. - -When `compatible=true`, a direct struct/class field can evolve between `list` and `array` for dense bool/numeric `T`. Integer list element encodings in the same signedness and width domain match the corresponding dense array element domain. This applies only to the immediate matched field schema. It does not apply to nested collection, map, array, union, or generic positions. A peer `list` schema can be read into a local `array` field when the actual payload has no null elements. If the payload carries a null element or ref-tracked element encoding, reading it into a local `array` field raises a compatible-read error. - -## Serialize Custom Types - -User-defined types must be registered using the register API to establish the mapping relationship between types in different languages. Use consistent type names across all languages. - -## Exact mappings - -The normative [xlang type mapping](../../specification/xlang_type_mapping.md) defines the exact -carrier mapping for every runtime. Runtime pages show the API syntax and examples for that mapping. diff --git a/docs/object-serialization/xlang/zero-copy.md b/docs/object-serialization/xlang/zero-copy.md deleted file mode 100644 index 49c6f07a64..0000000000 --- a/docs/object-serialization/xlang/zero-copy.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -title: Zero-Copy Serialization -sidebar_position: 7 -id: zero-copy -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Zero-copy serialization allows large binary data (byte arrays, numeric arrays) to be serialized out-of-band, avoiding memory copies and reducing serialization overhead. - -## When to Use Zero-Copy - -Use zero-copy serialization when: - -- Serializing large byte arrays or binary blobs -- Working with numeric arrays (int[], double[], etc.) -- Transferring data over high-performance networks -- Memory efficiency is critical - -## How It Works - -1. **Serialization**: Large buffers are extracted and returned separately via a callback -2. **Transport**: The main serialized data and buffer objects are transmitted separately -3. **Deserialization**: Buffers are provided back to reconstruct the original object - -This avoids copying large data into the main serialization buffer. - -## Java - -```java -import org.apache.fory.*; -import org.apache.fory.config.*; -import org.apache.fory.serializer.BufferObject; -import org.apache.fory.memory.MemoryBuffer; - -import java.util.*; -import java.util.stream.Collectors; - -public class ZeroCopyExample { - public static void main(String[] args) { - Fory fory = Fory.builder().withXlang(true).build(); - - // Data with large arrays - List list = List.of( - "str", - new byte[1000], // Large byte array - new int[100], // Large int array - new double[100] // Large double array - ); - - // Collect buffer objects during serialization - Collection bufferObjects = new ArrayList<>(); - byte[] bytes = fory.serialize(list, e -> !bufferObjects.add(e)); - - // Convert to buffers for transport - List buffers = bufferObjects.stream() - .map(BufferObject::toBuffer) - .collect(Collectors.toList()); - - // Deserialize with buffers - Object result = fory.deserialize(bytes, buffers); - System.out.println(result); - } -} -``` - -## Python - -```python -import array -import pyfory -import numpy as np - -fory = pyfory.Fory(xlang=True) - -# Data with large arrays -data = [ - "str", - bytes(bytearray(1000)), # Large byte array - array.array("i", range(100)), # Large int array - np.full(100, 0.0, dtype=np.double) # Large numpy array -] - -# Collect buffer objects during serialization -serialized_objects = [] -serialized_data = fory.serialize(data, buffer_callback=serialized_objects.append) - -# Convert to buffers for transport -buffers = [obj.to_buffer() for obj in serialized_objects] - -# Deserialize with buffers -result = fory.deserialize(serialized_data, buffers=buffers) -print(result) -``` - -## Go - -```go -package main - -import forygo "github.com/apache/fory/go/fory" -import "fmt" - -func main() { - serializer := forygo.NewFory(forygo.WithXlang(true)) - - // Data with large arrays - list := []any{ - "str", - make([]byte, 1000), // Large byte array - } - - buf := forygo.NewByteBuffer(nil) - var bufferObjects []forygo.BufferObject - - // Collect buffer objects during serialization - if err := serializer.SerializeWithCallback(buf, list, func(o forygo.BufferObject) bool { - bufferObjects = append(bufferObjects, o) - return false - }); err != nil { - panic(err) - } - - // Convert to buffers for transport - var buffers []*forygo.ByteBuffer - for _, o := range bufferObjects { - buffers = append(buffers, o.ToBuffer()) - } - - // Deserialize with buffers - var newList []any - if err := serializer.DeserializeWithCallbackBuffers(buf, &newList, buffers); err != nil { - panic(err) - } - fmt.Println(newList) -} -``` - -## Use Cases - -### High-Performance Data Transfer - -When sending large datasets over the network: - -```java -// Sender -Collection buffers = new ArrayList<>(); -byte[] metadata = fory.serialize(dataObject, e -> !buffers.add(e)); - -// Send metadata and buffers separately -network.sendMetadata(metadata); -for (BufferObject buf : buffers) { - network.sendBuffer(buf.toBuffer()); -} - -// Receiver -byte[] metadata = network.receiveMetadata(); -List buffers = network.receiveBuffers(); -Object data = fory.deserialize(metadata, buffers); -``` - -### Memory-Mapped Files - -Zero-copy works well with memory-mapped files: - -```java -// Write -Collection buffers = new ArrayList<>(); -byte[] data = fory.serialize(largeObject, e -> !buffers.add(e)); -writeToFile("data.bin", data); -for (int i = 0; i < buffers.size(); i++) { - writeToFile("buffer" + i + ".bin", buffers.get(i).toBuffer()); -} - -// Read -byte[] data = readFromFile("data.bin"); -List buffers = readBufferFiles(); -Object result = fory.deserialize(data, buffers); -``` - -## Performance Considerations - -1. **Threshold**: Small arrays may not benefit from zero-copy due to callback overhead -2. **Network**: Zero-copy is most beneficial when buffers can be sent without copying -3. **Memory**: Reduces peak memory usage by avoiding buffer copies - -## See Also - -- [Xlang Overview](index.md) - Standard serialization workflow and runtime guides -- [Python Out-of-Band Guide](../python/out-of-band.md) - Python-specific zero-copy details diff --git a/docs/row-format/compact.md b/docs/row-format/compact.md deleted file mode 100644 index dbe2104691..0000000000 --- a/docs/row-format/compact.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Compact Row -sidebar_position: 2 -id: compact -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Compact Row is a Java-only row encoding that reduces fixed-slot and null-bitmap overhead. It is not -wire-compatible with Standard Row. - -## Create a compact encoder - -```java -RowEncoder encoder = - Encoders.buildBeanCodec(MyBean.class) - .compactEncoding() - .build() - .get(); - -BinaryRow row = encoder.toRow(value); -MyBean decoded = encoder.fromRow(row); -``` - -Reuse the encoder within one thread. Create separate encoders for concurrent threads. - -## Layout tradeoffs - -- Fixed-size fields use their natural widths instead of eight-byte Standard Row slots. -- Fields are sorted by alignment to reduce padding. -- The null bitmap is omitted when no field is nullable. -- Fixed-size nested structs can be stored inline. - -Choose Compact Row only when every reader is Java and the space reduction justifies the -Java-specific layout. Use [Standard Row](standard.md) for Java/Python/C++/Rust interchange. - -See the [Row Format specification](../specification/row_format_spec.md) for the exact binary layout. diff --git a/docs/row-format/cpp.md b/docs/row-format/cpp.md index 9132a330c5..f34b25c6ea 100644 --- a/docs/row-format/cpp.md +++ b/docs/row-format/cpp.md @@ -504,6 +504,6 @@ int32_t id = row.get_int32(0); ## Related Topics - [C++ Row example](https://github.com/apache/fory/tree/main/examples/cpp/hello_row) - Complete runnable example -- [Basic Serialization](../object-serialization/cpp/core-api.md) - Object graph serialization +- [Basic Serialization](../object-serialization/cpp/basic-serialization.md) - Object graph serialization - [C++ Object Serialization Configuration](../object-serialization/cpp/configuration.md) - Builder options - [C++ Object Serialization Supported Types](../object-serialization/cpp/supported-types.md) - Object serialization types diff --git a/docs/row-format/index.md b/docs/row-format/index.md index 1ebcf6d913..8c901cd074 100644 --- a/docs/row-format/index.md +++ b/docs/row-format/index.md @@ -1,5 +1,5 @@ --- -title: Row Format +title: Introduction sidebar_position: 0 id: index license: | @@ -19,27 +19,81 @@ license: | limitations under the License. --- -Row Format stores typed values in a cache-friendly binary layout for random and partial access -without reconstructing a complete object graph. Use it for analytical and -in-memory data processing. +Row Format is a cache-friendly binary format for efficient random access and partial +deserialization. Unlike object graph serialization, it lets readers access individual fields +without reconstructing the complete object. -## Choose a row family +Row Format is intended only for trusted analytical data, including memory-mapped data, selective +field access, and data pipelines. Use [Object Serialization](../object-serialization/index.md) when +the application needs general object graphs, shared or circular references, or complete object +reconstruction as its primary access pattern. -| Family | Runtime support | Compatibility | -| --------------------------- | ----------------------- | -------------------------------- | -| [Standard Row](standard.md) | Java, Python, C++, Rust | Shared Standard Row layout | -| [Compact Row](compact.md) | Java | Java-only, space-oriented layout | +## Choose a Layout -Use Binary Object Serialization when the goal is complete object reconstruction, references, or -general application messaging. Use Row Format when a workload reads selected fields, nested arrays, -or maps directly from encoded data. +| Layout | Runtime support | Compatibility | +| ------------ | ----------------------- | -------------------------------- | +| Standard Row | Java, Python, C++, Rust | Shared Standard Row layout | +| Compact Row | Java | Java-only, space-oriented layout | -## Runtime guides +## Standard Row -- [Java](java.md) -- [Python](python.md) -- [C++](cpp.md) -- [Rust](rust.md) +Standard Row is the interoperable layout for Java, Python, C++, and Rust. -The normative [Row Format specification](../specification/row_format_spec.md) defines Standard and -Compact layouts. +### Features + +- **Zero-copy random access**: Read selected fields directly from encoded data. +- **Partial deserialization**: Reconstruct only the values an application needs. +- **Cross-language compatibility**: Share Standard Row bytes between Java, Python, C++, and Rust. +- **Apache Arrow integration**: Convert rows to Arrow data in Java and Python. + +### Layout + +Standard Row stores fixed-width values inline and variable-width values by offset and size. Rows, +arrays, and maps use a schema to resolve field positions and element types. The normative byte +layout, alignment rules, type table, and endianness are defined by the +[Row Format specification](../specification/row_format_spec.md). + +### Runtime Support + +| Runtime | Standard Row compatibility | Runtime guide | Additional integration | +| ------- | -------------------------- | ------------------- | ------------------------------------------------------ | +| Java | Compatible | [Java](java.md) | Arrow conversion; interface and extension-type mapping | +| Python | Compatible | [Python](python.md) | PyArrow schema and table conversion | +| C++ | Compatible | [C++](cpp.md) | Native row readers and writers | +| Rust | Compatible | [Rust](rust.md) | Borrowed struct, array, and map views | + +Use the runtime guides for installation, schema construction, encoding, random access, partial +reads, and language-specific integrations. + +## Compact Row + +Compact Row is a Java-only row encoding that reduces fixed-slot and null-bitmap overhead. It is not +wire-compatible with Standard Row. + +### Create a Compact Encoder + +```java +RowEncoder encoder = + Encoders.buildBeanCodec(MyBean.class) + .compactEncoding() + .build() + .get(); + +BinaryRow row = encoder.toRow(value); +MyBean decoded = encoder.fromRow(row); +``` + +Reuse the encoder within one thread. Create separate encoders for concurrent threads. + +### Layout Tradeoffs + +- Fixed-size fields use their natural widths instead of eight-byte Standard Row slots. +- Fields are sorted by alignment to reduce padding. +- The null bitmap is omitted when no field is nullable. +- Fixed-size nested structs can be stored inline. + +Choose Compact Row only when every reader is Java and the space reduction justifies the +Java-specific layout. Use Standard Row for Java, Python, C++, and Rust interchange. + +See the [Row Format specification](../specification/row_format_spec.md) for the exact Standard and +Compact binary layouts. diff --git a/docs/row-format/java.md b/docs/row-format/java.md index 8679cadd8c..04a8795514 100644 --- a/docs/row-format/java.md +++ b/docs/row-format/java.md @@ -279,6 +279,6 @@ Parent decoded = encoder.fromRow(row); ## Related Topics -- [Xlang Serialization](../object-serialization/java/xlang.md) - xlang mode +- [Cross-Language Interoperability](../object-serialization/java/basic-serialization.md#cross-language-interoperability) - xlang mode - [Java Advanced Features](../object-serialization/java/advanced-features.md) - Zero-copy object serialization - [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec) - Protocol details diff --git a/docs/row-format/python.md b/docs/row-format/python.md index 0963df2fb1..4852a16c56 100644 --- a/docs/row-format/python.md +++ b/docs/row-format/python.md @@ -195,6 +195,6 @@ pip install pyfory[format] ## Related Topics -- [Xlang Serialization](../object-serialization/python/xlang.md) - xlang mode -- [Basic Serialization](../object-serialization/python/core-api.md) - Object serialization +- [Cross-Language Interoperability](../object-serialization/python/basic-serialization.md#cross-language-interoperability) - xlang mode +- [Basic Serialization](../object-serialization/python/basic-serialization.md) - Object serialization - [Row Format Specification](https://fory.apache.org/docs/specification/row_format_spec) - Protocol details diff --git a/docs/row-format/rust.md b/docs/row-format/rust.md index 33efc184ac..debd04d7d1 100644 --- a/docs/row-format/rust.md +++ b/docs/row-format/rust.md @@ -185,6 +185,6 @@ assert!(scores.get(scores.len()).is_err()); ## Related Topics -- [Basic Serialization](../object-serialization/rust/core-api.md) - Object graph serialization -- [Standard Row Format](standard.md) - Shared layout for Java, Python, C++, and Rust +- [Basic Serialization](../object-serialization/rust/basic-serialization.md) - Object graph serialization +- [Standard Row Format](index.md#standard-row) - Shared layout for Java, Python, C++, and Rust - [Row Format Specification](../specification/row_format_spec.md) - Protocol details diff --git a/docs/row-format/standard.md b/docs/row-format/standard.md deleted file mode 100644 index f7d1b1aa62..0000000000 --- a/docs/row-format/standard.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -title: Standard Row Format -sidebar_position: 1 -id: standard -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. ---- - -Fory Row Format is a cache-friendly binary format for efficient random access -and partial deserialization. Unlike object graph serialization, it lets readers -access individual fields without reconstructing the complete object. - -## Features - -- **Zero-copy random access**: Read selected fields directly from encoded data. -- **Partial deserialization**: Reconstruct only the values an application needs. -- **Cross-language compatibility**: Share Standard Row bytes between Java, - Python, C++, and Rust. -- **Apache Arrow integration**: Convert rows to Arrow data in Java and Python. - -## Format Boundary - -Standard Row stores fixed-width values inline and variable-width values by -offset and size. Rows, arrays, and maps use a schema to resolve field positions -and element types. The normative byte layout, alignment rules, type table, and -endianness are defined by the -[Row Format Specification](../specification/row_format_spec.md). - -Row Format is intended for analytics, memory-mapped data, selective field -access, and data pipelines. Use [Object Serialization](../object-serialization/index.md) -when the application needs general object graphs, shared or circular references, -or complete object reconstruction as its primary access pattern. - -## Implementations - -| Runtime | Standard Row compatibility | Runtime guide | Additional integration | -| ------- | -------------------------- | ------------------- | ------------------------------------------------------ | -| Java | Compatible | [Java](java.md) | Arrow conversion; interface and extension-type mapping | -| Python | Compatible | [Python](python.md) | PyArrow schema and table conversion | -| C++ | Compatible | [C++](cpp.md) | Native row readers and writers | -| Rust | Compatible | [Rust](rust.md) | Borrowed struct, array, and map views | - -Use the runtime guides for installation, schema construction, encoding, random -access, partial reads, and language-specific integrations. diff --git a/docs/object-serialization/deserialization-security-model.md b/docs/security/deserialization.md similarity index 91% rename from docs/object-serialization/deserialization-security-model.md rename to docs/security/deserialization.md index 11ba637b17..bec502b9f4 100644 --- a/docs/object-serialization/deserialization-security-model.md +++ b/docs/security/deserialization.md @@ -1,22 +1,6 @@ --- title: Deserialization Security Model -sidebar_position: 99 -id: deserialization-security-model -license: | - Licensed to the Apache Software Foundation (ASF) under one or more - contributor license agreements. See the NOTICE file distributed with - this work for additional information regarding copyright ownership. - The ASF licenses this file to You under the Apache License, Version 2.0 - (the "License"); you may not use this file except in compliance with - the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. +sidebar_position: 3 --- This document defines the security model for Apache Fory deserialization. It is @@ -32,20 +16,23 @@ security boundary. ## Scope -This model applies only to deserializing Fory binary object-serialization data -from untrusted or partially trusted sources. Fory JSON has a separate -[security guide](../json/security.md) because it uses different readers, -policies, codecs, and resource-accounting rules. +This model applies to deserializing Fory binary data from untrusted or +partially trusted sources. Its resource, policy, and cleanup boundaries also +apply to Java Fory JSON. The Java Fory JSON subsection under +[Graph Memory Budget](#graph-memory-budget) defines that format's accounting +scope without changing binary Fory behavior. It does not treat the semantic content of a successfully deserialized value as a Fory security boundary. A sender can always construct protocol-valid data whose value is chosen by that sender. Application authorization, object-level business rules, and domain-specific validation remain application responsibilities. -The selected business invariant remains an application policy rather than a -Fory protocol security boundary. +Java Fory JSON applications can enforce those rules with `JsonValidator` or in +a `JsonCreator`, but the selected business invariant remains an application +policy rather than a Fory protocol security boundary. -This model does not govern memory-format paths unless a runtime explicitly -exposes such a path through an untrusted deserialization API. +This model also does not cover trusted in-memory formats. Row format and other +memory-format paths are trusted-data paths unless a runtime explicitly exposes +them as untrusted deserialization APIs. ## Trust Boundaries @@ -400,6 +387,42 @@ Float16 and BFloat16 dense-array carriers also include their wrapper's shallow o list conversion first decodes a primitive array, the array's reservation remains as credit toward the final list estimate, and the conversion reserves only a positive remaining difference. +### Java Fory JSON + +Java Fory JSON uses `ForyJsonBuilder.withMaxGraphMemoryBytes` to configure this per-root gate. The +default is the fixed `ForyJson.DEFAULT_MAX_GRAPH_MEMORY_BYTES` value of 128 MiB, and explicit values +must be positive. String and UTF-8 byte-array root reads use the same configured limit. Every root +read starts with the complete limit, and success or failure cannot reduce the next root operation's +budget. The limit is not derived from input length. + +Built-in Java JSON accounting includes shallow POJO and record storage, collections and sets plus +candidate element-reference slots, maps plus candidate key/value-reference slots, reference arrays +plus their slots, and Java primitive arrays plus their primitive storage. Natural `JsonObject` and +`JsonArray` values follow the same map and collection rules. Unknown-length collection, map, and +array storage is reserved in 1024-item batches before each batch's final child and at the tail. +Repeated set elements and duplicate or overwritten map members are therefore charged per input +occurrence. A reference array is charged even when its elements are leaves, and an object is charged +even when all of its properties are leaves. Primitive arrays decoded from JSON arrays reserve the +portable array header and actual Java primitive width using the same batch schedule. +`AtomicReference`, `AtomicReferenceArray`, and generic `Optional` values include wrapper and +reference storage; primitive optionals and atomic primitive values are leaves. + +Dedicated Java JSON leaf codecs are excluded from graph accounting: null, strings, characters, +booleans, numeric values including arbitrary-precision numbers, enums, temporal and other scalar +values, and binary values. A `byte[]` handled by a binary or Base64 codec remains a binary leaf; +the same Java carrier decoded from a JSON numeric array is a primitive-array owner. Byte-availability +and grammar checks still apply independently of graph accounting. + +A custom Java JSON codec that materializes composite graph owners must call +`JsonReader.reserveGraphMemory` with its application-defined byte estimate for each composite +application object, collection, map, or reference array. Unknown-length retained storage should be +reserved in bounded batches before each batch's final child and at the tail; a codec may use +stronger timing. A custom scalar or other dedicated leaf representation makes no reservation. The +budget cannot include custom allocations that the codec does not reserve, application constructor +or validator internals, temporary parsing storage, or unrelated process memory. Applications must +therefore combine this approximate gate with transport input limits, timeouts, and other resource +controls appropriate to their trust boundary. + ### Generated Structural Targets Wire members and physical storage are separate inputs. Properties, accessors, interfaces, and diff --git a/docs/security/index.md b/docs/security/index.md new file mode 100644 index 0000000000..66f8cc7e59 --- /dev/null +++ b/docs/security/index.md @@ -0,0 +1,22 @@ +--- +title: Security +sidebar_position: 1 +--- + +This directory documents Apache Fory security models and security invariants. It +is not a vulnerability disclosure area and does not list CVE details, exploit +samples, issue timelines, or implementation history. + +Security model documents describe how Fory should classify and prevent security +risks while preserving the performance characteristics expected from Fory +serialization runtimes. + +## Models + +- [Threat Model](threat-model.md): project-level trust boundaries, non-goals, + and downstream responsibilities. +- [Deserialization Security Model](deserialization.md): concrete rules for + classifying and preventing untrusted deserialization risks. + +For vulnerability reporting, see the repository +[security policy](../../SECURITY.md). diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md new file mode 100644 index 0000000000..5fea135056 --- /dev/null +++ b/docs/security/threat-model.md @@ -0,0 +1,95 @@ +--- +title: Threat Model +sidebar_position: 2 +--- + +This document describes Apache Fory's project-level security boundaries and +non-goals. It is the high-level entry point for Fory security models; concrete +untrusted deserialization classification rules live in the +[deserialization security model](deserialization.md). + +Fory is an in-process serialization library. Applications link Fory into their +own process, configure serializers and type policies, and call Fory APIs to +serialize application-owned objects or deserialize encoded Fory data. Fory does +not provide a standalone network service, daemon, authentication system, or +transport protocol. + +Fory can generate service companions for application-provided gRPC runtimes. +Those companions provide Fory serialization for request and response objects; +the application and gRPC stack still own listeners, channels, credentials, +authentication, authorization, deadlines, retries, and transport lifecycle. + +## Trust Boundaries + +Fory's primary security boundary is encoded bytes or streams passed to +deserialization APIs from untrusted or partially trusted sources. The embedding +application owns where those bytes come from and which Fory configuration, +registered types, schemas, and policies are used to read them. + +The adversary model for untrusted deserialization is a sender that can craft +encoded bytes or stream behavior presented to a Fory read API. It does not assume +the sender can change the embedding application's Fory configuration, registered +type set, `TypeChecker` or equivalent allow-list policy, schema definitions, +classloader, or other active policy objects unless the application itself exposes +those controls. + +Fory security boundaries include: + +- Runtime safety, including avoiding crashes, panics, undefined behavior, and + out-of-bounds memory access. +- Resource ownership, including memory, CPU progress, stream buffers, native + allocations, callbacks, and retained read-side state. +- Explicit Fory policy checks, such as class, type, function, method, + registration, or deserialization policies that restrict what may be + materialized. +- Cleanup boundaries, where state created during a failed root operation must + not leak into later operations. + +Runtime serializer code generation and JIT compilation are not paths for +executing encoded input. They operate on types and schemas after the active +registration check, `TypeChecker`, schema check, or policy check has accepted the +type surface. When class registration is disabled, `TypeChecker` or an +equivalent allow-list policy is the relevant gate. Generated serializer code is +derived from checked type descriptors rather than from attacker-controlled byte +contents. + +The [deserialization security model](deserialization.md) defines how to +classify these boundaries for untrusted deserialization paths. + +## Non-Goals + +Fory does not provide: + +- Encoded-data authenticity, integrity, confidentiality, signing, MACs, or + encryption. +- Transport security or protection for bytes while they are stored or moved + outside Fory, including transport security for generated service companions. +- Application-level authorization or validation for the business meaning of a + successfully deserialized value. +- A sandbox for user-registered classes, functions, constructors, setters, + finalizers, or other application-owned logic. + +Applications that receive Fory data from untrusted sources should authenticate +or integrity-check those bytes before passing them to Fory when authenticity or +tamper resistance matters. + +## Downstream Responsibilities + +Applications are responsible for: + +- Choosing whether a byte source is trusted enough for the configured + deserialization mode. +- Keeping class or type registration enabled for untrusted data unless another + explicit Fory policy owns the accepted type surface. +- Registering only types and serializers that are safe for the application's + trust boundary. +- Configuring depth and resource limits for the largest data shape the + application intends to accept. +- Treating cross-language peers and schemas as part of the application's trust + relationship. + +Disabling registration or using dynamic deserialization on trusted data is a +configuration choice. For untrusted data, bypassing an explicit Fory policy, +crashing, leaking resources, retaining attacker-controlled state, or allocating +disproportionately remains security-relevant as described in the +[deserialization security model](deserialization.md). diff --git a/docs/start/cpp.md b/docs/start/cpp.md index 837cb81bd9..bc8384134b 100644 --- a/docs/start/cpp.md +++ b/docs/start/cpp.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory C++ is consumed from the Apache Fory source tree. It requires a C++17 compiler and supports CMake 3.16 or later and Bazel 8 or later. Pin one Fory release or commit across every peer in an application. +Fory C++ provides binary Object Serialization, Row Format, generated models, +and Fory gRPC. It requires a C++17 compiler and supports CMake 3.16 or later +and Bazel 8 or later. ## Verify the Toolchain @@ -29,13 +31,75 @@ cmake --version # or: bazel --version ``` -## Choose a Capability +## Object Serialization -| Capability | Build target | Continue with | -| --------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Object Serialization | `fory::serialization` or `//cpp/fory/serialization` | [C++ object serialization](../object-serialization/cpp/index.md), then choose [xlang](../object-serialization/cpp/xlang.md) or [native](../object-serialization/cpp/native.md) | -| Row Format | C++ Row and encoder targets | [C++ Row Format](../row-format/cpp.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated companions plus gRPC C++ | [C++ gRPC](../grpc/cpp.md) | +Use xlang mode for data shared with other Fory runtimes or native mode for +C++-only data. Fetch a released source tree and link the serialization target: -The selected capability guide owns its exact CMake or Bazel setup and first runnable example. +```cmake title="CMakeLists.txt" +cmake_minimum_required(VERSION 3.16) +project(fory_example LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 17) +include(FetchContent) +FetchContent_Declare( + fory + GIT_REPOSITORY https://github.com/apache/fory.git + GIT_TAG v1.5.0 + SOURCE_SUBDIR cpp +) +FetchContent_MakeAvailable(fory) + +add_executable(fory_example main.cc) +target_link_libraries(fory_example PRIVATE fory::serialization) +``` + +```cpp title="main.cc" +#include +#include +#include +#include +#include + +#include "fory/serialization/fory.h" + +struct User { + int64_t id; + std::string name; + + bool operator==(const User &other) const { + return id == other.id && name == other.name; + } +}; +FORY_STRUCT(User, id, name); + +int main() { + auto fory = fory::serialization::Fory::builder().xlang(true).build(); + fory.register_struct(1); + + auto bytes = fory.serialize(User{1, "Alice"}); + assert(bytes.ok()); + std::vector data = std::move(bytes).value(); + auto decoded = fory.deserialize(data); + assert(decoded.ok()); + User user = std::move(decoded).value(); + assert(user.id == 1 && user.name == "Alice"); +} +``` + +```bash +cmake -S . -B build -DCMAKE_BUILD_TYPE=Release +cmake --build build --parallel +./build/fory_example +``` + +See [C++ Object Serialization](../object-serialization/cpp/index.md) for Bazel, +Windows, error handling, and thread-safe instances; then continue to +[xlang](../object-serialization/cpp/basic-serialization.md#cross-language-interoperability) or +[native mode](../object-serialization/cpp/native.md). + +## Other Capabilities + +- **Row Format** provides random and partial field access for trusted analytical data. See [C++ Row Format](../row-format/cpp.md). +- **Fory IDL and Compiler** generates C++ models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [C++ generated-code guide](../compiler/generated-code/cpp.md). +- **Fory gRPC** uses gRPC C++ transports with Fory-encoded messages. See [C++ gRPC](../grpc/cpp.md). diff --git a/docs/start/csharp.md b/docs/start/csharp.md index 545192663f..0167f6b0ad 100644 --- a/docs/start/csharp.md +++ b/docs/start/csharp.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory C# is published on NuGet as `Apache.Fory` and requires the .NET 8 SDK or later. The package includes the runtime and source generator for `ForyStruct` types. Use one compatible Fory release across every peer in an application. +Fory C# provides xlang Object Serialization, generated models, and Fory gRPC. +The `Apache.Fory` NuGet package requires .NET 8 or later and includes both the +runtime and source generator. ## Verify the Toolchain @@ -27,12 +29,53 @@ Fory C# is published on NuGet as `Apache.Fory` and requires the .NET 8 SDK or la dotnet --version ``` -## Choose a Capability +## Object Serialization -| Capability | Package or tool | Continue with | -| --------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `Apache.Fory` | [C# object serialization](../object-serialization/csharp/index.md) and [xlang](../object-serialization/csharp/xlang.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated companions plus gRPC .NET dependencies | [C# gRPC](../grpc/csharp.md) | +Create a console project and add the released package: -Each capability guide owns its exact package declaration and first runnable example. +```bash +dotnet new console -n ForyExample +cd ForyExample +dotnet add package Apache.Fory --version 1.5.0 +``` + +Replace `Program.cs` with: + +```csharp +using Apache.Fory; + +[ForyStruct] +public sealed class User +{ + public long Id { get; set; } + public string Name { get; set; } = string.Empty; +} + +public static class Program +{ + public static void Main() + { + Fory fory = Fory.Builder().Build(); + fory.Register(1); + + byte[] bytes = fory.Serialize(new User { Id = 1, Name = "Alice" }); + User decoded = fory.Deserialize(bytes); + Console.WriteLine(decoded.Name); + } +} +``` + +```bash +dotnet run +``` + +C# uses xlang mode. Continue with +[C# Object Serialization](../object-serialization/csharp/index.md), +[xlang types](../object-serialization/csharp/basic-serialization.md#cross-language-interoperability), +[configuration](../object-serialization/csharp/configuration.md), and +[schema evolution](../object-serialization/csharp/schema-evolution.md). + +## Other Capabilities + +- **Fory IDL and Compiler** generates C# models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [C# generated-code guide](../compiler/generated-code/csharp.md). +- **Fory gRPC** uses normal .NET gRPC transports with Fory-encoded messages. See [C# gRPC](../grpc/csharp.md). diff --git a/docs/start/dart.md b/docs/start/dart.md index b8f95f41b8..e68c918bbf 100644 --- a/docs/start/dart.md +++ b/docs/start/dart.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory Dart is published on pub.dev and requires Dart SDK 3.7 or later. Generated serializers use `build_runner`. Keep Fory packages and generated code in one application on compatible versions. +Fory Dart provides xlang Object Serialization, generated models, and Fory gRPC. +It is published on pub.dev, requires Dart 3.7 or later, and uses `build_runner` +to generate serializers. ## Verify the Toolchain @@ -28,12 +30,62 @@ dart --version dart pub --help ``` -## Choose a Capability +## Object Serialization -| Capability | Package or tool | Continue with | -| --------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `fory` plus `build_runner` | [Dart object serialization](../object-serialization/dart/index.md) and [xlang](../object-serialization/dart/xlang.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated companions plus `package:grpc` | [Dart gRPC](../grpc/dart.md) | +Add Fory and the generator to `pubspec.yaml`: -Each capability guide owns its exact `pubspec.yaml`, generation command, and first runnable example. +```yaml +dependencies: + fory: 1.5.0 + +dev_dependencies: + build_runner: ^2.4.0 +``` + +Create `lib/person.dart`: + +```dart +import 'package:fory/fory.dart'; + +part 'person.fory.dart'; + +@ForyStruct() +class Person { + Person(); + + @ForyField(type: Int64Type()) + int id = 0; + String name = ''; +} + +void main() { + final fory = Fory(); + PersonForyModule.register(fory, Person, name: 'example.Person'); + + final input = Person() + ..id = 1 + ..name = 'Alice'; + final bytes = fory.serialize(input); + final decoded = fory.deserialize(bytes); + print(decoded.name); +} +``` + +Generate the serializer and run the example: + +```bash +dart pub get +dart run build_runner build +dart run lib/person.dart +``` + +Dart uses xlang mode. Continue with +[Dart Object Serialization](../object-serialization/dart/index.md), +[code generation](../object-serialization/dart/code-generation.md), +[web support](../object-serialization/dart/web-platform-support.md), and +[schema evolution](../object-serialization/dart/schema-evolution.md). + +## Other Capabilities + +- **Fory IDL and Compiler** generates Dart models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [Dart generated-code guide](../compiler/generated-code/dart.md). +- **Fory gRPC** uses package:grpc transports with Fory-encoded messages. See [Dart gRPC](../grpc/dart.md). diff --git a/docs/start/go.md b/docs/start/go.md index fff98fe6ef..969fb106ee 100644 --- a/docs/start/go.md +++ b/docs/start/go.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory Go is published as the Go module `github.com/apache/fory/go/fory` and requires Go 1.24 or later. Use one compatible Fory release across every peer in an application. +Fory Go provides xlang and native Object Serialization, generated models, and +Fory gRPC. It is published as `github.com/apache/fory/go/fory` and requires Go +1.25 or later. ## Verify the Toolchain @@ -28,14 +30,64 @@ go version go env GOPROXY ``` -If a Go proxy has not picked up a new submodule tag yet, retry later or use `GOPROXY=direct` temporarily. +## Object Serialization -## Choose a Capability +Create a module and install the released Fory module: -| Capability | Module or tool | Continue with | -| --------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `github.com/apache/fory/go/fory` | [Go object serialization](../object-serialization/go/index.md), then choose [xlang](../object-serialization/go/xlang.md) or [native](../object-serialization/go/native.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated companions plus grpc-go | [Go gRPC](../grpc/go.md) | +```bash +mkdir fory-example +cd fory-example +go mod init example.com/fory-example +go get github.com/apache/fory/go/fory@v1.5.0 +``` + +If a Go proxy has not picked up a new submodule tag yet, retry later or use +`GOPROXY=direct` temporarily. + +```go title="main.go" +package main + +import ( + "fmt" + + "github.com/apache/fory/go/fory" +) + +type User struct { + ID int64 + Name string +} + +func main() { + f := fory.New(fory.WithXlang(true)) + if err := f.RegisterStruct(User{}, 1); err != nil { + panic(err) + } + + bytes, err := f.Serialize(&User{ID: 1, Name: "Alice"}) + if err != nil { + panic(err) + } + + var decoded User + if err := f.Deserialize(bytes, &decoded); err != nil { + panic(err) + } + fmt.Println(decoded.Name) +} +``` + +```bash +go run . +``` + +Use [xlang mode](../object-serialization/go/basic-serialization.md#cross-language-interoperability) for cross-language data +and [native mode](../object-serialization/go/native.md) for Go-only data. +Continue with [Go Object Serialization](../object-serialization/go/index.md), +[configuration](../object-serialization/go/configuration.md), and +[schema evolution](../object-serialization/go/schema-evolution.md). + +## Other Capabilities -Each capability guide owns its exact module command and first runnable example. +- **Fory IDL and Compiler** generates Go models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [Go generated-code guide](../compiler/generated-code/go.md). +- **Fory gRPC** uses grpc-go transports with Fory-encoded messages. See [Go gRPC](../grpc/go.md). diff --git a/docs/start/index.md b/docs/start/index.md index da24c38d57..ea48248b39 100644 --- a/docs/start/index.md +++ b/docs/start/index.md @@ -1,5 +1,5 @@ --- -title: Getting Started +title: Overview sidebar_position: 0 id: index license: | @@ -19,14 +19,27 @@ license: | limitations under the License. --- -Apache Fory™ releases are available both as source artifacts and language-specific packages. +Start with a runtime page below. Each page includes a release-pinned installation, +a runnable object-serialization round trip, the modes supported by that runtime, +and a short path into every additional Fory capability available there. -For source downloads, see the Apache Fory™ [download](https://fory.apache.org/download) page. +Apache Fory™ releases are available as source artifacts and language-specific +packages. For source downloads, see the Apache Fory™ +[download](https://fory.apache.org/download) page. + +## First Five Minutes + +1. Choose the runtime used by your application. +2. Install the package shown on that runtime page. +3. Run its minimal serialize/deserialize example. +4. Select xlang mode for cross-language data or native mode for same-runtime data + when both are available. +5. Continue to the capability guide for production configuration and advanced APIs. ## Choose a Runtime -Each runtime setup page checks the toolchain and routes to the capabilities that -runtime supports: +Each runtime page provides a release-pinned installation snippet, a minimal +round trip for an application project, and the next capability-specific steps: | Runtime | Setup | | --------------------- | -------------------------------------- | @@ -42,21 +55,17 @@ runtime supports: | Scala | [Scala](scala.md) | | Kotlin | [Kotlin](kotlin.md) | -## Choose a Capability - -| Need | Continue with | -| ----------------------------------- | -------------------------------------------------------- | -| Reconstruct object graphs | [Object Serialization](../object-serialization/index.md) | -| Random or partial analytical access | [Row Format](../row-format/index.md) | -| Exchange standard JSON from Java | [Fory JSON](../json/index.md) | -| Generate models from a schema | [Fory IDL and Compiler](../compiler/index.md) | -| Use generated models over gRPC | [Fory gRPC](../grpc/index.md) | +## What You Can Build -Object Serialization then asks you to choose xlang mode for portable -cross-language payloads or native mode for same-runtime payloads. Use -[Choose a Format](../introduction/choose-a-format.md) when that product decision -is not yet clear. +| Capability | Use it for | Available runtimes | Detailed guide | +| -------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------- | -------------------------------------------------------- | +| Object Serialization | Reconstruct object graphs, including shared references and schema changes | All runtimes | [Object Serialization](../object-serialization/index.md) | +| Row Format | Trusted analytical data with zero-copy, random, or partial field access | Java, Python, C++, Rust | [Row Format](../row-format/index.md) | +| Fory JSON | High-performance standard JSON mapping | Java | [Fory JSON](../json/index.md) | +| Fory IDL | Generate native models and serializers from Fory, protobuf, or FlatBuffers IDL | All runtimes | [Fory IDL and Compiler](../compiler/index.md) | +| Fory gRPC | Use generated models over normal gRPC transports with Fory-encoded messages | Java, Python, C++, Go, Rust, JavaScript, C#, Dart, Scala, Kotlin | [Fory gRPC](../grpc/index.md) | -The selected capability guide owns its exact dependency and first successful -task. Schema compatibility settings and production configuration are -runtime-specific. +Object Serialization uses xlang mode for portable cross-language data. Java, +Python, C++, Go, Rust, Scala, and Kotlin also offer native mode for +same-runtime data. Use [Choose a Format](../introduction/choose-a-format.md) +when you have not yet chosen a format. diff --git a/docs/start/java.md b/docs/start/java.md index 314b4f09b6..bc38eedafd 100644 --- a/docs/start/java.md +++ b/docs/start/java.md @@ -19,7 +19,11 @@ license: | limitations under the License. --- -Fory Java artifacts are published to Maven Central. Fory core supports Java 8 and later; Java Records require Java 17 or later, and Row Format requires Java 11 or later. Keep every Fory artifact in one application on the same version. +Fory Java provides binary Object Serialization, Fory JSON, Row Format, generated +models, and Fory gRPC. Artifacts are published to Maven Central. Fory core and +Fory JSON support Java 8 and later, Java Records require Java 17 or later, and +Row Format requires Java 11 or later. Keep every Fory artifact in one application +on the same version. ## Verify the Toolchain @@ -29,14 +33,100 @@ mvn -version # or: ./gradlew --version ``` -## Choose a Capability +## Object Serialization -| Capability | Artifact or tool | Continue with | -| --------------------------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `fory-core` | [Java object serialization](../object-serialization/java/index.md), then choose [xlang](../object-serialization/java/xlang.md) or [native](../object-serialization/java/native.md) | -| Fory JSON | `fory-json` | [Fory JSON Getting Started](../json/getting-started.md) | -| Row Format | `fory-format` | [Java Row Format](../row-format/java.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated companions plus gRPC Java dependencies | [Java gRPC](../grpc/java.md) | +Use Object Serialization for object graphs. Xlang mode produces data that other +Fory runtimes can read; native mode supports a broader JVM object surface. -Each capability guide owns its exact dependency declaration and first runnable example. +Maven: + +```xml + + org.apache.fory + fory-core + 1.5.0 + +``` + +Gradle: + +```kotlin +implementation("org.apache.fory:fory-core:1.5.0") +``` + +Run this complete xlang round trip: + +```java +import org.apache.fory.Fory; + +public final class ForyExample { + public static final class User { + public long id; + public String name; + + public User() {} + + public User(long id, String name) { + this.id = id; + this.name = name; + } + } + + public static void main(String[] args) { + Fory fory = Fory.builder().withXlang(true).build(); + fory.register(User.class, 1); + + byte[] bytes = fory.serialize(new User(1, "Alice")); + User decoded = (User) fory.deserialize(bytes); + System.out.println(decoded.name); + } +} +``` + +Reuse a `Fory` instance within one thread instead of rebuilding it for every +value. `Fory` is not thread-safe; use `ThreadSafeFory` for shared concurrent +access. Continue with +[Java Object Serialization](../object-serialization/java/index.md), +[xlang mode](../object-serialization/java/basic-serialization.md#cross-language-interoperability), +[native mode](../object-serialization/java/native.md), or +[configuration](../object-serialization/java/configuration.md). + +## Fory JSON + +Fory JSON maps Java objects to standard JSON text and UTF-8 bytes. Add +`fory-json` instead of `fory-core` when the application only needs JSON: + +```kotlin +implementation("org.apache.fory:fory-json:1.5.0") +``` + +Add the import to `ForyExample.java`: + +```java +import org.apache.fory.json.ForyJson; +``` + +Then place the JSON round trip inside `ForyExample.main`: + +```java +ForyJson json = ForyJson.builder().build(); +String text = json.toJson(new User(1, "Alice")); +User jsonDecoded = json.fromJson(text, User.class); +System.out.println(jsonDecoded.name); +``` + +See [Fory JSON Getting Started](../json/getting-started.md) for Maven setup, +object mapping, annotations, Android, GraalVM, and security. + +## Other Capabilities + +- **Row Format** provides random and partial field access for trusted analytical data. See [Java Row Format](../row-format/java.md). +- **Fory IDL and Compiler** generates Java models and registration helpers from Fory IDL, protobuf IDL, or FlatBuffers IDL. See [Compiler Getting Started](../compiler/getting-started.md) and the [Java generated-code guide](../compiler/generated-code/java.md). +- **Fory gRPC** uses normal grpc-java transports with Fory-encoded request and response objects. See [Java gRPC](../grpc/java.md). + +## Runtime Notes + +- On JDK 25 and later, follow the setup in + [Java Object Serialization](../object-serialization/java/index.md). +- For Android, see [Java Android support](../object-serialization/java/android.md). +- For native images, see [Java GraalVM support](../object-serialization/java/graalvm.md). diff --git a/docs/start/javascript.md b/docs/start/javascript.md index 144b41652e..c2577997d5 100644 --- a/docs/start/javascript.md +++ b/docs/start/javascript.md @@ -19,7 +19,10 @@ license: | limitations under the License. --- -Fory JavaScript/TypeScript packages are published on npm. The core package works without native acceleration; the optional `@apache-fory/hps` Node.js fast path requires Node.js 20 or later. Keep Fory packages in one application on compatible versions. +Fory JavaScript/TypeScript provides xlang Object Serialization, generated +models, Node.js gRPC, and browser gRPC-Web clients. Packages are published on +npm. The core package works without native acceleration; the optional +`@apache-fory/hps` Node.js fast path requires Node.js 20 or later. ## Verify the Toolchain @@ -28,12 +31,51 @@ node --version npm --version ``` -## Choose a Capability +## Object Serialization -| Capability | Package or tool | Continue with | -| --------------------------- | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `@apache-fory/core`; optional `@apache-fory/hps` | [JavaScript/TypeScript object serialization](../object-serialization/javascript/index.md) and [xlang](../object-serialization/javascript/xlang.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated Node.js or gRPC-Web companions | [JavaScript gRPC](../grpc/javascript.md) | +Install the core package: -Each capability guide owns its exact install command, environment-specific setup, and first runnable example. +```bash +npm install @apache-fory/core@1.5.0 +``` + +Define a schema and run an xlang round trip: + +```js title="example.cjs" +const { default: Fory, Type } = require("@apache-fory/core"); + +const userType = Type.struct( + { typeName: "example.User" }, + { + id: Type.int64(), + name: Type.string(), + }, +); + +const fory = new Fory(); +const { serialize, deserialize } = fory.register(userType); + +const bytes = serialize({ id: 1n, name: "Alice" }); +console.log(deserialize(bytes)); +``` + +```bash +node example.cjs +``` + +JavaScript uses xlang mode. Continue with +[JavaScript/TypeScript Object Serialization](../object-serialization/javascript/index.md), +[xlang types](../object-serialization/javascript/basic-serialization.md#cross-language-interoperability), +[configuration](../object-serialization/javascript/configuration.md), and +[schema evolution](../object-serialization/javascript/schema-evolution.md). + +For the optional Node.js string fast path, install the matching package version: + +```bash +npm install @apache-fory/core@1.5.0 @apache-fory/hps@1.5.0 +``` + +## Other Capabilities + +- **Fory IDL and Compiler** generates TypeScript interfaces, schemas, and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [JavaScript generated-code guide](../compiler/generated-code/javascript.md). +- **Fory gRPC** supports Node.js gRPC and browser gRPC-Web transports with Fory-encoded messages. See [JavaScript gRPC](../grpc/javascript.md). diff --git a/docs/start/kotlin.md b/docs/start/kotlin.md index 22139e7169..e2fe8f74ab 100644 --- a/docs/start/kotlin.md +++ b/docs/start/kotlin.md @@ -19,7 +19,8 @@ license: | limitations under the License. --- -Fory Kotlin artifacts are published to Maven Central and run on Fory Java. Fory core supports Java 8 and later. Keep Kotlin, Java core, and generated-code artifacts in one application on the same Fory release. +Fory Kotlin provides binary Object Serialization, generated models, Fory gRPC, +and Android support. It runs on Fory Java and supports Java 8 and later. ## Verify the Toolchain @@ -29,13 +30,51 @@ java -version # or: mvn -version ``` -## Choose a Capability +## Object Serialization -| Capability | Artifact or tool | Continue with | -| ---------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Object Serialization | `fory-kotlin` | [Kotlin object serialization](../object-serialization/kotlin/index.md), then choose [xlang](../object-serialization/kotlin/xlang.md) or [native](../object-serialization/kotlin/native.md) | -| Schema and generated models | `fory-compiler` and Kotlin KSP support | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated coroutine companions plus grpc-kotlin | [Kotlin gRPC](../grpc/kotlin.md) | -| Android object serialization | `fory-kotlin` plus generated serializers | [Android](../object-serialization/java/android.md) | +Add the runtime to the application module: -Each capability guide owns its exact Maven or Gradle declaration and first runnable example. +```kotlin title="build.gradle.kts" +dependencies { + implementation("org.apache.fory:fory-kotlin:1.5.0") +} +``` + +Create `src/main/kotlin/KotlinExample.kt`: + +```kotlin +import org.apache.fory.ThreadSafeFory +import org.apache.fory.kotlin.ForyKotlin + +data class User(val id: Long, val name: String) + +fun main() { + val fory: ThreadSafeFory = ForyKotlin.builder() + .withXlang(true) + .requireClassRegistration(true) + .buildThreadSafeFory() + fory.register(User::class.java, 1) + + val bytes = fory.serialize(User(1, "Alice")) + val decoded = fory.deserialize(bytes) as User + println(decoded.name) +} +``` + +If the project applies Gradle's `application` plugin, run its application task: + +```bash +./gradlew run +``` + +Use xlang mode for data shared with other Fory runtimes or native mode for +Kotlin/JVM-only data. Continue with +[Kotlin Object Serialization](../object-serialization/kotlin/index.md), +[xlang](../object-serialization/kotlin/basic-serialization.md#cross-language-interoperability), or +[native mode](../object-serialization/kotlin/native.md). + +## Other Capabilities + +- **Fory IDL and Compiler** generates Kotlin models and registration helpers through KSP. See [Compiler Getting Started](../compiler/getting-started.md) and the [Kotlin generated-code guide](../compiler/generated-code/kotlin.md). +- **Fory gRPC** uses grpc-kotlin and grpc-java transports with Fory-encoded messages. See [Kotlin gRPC](../grpc/kotlin.md). +- **Android** uses generated serializers with the same Fory Kotlin runtime. See [Android Object Serialization](../object-serialization/java/android.md). diff --git a/docs/start/python.md b/docs/start/python.md index cd9c233121..d7c7387bf9 100644 --- a/docs/start/python.md +++ b/docs/start/python.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Python packages are published on PyPI. `pyfory` supports Python 3.8 and later on Linux, macOS, and Windows. Use one compatible Fory release across every peer in an application. +`pyfory` provides binary Object Serialization and Row Format. Fory IDL can also +generate Python models and gRPC companions. The package supports Python 3.8 and +later on Linux, macOS, and Windows. ## Verify the Toolchain @@ -28,13 +30,45 @@ python --version python -m pip --version ``` -## Choose a Capability +## Object Serialization -| Capability | Package or extra | Continue with | -| --------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Object Serialization | `pyfory` | [Python object serialization](../object-serialization/python/index.md), then choose [xlang](../object-serialization/python/xlang.md) or [native](../object-serialization/python/native.md) | -| Row Format | `pyfory[format]` | [Python Row Format](../row-format/python.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | `pyfory` plus `grpcio` | [Python gRPC](../grpc/python.md) | +Install the released package: -Each capability guide owns its exact install command and first runnable example. +```bash +python -m pip install pyfory==1.5.0 +``` + +Run an xlang round trip: + +```python +from dataclasses import dataclass + +import pyfory + + +@dataclass +class User: + name: str + age: pyfory.Int32 + + +fory = pyfory.Fory(xlang=True, ref=True) +fory.register(User, type_id=1) + +data = fory.serialize(User("Alice", 30)) +decoded = fory.deserialize(data) +print(decoded) +``` + +Use [xlang mode](../object-serialization/python/basic-serialization.md#cross-language-interoperability) for cross-language +data. Use [native mode](../object-serialization/python/native.md) for Python-only +objects, including Python callables and serialization hooks. Continue with the +[Python guide](../object-serialization/python/index.md), +[configuration](../object-serialization/python/configuration.md), and +[type registration](../object-serialization/python/type-registration.md). + +## Other Capabilities + +- **Row Format** provides zero-copy field access for trusted analytical data. See [Python Row Format](../row-format/python.md). +- **Fory IDL and Compiler** generates Python models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [Python generated-code guide](../compiler/generated-code/python.md). +- **Fory gRPC** uses grpcio transports with Fory-encoded messages. See [Python gRPC](../grpc/python.md). diff --git a/docs/start/rust.md b/docs/start/rust.md index 9ae691e426..431c944159 100644 --- a/docs/start/rust.md +++ b/docs/start/rust.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory Rust is published on crates.io. The workspace minimum supported Rust version is 1.70 and uses the Rust 2021 edition. Use one compatible Fory release across every peer in an application. +Fory Rust provides xlang and native Object Serialization, standard Row Format, +generated models, and Fory gRPC. The public `fory` crate is published on +crates.io, supports Rust 1.70 or later, and uses the Rust 2021 edition. ## Verify the Toolchain @@ -28,13 +30,47 @@ rustc --version cargo --version ``` -## Choose a Capability +## Object Serialization -| Capability | Crate or tool | Continue with | -| --------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `fory` | [Rust object serialization](../object-serialization/rust/index.md), then choose [xlang](../object-serialization/rust/xlang.md) or [native](../object-serialization/rust/native.md) | -| Standard Row Format | `fory` | [Rust Row Format](../row-format/rust.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | generated companions plus `tonic` and `bytes` | [Rust gRPC](../grpc/rust.md) | +Add the public crate: -Each capability guide owns its exact dependency declaration and first runnable example. +```toml title="Cargo.toml" +[dependencies] +fory = "1.5.0" +``` + +```rust +use fory::{Error, Fory, ForyStruct}; + +#[derive(ForyStruct, Debug, PartialEq)] +struct User { + id: i64, + name: String, +} + +fn main() -> Result<(), Error> { + let mut fory = Fory::builder().xlang(true).build(); + fory.register::(1)?; + + let user = User { + id: 1, + name: "Alice".to_string(), + }; + let bytes = fory.serialize(&user)?; + let decoded: User = fory.deserialize(&bytes)?; + assert_eq!(user, decoded); + Ok(()) +} +``` + +Use [xlang mode](../object-serialization/rust/basic-serialization.md#cross-language-interoperability) for cross-language data +and [native mode](../object-serialization/rust/native.md) for Rust-only data. +Continue with [Rust Object Serialization](../object-serialization/rust/index.md), +[configuration](../object-serialization/rust/configuration.md), and +[type registration](../object-serialization/rust/type-registration.md). + +## Other Capabilities + +- **Row Format** provides zero-copy views over trusted analytical data using the standard Fory Row layout. See [Rust Row Format](../row-format/rust.md). +- **Fory IDL and Compiler** generates Rust models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [Rust generated-code guide](../compiler/generated-code/rust.md). +- **Fory gRPC** uses tonic transports with Fory-encoded messages. See [Rust gRPC](../grpc/rust.md). diff --git a/docs/start/scala.md b/docs/start/scala.md index 6101958a17..5e8a83c51b 100644 --- a/docs/start/scala.md +++ b/docs/start/scala.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory Scala artifacts are published to Maven Central for Scala 2.13 and Scala 3. Schema-generated Scala sources and macro-derived xlang serializers require Scala 3. The runtime uses Fory Java, so keep all Fory artifacts on the same release. +Fory Scala provides binary Object Serialization, generated models, and Fory +gRPC. The runtime artifact supports Scala 2.13 and Scala 3; generated Scala +models require Scala 3. ## Verify the Toolchain @@ -29,12 +31,48 @@ scala -version sbt --version ``` -## Choose a Capability +## Object Serialization -| Capability | Artifact or tool | Continue with | -| --------------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Object Serialization | `fory-scala_2.13` or `fory-scala_3` | [Scala object serialization](../object-serialization/scala/index.md), then choose [xlang](../object-serialization/scala/xlang.md) or [native](../object-serialization/scala/native.md) | -| Schema and generated models | `fory-compiler`; Scala 3 output | [Fory IDL and Compiler](../compiler/index.md) | -| Fory gRPC | Scala 3 generated companions plus grpc-java | [Scala gRPC](../grpc/scala.md) | +Add the runtime to `build.sbt`: -Each capability guide owns its exact Maven or sbt declaration and first runnable example. +```sbt +ThisBuild / scalaVersion := "3.3.1" +libraryDependencies += "org.apache.fory" %% "fory-scala" % "1.5.0" +``` + +Create `src/main/scala/ScalaExample.scala`: + +```scala +import org.apache.fory.Fory +import org.apache.fory.scala.ForyScala + +case class User(id: Long, name: String) + +object ScalaExample { + def main(args: Array[String]): Unit = { + val fory: Fory = ForyScala.builder() + .withXlang(true) + .build() + fory.register(classOf[User], 1) + + val bytes = fory.serialize(User(1, "Alice")) + val decoded = fory.deserialize(bytes).asInstanceOf[User] + println(decoded.name) + } +} +``` + +```bash +sbt run +``` + +Use xlang mode for data shared with other Fory runtimes or native mode for +Scala/JVM-only data. Continue with +[Scala Object Serialization](../object-serialization/scala/index.md), +[xlang](../object-serialization/scala/basic-serialization.md#cross-language-interoperability), or +[native mode](../object-serialization/scala/native.md). + +## Other Capabilities + +- **Fory IDL and Compiler** generates Scala 3 models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [Scala generated-code guide](../compiler/generated-code/scala.md). +- **Fory gRPC** uses grpc-java transports with Fory-encoded messages. See [Scala gRPC](../grpc/scala.md). diff --git a/docs/start/swift.md b/docs/start/swift.md index af494457c8..56bb072fd8 100644 --- a/docs/start/swift.md +++ b/docs/start/swift.md @@ -19,7 +19,9 @@ license: | limitations under the License. --- -Fory Swift is distributed through Swift Package Manager from the Apache Fory repository. The current package uses Swift tools 6.0 and targets macOS 13 or later and iOS 16 or later. Pin one Fory release across every peer in an application. +Fory Swift provides xlang Object Serialization and compiler-generated models. +It is distributed through Swift Package Manager, uses Swift tools 6.0, and +targets macOS 13 or later and iOS 16 or later. ## Verify the Toolchain @@ -27,11 +29,58 @@ Fory Swift is distributed through Swift Package Manager from the Apache Fory rep swift --version ``` -## Choose a Capability +## Object Serialization -| Capability | Package product or tool | Continue with | -| --------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -| Object Serialization | Swift package product `Fory` | [Swift object serialization](../object-serialization/swift/index.md) and [xlang](../object-serialization/swift/xlang.md) | -| Schema and generated models | `fory-compiler` | [Fory IDL and Compiler](../compiler/index.md) | +Create an executable package: -The selected capability guide owns its exact package declaration and first runnable example. +```bash +swift package init --type executable --name ForyExample +``` + +Add the released package and depend on its `Fory` library in the generated `Package.swift`: + +```swift title="Package.swift" +dependencies: [ + .package(url: "https://github.com/apache/fory.git", exact: "1.5.0") +], +targets: [ + .executableTarget( + name: "ForyExample", + dependencies: [.product(name: "Fory", package: "fory")] + ) +] +``` + +Replace `Sources/main.swift` with: + +```swift title="Sources/main.swift" +import Fory + +@ForyStruct +struct User: Equatable { + var id: Int64 = 0 + var name: String = "" +} + +let fory = Fory() +try fory.register(User.self, id: 1) + +let input = User(id: 1, name: "Alice") +let bytes = try fory.serialize(input) +let decoded: User = try fory.deserialize(bytes) +assert(input == decoded) +``` + +```bash +swift run +``` + +Swift uses xlang mode. Continue with +[Swift Object Serialization](../object-serialization/swift/index.md), +[xlang types](../object-serialization/swift/basic-serialization.md#cross-language-interoperability), +[configuration](../object-serialization/swift/configuration.md), and +[schema evolution](../object-serialization/swift/schema-evolution.md). + +## Other Capabilities + +- **Fory IDL and Compiler** generates Swift models and registration helpers. See [Compiler Getting Started](../compiler/getting-started.md) and the [Swift generated-code guide](../compiler/generated-code/swift.md). diff --git a/go/fory/README.md b/go/fory/README.md index d464d5994f..530ba62023 100644 --- a/go/fory/README.md +++ b/go/fory/README.md @@ -139,11 +139,11 @@ For comprehensive documentation, see the [Fory Go Guide](https://fory.apache.org Topics covered: - [Configuration](https://fory.apache.org/docs/object-serialization/go/configuration) - Options and settings -- [Basic Serialization](https://fory.apache.org/docs/object-serialization/go/core-api) - Core APIs and usage patterns +- [Basic Serialization](https://fory.apache.org/docs/object-serialization/go/basic-serialization) - Core APIs and usage patterns - [Type Registration](https://fory.apache.org/docs/object-serialization/go/type-registration) - Registering types for serialization - [Supported Types](https://fory.apache.org/docs/object-serialization/go/supported-types) - Complete type support reference - [References](https://fory.apache.org/docs/object-serialization/go/references) - Circular references and shared objects - [Schema Evolution](https://fory.apache.org/docs/object-serialization/go/schema-evolution) - Forward/backward compatibility -- [Xlang Serialization](https://fory.apache.org/docs/object-serialization/go/xlang) - Multi-language serialization +- [Basic Serialization](https://fory.apache.org/docs/object-serialization/go/basic-serialization) - Default xlang APIs and interoperability - [Thread Safety](https://fory.apache.org/docs/object-serialization/go/thread-safety) - Concurrent usage patterns - [Troubleshooting](https://fory.apache.org/docs/object-serialization/go/troubleshooting) - Common issues and solutions diff --git a/javascript/README.md b/javascript/README.md index 6aa8f0e359..094ee9285a 100644 --- a/javascript/README.md +++ b/javascript/README.md @@ -114,7 +114,7 @@ Readers can skip unknown fields and tolerate missing ones, supporting rolling up Full documentation is available at [fory.apache.org](https://fory.apache.org): - [JavaScript Serialization Guide](https://fory.apache.org/docs/object-serialization/javascript) -- [Xlang Serialization](https://fory.apache.org/docs/object-serialization/javascript/xlang) +- [Basic Serialization](https://fory.apache.org/docs/object-serialization/javascript/basic-serialization) - [Supported Types](https://fory.apache.org/docs/object-serialization/javascript/supported-types) - [Schema Evolution](https://fory.apache.org/docs/object-serialization/javascript/schema-evolution) - [Xlang Serialization Spec](https://fory.apache.org/docs/specification/xlang_serialization_spec) diff --git a/javascript/packages/core/README.md b/javascript/packages/core/README.md index 4f7b569d4b..c9e4f45b46 100644 --- a/javascript/packages/core/README.md +++ b/javascript/packages/core/README.md @@ -141,7 +141,7 @@ const bytes = serialize({ id: 1n, content: "hello from JavaScript" }); ## Documentation - [JavaScript Serialization Guide](https://fory.apache.org/docs/object-serialization/javascript) -- [Xlang Serialization](https://fory.apache.org/docs/object-serialization/javascript/xlang) +- [Basic Serialization](https://fory.apache.org/docs/object-serialization/javascript/basic-serialization) - [Supported Types](https://fory.apache.org/docs/object-serialization/javascript/supported-types) - [Schema Evolution](https://fory.apache.org/docs/object-serialization/javascript/schema-evolution) - [Xlang Serialization Spec](https://fory.apache.org/docs/specification/xlang_serialization_spec) diff --git a/swift/README.md b/swift/README.md index e03477e110..fc271469e6 100644 --- a/swift/README.md +++ b/swift/README.md @@ -486,7 +486,7 @@ ENABLE_FORY_DEBUG_OUTPUT=1 FORY_SWIFT_JAVA_CI=1 mvn -T16 test -Dtest=org.apache. - [Schema Evolution](../docs/object-serialization/swift/schema-evolution.md) - [External-Type Serialization](../docs/object-serialization/swift/external-types.md) - [Custom Serializers](../docs/object-serialization/swift/custom-serializers.md) -- [Xlang Serialization](../docs/object-serialization/swift/xlang.md) +- [Basic Serialization](../docs/object-serialization/swift/basic-serialization.md) - [Xlang Specification](../docs/specification/xlang_serialization_spec.md) - [Xlang Type Mapping](../docs/specification/xlang_type_mapping.md)