Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 32 additions & 20 deletions .agents/languages/java.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,27 +176,31 @@ Load this file when changing anything under `java/` or when Java drives a cross-
- If changes touch GraalVM bootstrap, serializer retention, native-image metadata, or `ObjectStreamSerializer` GraalVM behavior, verify the native-image build and run the produced binary; a plain Java compile is insufficient.
- Put latest-JDK or virtual-thread tests in the latest-JDK test modules with the matching compiler/profile floor, and centralize runtime-version probing in existing compatibility utilities.
- For JDK25+ zero-Unsafe work, preserve serializer-family selection by type and configuration. Do not switch a type from `ObjectStreamSerializer` or another Fory serializer family to `JavaSerializer`, a JDK stream fallback, or any broad `java.* Serializable` fallback by JDK version or no-arg-constructor shape.
- JDK25+ zero-Unsafe runtime support must distinguish launch shape. When Fory is on the module
path, use `--add-opens=java.base/java.lang.invoke=org.apache.fory.core`; when Fory is on the
classpath, use `--add-opens=java.base/java.lang.invoke=ALL-UNNAMED`. Missing this open is an
invalid access configuration, not a reason to open per-package JDK internals or switch
serializer/object-creation families. JPMS tests that validate named-module access should keep the
`org.apache.fory.core` target.
- Do not probe JDK25+ trusted-lookup availability and turn `_JDKAccess` field-access booleans false when the required `java.base/java.lang.invoke` open is missing. Keep those access flags true on JDK25+ and let the owning trusted-lookup path raise the configuration error.
- JDK25+ access must distinguish launch shape. Opening `java.base/java.lang.invoke` is not required
for normal launches while the current-JDK Unsafe fallback is available, but is recommended. It is
required for zero-Unsafe launches or when that fallback is unavailable. When Fory is on the
module path, target `org.apache.fory.core`; when Fory is on the classpath, target `ALL-UNNAMED`.
JPMS tests that validate named-module access should keep the `org.apache.fory.core` target.
- Do not probe JDK25+ trusted-lookup availability and turn `_JDKAccess` field-access booleans false
when the optional `java.base/java.lang.invoke` open is missing. Keep those access flags true on
JDK25+ and let `_Lookup` try the direct access path followed by the current-JDK Unsafe fallback.
Raise the access error only when both paths are unavailable.
- Keep JDK25+ unsafe-removal implementation invariants in agent/design docs and tests, not user guides. User guides should document user actions such as `--sun-misc-unsafe-memory-access=deny` and `java.base/java.lang.invoke` opens; do not expose internal serializer names, owner-model rationale, or avoided fallback strategies there.
- JDK25+ user docs must not require application module package opens for Fory private-field access.
The only required platform open is `java.base/java.lang.invoke`, targeted to `ALL-UNNAMED` for
classpath runs or `org.apache.fory.core` for module-path runs; application module package opens
are not part of this design.
The only platform open Fory recommends is `java.base/java.lang.invoke`, targeted to `ALL-UNNAMED`
for classpath runs or `org.apache.fory.core` for module-path runs. Describe it as not required but
recommended for normal launches, and required when Unsafe access is disabled or unavailable.
Application module package opens are not part of this design.
- JDK25+ final-field user docs must not tell ordinary classes to implement
`java.io.Serializable`. Fory supports ordinary non-Serializable classes; mention
`Serializable` only for JDK serialization hook examples or `java.*` serializability checks.
- JDK25+ final-field user docs must not include fallback advice such as switching unsupported
classes to records, no-arg constructors, or custom serializers. Keep user docs focused on the
supported runtime setup and normal class model.
- Do not create a separate JDK25+ support user-guide page for the Java runtime setup unless the
user explicitly asks for one. Keep the `java.base/java.lang.invoke` open in install-facing docs
such as the Java/Kotlin/Scala install sections and README.
user explicitly asks for one. Keep guidance that opening `java.base/java.lang.invoke` is not
required but recommended in install-facing docs such as the Java/Kotlin/Scala install sections
and README.
- JDK25+ zero-Unsafe final-field writes must use a true target-class trusted lookup from the original `IMPL_LOOKUP`, not `IMPL_LOOKUP.in(type)`. JDK26+ normal Fory final-field restoration must pass with `--illegal-final-field-mutation=deny` and must not require `--enable-final-field-mutation`.
- For JDK25+ object creation, do not use `sun.reflect.ReflectionFactory`, `jdk.unsupported`, or an
Unsafe-backed object instantiator. Normal JVM no-constructor construction must use the
Expand All @@ -205,8 +209,9 @@ Load this file when changing anything under `java/` or when Java drives a cross-
constructor validation. ObjectStream-compatible serializers own the separate
`ParentNoArgCtrInstantiator` path and must keep Java serialization parent-constructor rules. The
JDK25+ ReflectionFactory path uses trusted-lookup access to `jdk.internal.reflect.ReflectionFactory`
in `java.base` and must not require `--add-opens=java.base/jdk.internal.reflect=...`; the only
JDK25+ platform open remains `java.base/java.lang.invoke=org.apache.fory.core`. GraalVM JDK25+
in `java.base` and must not require `--add-opens=java.base/jdk.internal.reflect=...`;
`java.base/java.lang.invoke` remains the only platform open Fory may use, with the launch-shape
target described above. GraalVM JDK25+
native-image ordinary serializers may use an `ObjectStreamClass.newInstance` MethodHandle only
for the exact Serializable case where the serialization constructor class is `Object`; that
preserves normal empty-instance semantics because no user superclass constructor can run. For
Expand All @@ -225,7 +230,13 @@ Load this file when changing anything under `java/` or when Java drives a cross-
- `UnsafeObjectInstantiator` is the JDK8-24 Unsafe owner only. It must be a top-level instantiator
with a Java25 multi-release stub that contains no Unsafe, ObjectStream, ReflectionFactory, or
constructor-bypass implementation.
- Keep the Java25 `_Lookup` overlay unless a future refactor can merge it without exposing Unsafe to the JDK25 class graph. Root `_Lookup` uses Unsafe for the JDK8-24 trusted-lookup fast path, while Java25 `_Lookup` uses the required `java.lang.invoke` open. `DefineClass` is root-owned; when Java25+ generated serializers need hidden nestmate class definition, it must use cached method handles and reflective `Lookup.ClassOption.NESTMATE` loading so Java 8 through Java 14 can still load the root class safely.
- Keep the Java25 `_Lookup` overlay unless a future refactor can merge it without exposing Unsafe to
the JDK25 class graph. Root `_Lookup` uses Unsafe for the JDK8-24 trusted-lookup fast path, while
Java25 `_Lookup` first uses the optional `java.lang.invoke` open and falls back through the split
current-JDK Unsafe lookup when the open is absent. `DefineClass` is root-owned; when Java25+
generated serializers need hidden nestmate class definition, it must use cached method handles
and reflective `Lookup.ClassOption.NESTMATE` loading so Java 8 through Java 14 can still load the
root class safely.
- Treat `ByteArrayOutputStream` and `ByteArrayInputStream` as ordinary streams on every JDK. Do
not restore private-buffer wrapping for JDK8-24 performance, because that reintroduces
`java.base/java.io` private-field ownership and module-open requirements.
Expand Down Expand Up @@ -272,12 +283,13 @@ Load this file when changing anything under `java/` or when Java drives a cross-
focused on field and array access, keep serialization hook discovery in serializer-owned code,
and keep `_JDKAccess` limited to JDK lookup, module, function factory, and access-flag
primitives.
- JDK25+ serialization hook access must use the required trusted lookup from
`java.base/java.lang.invoke=org.apache.fory.core`. Keep `sun.reflect.ReflectionFactory` as a
JDK8-24 hook optimization only, and do not add per-type reflective escapes for hook invocation.
- JDK25+ serialization hook access must use the trusted lookup obtained through the optional
`java.lang.invoke` open or the existing current-JDK Unsafe fallback. Keep
`sun.reflect.ReflectionFactory` as a JDK8-24 hook optimization only, and do not add per-type
reflective escapes for hook invocation.
- JDK25+ `PlatformStringUtils` getter methods sit behind `StringSerializer` static-final access
gates. Do not add per-call access checks in those getters; missing module opens should fail at
trusted-lookup initialization or cold setup, not inside string hot paths.
gates. Do not add per-call access checks in those getters; failures after both direct lookup and
the current-JDK Unsafe fallback should surface during cold setup, not inside string hot paths.
- `FieldAccessor` owns field-accessor dispatch. `RecordFieldAccessors` owns record field access,
and `InstanceFieldAccessors` owns non-record instance field access. Do not reintroduce a
`FieldAccessorFactory` layer. Treat `InstanceFieldAccessors` as package-owned implementation
Expand Down
18 changes: 16 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down Expand Up @@ -2043,13 +2043,27 @@
ref: main
path: fory-site

- name: Set up Node.js 20.x
- name: Set up Node.js 24.x
uses: actions/setup-node@v4
with:
node-version: 20.x
node-version: 24.x
cache: npm
cache-dependency-path: fory-site/package-lock.json

- name: Install fory-site dependencies
working-directory: fory-site
run: npm ci --prefer-offline --fund=false

- name: Restore Docusaurus cache
uses: actions/cache@v4
with:
path: fory-site/node_modules/.cache
key: ${{ runner.os }}-node24-docusaurus-${{ hashFiles('fory-site/package-lock.json') }}

- name: Test fory-site validator
working-directory: fory
run: python3 -m unittest ci.test_validate_fory_site_sync

- name: Sync files and validate site build
run: |
python3 fory/ci/validate_fory_site_sync.py fory fory-site
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,9 @@ Gradle:
implementation "org.apache.fory:fory-core:1.5.0"
```

On JDK25+, open `java.lang.invoke` to Fory. Use `ALL-UNNAMED` when Fory is on
the classpath:
On JDK25+, opening `java.lang.invoke` to Fory core is not required, but is recommended. It avoids the
current-JDK Unsafe fallback and is required when Unsafe access is disabled or unavailable, including
with `--sun-misc-unsafe-memory-access=deny`. Use `ALL-UNNAMED` when Fory is on the classpath:

```bash
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
Expand Down
72 changes: 72 additions & 0 deletions ci/test_validate_fory_site_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import pathlib
import tempfile
import unittest
from unittest import mock

if __package__:
from . import validate_fory_site_sync
Expand All @@ -44,6 +45,77 @@ def test_rejects_forbidden_doc_roots(self):
with self.assertRaisesRegex(RuntimeError, "must not be synced"):
validate_fory_site_sync.parse_sync_mappings(sync_file)

def test_preserves_site_versions(self):
with tempfile.TemporaryDirectory() as directory:
root = pathlib.Path(directory)
fory_root = root / "fory"
site_root = root / "fory-site"
(fory_root / ".github").mkdir(parents=True)
(fory_root / "docs").mkdir()
(site_root / "docs").mkdir(parents=True)
(site_root / "versioned_docs" / "version-9.9.9").mkdir(parents=True)
(site_root / "versioned_sidebars").mkdir()

(fory_root / ".github" / "sync.yml").write_text(
"""apache/fory-site@main:
- source: docs/index.md
dest: docs/index.md
""",
encoding="utf-8",
)
(fory_root / "docs" / "index.md").write_text("current\n", encoding="utf-8")
(site_root / "docs" / "index.md").write_text("old\n", encoding="utf-8")
versioned_doc = site_root / "versioned_docs" / "version-9.9.9" / "index.md"
versioned_doc.write_text("released\n", encoding="utf-8")
versioned_sidebar = (
site_root / "versioned_sidebars" / "version-9.9.9-sidebars.json"
)
versioned_sidebar.write_text("{}\n", encoding="utf-8")
versions_json = site_root / "versions.json"
versions_json.write_text('["9.9.9"]\n', encoding="utf-8")
config = site_root / "docusaurus.config.ts"
config.write_text("lastVersion: '9.9.9'\n", encoding="utf-8")

with (
mock.patch.object(
validate_fory_site_sync,
"parse_args",
return_value=mock.Mock(
fory_root=fory_root, fory_site_root=site_root
),
),
mock.patch.object(validate_fory_site_sync, "run_site_commands") as run,
):
self.assertEqual(validate_fory_site_sync.main(), 0)

run.assert_called_once_with(site_root)
self.assertEqual((site_root / "docs" / "index.md").read_text(), "current\n")
self.assertEqual(versioned_doc.read_text(), "released\n")
self.assertEqual(versioned_sidebar.read_text(), "{}\n")
self.assertEqual(versions_json.read_text(), '["9.9.9"]\n')
self.assertEqual(config.read_text(), "lastVersion: '9.9.9'\n")

def test_uses_single_locale_build(self):
site_root = pathlib.Path("fory-site")
with mock.patch.object(validate_fory_site_sync.subprocess, "run") as run:
validate_fory_site_sync.run_site_commands(site_root)

self.assertEqual(
run.call_args_list,
[
mock.call(
("npm", "run", "lint", "--if-present"),
cwd=site_root,
check=True,
),
mock.call(
("npm", "run", "build", "--", "--locale", "en-US"),
cwd=site_root,
check=True,
),
],
)


if __name__ == "__main__":
unittest.main()
75 changes: 4 additions & 71 deletions ci/validate_fory_site_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,7 @@
from typing import List, Tuple

TARGET_REPO = "apache/fory-site@main"
FORBIDDEN_SYNC_ROOTS = (
pathlib.PurePosixPath("docs/security"),
)
FORBIDDEN_SYNC_ROOTS = (pathlib.PurePosixPath("docs/security"),)


def is_forbidden_sync_path(path: str) -> bool:
Expand Down Expand Up @@ -102,76 +100,12 @@ def sync_files(
print(f"synced {source} -> {dest}")


def rewrite_versions_block(text: str) -> str:
marker = "versions:"
idx = text.find(marker)
if idx == -1:
return text

brace_start = text.find("{", idx)
if brace_start == -1:
return text

depth = 0
end = -1
i = brace_start
while i < len(text):
ch = text[i]
if ch == "{":
depth += 1
elif ch == "}":
depth -= 1
if depth == 0:
end = i
break
i += 1

if end == -1:
return text

j = end + 1
while j < len(text) and text[j].isspace():
j += 1
if j < len(text) and text[j] == ",":
j += 1

replacement = (
"versions: {\n"
" current: {\n"
" label: 'dev',\n"
" },\n"
" },"
)
return text[:idx] + replacement + text[j:]


def patch_docusaurus_config(path: pathlib.Path) -> None:
if not path.exists():
return
text = path.read_text(encoding="utf-8")
text = re.sub(r"locales:\s*\[[^\]]*\]", "locales: ['en-US']", text, count=1)
text = re.sub(r"lastVersion:\s*'[^']*'", "lastVersion: 'current'", text)
text = rewrite_versions_block(text)
path.write_text(text, encoding="utf-8")


def prune_for_fast_build(site_root: pathlib.Path) -> None:
for directory in ("i18n", "versioned_docs", "versioned_sidebars"):
shutil.rmtree(site_root / directory, ignore_errors=True)

versions_json = site_root / "versions.json"
if versions_json.exists():
versions_json.write_text("[]\n", encoding="utf-8")

patch_docusaurus_config(site_root / "docusaurus.config.ts")
patch_docusaurus_config(site_root / "docusaurus.config.js")


def run_site_commands(site_root: pathlib.Path) -> None:
# Limit rendering without rewriting the site's released-version state. The normal build command
# also preserves site-owned Docusaurus acceleration such as `future.faster`.
for command in (
("npm", "install"),
("npm", "run", "lint", "--if-present"),
("npm", "run", "build"),
("npm", "run", "build", "--", "--locale", "en-US"),
):
subprocess.run(command, cwd=site_root, check=True)

Expand All @@ -197,7 +131,6 @@ def main() -> int:
raise FileNotFoundError(f"fory-site directory not found: {site_root}")

sync_files(fory_root, site_root, sync_file)
prune_for_fast_build(site_root)
run_site_commands(site_root)
return 0

Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,6 @@ Note: Results depend on hardware, dataset, and runtime versions. See the

## Read Results Responsibly

Start with [Methodology](methodology.md), then open the report whose capability, runtime, schema,
Start with [Methodology](methodology.md), then open the report whose capability, language, schema,
mode, and operation match your workload. The checked-in results are historical evidence from their
recorded environment, not a guarantee for a different application or current main branch.
1 change: 1 addition & 0 deletions docs/benchmarks/json/_category_.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"position": 4,
"key": "benchmark-json",
"label": "Fory JSON",
"collapsible": true,
"collapsed": true
Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarks/methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ comparing different semantics as if they were equivalent.

Compare one operation and equivalent data semantics. Do not compare an xlang payload with a native
payload or a full object reconstruction with Row Format field access without naming that semantic
difference. JIT- and codegen-based runtimes require representative warm-up.
difference. JIT- and codegen-based implementations require representative warm-up.

Run the active benchmark harness for decisions that depend on current code. Checked-in reports are
evidence for the commit and environment they record.
1 change: 1 addition & 0 deletions docs/benchmarks/object-serialization/_category_.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"position": 3,
"key": "benchmark-object-serialization",
"label": "Object Serialization",
"collapsible": true,
"collapsed": true
Expand Down
8 changes: 4 additions & 4 deletions docs/compiler/generated-code/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,12 +156,12 @@ union Wrapper {
| Swift | `toBytes` / `fromBytes` |
| Dart | (via `fory.serialize()`) |

## Runtime References
## Language References

Choose the generated-code reference for the output runtime. Generated models remain ordinary
runtime-owned types and use that runtime's supported Fory serialization APIs.
Choose the generated-code reference for the target language. Generated models remain ordinary
language-native types and use the supported Fory serialization APIs for that language.

| Runtime | Generated-code reference |
| Language | Generated-code reference |
| --------------------- | -------------------------------------- |
| Java | [Java](java.md) |
| Python | [Python](python.md) |
Expand Down
2 changes: 1 addition & 1 deletion docs/compiler/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ foryc animals.fdl --java_out=./generated/java --python_out=./generated/python --
```

The generated service code uses normal gRPC APIs, but request and response
objects are serialized with Fory. See [Fory gRPC](../grpc/index.md) for runtime
objects are serialized with Fory. See [Fory gRPC](../grpc/index.md) for language-specific
dependencies, server and client setup, streaming modes, browser support, and
interoperability boundaries.

Expand Down
Loading
Loading