Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/docs-and-formatting.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Load this file when changing documentation, public APIs, protocol specs, benchma
- Keep user-facing configuration text action focused. Do not put cache ownership,
cold/hot path, publish ordering, exact-local schema bypass, or TypeDef/TypeMeta routing
invariants there; put implementation invariants in `docs/specification/xlang_implementation_guide.md`,
security classifications in `docs/object-serialization/deserialization-security-model.md`, and agent-only workflow rules in
security classifications in `docs/security/deserialization.md`, and agent-only workflow rules in
`.agents/**`.
- Documentation examples should use normal explicit imports, avoid unused imports, and keep common Fory types unqualified where that is the idiom.
- Generated Markdown under `docs/benchmarks/**` should satisfy markdownlint blank-line rules at generation time: no repeated blank lines and no extra blank line after final content.
Expand Down
6 changes: 3 additions & 3 deletions .github/sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
apache/fory-site@main:
- source: docs/index.md
dest: docs/index.md
- source: docs/images/
dest: docs/images/
deleteOrphaned: true
- source: docs/introduction/
dest: docs/introduction/
deleteOrphaned: true
Expand Down Expand Up @@ -45,9 +48,6 @@ apache/fory-site@main:
- source: docs/development/
dest: docs/development/
deleteOrphaned: true
- source: docs/images/
dest: docs/images/
deleteOrphaned: true
- source: docs/specification/
dest: docs/specification/
deleteOrphaned: true
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
Expand Down Expand Up @@ -2011,6 +2011,8 @@
uses: actions/setup-python@v5
with:
python-version: 3.8
- name: Test release documentation updater
run: python -m unittest ci.test_release
- name: Set up .NET 8
uses: actions/setup-dotnet@v4
with:
Expand Down
19 changes: 10 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,11 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th
- `.agents/docs-and-formatting.md`: documentation, specification, and markdown rules.
- `.agents/ci-and-pr.md`: code review workflow, CI triage, PR expectations, and commit conventions.
- `.agents/testing/integration-tests.md`: `integration_tests/` prerequisites, regeneration rules, and commands.
- `docs/object-serialization/security.md`: user-facing security guidance for binary object
serialization.
- `docs/security/index.md`: contributor-facing security model index. This directory is internal
documentation and is intentionally excluded from the website sync.
- `docs/security/threat-model.md`: project trust boundaries and downstream responsibilities.
- `docs/security/deserialization.md`: implementation boundaries for untrusted deserialization.
- `docs/json/security.md`: user-facing security guidance for Fory JSON.
- `docs/object-serialization/deserialization-security-model.md`: implementation boundaries for
untrusted deserialization classification.
- `.agents/languages/java.md`
- `.agents/languages/csharp.md`
- `.agents/languages/cpp.md`
Expand All @@ -34,7 +34,7 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th
- Respect ownership. Keep logic, state, and helpers in their natural owner, and do not move serializer-local, context-local, runtime-type-local, or protocol-local problems into global utilities.
- Check the spec before implementation. For wire behavior and xlang mapping, use the specs as the source of truth and never copy one runtime's bug into another runtime just to make tests pass.
- Do not make assumptions about runtime behavior, ownership, registration, metadata construction, protocol semantics, or test coverage. Read the current code, owning docs/specs, and relevant tests before making a design judgment or implementation decision. If the evidence is incomplete, inspect more or state the uncertainty explicitly instead of filling gaps from memory or analogy with another runtime.
- For untrusted deserialization, read `docs/object-serialization/deserialization-security-model.md` before changing allocation, stream filling, skip, reference, metadata, or policy validation behavior. Variable-length deserialization must not allocate or reserve backing/output capacity from attacker-declared lengths or counts before the byte owner has proven proportional readable bytes with `checkReadableBytes` or the runtime equivalent. Root graph memory reservation is accounting only and may happen before that byte check, but it must not replace the byte check.
- For untrusted deserialization, read `docs/security/deserialization.md` before changing allocation, stream filling, skip, reference, metadata, or policy validation behavior. Variable-length deserialization must not allocate or reserve backing/output capacity from attacker-declared lengths or counts before the byte owner has proven proportional readable bytes with `checkReadableBytes` or the runtime equivalent. Root graph memory reservation is accounting only and may happen before that byte check, but it must not replace the byte check.
- Malformed input must surface as a controlled root-operation error and still run
root cleanup, but the exact exception type, error code, message, detection
layer, and detection point are not contracts unless a public API or
Expand Down Expand Up @@ -331,7 +331,8 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th

## Security

User-facing security guidance lives only under Object Serialization and Fory JSON. Read
`docs/object-serialization/security.md` or `docs/json/security.md` for the selected product. Before
reporting or changing allocation, stream filling, skip, reference, metadata, or policy validation
behavior, read `docs/object-serialization/deserialization-security-model.md`.
User-facing object-serialization security guidance lives in each runtime directory, such as
`docs/object-serialization/java/security.md`; Fory JSON guidance lives in `docs/json/security.md`.
Contributor-facing security models remain under `docs/security/` and are excluded from the website.
Before reporting or changing allocation, stream filling, skip, reference, metadata, or policy
validation behavior, read `docs/security/deserialization.md`.
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Key points:

## Testing

For environmental requirements, please check [DEVELOPMENT.md](./docs/development/building.md).
For environmental requirements, see the [Development guide](./docs/development/index.md).

### Python

Expand Down Expand Up @@ -229,7 +229,7 @@ To use Jetbrains IDEA IDE for Java Development, you need to configure the projec
And due to the usage of `sun.misc.Unsafe` API, which is not visible in Java 11+, you need to configure java compiler with `--releaese` option disabled.

<div align="center">
<img width="65%" alt="" src="docs/images/idea_jdk11.png"><br>
<img width="65%" alt="" src="docs/images/idea-jdk11.png"><br>
</div>

## Website
Expand All @@ -244,4 +244,4 @@ If you want write a blog, or update other contents about the website, please sub

## Development

For more information, please refer to [Development Guide](./docs/development/building.md).
For more information, see the [Development guide](./docs/development/index.md).
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<div align="center">
<img width="65%" alt="Apache Fory logo" src="docs/images/logo/fory-horizontal.png"><br>
<img width="65%" alt="Apache Fory logo" src="docs/images/fory-logo.png"><br>
</div>

[![Build Status](https://img.shields.io/github/actions/workflow/status/apache/fory/ci.yml?branch=main&style=for-the-badge&label=GITHUB%20ACTIONS&logo=github)](https://github.com/apache/fory/actions/workflows/ci.yml)
Expand Down Expand Up @@ -305,19 +305,19 @@ serializer setup.

**Development From Source**

See [docs/development/building.md](docs/development/building.md).
See [docs/development/index.md](docs/development/index.md).

Snapshots for Java, Scala, and Kotlin are available from
`https://repository.apache.org/snapshots/` with the matching `-SNAPSHOT` version.

## Choose a Serialization Format

| Format | Use it when | Start here |
| ------------- | ------------------------------------------------------------- | ------------------------------------------------------- |
| Xlang binary | Data crosses language boundaries | [Xlang guide](docs/object-serialization/xlang/index.md) |
| Native binary | Producer and consumer are in the same runtime family | [Native guide](docs/object-serialization/native.md) |
| Row format | You need random field access or analytics-style partial reads | [Row Format](docs/row-format/index.md) |
| Fory JSON | Java applications need high-performance standard JSON | [Fory JSON](java/fory-json/README.md) |
| Format | Use it when | Start here |
| ---------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- |
| Xlang binary (default) | Data crosses language boundaries | [Cross-language interoperability](docs/object-serialization/xlang.md) |
| Native binary | Producer and consumer are in the same runtime family | [Native guide](docs/object-serialization/native.md) |
| Row format | You need random field access or analytics-style partial reads | [Row Format](docs/row-format/index.md) |
| Fory JSON | Java applications need high-performance standard JSON | [Fory JSON](java/fory-json/README.md) |

For Java, Scala, Kotlin, Python, C++, Go, and Rust, use native mode for
same-language traffic. It avoids xlang's cross-language type mapping and
Expand Down Expand Up @@ -616,7 +616,7 @@ fun main() {
```

For shared/circular references, polymorphism, numeric IDs versus names, and
type-mapping rules, see the [cross-language guide](docs/object-serialization/xlang) and
type-mapping rules, see the [cross-language guide](docs/object-serialization/xlang.md) and
[type mapping specification](docs/specification/xlang_type_mapping.md).

## Native Serialization
Expand Down Expand Up @@ -909,11 +909,11 @@ security, and platform guides.
| Dart | [docs/object-serialization/dart](docs/object-serialization/dart) | [View](https://fory.apache.org/docs/object-serialization/dart/) |
| Scala | [docs/object-serialization/scala](docs/object-serialization/scala) | [View](https://fory.apache.org/docs/object-serialization/scala/) |
| Kotlin | [docs/object-serialization/kotlin](docs/object-serialization/kotlin) | [View](https://fory.apache.org/docs/object-serialization/kotlin/) |
| Cross-language xlang | [docs/object-serialization/xlang](docs/object-serialization/xlang) | [View](https://fory.apache.org/docs/object-serialization/xlang/) |
| Cross-language xlang | [docs/object-serialization/xlang.md](docs/object-serialization/xlang.md) | [View](https://fory.apache.org/docs/object-serialization/xlang) |
| Schema IDL/compiler | [docs/compiler](docs/compiler) | [View](https://fory.apache.org/docs/compiler/) |
| GraalVM native image | [docs/object-serialization/java/graalvm.md](docs/object-serialization/java/graalvm.md) | [View](https://fory.apache.org/docs/object-serialization/java/graalvm) |
| Android | [docs/object-serialization/java/android.md](docs/object-serialization/java/android.md) | [View](https://fory.apache.org/docs/object-serialization/java/android) |
| Development | [docs/development/building.md](docs/development/building.md) | [View](docs/development/building.md) |
| Development | [docs/development/index.md](docs/development/index.md) | [View](docs/development/index.md) |

**Specifications**

Expand All @@ -934,7 +934,7 @@ security, and platform guides.
## Contributing

Read [CONTRIBUTING.md](CONTRIBUTING.md) and
[docs/development/building.md](docs/development/building.md) before sending pull requests. Bug
[docs/development/index.md](docs/development/index.md) before sending pull requests. Bug
reports, docs fixes, tests, benchmarks, and implementation improvements are welcome.

## License
Expand Down
7 changes: 4 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,11 @@ GitHub issues or pull requests for security reports.

## Security Models

User-facing guidance is product-specific:
User-facing guidance is capability- and runtime-specific:

- [Object Serialization Security](docs/object-serialization/security.md)
- [Object Serialization runtime guides](docs/object-serialization/index.md) (each runtime section
ends with its own Security page)
- [Fory JSON Security](docs/json/security.md)

For detailed implementation classification rules for untrusted deserialization, see the
[Deserialization Security Model](docs/object-serialization/deserialization-security-model.md).
[Deserialization Security Model](docs/security/deserialization.md).
13 changes: 13 additions & 0 deletions ci/release.py
Original file line number Diff line number Diff line change
Expand Up @@ -1237,6 +1237,19 @@ def _update_release_doc_lines(lines, release_version):


def _update_release_doc_line(line, release_version):
scoped_patterns = (
r"(\bpyfory(?:\[[^\]]+\])?==)" + VERSION_PATTERN,
r"(\bgithub\.com/apache/fory/go/fory@v)" + VERSION_PATTERN,
r"(@apache-fory/(?:core|hps)@)" + VERSION_PATTERN,
)
scoped_update = line
has_scoped_dependency = False
for pattern in scoped_patterns:
if re.search(pattern, scoped_update):
has_scoped_dependency = True
scoped_update = re.sub(pattern, r"\g<1>" + release_version, scoped_update)
if has_scoped_dependency:
return scoped_update
if not _is_release_doc_line(line):
return line
if "crates.io-v" in line:
Expand Down
68 changes: 68 additions & 0 deletions ci/test_release.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

import unittest

if __package__:
from . import release
else:
import release


class ReleaseDocVersionTest(unittest.TestCase):
def test_updates_release_doc_dependencies(self):
lines = [
"python -m pip install pyfory==1.5.0\n",
'python -m pip install "pyfory[format]==1.5.0"\n',
"go get github.com/apache/fory/go/fory@v1.5.0\n",
"npm install @apache-fory/core@1.5.0 @apache-fory/hps@1.5.0\n",
"python -m pip install pyfory==1.5.0 grpcio==1.73.0 # Fory 1.4.0\n",
"go get github.com/apache/fory/go/fory@v1.5.0 # Fory 1.4.0\n",
"npm install @apache-fory/core@1.5.0 grpc@1.73.0 @apache-fory/hps@1.5.0\n",
'fory = "1.5.0"\n',
" fory: 1.5.0\n",
' .package(url: "https://github.com/apache/fory.git", exact: "1.5.0")\n',
'implementation("org.apache.fory:fory-core:1.5.0")\n',
'libraryDependencies += "org.apache.fory" %% "fory-scala" % "1.5.0"\n',
'implementation("io.grpc:grpc-api:1.73.0")\n',
"The wire format was introduced in version 1.5.0.\n",
]
expected = [
"python -m pip install pyfory==1.6.0\n",
'python -m pip install "pyfory[format]==1.6.0"\n',
"go get github.com/apache/fory/go/fory@v1.6.0\n",
"npm install @apache-fory/core@1.6.0 @apache-fory/hps@1.6.0\n",
"python -m pip install pyfory==1.6.0 grpcio==1.73.0 # Fory 1.4.0\n",
"go get github.com/apache/fory/go/fory@v1.6.0 # Fory 1.4.0\n",
"npm install @apache-fory/core@1.6.0 grpc@1.73.0 @apache-fory/hps@1.6.0\n",
'fory = "1.6.0"\n',
" fory: 1.6.0\n",
' .package(url: "https://github.com/apache/fory.git", exact: "1.6.0")\n',
'implementation("org.apache.fory:fory-core:1.6.0")\n',
'libraryDependencies += "org.apache.fory" %% "fory-scala" % "1.6.0"\n',
'implementation("io.grpc:grpc-api:1.73.0")\n',
"The wire format was introduced in version 1.5.0.\n",
]

updated = release._update_release_doc_lines(lines, "1.6.0")

self.assertEqual(expected, updated)
self.assertEqual(expected, release._update_release_doc_lines(updated, "1.6.0"))


if __name__ == "__main__":
unittest.main()
49 changes: 49 additions & 0 deletions ci/test_validate_fory_site_sync.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

import pathlib
import tempfile
import unittest

if __package__:
from . import validate_fory_site_sync
else:
import validate_fory_site_sync


class ForySiteSyncTest(unittest.TestCase):
def test_rejects_forbidden_doc_roots(self):
for source, dest in (
("docs/security/", "docs/security/"),
("docs/development/", "docs/security/"),
):
content = f"""apache/fory-site@main:
- source: {source}
dest: {dest}
"""
with (
self.subTest(source=source, dest=dest),
tempfile.TemporaryDirectory() as directory,
):
sync_file = pathlib.Path(directory) / "sync.yml"
sync_file.write_text(content, encoding="utf-8")
with self.assertRaisesRegex(RuntimeError, "must not be synced"):
validate_fory_site_sync.parse_sync_mappings(sync_file)


if __name__ == "__main__":
unittest.main()
14 changes: 14 additions & 0 deletions ci/validate_fory_site_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,16 @@
from typing import List, Tuple

TARGET_REPO = "apache/fory-site@main"
FORBIDDEN_SYNC_ROOTS = (
pathlib.PurePosixPath("docs/security"),
)


def is_forbidden_sync_path(path: str) -> bool:
candidate = pathlib.PurePosixPath(path.rstrip("/"))
return any(
candidate == root or root in candidate.parents for root in FORBIDDEN_SYNC_ROOTS
)


def parse_sync_mappings(sync_file: pathlib.Path) -> List[Tuple[str, str]]:
Expand Down Expand Up @@ -55,6 +65,10 @@ def parse_sync_mappings(sync_file: pathlib.Path) -> List[Tuple[str, str]]:
dest_match = re.match(r"^dest:\s*(.+)$", stripped)
if dest_match and source:
dest = dest_match.group(1).strip().strip("'\"")
if is_forbidden_sync_path(source) or is_forbidden_sync_path(dest):
raise RuntimeError(
f"path must not be synced to fory-site: {source} -> {dest}"
)
mappings.append((source, dest))
source = None

Expand Down
Loading
Loading