Skip to content

feat(go): use option for optional fields #3202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chaokunyang merged 22 commits into from
Jan 26, 2026
Merged

feat(go): use option for optional fields #3202

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
c394138
use option for optional fields
chaokunyang Jan 25, 2026
dc3b5d0
feat(go): add optional field support
chaokunyang Jan 26, 2026
21e6815
fix(python): restore bytes xlang out-of-band path
chaokunyang Jan 26, 2026
e7466f0
update DATE/TIMESTAMP
chaokunyang Jan 26, 2026
466b1b3
fix benchmark
chaokunyang Jan 26, 2026
346ddd9
refactor go struct serialization
chaokunyang Jan 26, 2026
bd09956
refactor optional
chaokunyang Jan 26, 2026
a689b65
fix refl
chaokunyang Jan 26, 2026
491e843
delete fory reflect in go
chaokunyang Jan 26, 2026
06ff6fd
refactor struct serializer
chaokunyang Jan 26, 2026
5d28d41
remove unrolloed int64 write
chaokunyang Jan 26, 2026
a368d47
fix test error
chaokunyang Jan 26, 2026
b4c44ee
fix(go): align nullable compatibility checks
chaokunyang Jan 26, 2026
3fc1a60
fix python tests
chaokunyang Jan 26, 2026
1c575a4
fix array group
chaokunyang Jan 26, 2026
cef791d
skip out-of-band buffer
chaokunyang Jan 26, 2026
5830601
Merge remote-tracking branch 'asf/main' into go_option_for_optional_f…
chaokunyang Jan 26, 2026
bb98db1
fix lint
chaokunyang Jan 26, 2026
26a1751
fix oob serialization in python
chaokunyang Jan 26, 2026
a3dbaa5
fix ci
chaokunyang Jan 26, 2026
5c0200e
fix(python): expose peer buffer flag for bytes read
chaokunyang Jan 26, 2026
e5045e6
ci: revert windows-2025 runners
chaokunyang Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

name: Fory CI

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

on:
push:
branches:
Expand Down
6 changes: 5 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,19 @@ integration_tests/idl_tests/cpp/generated/
integration_tests/idl_tests/go/addressbook*.go
integration_tests/idl_tests/go/complex_fbs/
integration_tests/idl_tests/go/monster/
integration_tests/idl_tests/go/optional_types/
integration_tests/idl_tests/java/src/main/java/addressbook/
integration_tests/idl_tests/java/src/main/java/complex_fbs/
integration_tests/idl_tests/java/src/main/java/monster/
integration_tests/idl_tests/java/src/main/java/optional_types/
integration_tests/idl_tests/python/src/addressbook.py
integration_tests/idl_tests/python/src/complex_fbs.py
integration_tests/idl_tests/python/src/monster.py
integration_tests/idl_tests/python/src/optional_types.py
integration_tests/idl_tests/rust/src/addressbook.rs
integration_tests/idl_tests/rust/src/complex_fbs.rs
integration_tests/idl_tests/rust/src/monster.rs
integration_tests/idl_tests/rust/src/optional_types.rs
javascript/**/dist/
javascript/**/node_modules/
javascript/**/build
Expand Down Expand Up @@ -115,4 +119,4 @@ examples/cpp/cmake_example/build
**/benchmark_*.png
**/results/
benchmarks/**/report/
ignored/**
ignored/**
5 changes: 5 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ While working on Fory, please remember:
- Fory java needs JDK `17+` installed.
- Modules target different bytecode levels (fory-core Java 8, fory-format Java 11); avoid using newer APIs in those modules.
- Use '.\*' form of import is not allowed.
- If you run temporary tests using `java -cp`, you must run `mvn -T16 install -DskipTests` to get latest jars for fory java library.

```bash
# Clean the build
Expand Down Expand Up @@ -302,6 +303,10 @@ sbt scalafmt
- All commands must be executed within the `integration_tests` directory.
- For java related integration tests, please install the java libraries first by `cd ../java && mvn -T16 install -DskipTests`. If no code changes after installed fory java, you can skip the installation step.
- For mac, graalvm is installed at `/Library/Java/JavaVirtualMachines/graalvm-xxx` by default.
- For `integration_tests/idl_tests`:
- you must install fory java first before runing tests under this dir if any code changes under `java` dir.
- you must install fory python first before runing tests under this dir if any code cython code changes under `python` dir.
- You are never allowed to manual edit generated code by fory compiler for `IDL` files, you must invoke fory compiler to regenerate code.

```bash
it_dir=$(pwd)
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -553,8 +553,8 @@ For more details on row format, see [Row Format Specification](docs/specificatio

### User Guides

| Guide | Description | Source | Website |
| -------------------------------- | ------------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------------------ |
| Guide | Description | Source | Website |
| -------------------------------- | ------------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------------- |
| **Java Serialization** | Comprehensive guide for Java serialization | [java](docs/guide/java) | [📖 View](https://fory.apache.org/docs/guide/java/) |
| **Python** | Python-specific features and usage | [python](docs/guide/python) | [📖 View](https://fory.apache.org/docs/guide/python/) |
| **Rust** | Rust implementation and patterns | [rust](docs/guide/rust) | [📖 View](https://fory.apache.org/docs/guide/rust/) |
Expand Down
28 changes: 21 additions & 7 deletions benchmarks/go_benchmark/benchmark_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,27 @@ func newFory() *fory.Fory {
fory.WithTrackRef(false),
)
// Register types with IDs matching C++ benchmark
f.Register(NumericStruct{}, 1)
f.Register(Sample{}, 2)
f.Register(Media{}, 3)
f.Register(Image{}, 4)
f.Register(MediaContent{}, 5)
f.RegisterEnum(Player(0), 6)
f.RegisterEnum(Size(0), 7)
if err := f.RegisterStruct(NumericStruct{}, 1); err != nil {
panic(err)
}
if err := f.RegisterStruct(Sample{}, 2); err != nil {
panic(err)
}
if err := f.RegisterStruct(Media{}, 3); err != nil {
panic(err)
}
if err := f.RegisterStruct(Image{}, 4); err != nil {
panic(err)
}
if err := f.RegisterStruct(MediaContent{}, 5); err != nil {
panic(err)
}
if err := f.RegisterEnum(Player(0), 6); err != nil {
panic(err)
}
if err := f.RegisterEnum(Size(0), 7); err != nil {
panic(err)
}
return f
}

Expand Down
2 changes: 1 addition & 1 deletion compiler/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ message Config { ... } // Registered as "package.Config"
| `float64` | `double` | `pyfory.float64` | `float64` | `f64` | `double` |
| `string` | `String` | `str` | `string` | `String` | `std::string` |
| `bytes` | `byte[]` | `bytes` | `[]byte` | `Vec<u8>` | `std::vector<uint8_t>` |
| `date` | `LocalDate` | `datetime.date` | `time.Time` | `chrono::NaiveDate` | `fory::LocalDate` |
| `date` | `LocalDate` | `datetime.date` | `time.Time` | `chrono::NaiveDate` | `fory::Date` |
| `timestamp` | `Instant` | `datetime.datetime` | `time.Time` | `chrono::NaiveDateTime` | `fory::Timestamp` |

### Collection Types
Expand Down
2 changes: 1 addition & 1 deletion compiler/fory_compiler/generators/cpp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ class CppGenerator(BaseGenerator):
PrimitiveKind.FLOAT64: "double",
PrimitiveKind.STRING: "std::string",
PrimitiveKind.BYTES: "std::vector<uint8_t>",
PrimitiveKind.DATE: "fory::serialization::LocalDate",
PrimitiveKind.DATE: "fory::serialization::Date",
PrimitiveKind.TIMESTAMP: "fory::serialization::Timestamp",
}

Expand Down
34 changes: 31 additions & 3 deletions compiler/fory_compiler/generators/go.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ def message_has_unions(self, message: Message) -> bool:
PrimitiveKind.FLOAT64: "float64",
PrimitiveKind.STRING: "string",
PrimitiveKind.BYTES: "[]byte",
PrimitiveKind.DATE: "time.Time",
PrimitiveKind.DATE: "fory.Date",
PrimitiveKind.TIMESTAMP: "time.Time",
}

Expand Down Expand Up @@ -268,6 +268,8 @@ def collect_message_imports(self, message: Message, imports: Set[str]):
"""Collect imports for a message and its nested types recursively."""
for field in message.fields:
self.collect_imports(field.field_type, imports)
if self.field_uses_option(field):
imports.add('optional "github.com/apache/fory/go/fory/optional"')
for nested_msg in message.nested_messages:
self.collect_message_imports(nested_msg, imports)
for nested_union in message.nested_unions:
Expand Down Expand Up @@ -458,7 +460,7 @@ def get_union_case_type_id_expr(
PrimitiveKind.FLOAT64: "fory.FLOAT64",
PrimitiveKind.STRING: "fory.STRING",
PrimitiveKind.BYTES: "fory.BINARY",
PrimitiveKind.DATE: "fory.LOCAL_DATE",
PrimitiveKind.DATE: "fory.DATE",
PrimitiveKind.TIMESTAMP: "fory.TIMESTAMP",
}
return primitive_type_ids.get(kind, "fory.UNKNOWN")
Expand Down Expand Up @@ -677,6 +679,18 @@ def get_array_type_tag(self, field: Field) -> Optional[str]:
return "type=uint8_array"
return None

def field_uses_option(self, field: Field) -> bool:
"""Return True if field should use optional.Optional in generated Go code."""
if not field.optional or field.ref:
return False
if isinstance(field.field_type, PrimitiveType):
base_type = self.PRIMITIVE_MAP[field.field_type.kind]
return base_type not in ("[]byte", "time.Time", "fory.Date")
if isinstance(field.field_type, NamedType):
named_type = self.schema.get_type(field.field_type.name)
return isinstance(named_type, Enum)
return False

def generate_type(
self,
field_type: FieldType,
Expand All @@ -685,17 +699,30 @@ def generate_type(
element_optional: bool = False,
element_ref: bool = False,
parent_stack: Optional[List[Message]] = None,
use_option: bool = True,
) -> str:
"""Generate Go type string."""
if isinstance(field_type, PrimitiveType):
base_type = self.PRIMITIVE_MAP[field_type.kind]
if nullable and base_type not in ("[]byte",):
if (
use_option
and not ref
and base_type not in ("time.Time", "fory.Date")
):
return f"optional.Optional[{base_type}]"
return f"*{base_type}"
return base_type

elif isinstance(field_type, NamedType):
type_name = self.resolve_nested_type_name(field_type.name, parent_stack)
if nullable or ref:
if nullable:
if use_option and not ref:
named_type = self.schema.get_type(field_type.name)
if isinstance(named_type, Enum):
return f"optional.Optional[{type_name}]"
return f"*{type_name}"
if ref:
return f"*{type_name}"
return type_name

Expand All @@ -707,6 +734,7 @@ def generate_type(
False,
False,
parent_stack,
use_option=False,
)
return f"[]{element_type}"

Expand Down
2 changes: 1 addition & 1 deletion compiler/fory_compiler/generators/java.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -645,7 +645,7 @@ def get_union_case_type_id_expr(
PrimitiveKind.FLOAT64: "Types.FLOAT64",
PrimitiveKind.STRING: "Types.STRING",
PrimitiveKind.BYTES: "Types.BINARY",
PrimitiveKind.DATE: "Types.LOCAL_DATE",
PrimitiveKind.DATE: "Types.DATE",
PrimitiveKind.TIMESTAMP: "Types.TIMESTAMP",
}
return primitive_type_ids.get(kind, "Types.UNKNOWN")
Expand Down
2 changes: 1 addition & 1 deletion cpp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ cpp/fory/
│ ├── collection_serializer.h # vector, set serializers
│ ├── map_serializer.h # map serializers
│ ├── smart_ptr_serializers.h # optional, shared_ptr, unique_ptr
│ ├── temporal_serializers.h # Duration, Timestamp, LocalDate
│ ├── temporal_serializers.h # Duration, Timestamp, Date
│ ├── variant_serializer.h # std::variant support
│ ├── type_resolver.h # Type resolution and registration
│ └── context.h # Read/Write context
Expand Down
Loading
Loading