fix(flow-php/etl): schema definition compatibility, matching and merging

documentation/components/core/schema.md

@@ -16,6 +16,20 @@ A schema consists of entry definitions that specify:
 - **Nullable**: Whether NULL values are permitted
 - **Metadata**: Key-value pairs for additional context
 
+## Arrays in a Schema
+
+`type_array()` declares a `json` column. The data layer has no array type - parquet, Spark and Floe all express an
+array as a JSON object or a JSON collection - so a declared `array<mixed>` is projected onto `json`, and the value is
+stored as a `Flow\Types\Value\Json`, which preserves whether it was an object or a collection.
+
+```php
+definition_from_type('tags', type_array())->type()->toString();   // "json"
+union_schema('tags', type_union(type_string(), type_array()))->type()->toString();   // "json|string"
+```
+
+Declare the concrete shape whenever it is known - `type_list()`, `type_map()` or `type_structure()` keep element typing
+that `json` throws away, and adapters can map them onto native nested types.
+
 ## Schema Validation Strategies
 
 Flow PHP provides two built-in validation strategies:

documentation/contributing.md

@@ -111,6 +111,6 @@ It's mandatory to follow all of them without any exceptions unless explicitly ov
 - [Development Guidelines](/documentation/contributing/guidelines.md)
 - [Benchmarks](/documentation/contributing/benchmarks.md)
 - Extension Development
-  - [Rust - Arrow Extension](/documentation/contributing/rust.md)
+  - [Rust - arrow-ext & flow-php-ext](/documentation/contributing/rust.md)
   - [C - pg-query Extension](/documentation/contributing/c.md)
   - [WASM - Interactive Playground](/documentation/contributing/wasm.md)

documentation/contributing/nix.md

@@ -132,13 +132,16 @@ nix-shell --arg with-pg-query-ext false --arg with-c true
 
 See [C - pg-query Extension Development](/documentation/contributing/c.md) for details.
 
-### arrow-ext Extension (Rust)
+### arrow-ext and flow-php-ext Extensions (Rust)
 
 ```shell
-nix-shell --arg with-arrow-ext false --arg with-rust true
+nix-shell --arg with-rust true
 ```
 
-See [Rust - Arrow Extension Development](/documentation/contributing/rust.md) for details.
+`with-arrow-ext` and `with-flow-php-ext` both default to `!with-rust`, so this single flag already turns the prebuilt
+extensions off for both.
+
+See [Rust - Extension Development](/documentation/contributing/rust.md) for details.
 
 ### Protobuf / gRPC Code Generation (protoc)
 

documentation/contributing/rust.md

@@ -1,26 +1,38 @@
-# Rust - Arrow Extension Development
+# Rust - Extension Development
 
 [TOC]
 
-This document describes how to develop the `arrow-ext` PHP extension, which is written in Rust
-using the [ext-php-rs](https://github.com/extphprs/ext-php-rs) framework.
+This document describes how to develop the two Rust PHP extensions in this monorepo, both written with the
+[ext-php-rs](https://github.com/extphprs/ext-php-rs) framework.
 
-## Overview
+| Extension | Package | What it provides |
+|---|---|---|
+| `arrow-ext` | `flow-php/arrow-ext` | Parquet reader and writer powered by the [Apache Arrow](https://arrow.apache.org/) Rust ecosystem, exposed as `Flow\Arrow\Parquet\Reader` and `Flow\Arrow\Parquet\Writer` |
+| `flow-php-ext` | `flow-php/flow-php-ext` | Native Floe binary frame encoding/decoding and row hydration/casting against a schema |
 
-The `arrow-ext` extension provides a high-performance Parquet reader and writer for PHP,
-powered by the [Apache Arrow](https://arrow.apache.org/) Rust ecosystem. It exposes
-`Flow\Arrow\Parquet\Reader` and `Flow\Arrow\Parquet\Writer` classes to PHP.
+Both are optional. The pure-PHP implementations in `flow-php/etl` remain the canonical behaviour reference, and Flow
+routes to the native code automatically when the extension is loaded.
 
-For usage documentation, see [Arrow Extension](/documentation/components/extensions/arrow-ext.md).
+For usage documentation, see [Arrow Extension](/documentation/components/extensions/arrow-ext.md) and
+[Flow PHP Extension](/documentation/components/extensions/flow-php-ext.md).
 
 ## Development Setup
 
 ```bash
-nix-shell --arg with-arrow-ext false --arg with-rust true
+nix-shell --arg with-rust true
 ```
 
-This provides the Rust toolchain, clang, libclang, and PHP dev headers for building the extension
-from source. After `make build`, the freshly compiled extension is loaded by PHP automatically.
+This provides the Rust toolchain, clang, libclang, and PHP dev headers for building either extension from source.
+
+`with-arrow-ext` and `with-flow-php-ext` both default to `!with-rust`, so `--arg with-rust true` already turns the
+prebuilt extensions off — you do not need to pass them yourself. Pass `--arg with-arrow-ext false` or
+`--arg with-flow-php-ext false` only to override an explicit `true`; `shell.nix` asserts when either is combined with
+`--arg with-rust true`, because a prebuilt extension and a source build would collide.
+
+> [!IMPORTANT]
+> `make build` does **not** make PHP pick up the freshly compiled extension. Each Makefile's `test` target loads the
+> binary explicitly with `php -d extension=...`. To use a new build from anything else, run `make install`, pass
+> `-d extension=` yourself, or re-enter `nix-shell` — see [Rebuilding after a source change](#rebuilding-after-a-source-change).
 
 ## Project Structure
 
@@ -37,49 +49,85 @@ src/extension/arrow-ext/
 ├── tests/
 │   ├── phpt/               # PHPT test files
 │   └── fixtures/           # Test parquet files
+└── ext/
+    └── config.m4           # PIE compatibility
+
+src/extension/flow-php-ext/
+├── Cargo.toml              # Rust dependencies and build config
+├── build.rs                # Build script
+├── Makefile                # Build orchestration
+├── src/                    # Rust source code
+│   ├── lib.rs              # Extension entry point, module registration
+│   ├── encode.rs           # Floe frame body encoder
+│   ├── format.rs           # Floe binary format primitives
+│   ├── hydrate.rs          # Row hydration against a schema
+│   ├── cast.rs             # Value casting
+│   ├── plan.rs             # Per-column plan resolved once per schema
+│   ├── ctx.rs              # Shared module context
+│   ├── values.rs           # Zval <-> PHP value helpers
+│   └── exception.rs        # Exception mapping
+├── php/                    # PHP stubs for static analysis
+│   └── Flow/
+├── tests/
+│   └── phpt/               # PHPT test files
 └── ext/
     └── config.m4           # PIE compatibility
 ```
 
 ## Commands
 
-Build the extension:
+Substitute `arrow-ext` or `flow-php-ext` for `<extension>`.
 
-```bash
-nix-shell --arg with-arrow-ext false --arg with-rust true --run "cd src/extension/arrow-ext && make build"
-```
-
-Run PHPT tests:
+Build:
 
 ```bash
-nix-shell --arg with-arrow-ext false --arg with-rust true --run "cd src/extension/arrow-ext && make test"
+nix-shell --arg with-rust true --run "cd src/extension/<extension> && make build"
 ```
 
-Build and run PHPT tests:
+Run PHPT tests (`test` depends on `build`, so this rebuilds first):
 
 ```bash
-nix-shell --arg with-arrow-ext false --arg with-rust true --run "cd src/extension/arrow-ext && make build && make test"
+nix-shell --arg with-rust true --run "cd src/extension/<extension> && make test"
 ```
 
-Run PHP-side parquet tests (uses the pre-built extension):
+Clean build artifacts:
 
 ```bash
-nix-shell --run "just test --testsuite=lib-parquet-integration"
-nix-shell --run "just test --testsuite=adapter-parquet-integration"
+nix-shell --arg with-rust true --run "cd src/extension/<extension> && make clean"
 ```
 
-Clean build artifacts:
+Run the PHP-side test suites against the prebuilt extension — note these use the **default** shell, not the Rust one:
 
 ```bash
-nix-shell --arg with-arrow-ext false --arg with-rust true --run "cd src/extension/arrow-ext && make clean"
+nix-shell --run "just test --testsuite=lib-parquet-integration"      # arrow-ext
+nix-shell --run "just test --testsuite=adapter-parquet-integration"  # arrow-ext
+nix-shell --run "just test --testsuite=etl-unit"                     # flow-php-ext
 ```
 
 ## Make Targets
 
 | Target    | Description                        |
 |-----------|------------------------------------|
 | `build`   | Build the extension (cargo + copy) |
-| `test`    | Run PHPT tests                     |
-| `install` | Install to system PHP              |
+| `test`    | Build, then run PHPT tests         |
+| `install` | Copy the built module into PHP's `extension_dir` |
 | `clean`   | Remove build artifacts             |
 | `rebuild` | Full clean + build                 |
+
+The two PHPT runners differ in two ways:
+
+- `arrow-ext` runs each test with `php -n`, so `php.ini` is ignored and no other extension is loaded.
+  `flow-php-ext` does not, so the ambient extensions load alongside it.
+- `flow-php-ext` honours `--SKIPIF--` blocks and reports a skipped count. `arrow-ext` ignores them.
+
+## Rebuilding after a source change
+
+`.nix/pkgs/php-arrow-ext` and `.nix/pkgs/php-flow-php-ext` build their extension from the local repository source, so a
+shell you entered before editing any `.rs` still embeds the **previous** build. Running `just test` in a stale shell
+produces failures that are artifacts of the old binary, not of your change.
+
+Re-enter `nix-shell` to rebuild the derivation, or build and test the extension directly:
+
+```bash
+nix-shell --arg with-rust true --run "cd src/extension/flow-php-ext && make build && make test"
+```

src/core/etl/src/Flow/ETL/Row/EntryFactory.php

@@ -8,6 +8,7 @@
 use Flow\ETL\Row\Entry\Instantiators;
 use Flow\ETL\Row\Entry\NullEntry;
 use Flow\ETL\Schema\Definition;
+use Flow\ETL\Schema\Definition\UnionDefinition;
 use Flow\ETL\Schema\Metadata;
 use Flow\Types\Exception\CastingException;
 use Flow\Types\Type;
@@ -22,6 +23,7 @@
 
 use function array_values;
 use function Flow\ETL\DSL\definition_from_type;
+use function Flow\Types\DSL\type_equals;
 use function Flow\Types\DSL\type_string;
 
 final class EntryFactory
@@ -74,6 +76,10 @@ public function cast(string $name, mixed $value, Type $type, ?Metadata $metadata
      */
     public function fromDefinition(Definition $definition, mixed $value): Entry
     {
+        if ($definition instanceof UnionDefinition) {
+            $definition = $definition->memberFor($value);
+        }
+
         $variant = $value === null && !$definition->isNullable() ? $definition->makeNullable() : $definition;
 
         return $this->instantiators->for($variant->entryClass())->instantiate(
@@ -90,6 +96,8 @@ public function fromDefinition(Definition $definition, mixed $value): Entry
      */
     private function build(string $name, mixed $value, Type $type, ?Metadata $metadata, bool $cast): Entry
     {
+        $declaredType = $type;
+
         try {
             if ($type instanceof OptionalType) {
                 $type = $type->base();
@@ -127,7 +135,12 @@ private function build(string $name, mixed $value, Type $type, ?Metadata $metada
 
             $definition = definition_from_type($name, $type, $value === null, $metadata);
 
-            return $this->fromDefinition($definition, $this->prepareValue($value, $definition->type(), $cast));
+            return $this->fromDefinition($definition, $this->prepareValue(
+                $value,
+                $declaredType,
+                $definition->type(),
+                $cast,
+            ));
 
             // @mago-ignore analysis:avoid-catching-error
         } catch (InvalidArgumentException|CastingException|TypeError $e) {
@@ -139,18 +152,23 @@ private function build(string $name, mixed $value, Type $type, ?Metadata $metada
     }
 
     /**
-     * @param Type<mixed> $type
+     * @param Type<mixed> $declaredType
+     * @param Type<mixed> $entryType
      */
-    private function prepareValue(mixed $value, Type $type, bool $cast): mixed
+    private function prepareValue(mixed $value, Type $declaredType, Type $entryType, bool $cast): mixed
     {
-        if ($value === null || !$cast) {
+        if ($value === null) {
+            return $value;
+        }
+
+        if (!$cast && type_equals($declaredType, $entryType)) {
             return $value;
         }
 
-        if ($type instanceof ListType) {
-            return array_values($type->cast($value));
+        if ($entryType instanceof ListType) {
+            return array_values($entryType->cast($value));
         }
 
-        return $type->cast($value);
+        return $entryType->cast($value);
     }
 }

src/core/etl/src/Flow/ETL/Schema/Definition/BooleanDefinition.php

@@ -92,14 +92,14 @@ public function makeNullable(bool $nullable = true): static
 
     public function matches(Entry $entry): bool
     {
-        if ($this->isNullable() && $entry->is($this->ref)) {
-            return true;
-        }
-
         if (!$entry->is($this->ref)) {
             return false;
         }
 
+        if ($entry->value() === null) {
+            return $this->isNullable();
+        }
+
         return $entry->type() instanceof BooleanType;
     }
 
@@ -133,7 +133,16 @@ public function merge(Definition $definition): Definition
             );
         }
 
-        throw new RuntimeException(sprintf('Cannot merge %s with %s', self::class, $definition::class));
+        if ($definition instanceof UnionDefinition && (new UnionMembers())->contains($definition, $this)) {
+            return new UnionDefinition(
+                $this->ref,
+                $definition->type(),
+                $this->nullable || $definition->isNullable(),
+                $this->metadata->merge($definition->metadata()),
+            );
+        }
+
+        return (new CommonType())->merge($this, $definition);
     }
 
     public function metadata(): Metadata

src/core/etl/src/Flow/ETL/Schema/Definition/CommonType.php

@@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Flow\ETL\Schema\Definition;
+
+use Flow\ETL\Schema\Definition;
+
+use function in_array;
+
+final readonly class CommonType
+{
+    private const CONTAINERS = [
+        JsonDefinition::class,
+        ListDefinition::class,
+        MapDefinition::class,
+        StructureDefinition::class,
+    ];
+
+    /**
+     * @param Definition<mixed> $left
+     * @param Definition<mixed> $right
+     *
+     * @return Definition<mixed>
+     */
+    public function merge(Definition $left, Definition $right): Definition
+    {
+        $nullable = $left->isNullable() || $right->isNullable();
+        $metadata = $left->metadata()->merge($right->metadata());
+
+        if (in_array($left::class, self::CONTAINERS, true) && in_array($right::class, self::CONTAINERS, true)) {
+            return new JsonDefinition($left->entry(), $nullable, $metadata);
+        }
+
+        return new StringDefinition($left->entry(), $nullable, $metadata);
+    }
+}