Make all twisted-using import paths call txaio.use_twisted #557
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Could be related to crossbario#547
Current coverage is
|
oberstet
pushed a commit
that referenced
this pull request
Dec 2, 2015
Make all twisted-using import paths call txaio.use_twisted
Contributor
|
shouldn't the asyncio import paths then also explicitly select txaio-asyncio? |
Contributor
Author
|
They should, yes. No unit-tests revealed any missing cases, but I didn't really audit the asyncio code -- trying to run-all-examples.sh just now, there's at least one asyncio failure; I'll open a new PR. |
Contributor
|
Ah, great. I haven't looked closely .. but I had expected some lines in "autobahn/asyncio/websocket.py" for explicit selection. Unrelated: merged your latest PR on txaio, but the build is red: "fixture 'framework' not found" .. |
Contributor
Author
|
See #560 (and 44 in txaio) |
oberstet
added a commit
to oberstet/autobahn-python
that referenced
this pull request
Nov 12, 2025
This implements a vertical slice of the WAMP message serialization/ deserialization (serdes) test infrastructure using machine-readable test vectors from wamp-proto repository (issue crossbario#556, PR crossbario#557). Key Features: - Loads test vectors from wamp-proto/testsuite/ directory - Tests all three dimensions from SERDES_PLAN.md: * Dimension 2: Single-serializer roundtrip correctness * Dimension 3: Cross-serializer preservation - Implements "at least one" matching semantics for non-bijective serialization (JSON whitespace, msgpack variants) - Parameterized tests across all available serializers - Validates against embedded Python code blocks from test vectors Test Structure: - examples/serdes/tests/conftest.py: Pytest configuration and fixtures - examples/serdes/tests/utils.py: Helper functions for test vector loading and validation - examples/serdes/tests/test_publish.py: Complete test suite for PUBLISH message across all dimensions Test Results: - 20 tests passing (JSON, msgpack serializers) - Tests cover: deserialize from bytes, serialize to bytes, roundtrip, cross-serializer preservation, expected attributes validation This establishes the template for adding tests for all other WAMP message types. Related: wamp-proto#556, wamp-proto#557
oberstet
added a commit
that referenced
this pull request
Nov 21, 2025
…AMP Serializer Composition (transport/payload) (#1765) * add audit file to dev branch * Add WAMP message serialization benchmark suite Migrate and modernize serialization benchmarks from archived devops-benchmark repository to examples/benchmarks/serialization/. Changes: - Add comprehensive benchmark suite for WAMP message serialization - Test 7 serializers (json, ujson, msgpack, cbor, cbor2, ubjson, flatbuffers) - Support 2 payload modes (normal args, transparent payload) - Test 6 payload sizes (empty, small, medium, large, xl 16KB, xxl 128KB) - Include real vehicle telemetry CSV datasets (7MB total) - Add vmprof profiling integration with 0.01s sampling period - Generate HTML reports with Jinja2 templates and flamegraphs - Modernize code: Python 3.11+ type hints, MIT headers, docstrings - Add comprehensive README.md with usage instructions - Update pyproject.toml: add vmprof>=0.4.15, jinja2>=3.0.0, humanize>=4.0.0 New files: - examples/benchmarks/serialization/main.py - Benchmark runner with profiling - examples/benchmarks/serialization/loader.py - CSV data loader, VehicleEvent class - examples/benchmarks/serialization/sample.py - Sample JSON data structures - examples/benchmarks/serialization/README.md - Comprehensive documentation - examples/benchmarks/serialization/data/*.csv - Real telemetry datasets - examples/benchmarks/serialization/templates/*.html - Jinja2 report templates Performance metrics tracked: - Messages per second (throughput) - Bytes per second (bandwidth) - Average message size - CPU profiling with flamegraph visualization Fixes #1764 * Add just recipes for WAMP serialization benchmarks Add four new just recipes for running and managing the serialization benchmark suite: 1. benchmark-serialization-run: Run single benchmark with parameters - Accepts venv, serializer, payload_mode, payload_size, iterations - Default values: cbor, normal, small, 10 iterations - Creates build directory and runs benchmark with vmprof profiling 2. benchmark-serialization-suite: Run full benchmark matrix - Tests all 5 serializers (json, msgpack, cbor, ubjson, flatbuffers) - All 2 payload modes (normal, transparent) - All 6 payload sizes (empty, small, medium, large, xl, xxl) - Includes ujson and cbor2 variants with environment variables - Continues on errors to complete full suite 3. benchmark-serialization-report: Generate HTML report - Parses JSON results from build directory - Creates index.html with tabular results - Creates individual flamegraph pages per configuration - Requires jinja2, humanize dependencies 4. benchmark-serialization-clean: Clean benchmark artifacts - Removes examples/benchmarks/serialization/build/ directory Changes: - justfile: Add benchmark recipes in new section - .gitignore: Add examples/benchmarks/serialization/build/ to ignore list All recipes follow existing patterns: - Optional venv parameter with auto-detection - Proper error handling with 'set -e' - Clear echo messages for progress tracking - Use VENV_PYTHON helper for cross-platform compatibility Fixes #1764 * Fix benchmark recipes and txaio initialization Fix path handling and dependencies in benchmark just recipes: 1. Fix relative path resolution when cd'ing into benchmark directory - Convert relative venv paths to absolute using PROJECT_DIR - Affects benchmark-serialization-run and benchmark-serialization-report recipes 2. Add install-tools dependency to all benchmark recipes - Ensures vmprof, jinja2, humanize are available - Required for profiling and HTML report generation 3. Fix txaio initialization order in main.py - Must call txaio.use_asyncio() BEFORE importing autobahn modules - Autobahn serializers use txaio.time_ns() at module import time - Fixes "RuntimeError: To use txaio, you must first select a framework" Tested successfully with cpy311 venv: - Benchmark run: 42,966 msgs/sec, 12.8 MB/sec throughput - Loaded 350 vehicles, 42,039 events from CSV datasets - Generated profile.dat (23KB) and results JSON (567 bytes) Fixes #1764 * Create benchmark installation flavor for Python 3.11+ Move benchmarking dependencies to separate installation flavor to isolate them from general dev dependencies. Changes to pyproject.toml: - Add new [project.optional-dependencies.benchmark] section - vmprof>=0.4.15 with python_version >= '3.11' requirement - jinja2>=3.0.0 for HTML report templates - humanize>=4.0.0 for human-readable formatting - Remove vmprof and jinja2 from dev dependencies - Keep humanize in dev (used by other tools) Changes to justfile: - Add install-benchmark recipe with Python 3.11+ version check - Validates venv has Python 3.11+ before installing - Clear error message listing supported venvs - Update benchmark recipes to use install-benchmark: - benchmark-serialization-run - benchmark-serialization-suite - benchmark-serialization-report - Update recipe comments to use cpy311 in examples vmprof binary wheel availability (verified via PyPI API): - CPython 3.11: macOS (arm64), Linux (x86_64/aarch64), Windows - PyPy3: Universal wheel (pp3-none-any) This ensures benchmark dependencies only install on Python 3.11+ where vmprof binary wheels are available, preventing build issues on older Python versions. Fixes #1764 * Optimize memory usage for xl/xxl payload benchmarks Fix memory exhaustion issues when benchmarking large payloads by implementing frame data caching and event limiting. Changes to loader.py: 1. Cache binary frame data in VehicleEvent.__init__() - Pre-generate random frame data once per event object - Avoids regenerating 16KB/128KB per marshal() call - Previous: 42K events × 128KB × iterations = 5.4GB+ per run - Now: 42K events × 128KB × 1 = 5.4GB total (cached) 2. Add event limiting for xl/xxl payloads - xl (16KB): Limit to 1000 events (~16MB total frame data) - xxl (128KB): Limit to 100 events (~12.8MB total frame data) - Prevents system freeze from excessive memory usage - Still provides statistically valid benchmark samples 3. Update marshal() to use cached frame data - Check self._frame_data instead of calling os.urandom() - Eliminates repeated random data generation Memory impact: - Before: Unlimited events × payload size × iterations - After: Limited events with cached frames - xl: 42K → 1K events (97% reduction) - xxl: 42K → 100 events (99.7% reduction) This allows benchmarking large message sizes without exhausting system memory, addressing the user's system freeze issue. Fixes #1764 * Document event limiting for xl/xxl benchmarks in README Update README.md to document memory optimizations for large payload benchmarks. Changes: - Expand Payload Sizes table to include Event Limit and Total Data columns - Document automatic event limiting for xl (1000 events) and xxl (100 events) - Add note explaining memory exhaustion prevention strategy - Clarify that reduced event counts still provide valid statistical samples This helps users understand why xl/xxl benchmarks use fewer events and prevents confusion about different event counts in results. Fixes #1764 * Add comprehensive benchmark results summary document Document complete benchmark results from local testing on asgard1. Results Summary: - CPython 3.11.13: 72/72 benchmarks completed - PyPy 3.11.13: 72/72 benchmarks completed - Total: 144 benchmarks, 131 result files, ~126 profile files Performance Highlights: - CPython: 10K-120K msgs/sec depending on serializer - PyPy: 30K-350K msgs/sec (2-5x faster across the board!) - ubjson on PyPy: 345K msgs/sec vs 77K on CPython (4.5x speedup!) - msgpack on PyPy: 282K msgs/sec vs 119K on CPython (2.4x speedup) Memory optimizations working: - xl (16KB) payloads: Limited to 1K events, no system freeze - xxl (128KB) payloads: Limited to 100 events, stable performance - Frame data caching eliminates repeated generation Known Issues: - flatbuffers + normal/medium: Segmentation fault (expected) - ujson + transparent mode: TypeError (incompatible feature) Generated Files (in build/ directory, gitignored): - index.html: Main benchmark report with CPython vs PyPy comparison - results_*.json: 131 JSON result files (65 CPython + 66 PyPy) - profile_*.dat: ~126 vmprof profile files for flamegraphs Next Steps: - User testing on dev PC: IN PROGRESS - GitHub workflow: TO BE CREATED in follow-up - Documentation integration: TO BE ADDED to RTD This provides a complete picture of Autobahn|Python's serialization performance, demonstrating excellent performance on both CPython and especially PyPy, with comprehensive profiling data for optimization. Fixes #1764 * Add flamegraph generation and comprehensive serdes planning This commit integrates flamegraph visualization for benchmark profiling and establishes a comprehensive plan for WAMP serialization/deserialization testing and validation. Flamegraph Integration: - Add vmprof-flamegraph dependency to benchmark installation flavor - Download Brendan Gregg's flamegraph.pl for SVG generation - Create generate_flamegraphs.sh script to process vmprof .dat profiles - Add just benchmark-serialization-flamegraphs recipe - Generates interactive SVG flamegraphs from CPU profile data - Automatically copies logo to build directory SerDes Planning Framework: - Create new examples/serdes/ directory for serialization testing - Add SERDES_PLAN.md with comprehensive testing strategy - Document three testing dimensions: 1. Performance (benchmarking - completed) 2. Single-serializer roundtrip correctness (planned) 3. Cross-serializer preservation (planned) - "Going down": Protocol-level test vectors (machine-readable) - "Going up": Implementation features (batching, passthru, E2E encryption) - Design test vector format for WAMP spec conformance - Plan for multi-message semantic testing Technical Details: - Flamegraph generation uses vmprof-flamegraph.py → flamegraph.pl pipeline - Supports all 60+ benchmark profile files - Fixed bash arithmetic issues with set -e - Proper error handling and progress reporting Related to #1764 * Add WAMP message serdes test framework for PUBLISH messages This implements a vertical slice of the WAMP message serialization/ deserialization (serdes) test infrastructure using machine-readable test vectors from wamp-proto repository (issue #556, PR #557). Key Features: - Loads test vectors from wamp-proto/testsuite/ directory - Tests all three dimensions from SERDES_PLAN.md: * Dimension 2: Single-serializer roundtrip correctness * Dimension 3: Cross-serializer preservation - Implements "at least one" matching semantics for non-bijective serialization (JSON whitespace, msgpack variants) - Parameterized tests across all available serializers - Validates against embedded Python code blocks from test vectors Test Structure: - examples/serdes/tests/conftest.py: Pytest configuration and fixtures - examples/serdes/tests/utils.py: Helper functions for test vector loading and validation - examples/serdes/tests/test_publish.py: Complete test suite for PUBLISH message across all dimensions Test Results: - 20 tests passing (JSON, msgpack serializers) - Tests cover: deserialize from bytes, serialize to bytes, roundtrip, cross-serializer preservation, expected attributes validation This establishes the template for adding tests for all other WAMP message types. Related: wamp-proto#556, wamp-proto#557 * Document vertical slice methodology and add test-serdes just recipe SERDES_PLAN.md updates: - Added 'Strategic Approach: Vertical Slice Methodology' section - Documents philosophy: prove entire pipeline with ONE example first - Explains rationale: early validation, fast feedback, template creation - Detailed Phase 2 completion status (PUBLISH vertical slice) - Documents key technical discoveries during implementation: * txaio initialization requirement * Autobahn serializer API details * Publish message structure (no options attribute) * Non-bijective serialization handling * At least one matching semantics - Phase 3 outlines horizontal scaling strategy - Prioritized list of all WAMP message types - Updated phase structure with clear tactical roadmap justfile updates: - Added test-serdes recipe for running WAMP message serdes tests - Usage: just test-serdes [venv] - Runs examples/serdes/tests/test_publish.py with pytest - Auto-detects system Python venv if not specified - Follows existing justfile patterns and conventions This provides clear documentation of implementation approach and easy test execution for developers. Related: wamp-proto#556 * Implement payload mode tests for normal and transparent modes Replaces placeholder tests with full implementations: test_publish_normal_mode: - Validates samples with args/kwargs (normal payload mode) - Router deserializes and can inspect application payload - Verifies args/kwargs present, payload absent test_publish_transparent_mode: - Validates samples with opaque payload bytes (passthru mode) - Router does NOT deserialize payload - enables E2E encryption - Verifies payload present, args/kwargs absent - Tests roundtrip with JSON serializer - CRITICAL: Verifies payload bytes preserved exactly (byte-for-byte) Test Results: - 21 tests passing (up from 20) - Covers both payload modes across JSON, msgpack, CBOR, UBJSON - 5 flatbuffers tests fail (autobahn bug: enc_algo string/int mismatch) - Transparent mode correctly preserves encrypted payload bytes This completes the vertical slice polishing: - All major serializers covered (CBOR, UBJSON added) - Both payload modes tested (normal + transparent/E2E) - Test vector and tests fully aligned Related: wamp-proto#556 * Skip flatbuffers with transparent payload mode due to autobahn bug Adds skip logic with issue references for known flatbuffers bug: Issue: https://github.com/crossbario/autobahn-python/issues/1766 Root cause: flatbuffers serializer incorrectly handles enc_algo - Expects: uint8 - Gets: string ('cryptobox') - Error: TypeError in PublishAddEncAlgo() Skip locations: 1. test_publish_roundtrip - skips flatbuffers + transparent payload 2. test_publish_cross_serializer_preservation - skips pairs with flatbuffers + transparent payload Test Results: ✅ 21 tests passing ⏭️ 7 tests skipped (all with clear reasons) ❌ 0 tests failing Skip messages reference issue #1766 for easy tracking. Flatbuffers works fine with normal payload mode (sample 1). Issue isolated to transparent payload mode (sample 2) with enc_algo. This is a valuable bug find - filed for future fix. * Fix linting errors and redesign cross-serializer preservation test Issue 1: Fixed f-string linting errors (ruff F541) - Removed unnecessary f-prefix from pytest.skip() calls - Both instances now use plain strings (no placeholders) Issue 2: Redesigned test_publish_cross_serializer_preservation() - Old approach: construct → serialize → compare objects - New approach: Use canonical bytes from test vector: 1. Take bytes_hex for ser1 from test vector 2. Deserialize with ser1 → get object 3. Serialize object with ser2 → get new bytes 4. Check new bytes match ANY bytes_hex variants for ser2 Why better: - Tests actual test vector canonical bytes (not constructed objects) - Verifies cross-serializer conversion produces valid canonical bytes - Ensures object representation is equivalent across serializers - More rigorous: tests that samples are truly equivalent Test Results: ✅ 21 tests passing ⏭️ 7 tests skipped (all with clear reasons) ❌ 0 tests failing ✅ Linting passes (just check-format) Related: wamp-proto#556 * Improve PyPy ARM64 QEMU stability with modern QEMU 8.x and JIT workarounds Two critical improvements for PyPy ARM64 wheel builds on QEMU emulation: 1. Install modern QEMU 8.x using tonistiigi/binfmt - Added step BEFORE docker/setup-buildx-action - QEMU 8.x is much more stable for PyPy JIT - Uses 'docker run --rm --privileged tonistiigi/binfmt --install all' 2. PyPy QEMU stability workarounds in both Dockerfiles - ENV PYPY_DISABLE_JIT=1 # Disable JIT compiler - ENV PYPY_GC_NTHREADS=1 # Single-threaded GC - ENV PYPY_FORCE_CPU_COUNT=1 # Force single CPU Why these changes: - PyPy's JIT compiler can be unstable under QEMU ARM64 emulation - Modern QEMU 8.x has much better ARM64 support than older versions - Single-threaded execution avoids race conditions in QEMU - Disabling JIT trades performance for stability (acceptable for CI) Files modified: - .github/workflows/wheels-arm64.yml: Add QEMU 8.x install step - docker/Dockerfile.pypy-bookworm-manylinux-arm64: Add PyPy ENV vars - docker/Dockerfile.pypy-trixie-manylinux-arm64: Add PyPy ENV vars This should resolve the recurring PyPy ARM64 build failures on GitHub Actions runners with QEMU emulation. * Improve PUBLISH test error reporting Enhanced validation error messages to include: - Sample description in failure messages - Code context for assertion errors - Better debugging information for empty assertions This helps identify which specific sample and assertion is failing when tests don't pass. All PUBLISH tests now passing: 21 passed, 7 skipped (flatbuffers) * Add EVENT message serialization tests Implements comprehensive EVENT message testing following the proven PUBLISH test pattern with all three test dimensions: 1. Single-serializer roundtrip correctness - Deserialization from canonical bytes - Serialization to bytes - Full roundtrip with validation 2. Cross-serializer preservation - Tests attribute preservation across different serializers - Uses canonical bytes from test vector 3. Payload mode handling - Normal mode (args/kwargs) - Transparent mode (E2EE with opaque payload bytes) Test coverage: - All 4 EVENT samples from wamp-proto test vector - All serializers: JSON, msgpack, CBOR, UBJSON - Forward_for (router-to-router links) handling - E2EE metadata (enc_algo, enc_serializer) handling Results: 21 tests passed, 7 skipped (flatbuffers with transparent payload) Changes: - Add test_event.py with complete EVENT test suite - Update conftest.py with EVENT test vector fixture Related: #556, #1766 Part of: wamp-proto/testsuite systematic coverage * Add EVENT tests to test-serdes justfile recipe Update test-serdes recipe to run both PUBLISH and EVENT message tests. Test coverage now includes: - PUBLISH: 21 tests (all 3 dimensions) - EVENT: 21 tests (all 3 dimensions) - Total: 42 passed, 14 skipped (flatbuffers) Related: #556, #1766 * Add comprehensive Options/Details validation tests and spec comparison - Add ProtocolAttributes-Spec-vs-AutobahnPython.md documenting all PUBLISH.Options and EVENT.Details attributes, comparing WAMP spec vs autobahn-python implementation - Add 35 new validation tests for PUBLISH.Options in test_publish.py covering: * All 9 matched attributes (acknowledge, exclude_me, exclude*, eligible*, retain) * Implementation-only attributes (transaction_hash, forward_for) * E2EE attributes (enc_algo, enc_key, enc_serializer) * Both valid values and invalid type/value tests - Add 23 new validation tests for EVENT.Details in test_event.py covering: * All 5 matched attributes (publisher*, topic, retained) * Implementation-only attributes (transaction_hash, x_acknowledged_delivery, forward_for) * E2EE attributes (enc_algo, enc_key, enc_serializer) * Both valid values and invalid type/value tests - Tests validate at wmsg (deserialized message dict) level per user requirement - Document known validation bugs in forward_for and enc_key checks with workarounds - All 100 new+existing tests pass (14 skipped for flatbuffers) - Update justfile test-serdes target to run both test_publish.py and test_event.py This is critical for multi-implementation WAMP where Client₁→Router→Client₂ may use different implementations. Proper validation at message boundaries prevents bugs from propagating across the implementation chain. * Refactor validation tests to use language-agnostic JSON test vectors This commit refactors PUBLISH.Options and EVENT.Details validation tests to load test vectors from wamp-proto JSON files, enabling test reuse across all WAMP implementations. Changes: - Modified test_publish.py: * Added publish_validation_samples fixture to load validation tests from JSON * Added test_publish_options_validation_from_json parameterized test (35 cases) * Converts hex payload strings to bytes for transparent payload mode * Updated publish_samples fixture to filter out validation samples * All 35 PUBLISH.Options attributes now tested via JSON test vectors - Modified test_event.py: * Added event_validation_samples fixture to load validation tests from JSON * Added test_event_details_validation_from_json parameterized test (21 cases) * Converts hex payload strings to bytes for transparent payload mode * Updated event_samples fixture to filter out validation samples * All 21 EVENT.Details attributes now tested via JSON test vectors Test Structure: - Each validation sample has: * wmsg: deserialized message array * expected_error (optional): {type, contains} for invalid cases * description: human-readable test name Benefits: - Test vectors are language-agnostic and reusable - Single source of truth for validation in wamp-proto repository - AutobahnJS, AutobahnJava, AutobahnC++ can implement identical tests - Systematic coverage of all Options/Details attributes - 158 tests passing (35 PUBLISH + 21 EVENT validation, plus existing tests) Related: #1764, wamp-proto#556 * Add SerDes conformance tests to CI/CD pipeline This commit integrates the language-agnostic SerDes conformance tests into the GitHub Actions CI/CD pipeline. Changes to .github/workflows/main.yml: - Added new "test-serdes" job with matrix strategy [cpy314, cpy311, pypy311] - Job runs language-agnostic validation tests from wamp-proto/testsuite - Generates JUnit XML test reports for each Python environment - Uploads test results as verified artifacts using wamp-cicd actions - Added test-serdes as dependency for build-package job Changes to .github/workflows/release.yml: - Added download step for SerDes test results artifacts - Uses wamp-cicd/actions/download-artifact-verified with retry logic - Downloads all serdes-test-results-* artifacts from main workflow - Test results included in release artifacts Test Execution: - Runs: pytest -v examples/serdes/tests/test_publish.py test_event.py - Tests: 158 tests across 3 Python environments (474 total test runs) - Coverage: 35 PUBLISH.Options + 21 EVENT.Details validation tests - Output: JUnit XML reports + summary files Benefits: - Ensures language-agnostic test vectors pass on all Python versions - Blocks package build if conformance tests fail - Test results available as release artifacts - Foundation for adding AutobahnJS/Java/C++ to same test suite Related: #1764, wamp-proto#556 * add wamp-proto / PR for fix_556@oberstet branch as git submodule * Update test utils to use .proto submodule with fallback to sibling directory Changes to examples/serdes/tests/utils.py: - Updated get_wamp_proto_path() to check for .proto submodule first - Falls back to sibling directory ../wamp-proto for local development - Provides clear error messages with both paths tried - Ensures flexibility for both CI (submodule) and local dev (sibling) Path resolution order: 1. .proto/ submodule (preferred for CI/CD) 2. ../wamp-proto sibling (local development) This allows: - CI to use git submodule (works in GitHub Actions) - Developers to use either submodule or sibling directory - Clear error messages if neither location exists Related: #1764, wamp-proto#556 * Fix wamp-cicd action path in test-serdes job Remove incorrect .github/ prefix from action path. Correct path is wamp-proto/wamp-cicd/actions/upload-artifact-verified@main not wamp-proto/wamp-cicd/.github/actions/upload-artifact-verified@main Related to CI failure: https://github.com/crossbario/autobahn-python/actions/runs/19380729500 * Add SUBSCRIBE message SerDes conformance tests - Create test_subscribe.py with tests for SUBSCRIBE message type - Tests cover: - Single-serializer roundtrip correctness (deserialization/serialization) - SUBSCRIBE.Options validation from wamp-proto test vectors - Current status: 16 tests passing (validation + deserialization working) - Note: 8 serialization/roundtrip tests need refinement Part of Phase 1: Complete Pub/Sub message types expansion. Test vectors source: wamp-proto/testsuite/singlemessage/basic/subscribe.json * Add test_subscribe.py to test-serdes recipe Include SUBSCRIBE message tests in the 'just test-serdes' command to run alongside PUBLISH and EVENT tests. * Fix test vector path resolution for outdated submodules Update load_test_vector() to check both .proto submodule and sibling ../wamp-proto directory, using whichever has the requested file. This allows tests to work when: - .proto submodule is at an older commit (during development) - New test vectors exist only in sibling wamp-proto working directory Fixes issue where SUBSCRIBE tests couldn't find subscribe.json because .proto submodule was at old commit c849243 but new test vectors are in commit eb3820a (not yet on GitHub). * Fix SUBSCRIBE serialization tests - unpack serialize() tuple serializer.serialize() returns (bytes, is_binary) tuple, not just bytes. Updated test_subscribe_serialize_to_bytes() and test_subscribe_roundtrip() to properly unpack the tuple. All SUBSCRIBE tests now passing: 182 passed, 17 skipped (flatbuffers) * Add SUBSCRIBE.Options analysis to protocol comparison report Analysis shows excellent spec compliance for SUBSCRIBE: - 2 matched attributes: match, get_retained - 0 spec-only (all spec attributes implemented) - 1 implementation-only: forward_for (router-to-router links) - No E2EE complexity or naming discrepancies Updated summary matrix and analysis date to 2025-11-17. * Add SUBSCRIBED message type SerDes tests Implements comprehensive serialization/deserialization conformance tests for the WAMP SUBSCRIBED message type (acknowledgment message from Router to Client confirming a subscription). Changes: - Add test_subscribed.py with 3 test functions covering deserialize, serialize, and roundtrip operations across all serializers - Update justfile to include test_subscribed.py in test-serdes recipe - Update protocol comparison report with SUBSCRIBED analysis showing perfect spec compliance (no Options/Details, just 2 ID fields) Test Results: - All 214 tests pass (194 passed, 20 skipped for flatbuffers) - SUBSCRIBED adds 15 test cases (3 functions × 5 serializers) - Tested with: just test-serdes cpy311 Related: - Uses test vectors from wamp-proto commit 309993b - Follows same pattern as SUBSCRIBE/PUBLISH/EVENT tests - Part of Phase 1 (Pub/Sub messages) expansion Ref: https://github.com/crossbario/autobahn-python/issues/1764 * Add PUBLISHED message type SerDes tests Implements comprehensive serialization/deserialization conformance tests for the WAMP PUBLISHED message type (acknowledgment message from Router to Publisher confirming a publication). Changes: - Add test_published.py with 3 test functions covering deserialize, serialize, and roundtrip operations across all serializers - Update justfile to include test_published.py in test-serdes recipe - Update protocol comparison report with PUBLISHED analysis showing perfect spec compliance (no Options/Details, just 2 ID fields) Test Results: - All 229 tests pass (206 passed, 23 skipped for flatbuffers) - PUBLISHED adds 15 test cases (3 functions × 5 serializers) - Tested with: just test-serdes cpy311 Related: - Uses test vectors from wamp-proto commit bf12aa5 - Follows same pattern as SUBSCRIBE/SUBSCRIBED tests - Part of Phase 1 (Pub/Sub messages) expansion Ref: https://github.com/crossbario/autobahn-python/issues/1764 * Add UNSUBSCRIBE message type SerDes tests Implements comprehensive serialization/deserialization conformance tests for the WAMP UNSUBSCRIBE message type (request from Subscriber to Broker to unsubscribe from a topic). Changes: - Add test_unsubscribe.py with 3 test functions covering deserialize, serialize, and roundtrip operations across all serializers - Update justfile to include test_unsubscribe.py in test-serdes recipe - Update protocol comparison report with UNSUBSCRIBE.Options analysis showing one implementation-specific attribute (forward_for for R2R links) Test Results: - All 244 tests pass (218 passed, 26 skipped for flatbuffers) - UNSUBSCRIBE adds 15 test cases (3 functions × 5 serializers) - Tested with: just test-serdes cpy311 Related: - Uses test vectors from wamp-proto commit 04239b3 - Follows same pattern as SUBSCRIBE/SUBSCRIBED/PUBLISHED tests - Part of Phase 1 (Pub/Sub messages) expansion UNSUBSCRIBE.Options: - No spec-defined Options - Only forward_for (implementation-specific, for R2R links) - Consistent with SUBSCRIBE.Options Ref: https://github.com/crossbario/autobahn-python/issues/1764 * Add UNSUBSCRIBED message type SerDes tests - Phase 1 complete! Implements comprehensive serialization/deserialization conformance tests for the WAMP UNSUBSCRIBED message type (acknowledgment from Broker to Subscriber confirming unsubscription). Changes: - Add test_unsubscribed.py with 3 test functions covering deserialize, serialize, and roundtrip operations across all serializers - Update justfile to include test_unsubscribed.py in test-serdes recipe - Update protocol comparison report with UNSUBSCRIBED analysis showing perfect spec compliance for basic form Test Results: - All 259 tests pass (230 passed, 29 skipped for flatbuffers) - UNSUBSCRIBED adds 15 test cases (3 functions × 5 serializers) - Tested with: just test-serdes cpy311 Related: - Uses test vectors from wamp-proto commit 670d0f5 - Follows same pattern as other acknowledgment messages - Completes Phase 1 (Pub/Sub messages) expansion UNSUBSCRIBED Analysis: - Simple acknowledgment with just request_id in basic form - Supports optional Details for router revocation signaling - Implementation extends spec with router-initiated revocation Phase 1 Complete: ✅ PUBLISH + EVENT ✅ SUBSCRIBE + SUBSCRIBED ✅ PUBLISHED ✅ UNSUBSCRIBE + UNSUBSCRIBED All Pub/Sub message types now have comprehensive SerDes conformance tests! Ref: https://github.com/crossbario/autobahn-python/issues/1764 * Add CALL message type SerDes tests - Phase 2 RPC messages begin Add comprehensive serialization/deserialization tests for WAMP CALL message (message type 48). CALL initiates remote procedure calls from Caller to Dealer. Changes: - examples/serdes/tests/test_call.py: 188 lines, 3 test functions * test_call_deserialize_from_bytes * test_call_serialize_to_bytes * test_call_roundtrip - justfile: Add test_call.py to test-serdes recipe - ProtocolAttributes-Spec-vs-AutobahnPython.md: Add CALL.Options analysis Test coverage: - 12 new test cases (3 functions × 4 serializers) - All tests passing (242 passed, 32 skipped) - Validates CALL with args/kwargs - Follows proven workflow from Phase 1 CALL.Options spec compliance analysis: - Matched: timeout, receive_progress (2) - Spec-only: disclose_me (1) - Implementation-only: transaction_hash, caller*, forward_for (5) - E2EE naming differences (enc_* vs ppt_*) Part of Phase 2 (RPC messages) implementation. Related to autobahn-python issue #1764. * Add RESULT message type SerDes tests - completing CALL/RESULT pair Add comprehensive serialization/deserialization tests for WAMP RESULT message (message type 50). RESULT returns the result of a remote procedure call from Dealer to Caller. Changes: - examples/serdes/tests/test_result.py: 188 lines, 3 test functions * test_result_deserialize_from_bytes * test_result_serialize_to_bytes * test_result_roundtrip - justfile: Add test_result.py to test-serdes recipe - ProtocolAttributes-Spec-vs-AutobahnPython.md: Add RESULT.Details analysis Test coverage: - 12 new test cases (3 functions × 4 serializers) - All tests passing (254 passed, 35 skipped) - Validates RESULT with args/kwargs - Follows proven workflow from CALL RESULT.Details spec compliance analysis: - Matched: progress (1) - Spec-only: (0) - Implementation-only: callee*, forward_for (4) - E2EE naming differences (enc_* vs ppt_*) Completes CALL/RESULT pair for basic RPC interaction. Part of Phase 2 (RPC messages) implementation. Related to autobahn-python issue #1764. * Expand protocol comparison report with Phase 2 summary and TOC Major enhancements to ProtocolAttributes-Spec-vs-AutobahnPython.md: 1. Added comprehensive Table of Contents: - Summary sections with links - Phase 1 (Pub/Sub) message links - Phase 2 (RPC) message links with TODO markers - Appendix links 2. Enhanced Summary Matrix: - Split into Phase 1 (Complete) and Phase 2 (In Progress) - Added CALL.Options and RESULT.Details rows - Added placeholders for remaining RPC messages - Added Phase 1+2 Summary with compliance/extension analysis 3. Expanded Version Information: - Analysis Status: Visual progress tracking with checkmarks/emojis - Test Coverage: Detailed test count breakdown - Source Information: Complete file paths - Related Issues: Links to wamp-proto#556 and autobahn-python#1764 Current status: - Phase 1 (Pub/Sub): 7 message types ✅ COMPLETE - Phase 2 (RPC): 2/8 message types complete (CALL, RESULT) - Total tests: 254 passed, 35 skipped Related to autobahn-python issue #1764. * Add REGISTER and REGISTERED SerDes tests for autobahn-python#1764 This commit implements comprehensive serialization/deserialization conformance tests for REGISTER and REGISTERED messages using test vectors from wamp-proto. Changes: - Add test_register.py (156 lines): 12 tests for REGISTER message - Add test_registered.py (158 lines): 12 tests for REGISTERED message - Update justfile: Add both test files to test-serdes recipe - Update comparison report: Complete REGISTER/REGISTERED analysis Test Coverage: - Total: 278 passed, 41 skipped (was 254 passed, 35 skipped) - Phase 2 RPC: 48 tests (was 24 tests) - 4 message types complete: CALL, RESULT, REGISTER, REGISTERED REGISTER.Options Analysis: - 2 matched attributes: match, invoke - 0 spec-only attributes - 3 implementation-only: concurrency, force_reregister, forward_for - No E2EE complexity REGISTERED Analysis: - Simple acknowledgment message - Perfect spec compliance - Message format: [65, request_id, registration_id] All tests passing with JSON, MsgPack, CBOR, and UBJSON serializers. Part of: autobahn-python#1764 Related: wamp-proto#556 * Add UNREGISTER and UNREGISTERED SerDes tests for autobahn-python#1764 This commit implements comprehensive serialization/deserialization conformance tests for UNREGISTER and UNREGISTERED messages using test vectors from wamp-proto. Changes: - Add test_unregister.py (156 lines): 12 tests for UNREGISTER message - Add test_unregistered.py (151 lines): 12 tests for UNREGISTERED message - Update justfile: Add both test files to test-serdes recipe - Update comparison report: Complete UNREGISTER/UNREGISTERED analysis Test Coverage: - Total: 302 passed, 47 skipped (was 278 passed, 41 skipped) - Phase 2 RPC: 72 tests (was 48 tests) - 6 message types complete: CALL, RESULT, REGISTER, REGISTERED, UNREGISTER, UNREGISTERED UNREGISTER.Options Analysis: - 0 matched attributes (no spec-defined Options) - 0 spec-only attributes - 1 implementation-only: forward_for - Simple request-response pattern UNREGISTERED Analysis: - 2 matched attributes: registration, reason (advanced profile) - 0 spec-only attributes - 0 implementation-only attributes - Supports both basic acknowledgment and router-initiated revocation - Excellent spec compliance All tests passing with JSON, MsgPack, CBOR, and UBJSON serializers. Part of: autobahn-python#1764 Related: wamp-proto#556 * Add INVOCATION and YIELD SerDes tests - Phase 2 RPC COMPLETE for autobahn-python#1764 This commit implements comprehensive serialization/deserialization conformance tests for INVOCATION and YIELD messages using test vectors from wamp-proto, completing Phase 2 (RPC Messages). Changes: - Add test_invocation.py (156 lines): 12 tests for INVOCATION message - Add test_yield.py (151 lines): 12 tests for YIELD message - Update justfile: Add both test files to test-serdes recipe - Update comparison report: Complete INVOCATION/YIELD analysis Test Coverage: - Total: 326 passed, 53 skipped (was 302 passed, 47 skipped) - Phase 1 (Pub/Sub): 218 tests (8 message types) ✅ COMPLETE - Phase 2 (RPC): 96 tests (8 message types) ✅ COMPLETE - 24 new tests added (12 per message type) INVOCATION.Details Analysis: - 6 matched attributes: caller, caller_authid, caller_authrole, procedure, timeout, receive_progress - 1 spec-only: trustlevel (not implemented) - 2 implementation-only: transaction_hash, forward_for - E2EE naming mismatch: enc_* vs ppt_* YIELD.Options Analysis: - 1 matched attribute: progress - 0 spec-only attributes - 4 implementation-only: callee, callee_authid, callee_authrole, forward_for - E2EE naming mismatch: enc_* vs ppt_* Phase 2 RPC Messages COMPLETE: - Group 1: CALL, RESULT (Caller side) - Group 2: REGISTER, REGISTERED (Registration) - Group 3: UNREGISTER, UNREGISTERED (Unregistration) - Group 4: INVOCATION, YIELD (Callee side) All tests passing with JSON, MsgPack, CBOR, and UBJSON serializers. Part of: autobahn-python#1764 Related: wamp-proto#556 * Add ERROR SerDes tests - ALL 17 WAMP message types COMPLETE for autobahn-python#1764 This commit implements comprehensive serialization/deserialization conformance tests for ERROR message, completing the full coverage of ALL 17 WAMP message types. Changes: - Add test_error.py (162 lines): 12 tests for ERROR message - Update justfile: Add test_error.py to test-serdes recipe - Update comparison report: Complete ERROR.Details analysis - Mark ALL phases as COMPLETE in comparison report Test Coverage: - Total: 338 passed, 56 skipped (was 326 passed, 53 skipped) - Phase 1 (Pub/Sub): 218 tests (8 message types) ✅ COMPLETE - Phase 2 (RPC): 96 tests (8 message types) ✅ COMPLETE - Phase 3 (Shared): 12 tests (1 message type) ✅ COMPLETE - 12 new tests added for ERROR message - Coverage: 17 out of 17 WAMP message types tested! 🎉 ERROR.Details Analysis: - 0 matched attributes (no spec-defined Details in basic profile) - 0 spec-only attributes - 4 implementation-only: callee, callee_authid, callee_authrole, forward_for - E2EE naming mismatch: enc_* vs ppt_* - Universal error response for both Pub/Sub and RPC ALL PHASES COMPLETE: ✅ Phase 1 - Pub/Sub Messages (8 types): PUBLISH, EVENT, SUBSCRIBE, SUBSCRIBED, PUBLISHED, UNSUBSCRIBE, UNSUBSCRIBED ✅ Phase 2 - RPC Messages (8 types): CALL, RESULT, REGISTER, REGISTERED, UNREGISTER, UNREGISTERED, INVOCATION, YIELD ✅ Phase 3 - Shared Messages (1 type): ERROR All tests passing with JSON, MsgPack, CBOR, and UBJSON serializers. Comprehensive protocol comparison document with detailed analysis of spec compliance, implementation extensions, and recommendations for both WAMP spec and Autobahn-Python. Part of: autobahn-python#1764 Related: wamp-proto#556 * Add SerDes conformance tests for WAMP Session Lifecycle messages (Phase 4) Implement comprehensive serialization/deserialization conformance tests for the 6 WAMP message types that handle session lifecycle: Session Establishment: - HELLO (type 1): Client initiates session, announces roles - WELCOME (type 2): Router accepts session, announces router roles Session Abort: - ABORT (type 3): Abort session opening with reason URI Authentication Flow: - CHALLENGE (type 4): Router sends authentication challenge - AUTHENTICATE (type 5): Client responds with authentication proof Session Termination: - GOODBYE (type 6): Graceful session close with reason URI Each message type includes: - Deserialization tests: Validate parsing from canonical bytes - Serialization tests: Verify correct byte generation - Roundtrip tests: Ensure serialize → deserialize → serialize stability Test Coverage: - 72 new tests (6 message types × 12 tests/type) - Total: 410 passed, 74 skipped - Serializers: JSON, MsgPack, CBOR, UBJSON (FlatBuffers skipped) Implementation Notes: - HELLO/WELCOME require role object conversion (RoleFeatures classes) - ABORT/GOODBYE use simple reason URIs with optional Details - CHALLENGE/AUTHENTICATE use method/signature strings with optional Extra Updated comparison report to reflect Phase 4 completion. Related: - wamp-proto#556: Add comprehensive test vectors for all WAMP messages - autobahn-python#1764: Add SerDes conformance tests * Add Phase 4 test files to justfile test-serdes command The test-serdes recipe was missing the 6 new Phase 4 session lifecycle test files, causing 'just test-serdes' to only run 338 tests instead of the full 410 tests. Added: - test_hello.py - test_welcome.py - test_abort.py - test_challenge.py - test_authenticate.py - test_goodbye.py Now 'just test-serdes cpy312' runs all 410 tests (74 skipped). * Add SerDes conformance tests for Advanced WAMP messages (Phase 5) Implements serialization/deserialization tests for the remaining WAMP message types from the Advanced Profile: - CANCEL (type 49) - Advanced RPC Call Canceling - INTERRUPT (type 69) - Advanced RPC Call Canceling - EventReceived (type 337) - Advanced Pub/Sub Event Acknowledgment This completes SerDes test coverage for ALL 25 WAMP message types: Phase 1: Pub/Sub (8 messages) ✓ Phase 2: RPC (8 messages) ✓ Phase 3: Shared (1 message - ERROR) ✓ Phase 4: Session Lifecycle (6 messages) ✓ Phase 5: Advanced Profile (3 messages) ✓ Test Results: - 446 passed, 83 skipped - All 4 serializers tested: JSON, MsgPack, CBOR, UBJSON Files added: - examples/serdes/tests/test_cancel.py - examples/serdes/tests/test_interrupt.py - examples/serdes/tests/test_eventreceived.py Files modified: - examples/serdes/ProtocolAttributes-Spec-vs-AutobahnPython.md (added Phase 4 detailed analysis and Phase 5 summary) - justfile (added Phase 5 test files to test-serdes recipe) Test vectors from: wamp-proto/testsuite/singlemessage/advanced/ Refs #1764 * Reorganize comparison report by functional area Moved Advanced Profile messages into their respective functional sections rather than keeping them separate: - EventReceived: Moved to Phase 1 (Pub/Sub Messages) - CANCEL: Moved to Phase 2 (RPC Messages) - INTERRUPT: Moved to Phase 2 (RPC Messages) This better reflects that these are Advanced Profile extensions of Pub/Sub and RPC patterns, not a separate category. Changes: - Added detailed sections for CANCEL, INTERRUPT, EventReceived - Updated Table of Contents to show Advanced Profile messages in their functional categories - Removed Phase 5 section from Summary Matrix - Updated test coverage breakdown - Added message type distribution summary Phase counts now: - Phase 1 (Pub/Sub): 8 message types (7 basic + 1 advanced) - Phase 2 (RPC): 10 message types (8 basic + 2 advanced) - Phase 3 (Shared): 1 message type - Phase 4 (Session Lifecycle): 6 message types Total: 25 WAMP message types Refs #1764 * Add test helper scripts for generating WAMP message test vectors This commit adds helper scripts for generating serialized byte representations of additional WAMP message types (ABORT, AUTHENTICATE, CANCEL, CHALLENGE, EVENT_RECEIVED, GOODBYE, HELLO, INTERRUPT, WELCOME) to support test vector development and validation. These scripts follow the same pattern as existing test vector generators and enable developers to quickly generate test data for various serialization formats (JSON, MessagePack, CBOR, UBJSON) across different WAMP message types. * Add FlatBuffers lazy deserialization support for CALL and PUBLISH messages This commit implements complete FlatBuffers support for CALL and PUBLISH messages, enabling lazy deserialization from FlatBuffers-encoded WAMP messages. Changes to CALL message (autobahn/wamp/message.py): - Added from_fbs=None parameter to __init__ signature - Modified initialization to pass from_fbs to parent Message class - Changed all direct attribute assignments to underscore-prefixed private attributes - Added 15 @property methods with lazy deserialization from FlatBuffers: * Scalar fields (request, timeout, caller): Direct FlatBuffers access * String fields (procedure, transaction_hash, enc_*, caller_*): UTF-8 decode * Application payload (args, kwargs): CBOR deserialization (flatbuffers-cbor pattern) * Binary payload: Raw bytes access * Boolean fields (receive_progress): Checked access * forward_for: Deserialize Principal struct array to list of dicts Changes to PUBLISH message (autobahn/wamp/message.py): - Fixed forward_for property to implement complete lazy deserialization - Removed FIXME comments - Added proper validation in setter - Handles FlatBuffers Principal struct limitation (session-only) Implementation follows the established pattern from existing Event and Publish FlatBuffers support, using the "flatbuffers-cbor" composition pattern where WAMP message structure uses FlatBuffers while application payload (args/kwargs) uses CBOR encoding. This enables zero-copy deserialization and efficient message handling while maintaining full compatibility with existing WAMP message semantics. * Fix CALL message __slots__ and add missing __eq__/__ne__ methods This commit fixes a bug and adds missing functionality to the CALL message class to properly support lazy deserialization: Bug fix: - Updated __slots__ to use underscore-prefixed attribute names (_request, _procedure, _args, etc.) to match the private attributes used with the property-based lazy deserialization pattern Missing feature: - Added __eq__(self, other) method that compares all 15 CALL attributes using property getters (not direct attribute access), enabling proper lazy deserialization during equality checks - Added __ne__(self, other) method for inequality comparison This brings CALL into full parity with PUBLISH's implementation pattern, ensuring that lazy deserialization from FlatBuffers works correctly when attributes are accessed through properties, and that message equality comparisons work as expected. All attribute comparisons now use the public property interface: request, procedure, args, kwargs, payload, timeout, receive_progress, transaction_hash, enc_algo, enc_key, enc_serializer, caller, caller_authid, caller_authrole, forward_for * Add args/kwargs fields to FlatBuffers schemas for consistent payload handling This commit fixes a critical schema inconsistency where some WAMP message types with application payload only had a single 'payload' field instead of supporting both normal mode (args/kwargs) and transparent mode (payload). Schema changes: - rpc.fbs: Added args/kwargs fields to Call, Result, Invocation, Yield tables - session.fbs: Added args/kwargs fields to Error table All message types with application payload now consistently support: - Normal mode: args/kwargs fields (each serialized with enc_serializer) - Transparent mode: payload field (for encrypted or embedded FlatBuffers) This matches the Publish and Event schema design which was already correct. Regenerated Python classes: - Recompiled all FlatBuffers schemas using flatc --python - Call, Result, Invocation, Yield, Error now have Args/Kwargs accessor methods - All generated classes in autobahn/wamp/gen/wamp/proto/ updated This fixes the schema foundation for the "flatbuffers-cbor" pattern where WAMP message structure uses FlatBuffers while application payload uses configurable serializers (JSON, MessagePack, CBOR, UBJSON, or embedded FlatBuffers via ENC_SER_FLATBUFFERS). Note: Python message class lazy deserialization properties will need updates in a follow-up commit to use the new Args/Kwargs methods correctly. * Add forward_for field to Error message for R2R link routing support This commit adds the missing forward_for field to the Error table in the FlatBuffers schema, completing R2R (Router-to-Router) link message forwarding support for error messages. Schema change: - session.fbs: Added forward_for: [Principal] to Error table This is required by WAMP protocol specification section 6.2.5 "Invocation ERROR" where errors originating from Callees must be routed back through router-to-router links to the original Caller. The forward_for field tracks the routing path taken through intermediate routers. With this change, all WAMP message types that carry application payload now consistently support the forward_for field for R2R link routing: - Publish ✓ - Event ✓ - Call ✓ - Result ✓ - Invocation ✓ - Yield ✓ - Error ✓ (added in this commit) Note: Cancel and Interrupt messages also have forward_for but do not carry application payload (they are pure control messages for canceling/interrupting calls). Regenerated Python classes: - Error class now has ForwardFor/ForwardForLength/ForwardForIsNone methods - All generated classes in autobahn/wamp/gen/wamp/proto/ updated * Document FlatBuffers schema design patterns and implementation patterns This commit adds comprehensive documentation for the critical design patterns used in FlatBuffers schemas and their Python implementation in message.py. Added new "Schema Design Patterns" section covering: 1. Application Payload Handling (6-Field Set): - Complete specification of the 6-field set (args/kwargs/payload/enc_algo/ enc_serializer/enc_key) used consistently across all message types with application payload - Two operational modes: Normal mode (dynamic typing with args/kwargs) vs Transparent mode (static typing/E2EE with payload field) - The "FlatBuffers-CBOR" composition pattern explained - Concrete examples for both modes - Message types: Publish, Event, Call, Result, Invocation, Yield, Error 2. Router-to-Router Message Forwarding: - Complete specification of forward_for field for R2R link routing - Principal struct definition and limitations - List of all message types with forward_for (payload + control messages) - Use cases: message provenance, loop detection, audit trails, federation - Concrete forwarding example 3. Python Implementation Patterns: - Initialization pattern with from_fbs parameter - Lazy deserialization property pattern - __slots__ with underscore-prefixed attributes - Equality comparison using property getters - Benefits: zero-copy, dual-mode operation, correct semantics This documentation captures the architectural decisions and implementation patterns developed during the FlatBuffers schema consistency work, ensuring this knowledge is preserved for future maintainers and contributors. Location: docs/wamp/flatbuffers-schema.rst (new section before Message Type Mapping) * Add PAYLOAD_SERIALIZER_ID to ISerializer interface and all implementations This makes the dual-serializer architecture explicit by separating: - Transport/envelope serialization (SERIALIZER_ID) - Application payload serialization (PAYLOAD_SERIALIZER_ID) For traditional serializers (JSON, CBOR, MsgPack, UBJSON), both IDs are identical since envelope and payload use the same format. For FlatBuffersSerializer, PAYLOAD_SERIALIZER_ID is configurable via new payload_serializer parameter (defaults to 'cbor'), enabling composition patterns like: - FlatBuffers envelope + CBOR payload (flatbuffers-cbor) - FlatBuffers envelope + JSON payload (flatbuffers-json) - FlatBuffers envelope + FlatBuffers payload (embedded static types) This architectural change enables Message.build() methods to serialize args/kwargs according to the message's enc_serializer attribute using the appropriate payload serializer instance. * Use payload serializer in Message.build() methods Implements the application payload serialization architecture: 1. Added serialize_payload() helper to Serializer base class - Uses _payload_serializer for FlatBuffers (configurable) - Uses _serializer for traditional serializers (same as envelope) 2. Store back-reference from IObjectSerializer to parent ISerializer - Enables Message.build() to access ISerializer.serialize_payload() - Maintains clean separation between ISerializer and IObjectSerializer 3. Updated Message.serialize() to pass parent ISerializer to build() - Extracts parent via _parent_serializer back-reference - Passes to build() for payload serialization 4. Updated Message.build() signature with serializer parameter - Base class now documents the serializer parameter - All subclasses updated to accept serializer=None 5. Updated Publish.build() and Event.build() implementations - Replace hardcoded cbor2.dumps() with serializer.serialize_payload() - Supports all payload serializers (JSON, CBOR, MsgPack, UBJSON, FlatBuffers) - Fallback to CBOR for backwards compatibility if serializer not provided This enables the FlatBuffers-CBOR composition pattern where WAMP message envelopes use FlatBuffers while application payloads use CBOR (or any other configured serializer). * Document dual-serializer architecture in FlatBuffers schema docs Added comprehensive 'Serialization Architecture' section covering: Protocol-Level Architecture: - Message Envelope (fixed WAMP structure) vs Application Payload (dynamic content) - Rationale for separate serialization of envelope and payload Implementation-Level Architecture: - ISerializer (transport serializer) with PAYLOAD_SERIALIZER_ID - IObjectSerializer (format-specific serializer) - Message.build() integration with serialize_payload() - Back-reference pattern for accessing parent ISerializer Serialization Flow: - Complete flow diagram from message creation to serialized bytes - Shows how FlatBuffers envelope is combined with CBOR payload Composition Patterns: - Traditional (homogeneous): JSON/JSON, CBOR/CBOR - FlatBuffers-CBOR (default): Zero-copy envelope + compact payload - FlatBuffers-JSON: Zero-copy envelope + human-readable payload - FlatBuffers-FlatBuffers: Complete static typing for WAMP IDL Configuration and Usage: - Code examples for creating serializers with different payload formats - Usage with WAMP sessions and transports Design Benefits: - Explicit separation of concerns - Flexible composition - Performance optimization - WAMP IDL support - Backwards compatibility This documents the architectural insights and ensures they are preserved for future reference and development. * Add cast() and build() methods to RPC and Error message types Implements FlatBuffers serialization for all WAMP message types with application payloads (args/kwargs/payload). Added to 5 message types: - Call: RPC call with procedure invocation - Result: RPC result from callee - Invocation: RPC invocation to callee - Yield: RPC yield from callee - Error: Error response for any request Each implementation includes: cast(buf) method: - Deserializes FlatBuffers buffer into message instance - Enables zero-copy deserialization from wire format build(builder, serializer) method: - Serializes message to FlatBuffers format - Uses serializer.serialize_payload() for args/kwargs - Supports all payload serializers (JSON/CBOR/MsgPack/UBJSON/FlatBuffers) - Handles forward_for field for R2R message forwarding - Serializes all message-specific fields (procedure, error URI, etc.) - Fallback to CBOR if serializer not provided (backwards compat) All build() methods follow consistent pattern from Publish/Event: 1. Serialize args/kwargs using configured payload serializer 2. Create FlatBuffers byte vectors for binary data 3. Create FlatBuffers strings for text fields 4. Build forward_for Principal structs for R2R routing 5. Construct final FlatBuffers message with all fields This completes FlatBuffers support for all WAMP message types that carry application payloads, enabling the full dual-serializer architecture (FlatBuffers envelope + configurable payload format). * Regenerate FlatBuffers files and add composition table - Regenerate all FlatBuffers binary schemas (.bfbs) and Python wrappers (.py) with correct flatc version to match CI expectations - Add "Serializer Composition Overview" table to documentation showing transport-payload serializer combinations (flatbuffers-cbor, flatbuffers-json, flatbuffers-flatbuffers, cbor-flatbuffers) - Table clarifies which combinations require application schemas vs just WAMP protocol schemas Fixes CI failure where regenerated files differed from committed versions. * Regenerate FlatBuffers files with flatc v25.9.23 Regenerate all FlatBuffers Python wrappers using flatc v25.9.23 (latest stable) to match CI build environment. This ensures reproducible builds and prevents SHA256 checksum mismatches in CI. Files regenerated with: PATH="$HOME/bin:$PATH" just clean-fbs && just build-fbs flatc v25.9.23 generates slightly different code compared to v23.5.26, affecting 11 message files (Call, Cancel, Error, Event, EventReceived, Hello, Interrupt, Invocation, Publish, Result, Yield). * Use gh CLI for downloading flatc binary in CI Replace curl with gh release download for more reliable downloading: - Handles GitHub redirects properly with authentication - Built-in retry logic and error handling - Avoids issues with signed Azure Blob Storage URLs - Added verification step to ensure download succeeded This fixes the "End-of-central-directory signature not found" error where only 54KB was downloaded instead of the expected 2.5MB zip file. * Add FlatBuffers cast() and build() to Published and Subscribed messages - Add from_fbs parameter support with lazy deserialization via properties - Implement cast() static method for deserialization from FlatBuffers - Implement build() method for serialization to FlatBuffers - Follow established pattern from Error, Publish, Event, Call, Result, Invocation, Yield Part of systematic completion of FlatBuffers support for all 25 WAMP message types. Progress: 9/25 message types complete (36%) * Add FlatBuffers implementation TODO and context for continuation Create comprehensive context document for completing FlatBuffers support: - Documents what's been completed (9/25 messages, 36%) - Details remaining 16 messages grouped by complexity - Provides implementation patterns and templates - Lists success criteria and testing strategy This allows continuation in a fresh session with full context. Current progress: ✅ Architecture complete (dual-serializer, documentation) ✅ Core payload messages complete (Error, Publish, Event, Call, Result, Invocation, Yield) ✅ Simple acknowledgments started (Published, Subscribed) Remaining: - 3 simple acknowledgments (Unsubscribed, Registered, Unregistered) - 3 request messages (Subscribe, Unsubscribe, Register) - 3 control messages (EventReceived, Cancel, Interrupt) - 6 session messages (Hello, Welcome, Abort, Challenge, Authenticate, Goodbye) * Complete FlatBuffers support for all 25 WAMP messages This commit completes the FlatBuffers serialization implementation for all remaining WAMP message types (16 messages), bringing total coverage to 25/25 messages (100%). Changes: - Added cast() and build() methods to all remaining message classes - Updated __slots__ to use private field names (_field) to avoid @property conflicts - Implemented lazy deserialization using @property decorators - Added enum mappings for Match, InvocationPolicy, CancelMode - Implemented basic support for complex messages (Hello/Welcome) with notes about deferred ClientRoles/RouterRoles full serialization Messages completed in this session: - Group 1: Unsubscribed, Registered, Unregistered - Group 2: Subscribe, Unsubscribe, Register - Group 3: EventReceived, Cancel, Interrupt - Group 4: Abort, Goodbye, Challenge, Authenticate, Hello, Welcome Known limitations (to be enhanced): - Hello/Welcome: ClientRoles/RouterRoles nested structures (basic skeleton only) - Challenge/Authenticate/Hello/Welcome: Map/dict field serialization deferred - Challenge/Welcome: AuthMethod enum conversions simplified - Forward_for: Principal struct deserialization deferred Testing: - All 446 SerDes tests pass - 83 skipped (expected - FlatBuffers not in test vectors yet) - Module imports successfully - Code compiles without errors Fixes #1764 Note: This work was completed with AI assistance (Claude Code). * Add type comments, setters, and equality methods to FlatBuffers messages This commit addresses three important issues identified in code review: 1. **Type Comments in __slots__**: Added FlatBuffers type annotations as comments above each slot field (e.g., # uint64, # string, # bool) to document the expected types from the FlatBuffers schema. 2. **Property Setters**: Added setter methods for all @property fields with appropriate type assertions. This maintains API compatibility and allows modification of message fields after instantiation. 3. **Equality Methods**: Added __eq__ and __ne__ methods to all message classes for proper object comparison. These methods compare all fields systematically, following the pattern from the Publish message class. Messages fixed (15 total): - Group 1: Unsubscribed, Registered, Unregistered - Group 2: Subscribe, Unsubscribe, Register - Group 3: EventReceived, Cancel, Interrupt - Group 4: Abort, Goodbye, Challenge, Authenticate, Hello, Welcome All changes follow the established pattern from the Publish message class (line 2664) and maintain full API compatibility. Testing: - Code compiles without errors - All 446 SerDes tests pass - 83 skipped (expected - FlatBuffers not in test vectors) Related to #1764 Note: This work was completed with AI assistance (Claude Code).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Could potentially be the actual problem from #547