Skip to content

Phlex v0.4.0

Choose a tag to compare

View all tags
@knoepfel knoepfel released this 30 Sep 15:57
· 75 commits to main since this release
v0.4.0
574340c

Important

This is a prototype release. It is known to have incomplete functionality with respect to the Phlex conceptual design; backwards compatibility of API is not guaranteed. The following API is intended for use:

  • PHLEX_REGISTER_ALGORITHMS(...) opener for registering physics algorithms
  • PHLEX_REGISTER_PROVIDERS(...) opener for registering explicit data-product providers
  • PHLEX_REGISTER_SOURCE(...) opener for registering a user-defined input source
  • PHLEX_REGISTER_DRIVER(...) opener for registering a user-defined framework-driver function
  • PHLEX_REGISTER_RESOURCES(...) opener for registering resource plugins
  • Anything in the top-level phlex namespace

Interfaces in phlex::experimental may change; interfaces containing internal or detail should not be directly used.

Phlex v0.4.0 is a prototype release of the Phlex framework. Building on Phlex v0.3.2, it introduces:

  • Framework-owned resources that algorithms can access, including serialized resources that limit concurrent execution, and configuration-driven resource plugins
  • Product selectors that may omit the data layer for eligible single-input algorithms, with layer checks applied as products travel through the graph
  • Stage-aware execution and product provenance
  • An experimental construction API for implicit providers in user-defined sources
  • Enum members in data-product type identifiers, and reworked fold-result accumulation

Instructions for building code that uses Phlex and running simple Phlex jobs can be found in the phlex-examples repository.

Warning

Breaking change: every Phlex invocation now requires a stage name.

Existing job configurations must be updated before they can run with Phlex v0.4.0. Specify the stage either as a top-level configuration parameter:

{
  stage: 'reconstruction',
  // Remaining job configuration
}

or on the command line:

phlex -c job.jsonnet --stage reconstruction

If both are specified, --stage takes precedence over the value in the configuration file. Stage names must be non-empty, and CURRENT is reserved and cannot be used as the invocation's stage name.

Product selectors that omit stage continue to select products from the current invocation, so most algorithm registrations do not need to change. Specify a stage in a product selector only when selecting a product from a different stage.

Note

Phlex v0.4.0 uses C++23. You will need a compiler with sufficient C++23 support to build and install this release (GCC 15, LLVM 21, etc.).

Specific Features

1. Framework-owned resources and resource-limited execution

Algorithms can request typed, non-data resources after their product selectors in input_family(...), using phlex::resource<T>{}. A graph can register an unlimited-access resource with add_unlimited_resource<T>(...) or a serialized resource with add_serialized_resource<T>(...). Unlimited resources do not constrain scheduling; serialized resources provide one access token at a time for the duration of an algorithm invocation. Resource dependencies are supported by observers, transforms, predicates, folds, and unfolds. Multi-token resources are not yet supported (#851, #850).

Resource plugins can instead use PHLEX_REGISTER_RESOURCES(...) to construct resources from job configuration. Phlex loads these plugins before algorithms, sources, and drivers so their dependencies can be resolved (#852, #905).

2. Layer-aware product selection

For a single-input computational operator, a product_selector may omit the layer, allowing provider resolution to determine the concrete data layer. This works with explicit and implicit providers and is also supported by Python algorithms. Multi-input operators must still specify all input layers; folds and unfolds also continue to require them (#875, #817, #889).

Product layers are now checked when products pass over graph edges. A consumer receives only products matching its selector's layer; graph construction can connect multiple producers that differ by layer, rather than prematurely choosing just one (#704).

3. Stage-aware execution and products

Every Phlex invocation now has a required stage name, supplied through the top-level stage configuration parameter or the --stage command-line option. The command-line value takes precedence.

The stage is part of a product's provenance and is available through phlex::handle<T>::stage() and phlex::product_store::stage(). A product selector may specify stage to consume a product from a particular processing stage; when the stage is omitted, the selector refers to the current invocation.

FORM storage and navigation are also stage-aware. Products are distinguished by creator and stage, allowing products with the same name and creator from different stages to coexist in a file. FORM sources use the configured stage when locating stored products.

This change removes the former implicit CURRENT fallback when constructing handles, product stores, and provider products directly. The configured stage is propagated to explicit and implicit providers that produce products for the current invocation. See #731, #897, #920, and #924.

4. Source-provider API with an explicit experimental boundary

phlex::provider_bundle and phlex::provider_bundles are available in the public namespace for sources that create implicit providers. The supporting product specifications, erased products, and provider-function types needed to construct those bundles are exposed under phlex::experimental, so source authors no longer need to name phlex::detail types. This construction API is provisional (#686, #883, #882, #891).

5. Enum members in data-product type identifiers

Type identifiers now accept enum types, including enums nested in structs passed between algorithms, using their underlying fundamental type for identification (#765, #874).

6. Fold-result accumulation and routing

Fold results now use accumulator-based partition processing and a dedicated fold join instead of the previous result-cache and store-counter machinery. This improves routing and flush handling for partitions, including empty partitions; associated follow-ups widen committed end-token counts and strengthen multi-partition tests (#359, #540, #829, #865).

7. Integration and correctness improvements

Installed CMake targets now locate their public dependencies automatically, avoiding the need for downstream projects to repeat Phlex's find_package(...) calls (#656, #827). The data_cell_cursor and other non-owning pointers express non-null requirements with gsl::not_null (#824, #839).

Python annotation processing now preserves errors when normalizing Numba signatures and avoids referencing a released Unicode buffer (#773, #889, #774, #836). Repeater cleanup also avoids a data race during the transition to pass-through operation (#884). Additional regression coverage exercises reuse of bound glue objects (#825).

Phlex now restores its framework logger after loading plugins, preventing resource, algorithm, source, or driver plugins from accidentally replacing or disabling framework logging (#919).

Release Scope

See the Prototype 0.4 milestone for the issues and pull requests considered for this release. The creator-optional selector work included in that milestone was already released in Phlex v0.3.x and is not a new v0.4.0 feature (#621).

Full Changelog: v0.3.2...v0.4.0

What's Changed

  • ci(coverage): temporarily disable push and pull_request triggers by @greenc-FNAL in #690
  • Catalog and loader cleanup by @marcpaterno in #659
  • Updates and improvements to pre-commit config by @greenc-FNAL in #662
  • style(ruff): bring scripts/git-ai-commit into conformance by @knoepfel in #696
  • Remove unused product_matcher utility by @knoepfel in #698
  • Adjust product_selector print format by @beojan in #694
  • use unique_ptr for FORM product_from_data to own buffer by @wwuoneway in #705
  • Post CodeQL comment only when PR creates a new alert or resolves an existing one by @knoepfel in #711
  • Forward instead of move in product_selector converting constructors by @olantwin in #714
  • clang-tidy: resolve readability-else-after-return by @knoepfel in #719
  • clang-tidy: resolve modernize-use-ranges by @knoepfel in #717
  • ci(coverage): re-enable push and pull_request triggers by @greenc-FNAL in #725
  • Sync main with v0.3.2 by @knoepfel in #724
  • feat: add dependency-aware release utilities by @greenc-FNAL in #652
  • build: update pre-commit hook versions by @greenc-FNAL in #722
  • CI container improvements inc. standalone Cetmodules by @greenc-FNAL in #661
  • feat: enhance git-ai-commit prompt and diff handling by @greenc-FNAL in #682
  • Migrate devcontainer to Docker Compose and update toolchain by @greenc-FNAL in #681
  • clang-tidy: resolve modernize-use-auto by @knoepfel in #733
  • clang-tidy: resolve modernize-use-std-numbers by @knoepfel in #734
  • chore: update workflow governance and runtime controls by @greenc-FNAL in #687
  • Make RNTuple Read All File Entries by @aolivier23 in #715
  • clang-tidy: resolve readability-redundant-string-init by @knoepfel in #716
  • clang-tidy: resolve modernize-redundant-void-arg by @knoepfel in #710
  • clang-tidy: resolve modernize-use-starts-ends-with by @knoepfel in #727
  • clang-tidy: resolve modernize-use-using by @knoepfel in #720
  • clang-tidy: resolve readability-redundant-control-flow by @knoepfel in #743
  • clang-tidy: resolve modernize-avoid-c-arrays by @knoepfel in #745
  • ci: fix indentation in actionlint-check.yaml workflow by @greenc-FNAL in #758
  • chore: use more informative target name in CodeQL debug log by @greenc-FNAL in #754
  • Fix FORM's RNTuple backend build against ROOT 6.40 by @olantwin in #749
  • clang-tidy: resolve readability-avoid-const-params-in-decls by @knoepfel in #747
  • clang-tidy: resolve readability-container-size-empty by @knoepfel in #748
  • clang-tidy: resolve modernize-make-shared by @knoepfel in #746
  • clang-tidy: resolve readability-static-definition-in-anonymous-namespace by @knoepfel in #742
  • clang-tidy: resolve readability-isolate-declaration by @knoepfel in #756
  • clang-tidy: resolve readability-inconsistent-ifelse-braces by @knoepfel in #761
  • clang-tidy: resolve readability-redundant-access-specifiers by @knoepfel in #757
  • chore(deps): update LLVM to 22.1.8 in CI spack environment by @greenc-FNAL in #767
  • clang-tidy: resolve readability-inconsistent-declaration-parameter-names by @knoepfel in #741
  • clang-tidy: resolve readability-redundant-typename by @knoepfel in #760
  • clang-tidy: resolve readability-qualified-auto by @knoepfel in #759
  • chore(devcontainer): update development container configuration by @greenc-FNAL in #753
  • ci: add mypy to pre-commit hooks by @greenc-FNAL in #738
  • FORM Apply Same Tests to RNTuple and TTree by @aolivier23 in #740
  • clang-tidy: resolve readability-redundant-member-init by @knoepfel in #763
  • clang-tidy: resolve readability-const-return-type by @knoepfel in #766
  • clang-tidy: resolve readability-const-return-type in virtual by @greenc-FNAL in #770
  • ci: fix indentation in actionlint-check workflow and update pre-commit hooks by @greenc-FNAL in #775
  • clang-tidy: resolve readability-use-concise-preprocessor-directives by @knoepfel in #772
  • FORM Unify TTree Testing by @aolivier23 in #777
  • maintenance/pre commit hook updates by @greenc-FNAL in #776
  • maintenance/workflows harden checkouts by @greenc-FNAL in #778
  • clang-tidy: resolve readability-container-contains by @knoepfel in #768
  • clang-tidy: resolve readability-math-missing-parentheses by @knoepfel in #769
  • clang-tidy: resolve readability-braces-around-statements by @knoepfel in #762
  • clang-tidy: resolve readability-redundant-casting by @knoepfel in #783
  • clang-tidy: resolve modernize-return-braced-init-list by @knoepfel in #788
  • clang-tidy: resolve modernize-use-scoped-lock by @knoepfel in #789
  • clang-tidy: resolve modernize-use-equals-default by @knoepfel in #784
  • clang-tidy: resolve modernize-avoid-c-style-cast by @knoepfel in #771
  • clang-tidy: resolve readability-redundant-string-cstr by @knoepfel in #792
  • Add some unit tests by @marcpaterno in #782
  • FORM Testing Use c++11 Random by @aolivier23 in #790
  • Fix name shadowing in storage_reader and storage_writer by @aolivier23 in #795
  • Form local naming readability and regression-test cleanup by @wwuoneway in #794
  • clang-tidy: resolve modernize-pass-by-value by @knoepfel in #797
  • Mark clang-analyzer-security.insecureAPI.rand as resolved by @knoepfel in #798
  • clang-tidy: resolve modernize-use-emplace by @knoepfel in #793
  • Fix set construction when deciding to insert repeaters by @beojan in #800
  • chore(deps): add npm release age constraint for @kilocode/cli by @greenc-FNAL in #801
  • Removed unused string_literal class by @knoepfel in #799
  • clang-tidy: resolve modernize-use-integer-sign-comparison by @knoepfel in #791
  • feat(core): add accumulator-based fold routing and edge-case hardening by @knoepfel in #540
  • Form redesign backend technology by @wwuoneway in #816
  • refactor: clean up sendability naming and flush-gate count typing by @knoepfel in #813
  • clang-tidy: resolve modernize-concat-nested-namespaces by @knoepfel in #804
  • Some improvements as identified by AI by @knoepfel in #708
  • Simplify construction of implicit providers by @knoepfel in #709
  • FORM Canonical Index Format by @wwuoneway in #823
  • clang-tidy: resolve performance-unnecessary-value-param by @knoepfel in #796
  • fix(cmake): propagate public dependencies to installed package config by @knoepfel in #827
  • Fix straggling clang-tidy errors by @knoepfel in #803
  • fix(core): preserve wide counts in indexed end tokens by @knoepfel in #829
  • clang-tidy fixes for exceptions that were created by not thrown by @wlav in #836
  • Form: Return a Token from registerWrite by @wwuoneway in #830
  • clang-tidy: resolve modernize-use-designated-initializers by @knoepfel in #835
  • test: add scoped shared ostream logger utility by @knoepfel in #828
  • test: standardize C++ test source and target names by @knoepfel in #842
  • FORM: apply naming conventions used as Phlex by @wwuoneway in #846
  • refactor(core): adopt gsl::not_null where appropriate by @knoepfel in #839
  • clang-tidy: resolve readability-use-anyofallof by @knoepfel in #805
  • chore(cleanup): revert unwanted commits by @greenc-FNAL in #848
  • Minor improvements to git-ai-commit by @greenc-FNAL in #849
  • clang-tidy: resolve readability-identifier-naming by @knoepfel in #858
  • fix: resolve dangling clang-tidy warnings in core and tests by @knoepfel in #837
  • clang-tidy: resolve readability-simplify-boolean-expr by @knoepfel in #859
  • test: cover fixed-hierarchy cursor traversal by @knoepfel in #838
  • test: improve multi-partition accumulator flush assertions by @knoepfel in #865
  • clang-tidy: resolve readability-static-accessed-through-instance by @knoepfel in #860
  • FORM write path: config-once, create container once, persistence owns the index by @wwuoneway in #864
  • test: add bound glue reuse regression coverage by @knoepfel in #866
  • Use struct for class with only public members by @pcanal in #862
  • clang-tidy: resolve readability-convert-member-functions-to-static by @knoepfel in #868
  • refactor(source): provide default source::indices() implementation by @knoepfel in #863
  • ci: make clang-tidy PR comments persistent by @knoepfel in #870
  • clang-tidy: resolve some warnings that slipped in by @knoepfel in #871
  • ci: improve clang-tidy PR reporting and scope checks by @knoepfel in #869
  • model: support enums in type identifiers by @knoepfel in #874
  • clang-tidy: reduce function sizes by @knoepfel in #872
  • Make layer specification optional by @beojan in #817
  • FORM RNTuple Accessor Functions by @aolivier23 in #880
  • refactor(python): split configuration and registration wrappers by @knoepfel in #861
  • style: sync phlex and phlex-examples configurations by @knoepfel in #877
  • feat: support openspec store in phlex-design by @marcpaterno in #890
  • refactor(api): expose provider bundles in phlex namespace by @knoepfel in #883
  • clang-tidy: resolve some straggling clang-tidy errors by @knoepfel in #892
  • ci: compare clang-tidy findings from PR merge base by @knoepfel in #894
  • refactor(api): rationalize phlex::experimental usage by @knoepfel in #891
  • Form write navigation layout by @wwuoneway in #893
  • form write: make technology part of write container identity by @wwuoneway in #899
  • feat: add resource-limited execution to graph nodes by @knoepfel in #850
  • fix(core): handle pass-through repeater entries safely by @knoepfel in #884
  • Python optional layer by @wlav in #889
  • Form: add DUNE-style hit example for navigation testing by @wwuoneway in #908
  • Remove unnecessary force_streamer_field configuration by @wwuoneway in #910
  • Layer checking as products pass over edges by @beojan in #704
  • feat(resources): add dynamically loaded resource plugins by @knoepfel in #905
  • style(headers): use angle brackets for third-party includes by @knoepfel in #911
  • feat: support 'stage' configuration parameter and API by @knoepfel in #897
  • FORM RNTuple exception handling by @aolivier23 in #909
  • fix(cmake): correct FORM test include path by @knoepfel in #914
  • Form: Make FORM's storage layout and navigation stage-aware by @wwuoneway in #920
  • fix(providers): propagate configured stage for provided products by @knoepfel in #924
  • fix(logging): restore Phlex logger after plugin setup by @knoepfel in #919
  • Resolve markdownlint config ambiguity and enhance config by @greenc-FNAL in #900
  • docs: update installation instructions for Phlex v0.4 by @knoepfel in #927
  • perf(handle): borrow stage and product specification by @knoepfel in #926
  • Enabled RNTuple by default when installing phlex. by @aolivier23 in #928
  • fix(app): make stage validation errors actionable by @knoepfel in #929
  • Form multiple output destinations by @wwuoneway in #930
  • fix(form): add missing include directory by @knoepfel in #931
  • clang-tidy(misc-include-cleaner): correct direct and unused includes by @knoepfel in #925

Full Changelog: v0.3.2...v0.4.0

Contributors

marcpaterno, greenc-FNAL, and 7 other contributors
Assets 2
Loading