Skip to content

Phlex v0.3.0

Choose a tag to compare

View all tags
@knoepfel knoepfel released this 30 Jun 18:08
· 272 commits to main since this release
v0.3.0
fa45b3a

Important

This is a prototype release. It is known to have incomplete functionality with respect to the Phlex conceptual design; backwards compatibility of API is not guaranteed. The following API is intended for use:

  • PHLEX_REGISTER_ALGORITHMS(...) opener for registering physics algorithms
  • PHLEX_REGISTER_PROVIDERS(...) opener for registering explicit data-product providers
  • PHLEX_REGISTER_SOURCE(...) opener for registering a user-defined input source
  • PHLEX_REGISTER_DRIVER(...) opener for registering a user-defined framework-driver function
  • Anything in the top-level phlex namespace

Any interface (namespaces, functions, classes, etc.) that contains the strings experimental, internal, or detail should not be directly used.

Phlex v0.3.0 is the third prototype release of the Phlex framework. The purpose of the v0.3.0 release is to allow early adopters of Phlex to continue exercising the functionalities provided by Phlex v0.2.0, while testing a few new features, such as:

  • Support for reading data products from FORM files
  • A user-defined API for registering input sources, from which data-product providers are implicitly registered
  • Algorithms that operate on a data layer downstream of an unfold and that receive data products produced by that unfold
  • A renamed and simplified product-selection grammar (product_selector) in which the creator name is optional
  • Native Numba support for transforms, enabling GIL-free execution of JIT-compiled algorithms

Instructions for building code that uses Phlex and running simple Phlex jobs can be found in the phlex-examples repository.

Note

Phlex v0.3.0 uses C++23. You will need a compiler with sufficient C++23 support to build and install this release (GCC 15, LLVM 21, etc.)

Specific Features

  1. FORM improvements

    • FORM gains a revised source infrastructure, writing of fundamental types through TTree, a technology loop across storage backends, output-metadata writing, reading of FORM input files (without metadata) via a phlex::source, and an initial RNTuple draft (#552, #597, #599, #612, #625, #647, #635, #674).

  2. API to register input sources and implicitly registered providers

    • Phlex v0.3.0 introduces a phlex::source base class, the PHLEX_REGISTER_SOURCE opener, and the g.add_source<T>(...) registration, along with a phlex::provider_bundle class (see #637).
    • Data-product providers can now be implicitly registered from a user-defined source, so users no longer need to register each provider explicitly (#630).
    • The phlex::index_generator type allows a source to lazily emit data-cell indices, supporting driver coroutines (#618).
    • Drivers can now access configured sources in a type-safe way, with driver functions receiving typed source parameters resolved from the registered sources (#658).

  3. Algorithms downstream of an unfold may consume that unfold's data products

    • Previously a transform operating on a hierarchy layer downstream of an unfold could not receive data products produced by that unfold.
    • Phlex v0.3.0 reworks the flush/routing pipeline so that multi-layer consumer nodes can receive input data products from unfold nodes (#550, #571).

  4. Simplified product-selection grammar

    • The product_query type has been renamed to product_selector (and product_queries to product_selectors), with assorted cleanups (#616).
    • The product creator name is now optional when selecting products (#621).

  5. Native Numba support

    • Numba-compiled transforms can now be called directly through a libffi-based dispatch, bypassing the Python interpreter (and the GIL) during graph execution and enabling strong scaling. This initial support covers transforms operating on builtin scalar types (#626).

  6. Experimental layer_path abstraction for hierarchy paths

    • A new phlex::experimental::layer_path class replaces the prior string-based representation of hierarchy paths used by the index_router and related infrastructure (#640).

  7. Hardened code base and tooling

    • Hundreds of clang-tidy findings (bugprone, cert, clang-analyzer, cppcoreguidelines, performance, portability, and modernize checks) have been resolved across the code base in preparation for tightening enforcement.
    • Shared libraries are now armored with explicit symbol visibility, and a Sphinx/Doxygen/Breathe documentation framework has been added for the phlex and phlex::experimental APIs (#322, #323).

Included Pull Requests

  • A more inclusive handling of Catch2's use of __COUNTER__ by @greenc-FNAL in #478
  • Metaprogramming using Boost.CallableTraits and Boost.MP11 by @beojan in #444
  • Update pre-commit hooks, including markdownlint-cli2 by @greenc-FNAL in #481
  • Add Perfetto profiling support to CI workflow by @greenc-FNAL in #341
  • Armor shared libraries with symbol visibility; add _internal companions and build-type-aware optimizations by @Copilot in #322
  • Apply pre-commit changes pursuant to #481 by @knoepfel in #482
  • Make generated macros consistent with UPPER_CASE convention by @knoepfel in #483
  • Add Sphinx/Doxygen/Breathe documentation framework for phlex and phlex::experimental API by @Copilot in #323
  • Fix clang-tidy --export-fixes race condition; use run-clang-tidy by @greenc-FNAL in #486
  • Use DISABLED test property for different_hierarchies test by @knoepfel in #492
  • clang-tidy: resolve bugprone-random-generator-seed by @greenc-FNAL in #490
  • clang-tidy: fix bugprone-reserved-identifier by @greenc-FNAL in #489
  • clang-tidy: resolve bugprone-throwing-static-initialization by @greenc-FNAL in #472
  • clang-tidy: resolve bugprone-exception-escape by @greenc-FNAL in #491
  • clang-tidy: resolve bugprone-multi-level-implicit-pointer-conversion by @greenc-FNAL in #494
  • clang-tidy: resolve bugprone-return-const-ref-from-parameter by @greenc-FNAL in #499
  • clang-tidy: resolve bugprone-implicit-widening-of-multiplication-result by @greenc-FNAL in #496
  • maintenance/clang tidy script improvements by @greenc-FNAL in #502
  • clang-tidy: resolve bugprone-unchecked-optional-access by @greenc-FNAL in #495
  • clang-tidy: resolve bugprone-macro-parentheses by @greenc-FNAL in #500
  • clang-tidy: resolve bugprone-narrowing-conversions by @greenc-FNAL in #497
  • improve error handling consistency by @wlav in #525
  • clang-tidy: resolve rand()-related cert-* issues by @greenc-FNAL in #527
  • clang-tidy: resolve cert-dcl50-cpp (modernize-avoid-variadic-functions) by @greenc-FNAL in #529
  • clang-tidy: resolve cert-err33-c (dangerous return discard from libc functions) by @greenc-FNAL in #528
  • clang-tidy: resolve cert-oop54-cpp by @greenc-FNAL in #526
  • Add branch push triggers for release/* branches by @greenc-FNAL in #531
  • clang-tidy: resolve clang-analyzer-core problems by @greenc-FNAL in #538
  • Test that repeaters never emit data products for flush counts of 0 by @knoepfel in #533
  • Provide actionable diagnostic data when the problem location is outside Phlex by @greenc-FNAL in #537
  • clang-tidy: resolve cppcoreguidelines-avoid-non-const-global-variables by @greenc-FNAL in #544
  • clang-tidy: resolve cppcoreguidelines-noexcept-move-operations by @greenc-FNAL in #549
  • Fix real heap leaks in py_callback registrations; suppress Boost false positive by @greenc-FNAL in #541
  • clang-tidy: resolve cppcoreguidelines-explicit-virtual-functions by @greenc-FNAL in #545
  • clang-tidy: resolve cppcoreguidelines-missing-std-forward by @greenc-FNAL in #547
  • clang-tidy: resolve cppcoreguidelines-avoid-const-or-ref-data-members by @greenc-FNAL in #543
  • Revised implementation of FORM source infrastructure by @wwuoneway in #552
  • clang-tidy: resolve cppcoreguidelines-init-variables by @greenc-FNAL in #546
  • clang-tidy: resolve cppcoreguidelines-no-malloc by @greenc-FNAL in #548
  • clang-tidy: resolve performance-avoid-endl by @greenc-FNAL in #563
  • clang-tidy: resolve cppcoreguidelines-rvalue-reference-param-not-moved by @greenc-FNAL in #559
  • clang-tidy: resolve performance-no-int-to-ptr by @greenc-FNAL in #564
  • clang-tidy: resolve cppcoreguidelines-pro-type-member-init by @greenc-FNAL in #555
  • clang-tidy: resolve cppcoreguidelines-special-member-functions by @greenc-FNAL in #560
  • clang-tidy: resolve cppcoreguidelines-use-default-member-init by @greenc-FNAL in #556
  • clang-tidy: resolve cppcoreguidelines-pro-type-vararg by @greenc-FNAL in #558
  • clang-tidy: resolve cppcoreguidelines-pro-type-union-access by @greenc-FNAL in #557
  • clang-tidy: resolve cppguidelines-pro-type-*cast* by @greenc-FNAL in #554
  • clang-tidy: resolve cppcoreguidelines-owning-memory by @greenc-FNAL in #551
  • Add clang-tidy new-issue detection for modified lines by @greenc-FNAL in #539
  • FORM Missing Branch Error Message by @aolivier23 in #570
  • Replace char const* with char* as appropriate by @knoepfel in #572
  • Add git-ai-commit script for AI-powered commit message generation by @greenc-FNAL in #567
  • Initial support for kilocode (fnal-litellm) by @greenc-FNAL in #568
  • Fix bugs in CI scripts and workflows; add test suites and improve documentation by @greenc-FNAL in #573
  • Update CI container and Spack configuration for improved build reliability by @greenc-FNAL in #575
  • Accommodate oneTBB 2023.0.0 resource-limiting interface by @knoepfel in #576
  • Avoid maybe-uninitialized GCC warnings by @knoepfel in #577
  • Improve scripts/ testing and ergonomics by @greenc-FNAL in #578
  • Improve error handling in git-ai-commit and add comprehensive tests by @greenc-FNAL in #579
  • clang-tidy: resolve performance-faster-string-find by @greenc-FNAL in #584
  • clang-tidy: resolve performance-move-const-arg by @greenc-FNAL in #587
  • clang-tidy: resolve performance-inefficient-string-vector-operation by @greenc-FNAL in #586
  • clang-tidy: resolve performance-enum-size by @greenc-FNAL in #583
  • clang-tidy: document incidental fix of performance-type-promotion-in-math-fn by @greenc-FNAL in #589
  • clang-tidy: resolve modernize-use-nullptr by @greenc-FNAL in #592
  • clang-tidy: resolve google-explicit-constructor by @greenc-FNAL in #581
  • Optimize FORM storage init by replacing double find with insert(...).first by @wwuoneway in #582
  • Adopt Claude-suggested optimization for async_driver by @knoepfel in #596
  • Provider adjustments in anticipation of v0.3 by @knoepfel in #580
  • FORM TTree Write Fundamental Types by @aolivier23 in #597
  • Form Test Writing Fundamental Types by @aolivier23 in #599
  • Fix compilation with GCC 16 by @beojan in #601
  • Exclude scripts/ path from unittests code coverage by @knoepfel in #598
  • clang-tidy: resolve portability-template-virtual-member-function by @greenc-FNAL in #591
  • clang-tidy: resolve performance-unnecessary-copy-initialization by @greenc-FNAL in #590
  • Move provider API away from misusing product_query for outputs by @beojan in #602
  • Support multi-layer algorithms that receive a data product from an unfold by @knoepfel in #571
  • Fix clang-tidy PR comments for fork PRs via workflow_run by @greenc-FNAL in #610
  • More workflow and documentation improvements by @greenc-FNAL in #611
  • FORM Testing Loop Over Technologies by @aolivier23 in #612
  • Optimize some hot paths by @knoepfel in #603
  • clang-tidy: resolve performance-inefficient-string-concatenation by @greenc-FNAL in #585
  • Add coverage monitoring analysis report for May 2026 by @greenc-FNAL in #593
  • Product query (now product_selector) cleanup and rename by @beojan in #616
  • clang-tidy: resolve performance-no-automatic-move by @greenc-FNAL in #588
  • Some cleanups per @beojan, etc. by @knoepfel in #615
  • Accommodate product_specification API change by @knoepfel in #620
  • Fix CodeQL PR comment: alert count collapse and non-filtered report link by @Copilot in #617
  • Make creator name optional by @beojan in #621
  • Constrain exposed API of graph-proxy objects by @knoepfel in #624
  • Introduce phlex::index_generator to support driver coroutines by @knoepfel in #618
  • FORM Test Technology Loop Part 2 by @aolivier23 in #625
  • Omit asserts from code-coverage results by @knoepfel in #634
  • Refactor edge-making into clearer stages by @knoepfel in #633
  • Change location of jsonnet image by @greenc-FNAL in #645
  • Support phlex::sources and implicit providers by @knoepfel in #637
  • fix(devcontainer): drop --format=docker from buildOptions by @Ahmed-Idani in #606
  • Improve AI integration with devcontainer and tools by @greenc-FNAL in #646
  • Introduce layer_path class by @beojan in #640
  • chore: use standalone action repositories by @greenc-FNAL in #639
  • Use zendesk/setup-jsonnet action for reliable go-jsonnet retrieval by @greenc-FNAL in #649
  • Form write output metadata by @wwuoneway in #647
  • Fix fold-result type by @knoepfel in #650
  • FORM RNTuple Draft by @aolivier23 in #635
  • Update CI container image to support Numba and libffi by @knoepfel in #657
  • Bump actions/checkout from 6.0.3 to 7.0.0 by @app/dependabot in #655
  • Bump Framework-R-D/action-configure-cmake from 1 to 2 by @app/dependabot in #654
  • Bump actions/setup-python from 6.2.0 to 6.3.0 by @app/dependabot in #660
  • Fix emplacement of implicit providers for multiple downstream nodes by @knoepfel in #664
  • Initial Numba support by @wlav in #626
  • Add stage name to handle by @beojan in #663
  • revert the metadata part introduced in PR647 by @wwuoneway in #665
  • Fix latent memory error exposed by fixed_hierarchy_test by @knoepfel in #668
  • Bump Framework-R-D/action-complete-pr-comment from 1 to 2 by @app/dependabot in #669
  • Bump Framework-R-D/action-run-change-detection from 1 to 2 by @app/dependabot in #670
  • Bump Framework-R-D/action-post-clang-tidy-results from 1 to 2 by @app/dependabot in #671
  • Bump Framework-R-D/action-handle-fix-commit from 1 to 2 by @app/dependabot in #672
  • Fix workflow-dispatch action triggering of code-coverage by @knoepfel in #673
  • FORM read files without metadata by @wwuoneway in #674
  • Allow drivers to access configured sources by @knoepfel in #658
  • workaround for numpy 2.5 typing origin change by @wlav in #675
  • Form read extend data type by @wwuoneway in #676
  • Explicitly include Cet CMake modules by @greenc-FNAL in #678
  • Rationalize namespaces by @knoepfel in #679

New Contributors

Full Changelog: v0.2.0...v0.3.0

Contributors

greenc-FNAL, beojan, and 5 other contributors
Assets 2
Loading