Add optional Rust backend with engine auto-dispatch#845
Merged
Conversation
Add vectorbt-rust crate (maturin/PyO3) with generic rolling kernels
(rolling_mean, rolling_std, rolling_min, rolling_max, diff) for both
1D and 2D arrays. Introduce _backend.py for centralized backend
resolution and per-module dispatch.py wrappers that route between
numba and Rust based on settings['backend'] ('auto'/'numba'/'rust').
Mark the Rust backend milestone as the 1.0.0 release. Updates version across vectorbt, vectorbt-rust, and the pinned dependency.
Reorganize .gitignore with categorized sections and proper glob patterns. Fix rust/pyproject.toml: remove missing readme reference, use license string instead of license-files (maturin disallows ..).
…with_rust Rewrite README with clearer tagline, bold feature labels, and concise example headings. Rename supports_rust() to array_compatible_with_rust() for clarity, trim redundant docstrings and comments in the backend module.
Add Rust implementations for shuffle, set_by_mask, fillna, shifts, diff, pct_change, fills, cumulative ops, rolling/expanding/EWM statistics, range detection, and drawdown detection. Introduce RustSupport dataclass for structured compatibility reporting with reasons. Wire all generic module entry points (accessors, ranges, drawdowns, plots, stats) through the dispatch layer. Add benchmarks and backend tests.
… cases Rename *_func_nb parameters to *_func across accessors and dispatch, add backend= support to reduce/squeeze_grouped, introduce backend-aware function validation (is_backend_func, assert_backend_func), fix Rust rolling_std when ddof >= window_len, add broadcasting to drawdown helpers, share RNG across shuffle columns, and handle empty drawdown arrays.
Add dedicated C-order implementations for 18 functions that operate directly on contiguous row-major memory, avoiding per-column extraction. Rewrite rolling/expanding mean and std to use online accumulators instead of cumsum arrays. Branch on is_standard_layout() at runtime to pick the optimal path.
…atch Port MA, MSTD, BBANDS, RSI, STOCH, MACD, ATR, and OBV to Rust via a new indicators module and dispatch layer. Expose generic helpers as pub(crate) for reuse, add ewm_mean/std 2D C-order kernels, and fix ffill/bfill 2D bugs. Rename is_backend_func to is_backend_compatible_func with stricter dispatch detection.
- Add `rust/src/signals.rs` with Rust implementations of clean_enex, rank (sig_pos_rank, part_pos_rank), nth_index, norm_avg_index, between_ranges, between_two_ranges, partition_ranges, and between_partition_ranges - Add `vectorbt/signals/dispatch.py` with backend-neutral dispatch wrappers that auto-select Rust or Numba - Wire signals accessor methods (clean, pos_rank, partition_pos_rank, first, nth, from_nth, nth_index, norm_avg_index, between_ranges, partition_ranges, between_partition_ranges) to dispatch layer with `backend=` parameter - Replace `as_slice()` with `Cow`-based `array1_as_slice_cow()` across all Rust 1D entry points to support strided/non-contiguous arrays - Remove contiguous-1D check from `_backend.array_compatible_with_rust` - Namespace benchmark function names (`generic.*`, `indicators.*`) and add per-config and overall statistics to the benchmark matrix
- Add Rust kernels in `rust/src/signals.rs` for generate_rand, generate_rand_by_prob, generate_rand_ex, generate_rand_ex_by_prob, generate_stop_ex, and generate_ohlc_stop_ex - Extend `vectorbt/signals/dispatch.py` with wrappers covering random generation, stop generation, and callback-based generate/generate_ex/ generate_enex (Numba-only for callbacks, uniform API) - Make signals accessor backend-agnostic: rename `choice_func_nb` to `choice_func`, route through dispatch, and add `backend=` parameter - Rename `cov` extra to `test` in `pyproject.toml`, bundle `vectorbt-rust`, and update CI workflow to install it - Extend benchmarks and tests to cover the new Rust signal kernels
Switch RAND, RANDX, RANDNX, RPROB, RPROBX, RPROBCX, RPROBNX, STX, STCX, OHLCSTX, and OHLCSTCX from `from_choice_func` to `from_apply_func` with new dispatch wrappers, so factory-built generators honor `backend=` and can run on the Rust engine. Thread `backend` through `SignalFactory` apply funcs, add `pass_seed` to the indicator factory, and add `resolve_random_backend` / `seed_for_rust` plus broadcast helpers in `_backend` to share flex-2d broadcasting between accessors and dispatch.
Add Rust implementations for core records operations (col_range, col_map, sorting checks, expansion, value counts, top/bottom-N masks) and wire them through dispatch wrappers in Records, ColumnMapper, and MappedArray. Also fix circular import in _settings.py by using string-based class references for CacheCondition whitelist entries.
Remove hardcoded byte-offset requirement for the `col` field — now dynamically resolved from the dtype, so record arrays with extra leading fields or reordered layouts work correctly. Add `normalize_col` helper for single-column edge cases and use bulk `copy_from_slice`/`fill` in col_map_select. Update benchmarks to include records functions.
…ispatch Port core portfolio simulation, order execution, cash/asset flow, performance metrics, and trade record functions to Rust with backend-neutral dispatch wrappers that auto-select the optimal engine.
Add `backend` parameter to public-facing methods in generic, indicators, portfolio, records, and mapped_array modules so users can select the execution backend (e.g., Rust) at call sites.
Rename _backend module to _engine, update all references in dispatch modules, accessors, tests, benchmarks, and settings to use the "engine" terminology consistently.
Store engine preference on accessor, records, mapped array, portfolio, and column mapper instances so dispatch calls inherit it automatically. Per-call engine parameter still takes precedence when provided.
Refactor Rust portfolio simulation with proper error types, NaN defaults for NO_ORDER fields, and OrderResult conversion for cross-engine compat. Route random-dependent dispatch functions (call seq, signals) through resolve_random_engine to fall back to Numba when needed. Add portfolio engine tests.
Add fast path for non-free cash flow, optimize assets allocation, extract col_start_idxs_usize helper. Add benchmark cases for grouped operations, sim-order functions, trade streaks, and positions. Update benchmark results.
Add specialized simulate_from_orders_non_shared_inner for the common case without cash sharing or auto call seq, avoiding group iteration overhead. Use Cow for call_seq to skip copies, lazily allocate temp_order_value and call_seq_mut only when needed.
Implement simulate_from_signals in Rust with full stop-loss/take-profit support, signal conflict resolution, and accumulation modes. Add corresponding Numba function, dispatch wrapper, and engine parameter threading through Portfolio.from_signals. Expand engine tests and benchmarks.
…ispatch Introduce RustConversion tracking and prepare_array_for_rust to auto-cast safely-compatible arrays before Rust calls instead of rejecting them. Add flex_array_compatible_with_rust for broadcast validation, exact_array_compatible_with_rust for mutable args, and array_shape_compatible_with_rust. Apply prepare_array_for_rust across all dispatch modules. Remove unused imports.
Reorder dispatch functions, Rust PyO3 exports, and module registrations to match canonical numba function order. Add ordering consistency tests. Expand engine dispatch to OHLCV stats, returns daily/annual, and rolling return functions. Rename order_nb_rs to order_rs.
Remove ordering/inventory tests superseded by dispatch coverage, consolidate individual portfolio parity tests into a single parametrized test, shorten verbose test names, and harden ray.init in test setup. Fix bare f-strings, remove unused imports/variables, prefix unused params with underscore, and move lazy import to module level.
Add single-column delegation in Rust shift/diff/pct_change and standard-layout fast path for find_ranges. Overhaul benchmark harness with layout modes (contiguous/view/copy-included), suite filtering (core/extended), per-case tags, and separate Numba/Rust result tables. Add bench_matrix test.
Tests covered by the overhauled benchmark harness introduced in d354dd0.
Replace x==x NaN idiom with idiomatic is_nan() in Rust EWM functions, extract uninit_f64_vec helper to DRY unsafe allocation, precompute benchmark_value factors, add flatten_forder fast path, rewrite true_range_nb as single-pass, and replace O(n²) between_two_ranges_nb with O(n) two-pointer scan. Update benchmark results.
Update the getting-started page to describe the Rust acceleration layer alongside Numba, add README files for the rust/ and benchmarks/ directories, and expose the _engine module in pdoc.
…thon-side broadcast Add a FlexArray enum in Rust that handles scalar, 1D, and 2D inputs with on-the-fly broadcasting, so the Python dispatch layer can pass compact arrays directly instead of materializing full 2D copies via flex_broadcast_to_shape. Both simulate_from_orders and simulate_from_signals now accept PyReadonlyArrayDyn inputs and resolve element access through FlexArray::get(row, col).
…e broadcast Adopt FlexArray in Rust signals (prob, stop, trailing, OHLC stop params) and labels (pos_th, neg_th) so these modules also skip Python-side flex_broadcast_to_shape. Add FlexArray::as_full_2d() fast-path for the common pre-broadcast case. Remove the now-unused flex_broadcast_to_shape helper. Split benchmark cases into _full/_flex variants and regenerate reports.
Add rust-engine test matrix to CI, vectorbt-rust sdist/wheel build jobs and PyPI publish step, Docker rust and full-rust image variants, and dry-run support for the release workflow. Restructure optional dependencies so full extends full-no-talib and add rust and all extras. Track Cargo.lock. Update README, installation, and feature docs to describe Rust engine installation and capabilities.
Move Rust engine to a prominent position in the features list, tighten feature descriptions, and clarify the [full] extra.
Add rustfmt.toml (max_width = 120) and [tool.black] config, then reformat all Rust sources and fix one overlong Python line.
Add `test-rust` optional dependency group that extends `test` with maturin. Update CI workflow to install `test-rust` extras and run all tests under `tests/` instead of only `test_engine.py`. Add conftest that auto-skips Rust engine tests when the extension is unavailable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR adds an optional Rust execution engine to vectorbt and wires it into the Python API through engine-aware dispatch. The Rust backend can be selected globally, per instance, or per call while preserving the existing Numba/Python behavior as the default compatibility path.
What Changed
Testing
Review Notes
This is a broad feature branch with both functional changes and formatting updates. The main review areas are engine dispatch semantics, Rust/Python parity, packaging workflow changes, and portfolio simulation behavior.