Enhance project structure, documentation, and dependency management - #87
Merged
Conversation
…47) - Updated .gitignore to exclude new directories and files related to Python probing extensions and added patterns for temporary files. - Modified .pre-commit-config.yaml to enable `cargo-clippy` and `cargo-fmt` hooks for Rust code linting and formatting. - Enhanced Cargo.toml and Cargo.lock to include new dependencies and features, improving the overall project structure. - Introduced a new clippy.toml file for shared Clippy settings to standardize linting across the project. - Updated documentation to reflect changes in the development setup and project structure.
* Remove .cargo/config.toml and update setup-build-env action to include clippy; streamline contributing documentation links * Refactor InjectionTrait implementation and remove GenericInjection struct - Removed the GenericInjection struct from injection_trait.rs to simplify the codebase. - Updated the InjectionTrait implementation in injection.rs to use a more concise lifetime annotation. - Improved safety comments regarding pointer conversions in the inject method for clarity. * Update Cargo.toml and .cargo/config.toml for improved build performance and debugging - Removed unused cargo-features from Cargo.toml. - Added optimized debug profile settings for faster compilation and smaller artifacts. - Introduced a new .cargo/config.toml file to specify custom Rust flags for specific targets, enhancing build configurations for both native and WASM environments. - Updated nccl-profiler source files to improve code safety and clarity, including adjustments to pointer handling and struct definitions. - Refactored Python extension features to utilize thread-local storage for better performance and safety in concurrent environments. * Update Rust toolchain from nightly to stable in GitHub workflows and setup action - Changed the Rust toolchain from nightly to stable in the setup-build-env action. - Updated pypi.yml and test.yml workflows to use the stable toolchain for consistency across builds. * Update contributing and installation documentation to clarify Rust stable requirements - Revised contributing.md and contributing.zh.md to specify the need for Rust stable and provided installation instructions. - Updated installation.md and installation.zh.md to reflect Rust stable as a prerequisite for building from source and included installation commands. - Enhanced clarity in documentation regarding environment setup for contributors. * Update .cargo/config.toml to comment out optional Rust flags for Linux target - Commented out the Rust flags for the x86_64-unknown-linux-gnu target to prevent potential CI issues, while retaining the option for local speedup during development. * Refactor syscall handling in current_thread_id function for Linux target - Simplified the return statement in the current_thread_id function by removing the explicit return keyword, enhancing code clarity and consistency across platforms. * Refactor CPU and GPU backend discovery for improved clarity and performance - Updated the CPU sampler implementation to remove unnecessary type casting, enhancing code readability. - Simplified the backend discovery logic for GPU by using option types to handle CUDA and Apple Silicon backends, improving maintainability and clarity. - Refactored the `host_sampler` function to streamline the return statements, ensuring consistency across different target platforms. * Refactor signal handling in stack_tracer.rs for improved clarity - Removed unnecessary thread ID parameter from the trace_thread_signal function, simplifying the logic for signal handling. - Updated conditional checks to streamline signal processing for the current thread, enhancing code readability and maintainability. - Cleaned up unused variables to improve overall code clarity. * Update Makefile and GitHub workflow for improved build output and test execution - Modified the Makefile to specify output directory for the wheel build, enhancing clarity in build artifacts. - Removed redundant Clippy linting step from the GitHub workflow to streamline the CI process. - Adjusted Rust test execution in the workflow to avoid unnecessary environment sourcing, improving efficiency. * Update Makefile and GitHub actions for enhanced build features and code clarity - Updated the Makefile to include new features for the Maturin build process, improving support for extension modules and kmsg. - Modified the GitHub actions workflow to cache Rust artifacts more effectively and streamline code style checks. - Refactored imports in Rust source files for improved readability and consistency across the codebase. * Enhance GitHub workflows with caching and build optimizations - Added a new input for enabling Rust caching in the setup-build-env action to improve build performance. - Implemented caching for web UI builds in multiple workflows to speed up the build process. - Updated the Python setup to cache pip dependencies, enhancing efficiency in test workflows. - Refactored the wheel bundle preparation steps to skip frontend cleaning when the build output already exists, streamlining the process. * Update GitHub workflow to improve code style checking command - Modified the code style checking step in the GitHub Actions workflow to use a subshell for the `cd` command, enhancing clarity and ensuring the command executes in the correct directory context. * Enhance Makefile and GitHub workflows for improved build and testing processes - Added a new target in the Makefile for creating a Python virtual environment, streamlining dependency management. - Updated the installation steps in the Makefile to ensure build dependencies are installed in the virtual environment. - Modified GitHub workflows to utilize the new virtual environment setup, improving consistency in the build process. - Adjusted caching strategies in workflows to optimize performance and reduce redundant installations. * Refactor CUDA backend handling for improved stability and clarity - Updated the CudaBackend implementation to avoid panics when libcuda is absent, enhancing robustness in CI and CPU-only environments. - Introduced a new probing mechanism to check for CUDA availability without causing fatal errors. - Added tests to ensure backend discovery is idempotent and does not abort when CUDA is unavailable. - Adjusted Makefile to refine Python wheel testing arguments and coverage reporting, improving test execution consistency. * Fix CString creation in CUDA backend to correctly dereference names - Updated the CString creation in the CUDA backend to dereference the name variable, ensuring proper handling of string literals. - This change enhances the robustness of the libcuda driver readiness check by preventing potential runtime errors. * Enhance probing package structure and validation checks - Updated the Makefile to include additional validation for bundled skills and web assets during installation, ensuring critical files are present. - Refactored the web_assets and skills paths modules to improve directory resolution and resource handling, enhancing the reliability of asset loading. - Improved the README documentation for diagnostic skills, clarifying the authoring process and installation instructions. - Added tests to verify the existence of bundled skills and web assets, ensuring proper functionality in both wheel and editable installations. * Update Makefile and GitHub workflows for enhanced dependency management and skill bundling - Added a new target in the Makefile for installing wheel test dependencies, improving testing capabilities. - Refactored paths in the Makefile to use 'bundled_skills' and 'bundled_web' for better organization of assets. - Updated GitHub workflows to utilize caching for Python virtual environments and web UI builds, optimizing build performance. - Introduced new skills and their corresponding YAML configurations for improved diagnostics and monitoring capabilities. * Update .gitignore to remove unnecessary probing directories - Removed the _skills and _web directories from the .gitignore file, allowing for better tracking of relevant files in the probing package. - This change enhances the visibility of important assets during development and testing. * Refactor global federation tests to use fixtures for cluster node checks - Replaced the `@pytest.mark.skipif` decorator with a new fixture `_require_cluster_nodes` to defer engine probe until fixtures run, improving test clarity and execution flow. - This change ensures that tests only skip when necessary, enhancing the robustness of the testing framework. * Update Makefile and GitHub workflows for improved coverage reporting - Refactored the `coverage-rust` target in the Makefile to streamline the command for running Rust coverage, enhancing clarity and consistency. - Updated the GitHub workflow to align with the changes in the Makefile, ensuring that the coverage reporting process is more efficient and easier to maintain. * Enhance probing engine readiness check in regression tests - Updated the `_wait_for_probing_engine` fixture to implement a more robust waiting mechanism, ensuring the in-process probing engine is ready before executing tests. This includes a 30-second timeout with error handling for better reliability. - Removed the `_table_exists` function and `_require_cluster_nodes` fixture from global federation tests, simplifying the test setup. - Adjusted imports in REPL test files to ensure necessary modules are only imported when required, improving test efficiency. * Refactor GitHub Actions for improved caching and environment setup - Updated the setup-build-env action to clarify caching behavior, focusing on environment caching without compile artifacts. - Removed unnecessary caching options for web UI and Rust build artifacts to streamline the workflow. - Adjusted workflow configurations in pypi.yml and test.yml to disable web UI caching and ensure consistent virtual environment caching settings.
…49) * Update dependencies, enhance documentation, and add new features for probing extensions - Added new dependencies including `probing-hccl-shim`, `probing-nccl-profiler`, and `serde_yaml` to `Cargo.lock` and `Cargo.toml`. - Updated the Makefile to include new targets for HCCL shim library and improved frontend build instructions. - Enhanced documentation to include a new section for the Federated Query Engine and updated existing design documents. - Refactored examples to replace deprecated scripts and improve clarity in tracing examples. - Improved the overall structure of the project by organizing new features and dependencies effectively. * Implement Drop trait for RealLib to ensure proper resource cleanup and remove shutdown function from public API. Update context ID info validation in msprof.rs for improved safety. * Enhance probing extensions with new libraries and update build configurations - Added `probing-hccl-profapi` and `probing-nccl-profiler-cdylib` as new packages in the workspace, including their dependencies in `Cargo.toml` and `Cargo.lock`. - Updated the Makefile to build the new libraries and adjusted targets for HCCL and NCCL profiling. - Refactored the `builtin-schema-docs` feature to include the new HCCL shim library. - Improved coverage reporting by excluding the new cdylib libraries from coverage checks in the Makefile and GitHub workflows. - Updated documentation and project structure to reflect the addition of new features and libraries. * Update SQL queries in documentation to use 'local_step' for consistency - Changed SQL queries in the quickstart and reference documentation to replace 'step' with 'local_step' for clarity and consistency across examples. - Updated related documentation in both English and Chinese to reflect these changes. - Adjusted descriptions in the concepts and SQL tables documentation to align with the new terminology. * Update documentation for SQL analytics and add new environment variable references - Revised SQL queries in the README and quickstart documentation to enhance clarity and consistency, particularly focusing on the use of 'local_step'. - Expanded the documentation to include new sections on environment variables, detailing their usage and configuration for Probing. - Updated both English and Chinese documentation to reflect these changes, ensuring accessibility for a wider audience. - Improved the overall structure and navigation of the documentation to facilitate easier access to key information.
#50) * Enhance probing features with new crash demos and improved error handling - Added two new Python scripts: `crash_demo.py` and `crash_torchrun_demo.py` for demonstrating crash handling in single and distributed environments. - Introduced a shell script `run_crash_torchrun.sh` to facilitate running the distributed crash demo. - Updated `Cargo.lock` to include new dependencies such as `log`, `serde`, `sha2`, and `uuid` for enhanced functionality. - Refactored mutex handling in `memtable_sql.rs` and `step.rs` to improve error recovery and thread safety. - Added new functions for crash reporting and handling in the Python extension, enhancing the robustness of the crash capture mechanism. * Enhance Makefile with formatting targets and update component imports - Added new `fmt` and `fmt-check` targets to the Makefile for Rust code formatting using `cargo fmt`. - Updated the help section in the Makefile to include the new formatting commands. - Reordered imports in `traces.rs` for consistency and clarity, moving `PageContainer` and `RefreshButton` to the correct position. - Restored the `span_timeline` module import in `mod.rs` to ensure proper component usage.
* Update documentation and add torchrun cluster heartbeat support - Added a new `torchrun-cluster.md` file detailing the hierarchical cluster heartbeat mechanism for multi-process `torchrun` jobs. - Updated existing documentation to reference the new heartbeat feature, including changes in `distributed.md`, `index.md`, and their Chinese counterparts. - Introduced environment variable references for configuring the cluster heartbeat in `env-vars.md`. - Added a demo script `cluster_multinode_demo.py` to showcase the new functionality. - Updated `Cargo.lock` to include the new `probing-store` dependency for enhanced functionality. * Update documentation to include Torchrun Cluster Heartbeat and improve navigation - Added a new entry for "Torchrun Cluster Heartbeat" in the plugins section of `mkdocs.yml`. - Updated the navigation in `mkdocs.yml` to include a link to the new `torchrun-cluster.md` design document. - Revised links in `quickstart.md` and `quickstart.zh.md` to point to the correct paths for SQL Analytics and Distributed documentation.
Use poison-safe lock_mutex in torchrun cluster paths, rewrite federation catalog prefixes via AST (with legacy fallback), update modularity docs for known boundary debt and torchrun vs Pulsing membership, soften README overhead claims, and track uv.lock while gitignoring generated bundled_web. Co-authored-by: Cursor <cursoragent@cursor.com>
* Harden cluster mutex handling and align docs with refactor reality. Use poison-safe lock_mutex in torchrun cluster paths, rewrite federation catalog prefixes via AST (with legacy fallback), update modularity docs for known boundary debt and torchrun vs Pulsing membership, soften README overhead claims, and track uv.lock while gitignoring generated bundled_web. Co-authored-by: Cursor <cursoragent@cursor.com> * Update dependencies and enhance error handling across modules - Added `pyo3` and updated `thiserror` to version `2.0.12` in `Cargo.lock` for improved Python integration and error management. - Refactored error handling in various modules to utilize `Context` for better error context propagation. - Updated documentation to reflect changes in modularity and signal handling, including the addition of macOS-specific signal delivery methods. - Improved overall code clarity and consistency by simplifying function signatures and enhancing comments across multiple files. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* Harden cluster mutex handling and align docs with refactor reality. Use poison-safe lock_mutex in torchrun cluster paths, rewrite federation catalog prefixes via AST (with legacy fallback), update modularity docs for known boundary debt and torchrun vs Pulsing membership, soften README overhead claims, and track uv.lock while gitignoring generated bundled_web. Co-authored-by: Cursor <cursoragent@cursor.com> * Enhance documentation and error handling for Probing - Added a new section on error handling and runtime practices in AGENTS.md, emphasizing clean error propagation and logging standards. - Updated README files to reflect the new focus on agent-native performance analysis and added details on using Probing with coding agents. - Improved navigation and structure in mkdocs.yml, including new entries for modularity, architecture, and CLI command tree. - Introduced WRITING.md to standardize documentation style and tone across the project. - Updated Cargo.toml to include new dependencies and features, enhancing overall functionality and integration capabilities. * fix lint errors --------- Co-authored-by: Cursor <cursoragent@cursor.com>
- Added the `probing.timing` package for stream-value-gated CUDA-event timing, exposing the `timing` decorator and `timing_context`, an `AcceleratorTimer`, backend detection, a per-device stream-value gate, and a ctypes CUDA-driver FFI bridge. - Added hardware-independent unit tests under `tests/unit/probing/timing/` using fake torch/gate stand-ins: backend detection and event helpers, the gate's CUDA guard and per-device cache, `AcceleratorTimer` ordering and gate-release-on-error, the `@timing`/`TimingContext` bookkeeping and contextvar isolation, and the FFI symbol resolver, constants, and singleton loader. - Added a CUDA-gated end-to-end regression test `tests/regression/timing/test_timing_gpu.py` exercising the stream-value-gated CUDA-event path, with a cuBLAS warm-up to avoid deadlocking behind the closed gate.
Add accelerator timing module with unit and GPU tests
#54) * Harden cluster mutex handling and align docs with refactor reality. Use poison-safe lock_mutex in torchrun cluster paths, rewrite federation catalog prefixes via AST (with legacy fallback), update modularity docs for known boundary debt and torchrun vs Pulsing membership, soften README overhead claims, and track uv.lock while gitignoring generated bundled_web. Co-authored-by: Cursor <cursoragent@cursor.com> * Update dependencies and enhance documentation for NCCL profiling and cluster management - Added new dependencies including `env_logger`, `log`, and `criterion` to improve logging and performance analysis capabilities. - Updated the default `NCCL_PROFILE_EVENT_MASK` to `94` for enhanced profiling granularity. - Revised documentation across multiple files to reflect changes in environment variables, SQL tables, and cluster query handling. - Improved error handling and metadata reporting in cluster queries to provide better insights into node performance and query integrity. - Enhanced modularity documentation to clarify the integration of new features and their usage in the context of Probing. This commit aims to improve the overall functionality and usability of the Probing framework, particularly in relation to NCCL profiling and cluster operations. * Remove NCCL_LOG_TRACE constant and associated debug function from NCCL profiler logging module to streamline logging functionality. This change simplifies the logging interface by eliminating unused trace logging capabilities. * Refactor memory snapshot handling in Linux to simplify RSS byte retrieval - Updated the `read_host_rss_bytes_signal_safe` function to directly return the result of `parse_proc_vmrss_signal_safe`, improving code clarity. - Simplified the file descriptor opening process in `parse_proc_vmrss_signal_safe` by using a byte string for the path, enhancing safety and readability. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
- Added support for bundled skills in `python/probing/bundled_skills/` and updated the symlink in `skills/` to point to this new location. - Enhanced `.gitignore` to exclude `*.egg-info/` directories. - Updated `Cargo.toml` and `Cargo.lock` to include the new `probing-skills` crate, which centralizes skill loading and execution. - Revised documentation in `AGENTS.md`, `api-reference.md`, and `extensibility.md` to reflect changes in skill discovery and execution paths. - Improved Makefile for better handling of Python virtual environments and build processes. - Removed outdated `skills/README.md` and `skills/catalog.yaml` files as part of the restructuring. This commit aims to streamline the skill management process and improve the overall clarity of the project's documentation.
#56) * Enhance Makefile and GitHub Actions for Python virtual environment management - Updated the Makefile to improve handling of Python virtual environments, including better detection and recreation logic for `.venv`. - Modified GitHub Actions to set the Python executable path in the environment after creating the virtual environment. - Adjusted `.gitignore` to include `.venv/` for better project cleanliness. - Updated documentation Makefile to use the system Python command for improved compatibility. These changes aim to streamline the development workflow and ensure consistent environment setup across different systems. * Enhance skills management and debug console initialization - Updated the `bundled_skills_dir` function to prioritize the source tree over wheel copies when running in a probing git checkout, ensuring consistency with discovery results. - Introduced a lazy initialization for the `DebugConsole` to prevent conflicts with the native engine during import time, allowing for a more efficient debug console setup. - Added a `get_debug_console` function to manage the creation and retrieval of the debug console instance, improving resource management. These changes aim to enhance the functionality and usability of the skills management and debugging processes within the Probing framework. * Update debug console initialization in Python REPL - Modified the import statement in `console.rs` to use `get_debug_console()` for initializing the debug console, enhancing the setup process. - Updated the `pythonext.py` handler to reflect the new import structure, ensuring compatibility with the revised debug console management. These changes aim to streamline the debug console initialization process and improve overall resource management within the Probing framework. * Update Makefile and enhance Rust core functionality - Modified the Makefile to disable the VM tracer during Python wheel tests when the `PROBING_VM_TRACER` environment variable is set to `0`, improving test reliability. - Introduced a new function `on_native_bridge_thread` in `runtime.rs` to check if the current thread is the dedicated native bridge worker. - Updated the Rust core library to utilize the new `on_native_bridge_thread` function, enhancing thread management during Rust/Python bridge operations. - Improved the `with_detached_native` function in `native_bridge.rs` to avoid unnecessary thread creation when already on the native bridge worker, preventing potential SIGABRT issues in CI environments. - Enhanced the Python test setup in `conftest.py` to disable the tracer when conflicts are detected, ensuring smoother test execution. These changes aim to improve the stability and performance of the Probing framework during testing and runtime operations.
…ng (#57) * Update dependencies and enhance Makefile for benchmarking and profiling - Added `tokio` as a dependency in `Cargo.lock` to support asynchronous operations. - Enhanced the Makefile to include new benchmarking scripts for instrumentation overhead, allowing for better performance analysis. - Updated help commands in the Makefile to include new benchmarking options. - Improved Python dependency installation checks to ensure `ruff` is available for linting. - Revised documentation to reflect changes in the profiling and benchmarking processes. These changes aim to improve the development workflow and provide better tools for performance monitoring within the Probing framework. * Refactor imports and improve code readability in overlay components - Consolidated import statements in `app_overlays.rs` and `mod.rs` for better organization. - Enhanced readability of the `sample_mode` extraction logic in `panel.rs` by restructuring the code for clarity. - Simplified the `window_start` function in `sql.rs` to improve code conciseness. These changes aim to streamline the codebase and enhance maintainability across the overlay components.
…#60) * Enhance soak testing capabilities and update documentation - Added new soak testing scripts (`run_soak.sh` and `soak_assert.py`) for long-running training with assertions, improving performance validation. - Updated the Makefile to include new soak commands and enhance help documentation. - Enhanced `.gitignore` to include coverage files and improve project cleanliness. - Introduced `SoakLimits` class in `imagenet_with_span.py` to manage training duration and step limits. - Updated `README.md` to document new soak testing features and usage instructions. These changes aim to improve the testing framework and provide better tools for performance monitoring within the Probing project. * Remove dead code audit documentation file - Deleted the `code-review-implementation-quality.md` file, which contained a comprehensive audit of implementation quality and dead code across the project. This file was deemed unnecessary as the information is no longer relevant or required for ongoing development. This change aims to streamline the documentation and remove outdated content from the repository. * Update pre-commit configuration, dependencies, and enhance Makefile for improved development workflow - Removed the `black` hook from the pre-commit configuration and replaced it with `ruff-format` for code formatting. - Updated several dependencies in `Cargo.lock`, including `anyhow`, `crossbeam-epoch`, and `rand`, to their latest versions for better performance and security. - Introduced a new `deny.toml` file for managing supply-chain policies and advisories. - Enhanced the Makefile to include a new target for formatting all configurations and improved help documentation for clarity. - Added a script `fmt-all-cfgs.sh` to format code across multiple compilation targets, ensuring consistent styling across platforms. These changes aim to streamline the development process, improve code quality, and ensure better dependency management. * Add uv.lock file for dependency management and update setup-build-env action - Introduced a new `uv.lock` file to manage Python package dependencies, specifying versions and sources for packages like `appnope`, `asttokens`, and `black`. - Updated the GitHub Actions setup-build-env action to replace spaces with hyphens in the `uv-sync-groups` input, improving output formatting. These changes aim to enhance dependency management and streamline the build environment setup process. * Update uv.lock verification in GitHub Actions for improved error handling - Enhanced the verification step for the `uv.lock` file in both the setup-build-env action and the supply-chain workflow. The updated checks now ensure that the `uv.lock` file is present before proceeding, providing clearer error messages if it is missing. - This change aims to improve the reliability of dependency management by ensuring that the necessary lock file is available during the build process. * Add sync-uv-groups target to Makefile and update GitHub Actions for improved dependency synchronization - Introduced a new `sync-uv-groups` target in the Makefile to facilitate synchronization of locked Python dependencies based on specified groups. - Updated the GitHub Actions workflow to utilize the new Makefile target for syncing dependencies, enhancing error handling and ensuring the presence of the `uv.lock` file. - Improved caching strategy in the setup-build-env action by updating the cache key format for better versioning and architecture support. These changes aim to streamline the dependency management process and improve the reliability of the build environment. * Refactor error messages in process.rs for improved clarity - Updated error messages in the `Process` implementation to use formatted string syntax for better readability and consistency. This change enhances the clarity of error reporting when no processes or multiple processes are found matching the command line pattern. These modifications aim to improve the user experience by providing clearer feedback during process injection operations. * Enhance Makefile for Python linting and formatting integration - Updated the `fmt` target in the Makefile to include `ruff` for Python code formatting, ensuring it passes lint checks. - Added a new `FMT_PYTHON` variable to streamline the linting and formatting process for Python files. - Enhanced the `fmt-check` target to verify Python code formatting using `ruff`, improving code quality assurance. These changes aim to improve the development workflow by integrating Python linting and formatting into the existing Makefile structure. * Enhance GitHub Actions workflows for improved testing and wheel preparation - Updated the `pypi.yml` and `test.yml` workflows to streamline the wheel preparation process by ensuring the bundled web assets are correctly copied and organized. - Introduced a new `windows-test` job in `test.yml` to run Rust and Python tests on Windows, enhancing cross-platform testing capabilities. - Improved error handling and asset management during the wheel build process, ensuring necessary files are present and correctly structured. These changes aim to enhance the reliability and efficiency of the CI/CD pipeline, improving the overall development workflow. * Update pre-commit configuration and Makefile for improved Python linting and formatting - Upgraded `ruff` version in the pre-commit configuration to `v0.11.11` for enhanced linting capabilities. - Refined the Makefile to streamline Python linting and formatting processes, consolidating commands for better usability. - Updated documentation to reflect changes in the Python linting and formatting workflow. These modifications aim to enhance code quality and simplify the development process for Python projects. * Remove pre-commit configuration and streamline Makefile setup - Deleted the `.pre-commit-config.yaml` file to simplify the project setup and remove unnecessary dependencies. - Updated the Makefile to remove the `setup` target related to pre-commit installation, focusing on essential development dependencies. - Adjusted documentation to reflect the changes in the development workflow, ensuring clarity on the current setup process. These modifications aim to enhance the development experience by reducing complexity and focusing on core tools.
#65) * Update project structure, enhance documentation, and improve dependency management - Updated `.gitignore` to include additional directories for cleaner project management. - Expanded `AGENTS.md` with new coding principles and guidelines to improve code quality and maintainability. - Modified `Cargo.toml` and `Cargo.lock` to include the `probing-skills` crate, centralizing skill loading and execution. - Enhanced `Makefile` for better handling of Python virtual environments and streamlined build processes. - Added new documentation files for overhead measurement and invariants, ensuring clarity on performance metrics. - Revised existing documentation to reflect changes in skill discovery, execution paths, and overhead measurement. These changes aim to improve the overall structure and usability of the Probing project, enhancing both developer experience and documentation clarity. * Refactor import statements in inject.rs for improved clarity - Removed unused `Context` import from `anyhow`, streamlining the code and enhancing readability in the `inject.rs` file. This change aims to simplify the codebase and improve maintainability. * Update `.gitignore` and fix thread ID retrieval in `span.rs` - Removed the `python/probing/bundled_web/` entry from `.gitignore` for better project management. - Updated the thread ID retrieval in `span.rs` to use the fully qualified path for `std::thread::current().id()`, enhancing code clarity and consistency. These changes aim to improve project cleanliness and code readability. * Enhance probing wheel verification and update Makefile for improved build process - Introduced a new script `verify_wheel_contents.py` to ensure that the probing wheel contains all required assets and modules, enhancing the reliability of the build process. - Updated the Makefile to include a new target for verifying wheel contents before installation, ensuring that necessary files are present and correctly structured. - Refactored the logic in `conftest.py` to check for the completeness of the installed probing package, preventing conflicts with site-packages during testing. - Modified `pyproject.toml` to include additional Python files in the package distribution, improving the completeness of the wheel. These changes aim to improve the integrity of the wheel installation process and enhance the overall development workflow. * Refactor probing module checks and update Makefile for installation process - Renamed the function in `conftest.py` to `_missing_probing_modules` for clarity, returning a list of missing modules instead of a boolean value. - Introduced a new error message format in `conftest.py` to provide clearer feedback on missing modules during installation. - Updated the Makefile to ensure the `PROBING` environment variable is set to `0` during wheel installation, improving the installation process. - Cleaned up `.gitignore` by removing unnecessary entries related to probing libraries. These changes aim to enhance the reliability of the probing module checks and streamline the installation process.
#66) * Enhance documentation and add new examples for Megatron-Core and vLLM integration - Updated `env-vars.md` to include new environment variables for `PROBING_TABLE_DEFAULT_MB` and vLLM autostart configurations. - Introduced `megatron_mcore_train_loop.py` for a real Megatron-Core training loop with probing support. - Added `run_megatron_soak.sh` and `run_vllm_soak.sh` scripts for soak testing with Megatron-Core and vLLM, respectively. - Enhanced `README.md` to document new soak testing features and usage instructions. - Updated `vllm_offline_soak.py` to support offline inference with probing and improved observability hints. These changes aim to improve the usability and clarity of the Probing project, particularly for users integrating with Megatron-Core and vLLM. * Refactor import hook registration and enhance testing setup - Introduced a `_default_register` function to encapsulate the module registration logic, improving code organization and readability. - Added a `reset_register` function to restore the import hook state, facilitating cleaner test setups and ensuring isolation between tests. - Updated test fixtures to utilize the new `reset_register` function, enhancing the reliability of tests by resetting the import hook state before and after each test. These changes aim to improve the maintainability of the import hook system and ensure consistent test behavior.
* Update documentation and add Torch Profiler SQL support - Added new entries for "Torch Profiler SQL" in the English and Chinese documentation, enhancing the navigation and vocabulary sections. - Introduced a new design document for "Torch Profiler SQL," detailing its functionality and integration with existing profiling tools. - Updated the `mkdocs.yml` configuration to include the new documentation links and ensure proper navigation. - Enhanced the `profiling.md` files to reference the new SQL capabilities, providing users with clearer guidance on using the profiler effectively. These changes aim to improve the clarity and accessibility of the documentation related to the Torch Profiler SQL feature, facilitating better user understanding and adoption. * Refactor integration tests for static catalog documentation - Updated the `test_describe_builtin_hccl_via_query_subprocess` to validate catalog rows instead of relying on the `DESCRIBE` command, ensuring tests are more robust against CI subprocess issues. - Modified `test_describe_json_shape_subprocess` to check for `description` and `table_name` columns, enhancing the validation of catalog table documentation. - Removed assertions related to `comment` and `table_comment` to align with the new testing approach. These changes aim to improve the reliability and clarity of the integration tests for the static catalog documentation.
- Added 'frontend/' to .gitignore for improved project cleanliness. - Removed 'signal-hook-registry' from Cargo.lock and added 'ureq' to dependencies. - Enhanced profiling documentation in both English and Chinese to clarify stack capture mechanisms and improve user guidance on using the TorchProfiler. - Introduced new functions in `imagenet_with_span.py` for distributed training argument configuration and initialization, improving the setup for distributed demos. These changes aim to streamline project management and enhance the usability of profiling features.
) * Refactor profiling documentation and update stack capture features - Enhanced the profiling documentation in both English and Chinese, providing clearer guidance on stack capture mechanisms and the TorchProfiler. - Introduced a new directory structure under `probing/extensions/python/src/features/` to better organize stack capture and profiling modules. - Updated the `setup.rs` and `pprof.rs` files to reflect changes in the stack capture pipeline, ensuring proper integration with the new structure. - Removed deprecated modules and functions related to stack tracing, streamlining the codebase and improving maintainability. These changes aim to improve the clarity and usability of the profiling features, facilitating better user understanding and adoption. * update tracer * Refactor stacktrace capture and logging for improved readability - Reformatted code in `capture.rs` to enhance clarity by aligning tuple returns and simplifying function signatures. - Updated logging in `dynamic.rs` to streamline debug messages for concurrent stack captures. - Adjusted formatting in `pprof.rs` to improve readability of flag manipulations. These changes aim to enhance code maintainability and improve the overall readability of the stacktrace features. * Enhance safety documentation for stacktrace capture functions - Added safety documentation comments to `fill_raw_snapshot_with` and `capture_raw_snapshot` functions, clarifying the requirements for `uctx` and `out` parameters. - Improved code clarity by ensuring that safety considerations are explicitly stated, aiding developers in understanding the usage constraints of these functions. These changes aim to enhance the safety and usability of the stacktrace capture features.
* feat: add Probing-RL online observability for agentic RL training [WIP] * support RL-slime perference analysis
…73) * Update Makefile and documentation for example paths and soak scripts - Updated the Makefile to change the benchmark script path to `examples/overhead/bench_instrumentation.py` and adjusted soak script paths to `examples/imagenet/run_soak.sh`. - Modified soak-related commands in the Makefile to reflect the new script locations. - Enhanced documentation to clarify the new paths for example scripts and their usage in soak testing. These changes aim to improve the organization of example scripts and ensure accurate references in the documentation. * Refactor probing repository root detection and enhance CLI script handling - Updated the `probing_repo_root` function to improve the detection of the probing repository by checking for the presence of `pyproject.toml` and the `python/probing` directory. - Modified the main script to utilize the new `probing_repo_root` function for determining the root path when executing the `pretrain_gpt` command, ensuring better compatibility with different project layouts. - Added a new test for the `_is_probing_repo` function to validate its behavior under various directory structures, enhancing test coverage and reliability. These changes aim to streamline the probing repository detection process and improve the overall robustness of the CLI scripts.
- Updated the extensibility documentation to clarify the handling of column types and schema stability in Python, ensuring users understand the implications of dtype variations. - Improved the profiling documentation to delineate the independent paths of TorchProbe and Torch Profiler, emphasizing their distinct roles and operational characteristics. - Refactored the fanout statistics handling in the federation module, introducing a new `FanoutStatsHandle` for better concurrency management and encapsulation of fanout statistics. - Streamlined the federated scan execution process to utilize the new fanout statistics handling, enhancing the clarity and maintainability of the code. These changes aim to improve user understanding of profiling tools and enhance the robustness of the federation features.
* Fix hccl-shim deadlock and expand Msprof passthrough exports. Replace the reentrant Mutex around real libprofapi with OnceCell so HCCL/MSProf callbacks cannot hang, and forward the additional Msprof/prof* symbols Ascend needs when the shim is first on LD_LIBRARY_PATH. Co-authored-by: Cursor <cursoragent@cursor.com> * Apply rustfmt and fix clippy lints in hccl-shim and aarch64 inject. - rustfmt hccl-shim forward.rs / lib.rs (wrap long fn-pointer types). - hccl-shim names.rs: use ptr::cast to avoid unnecessary_cast on aarch64. - cli aarch64 inject: from_mut for ref_as_ptr, add digit separators to hex literals. - aarch64 inject regression test: wait() the spawned child (zombie_processes). --------- Co-authored-by: Vincent <zhangwensheng@pjlab.org.cn> Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve signal capture reliability * Add payload length validation for blob headers and update capture functions
* Refactor Next shell and diagnostic contracts * Fix report module item ordering * Align hierarchical fan-out regression counts * Fix Web UI Clippy warnings * Keep local auth test client connected
* Add end-to-end NCCL profiling support Make distributed attribution reliable and preserve Megatron topology metadata, with validated overhead workflows for production diagnosis. Co-authored-by: Cursor <cursoragent@cursor.com> * Exclude local NCCL overhead experiment notes Keep environment-specific benchmark results outside the upstream PR while retaining the local document. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Rust formatting in cluster query errors Keep the new response decoding path compatible with the repository's rustfmt CI check. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix NCCL role resnapshot Clippy failure Use the standard integer multiple check so Linux wheel CI can enforce warnings as errors. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Vincent <zhangwensheng@pjlab.org.cn> Co-authored-by: Cursor <cursoragent@cursor.com>
reiase
force-pushed
the
codex/harden-probing-runtime
branch
from
August 5, 2026 14:29
dbe58c4 to
f734107
Compare
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.