Integrate pdex2

[noamteyssier]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @noamteyssier, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces pdex2, a substantial refactor of the pdex library for parallel differential expression analysis in single-cell genomics. The primary goal was to streamline the API, enhance performance, and provide more flexible comparison modes. The changes consolidate previous functionalities into a single, powerful pdex() function, backed by Numba-accelerated statistical tests and Polars for data handling. This overhaul simplifies usage, improves computational efficiency, and expands the analytical capabilities of the library.

Highlights

• Major API Refactor: The core differential expression functionality has been completely rewritten and unified under a single pdex() function, replacing the previous parallel_differential_expression and pseudobulk_dex functions.
• New Comparison Modes: Introduced three distinct comparison modes: "ref" (group vs. reference), "all" (1-vs-rest), and "on_target" (group vs. reference for a specific gene).
• Performance Enhancements: Leveraged numba-mwu for Numba-accelerated Mann-Whitney U tests and Polars for efficient DataFrame operations, significantly improving performance and memory efficiency.
• Simplified Pseudobulk Logic: Consolidated pseudobulk calculations and statistical tests into a new _math.py module, supporting both arithmetic and geometric means with flexible log1p handling.
• Updated Python and Dependencies: The project now requires Python 3.11+ and has updated its dependencies to newer versions, including the addition of numba-mwu and ty, while removing adpbulk, pydeseq2, numpy, and pandas as direct dependencies in favor of polars.
• Comprehensive Documentation: Added a detailed CLAUDE.md file for AI guidance and significantly updated the README.md to reflect the new API, modes, parameters, and output schema.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• .python-version
  • Updated the required Python version from 3.10 to 3.13.
• CLAUDE.md
  • Added a new file providing comprehensive guidance for Claude Code, detailing project overview, commands, architecture, performance, output schema, public API, and dependencies.
• README.md
  • Completely rewritten to reflect the new pdex API, including new sections for Overview, Modes, Usage, Parameters, and Output, replacing previous sections on installation, summary, backed vs in-memory AnnData, and parallelization.
• pyproject.toml
  • Updated project version to 0.2.0 and Python requirement to >=3.11.
  • Revised dependencies, adding numba-mwu and ty, and updating versions for anndata, numba, polars, pyarrow, tqdm, while removing adpbulk, numpy, pandas, pydeseq2, and scipy as direct dependencies.
• ruff.toml
  • Removed the ruff configuration file.
• src/pdex/init.py
  • Rewrote the main __init__.py to introduce the new pdex entry point, incorporating logging, warning handling, and internal helper functions for group validation, reference identification, and matrix isolation.
  • Implemented core logic for _pdex_ref, _pdex_all, and _pdex_on_target modes, utilizing polars for data structures and numba_mwu for statistical tests.
• src/pdex/_math.py
  • Added a new module to centralize Numba-jitted mathematical operations for pseudobulk calculation, fold change, percent change, and Mann-Whitney U tests, supporting both dense and sparse matrices.
• src/pdex/_parallel.py
  • Removed the module containing previous parallelization helpers and Wilcoxon ranksum implementations.
• src/pdex/_pseudobulk.py
  • Removed the module containing the pseudobulk_dex function and its related DESeq2 integration.
• src/pdex/_single_cell.py
  • Removed the module containing the parallel_differential_expression function and its associated chunked and shared-memory implementations.
• src/pdex/_utils.py
  • Refactored utility functions, replacing guess_is_log with _detect_is_log1p heuristic and adding set_numba_threadpool for Numba thread management.
• tests/bench_expr.py
  • Removed benchmarking tests for the old experimental modes.
• tests/conftest.py
  • Added new pytest fixtures for generating synthetic AnnData objects, including sparse, log1p, and on-target configurations, to support the new testing framework.
• tests/test_integration.py
  • Removed integration tests for the previous parallel_differential_expression function.
• tests/test_internals.py
  • Added new tests for internal helper functions within pdex/__init__.py, such as group validation and reference identification.
• tests/test_math.py
  • Added new tests specifically for the mathematical functions implemented in src/pdex/_math.py, covering fold change, percent change, and geometric mean calculations.
• tests/test_parallel.py
  • Removed tests for the old _parallel.py module.
• tests/test_pbdex.py
  • Removed tests for the pseudobulk_dex function.
• tests/test_pdex.py
  • Completely rewritten to provide comprehensive integration tests for the new pdex function, covering all three modes (ref, all, on_target), sparse data handling, geometric mean options, and validation scenarios.
• tests/test_utils.py
  • Rewrote tests for utility functions, specifically focusing on set_numba_threadpool.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request represents a major refactoring of the pdex library, simplifying its API and core logic. The previous implementation with multiple backends and metrics has been replaced with a streamlined approach centered around a single pdex function with different modes (ref, all, on_target), leveraging numba-mwu for efficient Mann-Whitney U tests. The changes significantly improve code clarity and maintainability. My review focuses on the new implementation's correctness, efficiency, and documentation clarity. I've identified a minor inefficiency, an inconsistency in project configuration, and opportunities to clarify the documentation regarding FDR correction. Overall, this is an excellent and well-executed refactor.

[gemini-code-assist]

The .python-version file specifies Python 3.13, while this file allows >=3.11. For consistency and to prevent potential issues with environment setup tools like pyenv, it's recommended to align these. If the project is intended to be developed and tested primarily on 3.13, consider changing this to >=3.13. If Python 3.11+ is supported, consider changing .python-version to 3.11 to ensure development and testing against the minimum supported version.

[gemini-code-assist]

Calling the Numba-jitted fold_change and percent_change functions for single scalar values is inefficient as it involves creating temporary NumPy arrays in each loop iteration. Since the underlying calculations are simple, they can be performed directly on the scalar target_mean and ref_mean values. This will be more performant and readable.

Suggested change

	fc = float(fold_change(np.array([target_mean]), np.array([ref_mean]))[0])
	pc = float(percent_change(np.array([target_mean]), np.array([ref_mean]))[0])
	with np.errstate(divide="ignore", invalid="ignore"):
	fc = float(np.log2(target_mean / ref_mean))
	pc = float((target_mean - ref_mean) / ref_mean)

[gemini-code-assist]

The description for FDR correction is accurate for ref and all modes, but can be misleading for on_target mode. In on_target mode, FDR is corrected across groups, not per-group across genes (since there's only one gene per group). It would be clearer to specify this distinction.

Suggested change

	| `fdr` | FDR-corrected p-value (per-group, across genes) |
	| `fdr` | FDR-corrected p-value. For `ref` and `all` modes, this is applied per-group across genes. For `on_target` mode, it is applied across groups. |

[gemini-code-assist]

This statement about FDR correction is accurate for ref and all modes, but not for on_target mode, where correction happens across groups. To avoid confusion, it would be helpful to clarify how FDR is handled in each case.

Suggested change

	FDR is corrected within each group (across genes), not globally across all (group, gene) pairs.
	FDR is corrected within each group (across genes) for `"ref"` and `"all"` modes, not globally across all (group, gene) pairs. For `"on_target"` mode, FDR is corrected across all tested groups.