[Epic] Restore CI, raise the SE baseline, and extend modeling (bioreactors, crystallizers, scheduling)

[bernalde]

Summary

This is the tracking issue for a coordinated effort to (1) restore CI and a working test suite, (2) raise the software-engineering baseline of the codebase, and (3) extend PharmaPy with new modeling capabilities for bioreactors, crystallizers, and scheduling.

Context: a CI/CD pipeline and test framework were added in PR #106 (CryPTSys) and reverted in PR #107 (commit 9d3ab8c). The revert was driven largely by the PR bundling the CI work together with a repo-wide black reformatting pass and test-path changes, which made it hard to review and noisy to merge. The plan below re-introduces the valuable pieces incrementally and with isolated concerns, so each change is reviewable on its own.

Current Recommended Sequence (June 30, 2026)

Reverted PR #106 fork re-introduction

Review and merge the fork PRs in this order. The branches are stacked, so each later PR depends on the previous base branch until the stack is merged.

1. Adopt pytest discovery for existing tests #90 — test/pytest-discovery-fixtures, closes Adopt pytest for the existing test suite (discovery + fixtures) #4. Restores pytest discovery, stable fixtures, test-data paths, and Assimulo markers. Local checks are recorded in the PR body. This PR itself has no Actions run because the workflow does not exist until the next PR.
2. Add minimal core GitHub Actions workflow #91 — ci/minimal-core-tests, closes Add a minimal, assimulo-free GitHub Actions CI workflow #5, base test/pytest-discovery-fixtures. Adds the minimal core GitHub Actions workflow. Actions proof: Core tests passed on both push and pull_request runs.
3. Modernize packaging metadata #92 — packaging/pyproject-metadata, closes Modernize packaging and project metadata (pyproject.toml, drop egg-info) #8, base ci/minimal-core-tests. Adds pyproject.toml/MANIFEST.in, simplifies setup.py, and removes tracked egg-info metadata. Actions proof: Core tests passed on both push and pull_request runs.
4. Relax dependency bounds and enable Dependabot #93 — packaging/dependency-bounds, closes Relax dependency pins and enable Dependabot #9, base packaging/pyproject-metadata. Relaxes core dependency metadata, makes Assimulo optional, adds Dependabot, and documents bounds. Actions proof: Core tests passed on both push and pull_request runs.
5. Add optional Assimulo integration CI #94 — ci/assimulo-integration, closes Add an opt-in assimulo integration CI job (conda) #6, base packaging/dependency-bounds. Adds the conda-forge Assimulo integration job as continue-on-error: true. Actions proof: Core tests and Assimulo integration tests passed on both push and pull_request runs.
6. Add initial core helper smoke tests #95 — test/core-smoke-coverage, refs Expand test coverage across modules (unit + integration) #7, base ci/assimulo-integration. Adds the first small fast core helper tests. Actions proof: Core tests and Assimulo integration tests passed on both push and pull_request runs.

PR #91 is the first PR where GitHub Actions can execute, and it validates the stacked #4 + #5 changes together. Subsequent PRs all have non-empty Actions runs through gh pr checks. Keep the stack draft until reviewers are ready to merge it in order.

After the CI/packaging stack

1. Merge the stack above in order so the fork has test discovery, CI, packaging metadata, dependency hygiene, optional Assimulo CI, and an initial unit-test baseline.
2. Stabilize verified correctness bugs before new features: handle the P1/P2 queues in [Epic] Verified model and numerical correctness defects (#18-#66) #17 and [Epic] Verified runnability and correctness defects from June 2026 audit (#68-#89) #67 first, starting with Critical: [SimExec] Stream handoff routes nonlinear flowsheets to the next execution item #19, Critical: [Phases] VaporPhase mass-basis enthalpy weights latent heat by mole fraction #18, Critical: [Drying] Liquid saturation balance mixes molar drying rate with mass density #20, Critical: [Drying] Gas species balance adds molar source terms to mass-fraction states #21, Critical: [Distillation] DistillationColumn.solve_unit calls retrieve_results without required arguments #49, Critical: [Crystallizers] Energy balances crash in adiabatic/jacket modes and mix enthalpy bases #68, and Critical: [Evaporators] Adiabatic ContinuousEvaporator DAE residual uses unbound h_vap #69.
3. Clear documented-mode crashes and tooling breakages next: follow the P3 queues in [Epic] Verified model and numerical correctness defects (#18-#66) #17 and [Epic] Verified runnability and correctness defects from June 2026 audit (#68-#89) #67, especially High: [Containers] Mixer.balances_solids Cake branch leaves Outlet and distrib unset #88, High: [Evaporators] ContinuousEvaporator steady_state calls unit_model with missing arguments #85, High: [SolidLiquidSep] DisplacementWashing static solve indexes 2D output as time series #86, High: [Reactors] Coil heat transfer, sensitivity estimation, and steady-state PFR modes crash #70, High: [Kinetics] Custom rate-law setup and vectorized equilibrium reverse term crash #71, High: [SimExec] Raw-material and OPEX accounting use wrong arguments, basis, and phase checks #76, High: [ParamEstim/Calibration] IPOPT result assembly crashes and PCR predict recenters on new data #78, High: [ParamEstim] Fixed-parameter gradients and IPOPT residual weighting are broken #83, and High: [MetaModeler] Template generator emits invalid concatenate/hstack calls and np.acum_len #82.
4. Work through remaining high-impact numerical/model defects, then medium and low cleanup, using the ordering recorded in [Epic] Verified model and numerical correctness defects (#18-#66) #17 and [Epic] Verified runnability and correctness defects from June 2026 audit (#68-#89) #67.
5. Resume broader software-engineering work after the bug backlog is under control: Decouple the ODE/DAE solver behind an interface (assimulo optional, scipy fallback) #10, Code hygiene: logging instead of print, bare excepts, hardcoded paths, dead code #11, Introduce a shared UnitOperation base class and constants module #12, and Type hints, docstring standardization, and auto-generated API docs #13.
6. Start new modeling capabilities last: [Scoping] Bioreactor module (Monod kinetics, fed-batch, chemostat) #14, [Scoping] Crystallizer model extensions (agglomeration, breakage, polymorphism, 2-D PBE) #15, and [Scoping] Scheduling layer (campaigns, equipment allocation, greedy + MILP) #16 should wait until the verified correctness backlog has an active test and CI baseline.

Workstreams

A. CI & testing (do first)

• Adopt pytest: conftest.py, pytest.ini, fixtures, absolute test-data paths — make the existing unittest tests discoverable without reformatting code
• Add a minimal, fast GitHub Actions workflow (no assimulo; Ubuntu + one Python)
• Add an opt-in assimulo integration job (conda, continue-on-error)
• Expand per-module test coverage (reactors, crystallizers, separations, thermo, flowsheet)

B. Software-engineering health

• Modernize packaging (pyproject.toml, real metadata, drop committed egg-info)
• Relax over-tight dependency pins; enable Dependabot (see Relax dependency pins and enable Dependabot #9 / PR Relax dependency bounds and enable Dependabot #93)
• Decouple the ODE/DAE solver behind an interface (assimulo optional, scipy fallback)
• Logging instead of print, remove bare except:, fix hardcoded relative paths
• Shared UnitOperation base + constants module to cut duplication
• Type hints + docstring standardization + auto-generated API docs

C. New modeling capabilities

• Bioreactors module (Monod/inhibition kinetics; fed-batch & chemostat)
• Crystallizer extensions (agglomeration, breakage, polymorphism, 2-D PBE)
• Scheduling layer (recipes/campaigns/equipment; greedy + MILP via Pyomo)

Principles

1. One concern per PR. Never mix CI/test infrastructure with formatting or logic refactors.
2. No big-bang black pass. If formatting is desired, do it in a dedicated, clearly-labeled PR with a .git-blame-ignore-revs entry.
3. assimulo stays optional in CI. It is a hard-to-install Sundials/Fortran dependency; the core test job must run without it.
4. New modules subclass existing abstractions (_BaseReactor, _BaseCryst, SimulationExec) rather than duplicating them.

Individual issues are linked below as they are filed.

---

Filed issues

A. CI & testing

• Adopt pytest for the existing test suite (discovery + fixtures) #4 — Adopt pytest for the existing test suite
• Add a minimal, assimulo-free GitHub Actions CI workflow #5 — Minimal, assimulo-free GitHub Actions CI
• Add an opt-in assimulo integration CI job (conda) #6 — Opt-in assimulo integration CI job
• Expand test coverage across modules (unit + integration) #7 — Expand test coverage across modules

B. Software-engineering health

• Modernize packaging and project metadata (pyproject.toml, drop egg-info) #8 — Modernize packaging & metadata
• Relax dependency pins and enable Dependabot #9 — Relax dependency pins + Dependabot
• Decouple the ODE/DAE solver behind an interface (assimulo optional, scipy fallback) #10 — Decouple ODE/DAE solver behind an interface
• Code hygiene: logging instead of print, bare excepts, hardcoded paths, dead code #11 — Code hygiene (logging, bare excepts, paths, dead code)
• Introduce a shared UnitOperation base class and constants module #12 — Shared UnitOperation base + constants module
• Type hints, docstring standardization, and auto-generated API docs #13 — Type hints, docstrings, API docs

C. New modeling capabilities

• [Scoping] Bioreactor module (Monod kinetics, fed-batch, chemostat) #14 — Bioreactor module (scoping)
• [Scoping] Crystallizer model extensions (agglomeration, breakage, polymorphism, 2-D PBE) #15 — Crystallizer extensions (scoping)
• [Scoping] Scheduling layer (campaigns, equipment allocation, greedy + MILP) #16 — Scheduling layer (scoping)

---

Upstream contribution strategy (CryPTSys/PharmaPy)

These issues live on the SECQUOIA fork, but most of the work is intended to land upstream at CryPTSys/PharmaPy. The last attempt (PR #106) was reverted (PR #107) because it bundled CI + a repo-wide black reformat + test-path changes into one un-reviewable PR. So the upstream plan is many small, single-concern PRs, sequenced by risk.

Ground rules for every upstream PR

1. One concern per PR. Never mix infra/CI with formatting or logic refactors.
2. No big-bang black. If formatting is wanted, a separate, clearly-labeled PR + .git-blame-ignore-revs.
3. Core CI stays assimulo-free (see [Decouple the ODE/DAE solver behind an interface (assimulo optional, scipy fallback) #10] and the assimulo constraint).
4. Given the prior revert, give maintainers a heads-up issue/comment upstream before re-attempting CI, and keep the first PRs tiny to rebuild trust.

Wave 1 — low-risk, high-acceptance (open upstream first)

• Adopt pytest for the existing test suite (discovery + fixtures) #4 pytest adoption (test-only, no source touched)
• Modernize packaging and project metadata (pyproject.toml, drop egg-info) #8 packaging / drop committed egg-info (isolated)
• Relax dependency pins and enable Dependabot #9 dependency pins + Dependabot (config-only; closes upstream CryPTSys#94)
• Add a minimal, assimulo-free GitHub Actions CI workflow #5 minimal assimulo-free CI (only .github/ + README badge) — after a heads-up given Fix reactor mode crash paths #107

Wave 2 — incremental, after CI is green upstream

• Add an opt-in assimulo integration CI job (conda) #6 opt-in assimulo CI job
• Expand test coverage across modules (unit + integration) #7 coverage (small test-only PRs per module)
• Code hygiene: logging instead of print, bare excepts, hardcoded paths, dead code #11 hygiene (separate PRs: logging / excepts / paths / dead code — never with formatting)
• Type hints, docstring standardization, and auto-generated API docs #13 typing + docs (incremental)

Wave 3 — behavior-touching refactors (validate on fork first)

• Decouple the ODE/DAE solver behind an interface (assimulo optional, scipy fallback) #10 solver decoupling and Introduce a shared UnitOperation base class and constants module #12 shared base class. Prototype on the fork behind passing CI; upstream PRs must demonstrate identical results on the existing integration tests, landed in small steps.

Wave 4 — new features (propose before building)

• [Scoping] Bioreactor module (Monod kinetics, fed-batch, chemostat) #14 bioreactors, [Scoping] Crystallizer model extensions (agglomeration, breakage, polymorphism, 2-D PBE) #15 crystallizer extensions, [Scoping] Scheduling layer (campaigns, equipment allocation, greedy + MILP) #16 scheduling. Open an upstream proposal/discussion issue to align on scope and API (and, for [Scoping] Scheduling layer (campaigns, equipment allocation, greedy + MILP) #16, the new optional Pyomo dependency) before writing the module.