Skip to content

chore(release): promote develop → main - #360

Merged
ywatanabe1989 merged 19 commits into
mainfrom
develop
Sep 1, 2026
Merged

chore(release): promote develop → main#360
ywatanabe1989 merged 19 commits into
mainfrom
develop

Conversation

@ywatanabe1989

Copy link
Copy Markdown
Collaborator

What this is

A develop → main promotion PR, opened by scitex-dev as part of a fleet-wide sweep. Your default branch is behind your own develop, which means main no longer reflects what has been released.

You decide whether to merge it. I do not merge other packages' default branches — opening a PR is reversible and CI-gated; merging is neither. If this is wrong for your repo, close it and nothing is lost.

Why it was opened

Measured across the fleet: 22 repos had main behind their newest tag. That gap is not cosmetic — it means:

  1. Anyone reading your default branch sees stale code.
  2. A published release on PyPI can have no corresponding commit on main.
  3. The next promotion accumulates, so the diff only gets harder to review.

On the "behind" count

If the compare view says main is also behind develop by a few commits, that is expected and not divergence: merge-based promotion makes main accumulate merge commits that develop never sees. It is the normal steady state for this workflow.

Authorisation

Operator ruling, 2026-08-04: PRs against repos scitex-dev does not own are explicitly authorized. This sweep was held for six days waiting on exactly that word, and it applies here.

Tracked on card default-branch-behind-published-release-promotion-gap-20260729.

ywatanabe1989 and others added 15 commits June 22, 2026 12:19
Replace the GitHub-hosted release workflow with the proven self-hosted
SIF pipeline (test->build->publish->release inside the reused scitex-ci
ci-cpu.sif via apptainer exec), matching the fleet canary
(openalex-local). The bare Spartan compute node has no Python, so
actions/setup-python and the Docker-based pypa/gh-action-pypi-publish
cannot run there; publish now does MANUAL OIDC trusted publishing inside
the SIF (GitHub OIDC JWT -> PyPI mint-token -> twine upload). Trust
config on PyPI is unchanged.

- KEEP trust-bound filename pypi-publish-and-github-release-on-tag.yml
  (PyPI trusted publisher is bound to the workflow filename).
- Add the 4 .github/ci/{exec,run,build,publish}-in-sif.sh helpers.
- Strip the auto sync-main job (develop->main is a deliberate PR).
- runs-on from Actions Variable CI_RUNS_ON; SIF + apptainer paths from
  SCITEX_CI_SIF / SCITEX_CI_APPTAINER (same as pytest-matrix).
- --cov=src/scitex; environment url https://pypi.org/p/scitex;
  unique-per-run TMPDIR; matplotlib Agg + warm font cache guarded.
- fail-loud: a missing SIF/shim/interpreter is a HARD error.
…d (pinned-in-core) (#340)

Complete the mid-release 2.30.2 reconcile to TRUE current-latest PyPI
versions, pinned in CORE [project.dependencies] (not extras-only) so
`pip install scitex` and `uv pip install scitex[all]` resolve the same
stable set.

Leaf bumps (current -> latest):
- figrecipe       0.28.20 -> 0.29.5
- scitex-audio    0.2.13  -> 0.3.0
- scitex-cv       0.1.5   -> 0.2.0
- scitex-dev      0.18.1  -> 0.20.1
- scitex-io       0.3.2   -> 0.3.3
- scitex-writer   2.17.5  -> 2.22.0

- Add scitex-pd==0.1.6 to core deps AND the [pd] extra (which flows into
  [all]) so `scitex.pd` works out of the box.
- Bump umbrella version 2.30.2 -> 2.30.3 (2.30.2 already published).

Conflict-verified: `uv pip compile` of the full resulting pin set exits 0
(732-line lock, no conflicts).
)

The self-hosted SIF release pipeline (run-in-sif.sh) drifted out of parity with
the hardened ubuntu workflow, breaking the v2.30.3 release at the 3.13 test gate:

- It installed only `.[all,dev]`, but the umbrella-aliased PEER standalones
  (scitex-gen, scitex-dsp, ...) are NOT in [all]. So scitex_gen was never
  refreshed and the test used the SIF's baked, stale copy (missing `to_even`),
  failing tests/integration/test_cross_package_imports.py. Reinstall the peers
  FRESH into the same --target so they shadow the baked copies on PYTHONPATH.
- It ran bare `pytest tests/` with no quarantine. Restore the ignores the
  hardened workflow uses: tests/e2e + the cross-package gate (segfault-prone),
  and tests/examples/test_09_dev.py (hard-requires scitex_hub, which is
  umbrella_skip by design and never in the [all] SIF).
figrecipe 0.29.5->0.29.9, scitex-writer 2.22.0->2.23.0, scitex-clew 0.2.17->0.2.18
(PS-170 freshness; strict pin audit green). figrecipe 0.29.9 lets downstream
research projects pin figrecipe==0.29.9 consistently alongside scitex==2.30.4.
…eterministic (#343)

The v2.30.4 release SIF flaked on test_bundle_submodule_is_a_package:
`import scitex.io as sio; sio.bundle` asserts attribute access WITHOUT
importing the submodule, so it only passes when a sibling test imported
scitex.io.bundle first — nondeterministic under pytest-xdist worker
distribution (reproduces back to 2.30.0; not a regression, not the pin bump).
Product is fine: `import scitex.io.bundle` is the supported path and works.
Import it explicitly so the assertions are deterministic on every worker.
scitex-writer 2.23.0->2.24.1 (post-compile fail-loud gate, incident fix),
scitex-dev 0.20.1->0.21.0 (compliance-linter trio), scitex-clew 0.2.18->0.2.22.
Strict pin audit green.
…for SIF rebuild (#346)

* chore(umbrella): scitex 2.30.6 — re-pin members to current-published for SIF rebuild

* chore(umbrella): narrow 2.30.6 to minimal set (clew 0.16.0 / notification 0.2.9 / notebook 0.1.2)

Revert the broad fleet-latest bumps (writer/session/dev/container/dataset/
scholar/ssh) that broke the umbrella thin-wrapper MCP-parity test
(test_umbrella_exposes_writer_mcp_tools) and segfaulted pytest-matrix.
The SIF rebuild only needs the minimal alignment set; broad 'fleet on
latest' re-pins require per-member thin-wrapper regen and land as a
follow-up.
…gregator (#347)

* rescue: pre-stop autosave scitex-python@20260703T223658Z

* fix(mcp): bounded per-peer mount so one hung peer can't darken the aggregator

The umbrella `scitex serve` aggregator fronts ~33 packages' MCP tools by
importing each peer's `_mcp_server` and mounting its FastMCP at module-load
time. Previously each peer was resolved SEQUENTIALLY and SYNCHRONOUSLY. If a
single peer's import hangs at init (real case: scitex-todo store-wedge stalls
20s+), the whole `scitex._mcp` import never completes and EVERY peer's tools
go dark — the failure concentrates 33x vs the per-server split.

Fix: resolve each peer's FastMCP concurrently in a bounded DAEMON thread
(`_resolve_peers_bounded`). Threads start at once and are joined against one
shared deadline, so total load time is ~max(peer) bounded by the per-peer
budget — never the sum, and a hung thread can't block interpreter exit. On
timeout / exception the peer is SKIPPED with a warning ("... unavailable this
session") and the healthy peers still mount. Budget is configurable via
`SCITEX_MCP_PEER_TIMEOUT` (default 8s).

Mounting stays on the MAIN thread: only the peer IMPORT is slow/hang-prone;
`safe_mount` is fast and mutates shared parent-server state, so it is kept
single-threaded to avoid concurrent-mutation races.

Preserves all prior behavior: `_is_enabled` env gate, `_NAMESPACE_ALIASES`,
`_SKIP_CATEGORIES`, `register_peer_extras`, `register_umbrella_tools`, the
mounted-count log, and `SCITEX_MCP_USE_<NS>=0` gating. `register_all_tools`
gains injectable `iter_registry` / `peer_timeout` params (defaults unchanged)
so tests drive the real path without patching module globals.

Tests (tests/scitex/test_mcp_bounded_mount.py): real fixture peers written to
disk — a 30s-sleep hung peer, an ImportError peer, a sys.exit peer, fast
peers. Assert the bounded resolve returns within the budget, the hung peer is
skipped + warned, the fast peers still mount, and the env gate holds.

* fix(mcp): serialize bounded peer resolve — concurrent imports deadlocked CI collection

PR #347's pytest-matrix hung for 1h30m (job timeout, all 3 py versions) with
"test session starts" printed and then nothing — i.e. a hang during COLLECTION,
before any test ran, where pytest-timeout's per-test SIGALRM cannot fire.

Root cause: the first cut resolved every peer's `_mcp_server` import CONCURRENTLY
(one daemon thread per peer, ~33 fired at once). Importing a large, interdependent
package set from many threads simultaneously races the CPython import machinery and
can deadlock via cross-thread circular imports that never occur when modules are
imported one at a time. That deadlock strikes at `from scitex import _mcp` import
(register_all_tools runs at module load) — which every test module triggers at
collection. `develop` passes because its resolve is sequential; the concurrency
was the regression.

Fix: resolve peers SERIALLY. Each peer still imports in its own bounded daemon
thread (`_resolve_one_peer_bounded`, joined for at most `SCITEX_MCP_PEER_TIMEOUT`,
default 8s) so a hung/wedged import is abandoned + skipped and can never block
interpreter exit — but only ONE import is ever in flight, so imports stay as safe
as develop's sequential loop while gaining the per-peer hang bound. Wall cost is
the sum of the healthy peers' (fast) import times plus the timeout per wedged peer
— a one-time startup cost paid for reliability (the original "~max(peer)"
concurrency goal is knowingly traded away; correctness first).

Verified locally: real `from scitex import _mcp` completes in ~8s and leaves NO
lingering non-daemon threads (so the main-thread timeout signal can always fire);
`test_mcp_entrypoint.py` matches the develop baseline (no new failures, no hang).

Tests: added `_INFINITE_PEER` (import blocks on `threading.Event().wait()`, never
set — the real store-wedge shape, not a bounded sleep) with a fixture asserting the
bounded resolve returns within the budget, skips the wedged peer, and still mounts
the co-resident fast peer. All hang fixtures are written to tmp dirs and imported
ONLY inside the bounded code path at test time — never at module/collection time,
so pytest can never auto-collect or import them into a collection hang.
25 tests pass.
…ned MCP aggregator (#348)

Bundles the merged #347 bounded-per-peer mount hardening (aggregator no
longer darkened by one hung peer) with the clew 0.17.0 upgrade
(supersedes the 2.30.6 clew-0.16.0 stopgap). This is the version sac
floors (scitex>=2.30.7) for the .mcp.json single-aggregator swap.
… guard (#349)

* fix(mcp): skip orochi orchestrator in umbrella aggregator cold-start

The umbrella MCP aggregator serially resolves each ecosystem peer's
_mcp_server at import time. scitex-orochi is the single-instance
agent-communication ORCHESTRATOR — its mcp_server guards sys.exit when a
Telegram bot token / telegram agent-role is present, and it is not a
per-agent tool provider. Auto-mounting it is both semantically wrong and
a cold-start hazard (its heavy import blocks the serial resolution loop).

_iter_registry now delegates the skip decision to a new _mount_skip()
helper which prefers scitex-dev's is_mcp_mountable() SSoT and falls back
to the local archived/category/mcp_mountable-field checks plus a
_LOCAL_UNMOUNTABLE guard (scitex-orochi) so the orchestrator is skipped
even against an older scitex-dev that predates the SSoT helper.

Coordinated two-repo change with scitex-dev
_ecosystem._core.is_mcp_mountable / _MCP_UNMOUNTABLE.

Adds 4 focused tests to test_mcp_entrypoint.py.

* fix(mcp): also skip scitex-types (zero tools, heavy import) in aggregator

scitex-types ships no _mcp_server (zero MCP tools) but importing it pulls
the heavy scientific stack (numpy/torch/…). Probing it for a non-existent
FastMCP is pure cold-start waste. Add it to _LOCAL_UNMOUNTABLE (kept in
sync with scitex-dev _core._MCP_UNMOUNTABLE).

This rides on top of the #347/#348 bounded per-peer mount: skipped peers
are removed from the peer list before resolution, so no bounded daemon
thread is even spawned for them.

Adds 2 tests (types excluded from _iter_registry / _mount_skip True).
The umbrella aggregator's _mount_skip prefers scitex-dev's is_mcp_mountable
SSoT and falls back to the local _LOCAL_UNMOUNTABLE set when an older
scitex-dev (without that helper) is installed. scitex-str ships no _mcp
server (zero tools) yet importing scitex_str eagerly pulls pandas + numpy
via its _search / _plot submodules — pure cold-start waste, exactly like
scitex-types. Add it to the fallback set so the skip holds across the
coordinated two-repo rollout even before the scitex-dev SSoT bump is
installed.

Diagnosed in sac's real-SIF re-verify of umbrella 2.30.8. Pairs with
scitex-dev _MCP_UNMOUNTABLE bump and scitex-resource lazy-import PR.

Tests: assert _LOCAL_UNMOUNTABLE covers orochi/types/str and that
_mount_skip returns True for scitex-str via the forced fallback path.
…letion group, categorized help (#352)

* fix(cli): repair writer mount crash and mount scholar

- scitex writer --help crashed with AttributeError: the registry probe
  picked up scitex_writer._cli.main (a plain console-script function)
  instead of the click group. Add the doctrine 5b re-export shim
  (scitex/cli/writer.py -> scitex_writer._cli.main_group) and make the
  lazy loader skip non-click-command attributes so a probe can never
  mount an uncallable object again.
- scitex scholar was not mounted: no probe matched the standalone's
  entry point scitex_scholar._cli_main:cli. Add that probe shape.

* refactor(cli): retire duplicate umbrella namespaces via warn-phase aliases

notify->notification, verify->clew, events->event, socialia->social.
The old names are no longer registered as lazy subcommands; when
scitex-dev's click_compat helper is importable (scitex-dev > 0.21.0)
they come back as hidden warn-phase deprecated aliases that forward to
the canonical command (doctrine 3-phase ladder, removed in v3.0); with
an older scitex-dev they are simply excluded.

figrecipe/plt is intentionally NOT deduplicated: scitex-plt is a
published identity-alias package for figrecipe and doctrine 5b's brand
table documents 'scitex plt' as a figrecipe mount.

Hidden commands (the aliases) are now skipped in root help output.

* refactor(cli): make completion the canonical noun group

Doctrine 1b (04_exceptions.md, amended 2026-07-07): a bare completion
command is banned; the completion GROUP with verb subcommands is the
canon. Changes:

- move the group out of main.py into scitex/cli/completion.py (lazily
  mounted like every other wrapper; main.py shrinks well under the
  512-line cap)
- bare 'scitex completion' now shows group help instead of silently
  auto-installing
- 'completion install' gains --dry-run: prints the target rc file and
  the completion script without touching the filesystem (subsumes the
  old bash/zsh/fish script-dump leaves)
- 'completion bash|zsh|fish' become hidden warn-phase deprecated
  leaves: still print the script, warn on stderr, removed in v3.0
- root help examples updated ('scitex completion --show' never existed)

* feat(cli): categorized root help, -V flag, real peer one-liners

- root help now renders the doctrine 4a fixed ordered category headers
  (Core / Data & Sync / Service / Diagnostics / Introspection / Shell;
  Other is the catch-all and stays empty) — implemented inside
  LazyGroup.format_commands so no lazy subcommand is imported for help
  (scitex-dev's CategorizedGroup resolves real command objects, which
  would defeat the lazy mount and reintroduce the ~45s cold start)
- add -V as the short form of --version (doctrine 08 universal flags)
- ~25 mounted groups whose one-liner degraded to the bare package name
  when the peer was not installed (dataset, git, hpc, newb, datetime,
  ...) now fall back to a curated one-liner sourced from each package's
  own pyproject description (_INTERNAL_HELP renamed _FALLBACK_HELP)

* test(cli): cover slice-5 umbrella CLI standardization

- writer/scholar mount smoke (--help exits 0)
- retired duplicates hidden from help; warn-phase alias help page when
  scitex-dev click_compat is importable, unknown-command otherwise
- completion noun group: verbs listed, deprecated bash/zsh/fish leaves
  hidden but still printing the script with a stderr warning,
  install --dry-run prints plan and never writes
- -V short flag; 4a category headers, order, and empty Other
- builder unit tests: alias exclusion, scholar probe shape, writer
  wrapper override, fallback one-liners
Pilot org-level reusable-workflow migration (operator-approved
2026-07-10 incident remediation), following the pattern merged in
scitex-ai/scitex-stats#69.

Converted (clean mechanical match — job body verified byte-identical
to the reusable workflow before conversion):
- auto-merge-to-develop.yaml -> caller stub for
  scitex-ai/.github/.github/workflows/auto-merge-to-develop.yml@main

Left AS-IS (do not cleanly match the current reusable-workflow catalog
today — converting would silently change behavior or target an
unverified runner; see PR description for detail on each):
- pytest-matrix-on-ubuntu-py3-11-3-12-3-13.yml: runs on ubuntu-latest
  (not self-hosted spartan-cpu) with CPU-torch pre-pin, peer-standalone
  installs, a PS-170 umbrella-pin audit step, and a segfault-retry +
  JUnit-report pass/fail decoupling scheme with none of that present in
  the reusable pytest-matrix.yml.
- import-smoke-on-ubuntu-py3-12.yml: runs on ubuntu-latest with plain
  pip/venv and a hardcoded package name, vs. the reusable workflow's
  self-hosted/uv/derived-package-name approach.
- scitex-quality-audit-on-ubuntu-latest.yml: audits the WHOLE
  ecosystem (clones every scitex-* repo) rather than auditing only the
  calling repo, which is what the reusable quality-audit.yml does —
  not the same job semantically.
- rtd-sphinx-build-on-ubuntu-latest.yml: also commits the built HTML
  bundle back into src/scitex/_sphinx_html/ on develop pushes, a
  repo-specific step with no equivalent in the reusable workflow.
- cla.yml: no `push` trigger / no owner-bypass job (structurally
  simpler than the reusable cla.yml), and uses a differently-named
  secret (CLA_PERSONAL_ACCESS_TOKEN vs. the reusable workflow's
  hardcoded GH_PERSONAL_ACCESS_TOKEN) — converting risked silently
  breaking the CLA-signature push.
- newb.yml / newb-docs-quality-on-ubuntu-latest.yml: no reusable
  workflow exists for this job yet (two near-duplicate files already).
- pypi-publish-and-github-release-on-tag.yml: untouched per explicit
  instruction (PyPI OIDC trusted publishing does not support
  workflow_call).

Branch protection: confirmed via `gh api .../branches/{develop,main}/
protection` that neither branch has protection configured (404 on
both) — nothing to update.
chore(ci): migrate to scitex-ai/.github reusable workflows (pilot follow-up)
@ywatanabe1989

Copy link
Copy Markdown
Collaborator Author

Left open deliberately — fleet PR/Issue backlog sweep, 2026-08-11.

This PR was reviewed in a sweep of all 452 open PRs across the SciTeX repos. It was not closed: it is still considered live work.

State read live on 2026-08-11: mergeStateStatus=BLOCKED.

What it is waiting on: an explicit decision to cut a release for this package. Merging a developmain promotion publishes — it can fire a PyPI upload and a GitHub Release — so it is a decision about when to ship, not a code-review verdict. A backlog sweep is the wrong actor to make that call, so this stays open.

If a release is not wanted here, close this PR: the commits stay on develop either way.

* fix(cla): point the signature store at the renamed branch

Every PR in this repository currently fails its CLA check with

    Error occurred when creating the signed contributors file:
    Branch cla-signatures not found.
    Committers of pull request N have to sign the CLA

and SIGNING CANNOT CLEAR IT — the branch the action writes to no longer
exists. Whoever opens a PR is told they failed to sign something they did
sign, which is a false accusation aimed at contributors.

The signature branch was renamed `cla-signatures` -> `cla` across the fleet
and the old name deleted. This repository calls
`contributor-assistant/github-action` DIRECTLY rather than through the org
reusable, with the branch name hardcoded in this file, so it never picked
the rename up and no bump of the reusable could have reached it.

Points `branch:` at `cla`, which exists here and already holds
`signatures/cla.json`.

* fix(cla)!: call the org reusable instead of inlining the CLA config

Every PR in this repository was failing its CLA check with

    Error occurred when creating the signed contributors file:
    Branch cla-signatures not found.
    Committers of pull request N have to sign the CLA

and SIGNING COULD NOT CLEAR IT — the branch the action writes to had been
deleted. Contributors were told they had failed to sign something they did
sign.

WHY THE ONE-LINE FIX WAS NOT THE RIGHT FIX
------------------------------------------
The signature branch was renamed `cla-signatures` -> `cla` fleet-wide. This
repository called `contributor-assistant/github-action` DIRECTLY, with the
branch name — and the signature path, the document URL, the allowlist and both
comment templates — hardcoded in its own workflow. TWENTY repositories carried
a byte-identical copy of that block, so twenty of them broke together while
every repository already calling the org reusable picked the rename up for
free and was never affected.

The first version of this change simply rewrote the string here. That restores
service and leaves the duplication in place, which is the same twenty-way break
waiting for the next change to the CLA.

WHAT THIS DOES INSTEAD
----------------------
Deletes the inline job and calls the org reusable
`scitex-ai/.github/.github/workflows/cla.yml`, pinned to a SHA. The branch
name, the document URL and the comment wording now live in exactly one file
for the whole fleet. Single source of truth.

Two settings are carried over rather than defaulted:

  * `owner_allowlist` — the reusable defaults to `bot*,ywatanabe1989`; the
    inline version here also allowlisted `LLEmacs`. Taking the default would
    silently start demanding a CLA signature from an already-allowlisted
    contributor.

`default_branch` (main) and `runs_on` (ubuntu-latest) already match what the
inline version did and are left at their defaults.

The `uses:` ref is a SHA rather than `@main` on purpose: this workflow runs on
`pull_request_target` and `issue_comment`, which unauthenticated strangers can
fire on a public repo while it holds a write token and a PAT. A mutable
org-level ref would land an unreviewed change in every calling repo at once,
which is least acceptable exactly here.

`secrets:` is mapped explicitly rather than `inherit`, so only the one secret
the callee declares is forwarded into an attacker-startable job.

NOTE ON TIMING: `pull_request_target` takes its workflow definition from the
BASE ref, so the CLA check on the PR carrying this change still runs the old
inline version and still fails. The fix takes effect once merged.
…> 0), and two committed databases go with it (#372)

* chore(sqlite-out): the retired engine's name reaches zero outside docs/adr

We do not use the retired embedded engine; runtime state lives in
PostgreSQL. Its name survived in 20 tracked files. This clears all of
them, keeping only docs/adr/ (untouched) and the Zotero/Optuna
integrations (which carry none).

Deleted outright:

- `.scitex/clew/runtime/db.sqlite` and
  `examples/_legacy/scitex/clew/multi_parent/scitex/clew.db` — two live
  database files committed into the tree.
- `examples/_legacy/notebooks/19_scitex_db.ipynb` — a tutorial for the
  retired engine end to end (raw client connections, `sqlite_master`
  introspection). Nothing survives removing the engine from it, so the
  notebook goes and the master index's two links to it go with it.

Rewritten to name PostgreSQL, which is what we actually run:

- README's `scitex.db` section, `docs/05_ADDITIONAL_MODULES.md`, and the
  clew provenance prose in `docs/sphinx/core_concepts.rst` /
  `docs/sphinx/modules/io.rst`.
- `examples/_legacy/scitex/io/schematic_io_concept.py` and its recorded
  `_out/` YAML — the schematic's store box.

Also:

- `tests/integration/test_integration.py` no longer asserts the engine's
  wrapper class is exported; it still asserts `PostgreSQL` and
  `check_health`.
- `tests/integration/test_cross_package_imports.py` drops
  `scitex_io._load_modules._sqlite3` — that module was deleted in
  scitex-io 0.3.0, so the entry has silently skipped ever since.
- The engine's name leaves `.gitignore` (both copies), `pyproject.toml`'s
  clew comment, the stdlib table in `scripts/maintenance/_pypi_packages.py`,
  the CrossRef env example, and the legacy io-module guideline's format
  table.
- `examples/_legacy/notebooks/27_scitex_scholar.ipynb` no longer opens a
  raw client against the third-party impact-factor database; Scholar's
  own `enrich_by_default=True` already does that enrichment.

No guard test asserting the name's absence is added, deliberately.

* chore: stop the clew runtime store from being committed again

Running the suite after deleting the two committed database files
re-created `.scitex/clew/runtime/clew.db` (84 KB, magic bytes
`SQLite format 3`). The writer is the `scitex-clew` package pinned in
`pyproject.toml`, so this repo cannot stop the write — but it can stop
the file from being re-committed, which is how it got in.

Nothing was ignoring that path: the previous file was TRACKED, so it
never showed up as untracked noise and nobody added a rule.
…importing (#373)

check_optional_deps answered "is this installed?" by running
importlib.import_module(name) -- executing the dependency's top-level code
just to find out that it exists.

Walking MODULE_REQUIREMENTS once (what check_module_deps, require_module,
show_install_guide and the install-guide tests all do) therefore imported the
entire optional stack into the calling process. Measured on this repo's own
.venv: 5912 modules and 233 C extensions loaded, purely as the side effect of
a predicate.

That is the reachability path to CI's py3.13 SIGSEGV. Job 99189761473 died
inside openai/types/responses/response_compaction_item_param_param.py building
a TypedDict, in a GC pass, in a process holding 199 C extensions. On CPython
3.13.15 with openai 3.6.0, the availability walk loads that exact module;
after this change it does not.

The old body also caught only ImportError, so a package that is installed but
raises anything else did not report False -- it crashed the caller. Probing
the `dsp` module on a host without PortAudio raised
OSError('PortAudio library not found') straight out of check_optional_deps.
That is reproducible here and is fixed too.

importlib.util.find_spec resolves the module's spec from sys.meta_path without
executing it, which is the correct question for an availability probe, is much
faster, and cannot fail in the dependency's own code.

Semantics: this now answers "is it installed", not "does it import cleanly".
A package present but broken at import reports True. Callers needing the module
itself already use optional_import, which is unchanged. The `requires`
decorator in _install_guide.py deliberately keeps import_module: it guards a
function that is about to use the package, so "imports cleanly" is the honest
question at that site.

find_spec raises rather than returning None for a dotted name whose parent is
absent, so ModuleNotFoundError, AttributeError and ValueError are all treated
as "not installed", mirroring the handling already used in scitex.re_export.

Tests: four regression tests build real packages on disk (no mocks) pinning
that the probe does not execute the module body, reports an
installed-but-raising package instead of propagating, returns plain bools
(has_stats compares the dict to a literal), and reports False for a dotted
name with a missing parent.
* chore(ci): remove the newb workflow (abolished fleet-wide)

* chore(ci): remove the newb workflow (abolished fleet-wide)
@ywatanabe1989
ywatanabe1989 merged commit 5fdcc72 into main Sep 1, 2026
10 of 11 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant