[ci/doc] Guard against annotated subpackages that escape the API walk

[dstrodtman]

Why

The API doc consistency check (ci/ray_ci/doc/cmd_check_api_discrepancy.py)
only sees APIs that its module walk reaches, and the walk only follows
submodules a parent __init__ actually imports. A public,
@PublicAPI-annotated subpackage that its parent does not import is invisible
to the check — its APIs can silently rot undocumented.

ray.serve.llm and ray.data.llm are both such surfaces today: documented and
annotated, but reached by no walk.

What

• Onboard ray.serve.llm by adding it to the serve head_modules. It is
  import-safe in the docbuild image (transformers is a soft try_import, and
  vLLM is only imported lazily), so it can be its own walk root.
• Add a cross-team coverage guard that enumerates one level of subpackages
  under every configured head module and fails when an annotated subpackage is
  neither reached by any walk nor knowingly excluded. A reviewed allowlist
  records the intentional exclusions and the reason for each.
• Allowlist ray.data.llm. Unlike ray.serve.llm, it eagerly imports the
  vLLM/SGLang engine processor configs, which do a hard import transformers
  at module load. transformers is absent from the docbuild image
  (python/deplocks/docs/docbuild_depset_py3.11.lock), so ray.data.llm
  cannot be imported by this check. Its surface stays documented in
  doc/source/data/api/llm.rst.
• Track reachable module names on Module so the guard distinguishes "actually
  walked" from "merely name-prefixed" (ray.data.llm is prefixed by ray.data
  yet never imported by ray/data/__init__.py).
• Unit-test the guard's decision core and helpers against the mock module.

Stacking

This PR is stacked on top of the API-ref consistency-policy work and should be
reviewed/merged after it (its diff transitively shows those commits until they
land):

• depends on [ci/doc] Generate API stubs without a full doc build; fail loudly #64423 (generate API stubs without a full doc build)
• which depends on [ci/doc] Add reverse and dedup API-ref consistency policies #64420 (reverse and dedup consistency policies)

Kept as a draft until the base PRs merge; it will be rebased onto master
afterward.

Checks

The decisive check is doc: check API doc consistency — it exercises the new
guard, ray.serve.llm in head_modules, and the ray.data.llm allowlist
against the real docbuild image.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request optimizes the API documentation consistency check by generating only the necessary autosummary stub files via a new standalone script (api_autogen.py), eliminating the need to build the entire HTML documentation. It also introduces Policy 02 (ensuring documented APIs resolve to public code) and Policy 04 (preventing duplicate documentation), along with a coverage guard to catch unwalked annotated subpackages. The feedback suggests adding exception handling around dynamic attribute inspection and module iteration in cmd_check_api_discrepancy.py to prevent crashes from lazy imports or invalid paths.

[gemini-code-assist]

Accessing attributes dynamically via getattr on a module can trigger lazy imports or properties that raise exceptions (such as ModuleNotFoundError for missing optional dependencies). Wrapping the attribute inspection in a try-except block ensures that optional dependency failures or buggy attributes do not crash the entire API consistency check.

Suggested change

	for attr in dir(mod):
	obj = getattr(mod, attr, None)
	origin = getattr(obj, "__module__", None)
	if not origin or (origin != module and not origin.startswith(f"{module}.")):
	continue
	if hasattr(obj, "_annotated"):
	return (True, True)
	for attr in dir(mod):
	try:
	obj = getattr(mod, attr, None)
	origin = getattr(obj, "__module__", None)
	if not origin or (origin != module and not origin.startswith(f"{module}.")):
	continue
	if hasattr(obj, "_annotated"):
	return (True, True)
	except Exception:
	continue

[gemini-code-assist]

Calling pkgutil.iter_modules on pkg.__path__ can raise exceptions if any of the paths in __path__ are invalid, non-existent, or inaccessible (e.g., due to permission issues or virtual environment configurations). Wrapping this in a try-except block makes the subpackage discovery more robust.

Suggested change

	children = []
	for info in pkgutil.iter_modules(pkg.__path__, prefix=f"{package}."):
	if info.name.rsplit(".", 1)[-1].startswith("_"):
	continue
	children.append(info.name)
	children = []
	try:
	for info in pkgutil.iter_modules(pkg.__path__, prefix=f"{package}."):
	if info.name.rsplit(".", 1)[-1].startswith("_"):
	continue
	children.append(info.name)
	except Exception:
	pass

[dstrodtman]

Thanks for the review. Two notes:

Scope: this PR only adds three things — ray.serve.llm to the serve head_modules, an UNWALKED_ANNOTATED_ALLOWLIST entry for ray.data.llm, and the cross-team coverage guard (_check_unwalked_annotated_subpackages) plus its tests. The api_autogen.py script and Policies 02/04 mentioned in the summary belong to the PRs this one is stacked on (#64423 → #64420); they show up here only because GitHub can't base a fork PR on another fork branch, so the diff is against master until those land.

Both inline suggestions adopted in the latest push:

• _import_status: wrapped the per-attribute getattr/inspection in try/except … continue. This is exactly the right call for this guard — a module with a PEP 562 __getattr__ (e.g. the batch processor package) can trigger a heavy optional-dep import on attribute access, which would otherwise crash the check for every team. Added a regression test with an "exploding" lazy attribute.
• _immediate_child_modules: wrapped pkgutil.iter_modules(pkg.__path__, …) so a bad/inaccessible __path__ entry yields no children instead of raising.

Both are consistent with the guard's own principle: an optional-dependency failure should downgrade coverage of one module, never crash the whole consistency check.