A rule for introducing new public APIs

[TaiSakuma]

The problem

Awkward is under active development. New public APIs keep arriving, and the accumulated change across many releases can be large — but it arrives incrementally and progressively, never as one major breaking upgrade. Under the current semantic-versioning scheme there will be no Awkward 3, and therefore no major release for a pre-release or release candidate to stage. Every release is a production release, so a new API is in production the day it ships.

That closes off the usual way to gather feedback on an unfinished API. In the ordinary release pipeline an alpha or beta build goes out before the production release, feedback comes back, and the build is discarded. We have no such phase. Whatever feedback we get on a new API, we get from people running it in production — which means the signal that an API is unfinished has to reach them, at runtime, in a production release.

Not every new API is settled when it ships. Sometimes we publish a proof of concept; sometimes we promote an internal API to public, as in #2085. Once the name appears in ak.*, nothing distinguishes it from the more than 160 other public names. Fourteen new public operations arrived between 2.6.3 and 2.10.0. Most are surely settled by now — but nothing in the code or the documentation tells a reader which, and neither the author nor the reviewer had any way to record the answer at the time.

We have no way to say that an API is public but not yet settled:

• No @experimental decorator, no experimental warning class, no experimental marker in src/awkward/, and no mention of experimental status in docs/.
• Nothing anywhere states that some public APIs are exempt from the stability a user may reasonably expect of the rest.

This discussion proposes a rule for introducing new public APIs, and specifically for signalling their maturity. Deprecating and removing existing APIs is the natural next topic, and deserves its own discussion.

Prior discussion, for context

In discussion #1969 (2022), jpivarski set out "three formal levels of scoping": public high-level, public low-level, and internal. That model still describes how we scope APIs today, and it works. But it is a model of scope — who an API is for — and scope is orthogonal to maturity: an API can sit at any of the three levels while still being unfinished.

One argument from that thread is worth reusing. On why a boundary cannot rest on documentation alone, jpivarski wrote: "we can't assume that anybody reads a warning we put in the documentation before they start depending on our public API... discovery is hard and they might not find it."

CPython reached the same conclusion by experiment. PEP 411, which gave the standard library "provisional" APIs marked only in the docs, is now Superseded; its own note says the status "has also not helped prevent people from relying too heavily on provisional modules".

What other projects do

Project	Mechanism	Cost when it graduates
CPython (PEP 411)	Docs note only	none (superseded as ineffective)
pandas	Docs admonition + policy carve-out	none
PyTorch	Docs tiers: Stable, Beta, Prototype	none
polars	@unstable decorator; warning off unless POLARS_WARN_UNSTABLE	none
pydantic	pydantic.experimental and a default-on PydanticExperimentalWarning	import path changes
scikit-learn	sklearn.experimental with enable_* opt-in imports	enabler becomes a no-op shim
JAX, TensorFlow	jax.experimental; any TF name containing experimental	import path or name changes
Strawberry	strawberry.experimental, with no warning and no stated guarantee	import path changes
NumPy, matplotlib, astropy, Django	no experimental tier at all	—

Sort them by one question — must the user edit their code when the API graduates? — and they split cleanly. A separate location (jax.experimental, std::experimental, golang.org/x/exp, jdk.incubator.*) gives a signal nobody can miss, and charges the early adopter a migration when the API moves. An in-place marker (Node.js's stability index, .NET's [Experimental], Guava's @Beta, polars' @unstable) is quieter, and free.

The loudest markers fail the build: Rust's feature gates, Kotlin's @RequiresOptIn, .NET's [Experimental(diagnosticId)]. Python cannot, and that may be no loss — SQLite's SQLITE_EXPERIMENTAL macro once expanded to compiler warnings, and the project reduced it to a no-op because they "ended up generating such a flurry of bug reports". One idea from .NET is still worth taking: per-feature granularity, so that silencing one experimental API is not consent to another.

SQLite is also the closest precedent for the graduation model below. It publishes three tiers — stable, experimental, deprecated — promising that "stable interfaces will be maintained indefinitely in a backwards compatible way" while "experimental interfaces are subject to change". An interface graduates when the marker is deleted; the name never moves. sqlite3_snapshot_get, sqlite3_vtab_collation, and sqlite3_blob_reopen all became stable that way, and no caller changed — from a project that pledges C API compatibility until 2050. Its signal, though, is documentation plus an inert annotation, and that is where we should depart from it.

The counter-position deserves a hearing. Django has no experimental tier by design: everything documented is stable, everything internal is underscored, and public APIs "will not be moved or renamed without providing backwards-compatible aliases". Even Django 6.0's new Tasks framework ships with no caveat. NumPy, matplotlib, and astropy, the projects we most resemble, arrive at the same place. One can decline to make an API public until it is finished.

Four constraints

There is no migration boundary. With no single major breaking upgrade, we lose the boundary at which most semantic-versioning projects batch their breaking changes, and the pre-release that would stage one: an rc stages a release, not an API. Changing a public API therefore means breaking users outright or running a deprecation cycle — the right price for an API we mean to keep, the wrong price for one we are still designing. An experimental marker is what exempts a new API from that choice, which makes clause 3 essential rather than a formality.

A maturity warning must subclass UserWarning. warnings.warn records a warning's module as the __name__ of the frame chosen by stacklevel, not the file name, so our import-time filterwarnings("default", module="awkward.*") matches only warnings attributed to a frame inside Awkward. The call in src/awkward/jax.py passes stacklevel=2, attributing its warning to the caller. Measured against the installed package:

stacklevel	from __main__	from a library
1 (attributed to Awkward)	visible	visible
2 (attributed to the caller)	visible	hidden

That __main__ column is visible only because CPython makes an exception for it, not because of our filter. Such a warning reaches someone working in a notebook and nobody inside uproot or coffea. UserWarning subclasses are visible in every column. (FutureWarning is too, but it means "deprecated", which would misdescribe a new API.)

A runtime marker will not reach the docs. sphinx-autoapi parses the source statically and never imports it, so a decorator that warns at call time is invisible to the API reference. The label must also be in the docstring.

The rule governs new APIs only. Nothing already public gets reclassified. dask-awkward and uproot import awkward._nplikes, and coffea imports awkward._util; those internals are already essential to them, underscore or not. Relabelling that surface would break all three at once. So the rule applies from the next API onward, and everything already public keeps the stability it has.

Proposal

Keep the API at its final name, mark it, and remove the mark when it stabilizes. The alternative — Django's, and ours today by default — is to hold a new API private until it is finished. That is a coherent position, and its cost is that useful work waits behind an underscore, or goes public with no way to say it is provisional. A new public API may ship as experimental if and only if all three of the following hold. (ak.stupendous_reduce, ak.astounding_zip, ak.gizmo, and the issue number NNNN are invented placeholders.)

1. It emits an ExperimentalWarning on first use — a new ak.errors.ExperimentalWarning subclassing UserWarning, fired once per marked name per process. It names the API, links its tracking issue, and says how to silence it.

Each experimental API gets one tracking issue, opened with it. That is where users report on it, and where we later decide to graduate or remove it. Rust requires the same thing: #[unstable(feature = "...", issue = "...")] will not compile without an issue number, and stabilization is settled on that issue. It costs one issue per experimental API and gives anyone who meets the warning somewhere to go.

Between them, the warning and the issue are a beta programme run inside production releases: the warning tells a production user that the API is unfinished, and the issue is where their feedback lands.

>>> ak.stupendous_reduce(arr)
ExperimentalWarning: ak.stupendous_reduce is experimental. Its API and behaviour
may change, or it may be removed, in any release, without a deprecation period.
Feedback: https://github.com/scikit-hep/awkward/issues/NNNN
Silence with:
    warnings.filterwarnings("ignore", "ak.stupendous_reduce is experimental",
                            ak.errors.ExperimentalWarning)

Filtering on the message, not the class, gives .NET's per-feature granularity: silencing ak.stupendous_reduce does not also silence ak.astounding_zip.

2. It is marked in the source, and its docstring carries the admonition.

@experimental("ak.stupendous_reduce", issue=NNNN)
def stupendous_reduce(array, axis=-1):
    """Reduce ``array`` along ``axis``.

    .. warning::
        This API is experimental. It may change or be removed in any
        release, without a deprecation period.
    """

experimental(...) returns an object usable either as a decorator or as a value, the way pytest.mark.skipif is — our own test suite already assigns it to pytestmark. So the mark is not only for functions:

Marked object	Warning fires on	Admonition lives in
function, method	first call	its own docstring
class	first instantiation	the class docstring
property	first access	the getter's docstring
module, namespace	first attribute access on the parent	the module docstring

A module cannot be decorated, so it is marked by assignment, carrying the same admonition in its module docstring:

# src/awkward/gizmo/__init__.py
__experimental__ = experimental("ak.gizmo", issue=NNNN)

awkward/__init__.py honours that from a module-level __getattr__ (PEP 562). The hook runs only for names not already bound, so an experimental submodule must be imported lazily — which SPEC 1 recommends anyway, and which makes the warning fire exactly once, since importing a submodule binds it on its parent.

3. It is covered by a stated guarantee — one sentence, wherever we document API stability: public APIs are stable except those marked experimental, which "may change at any point without it being considered a breaking change". That wording is settled across scikit-learn, polars, and TensorFlow, and there is no reason to invent our own.

Graduation is then: delete the mark, delete the admonition, note it in the release notes. Nobody's code breaks. Removal is: delete it, note it. Nobody was promised otherwise.

What it would cost us

Little, because each piece already has a precedent in this repository.

ak.errors is already a public module with an __all__; ExperimentalWarning joins AxisError and FieldNotFoundError. The 174 operation modules under src/awkward/operations/ already stack @high_level_function(), so @experimental(...) sits above a decorator we apply everywhere. And clause 1's per-message filtering is a pattern we already rely on: pyproject.toml carries "ignore:The JAX backend is deprecated:DeprecationWarning" today.

That last file also sets filterwarnings = ["error"], which is the one detail worth settling before anyone writes code. Under warnings-as-errors, a test that calls an experimental API would fail unless its module opts out — one pytestmark = pytest.mark.filterwarnings("ignore:ak.gizmo is experimental") line per test module, in the same idiom our tests already use for skipif. The alternative is for the mark to stay silent when the caller is inside awkward itself, which is what .NET does for references from one experimental member to another. Either works; we should choose deliberately.

Open questions

• How strongly should the rule bind? Enforced in CI by a test over the public API surface; written into CONTRIBUTING.md and applied at review; or offered as advice. The first costs a check to write and maintain. The second gives a reviewer something to point at, which is most of the value. The third asks nothing of anyone.
• Can a part of a settled API be experimental? Any public name can be marked, but a new keyword argument has no name of its own to carry a mark and no docstring of its own to carry an admonition. TensorFlow handles this by spelling experimental inside the argument's name. We add keyword arguments to stable functions often, so this needs an answer.
• One tier, or two? The mark proposed here withdraws two promises at once: that the API will survive, and that its shape will not change. Kubernetes and PyTorch separate them — for a beta API, "the support for a feature will not be dropped, though the details may change". Do we want a second tier that says "this is staying, but its signature may still move"?
• Should experimental status expire? Java removes incubator modules not standardized "after a small number of JDK feature releases", and Kubernetes gives a beta API "a maximum lifetime of 9 months or 3 minor releases". Do we want a review point — still experimental after N minor releases means graduate, remove, or renew? The tracking issue is the natural place to hold that review.
• What about downstream libraries? If uproot or coffea call an experimental API internally, their users see a warning they cannot act on. Choosing stacklevel puts it in the downstream maintainer's test suite instead, which is the intent — but we should decide it, not discover it.

---

Awkward claims were checked against the repository and the GitHub API; the warning table was measured on CPython 3.10. Other projects were checked against primary sources, except polars' decorator internals, which come from its documentation and a code search.

🤖 Drafted with Claude Code

[ianna]

@TaiSakuma - thanks for summarizing the discussion. Personally I like experimental. As @henryiii poined out -- we need it to be somewhat flexible/configurable.

[TaiSakuma]

A scoping note toward implementation: the kinds of objects the experimental mark needs to cover.

• module (including a package)
• class
• method (including a class method and a static method)
• function
• property
• attribute (e.g., a module-level constant)
• keyword argument

Possible implementation order, easiest first:

1. function, method, property — one shared wrapper; the mark sits directly above the def, so at decoration time all three are plain functions
2. class — one extra branch: wrap __init__ and return the same class
3. module — parent-side machinery: a PEP 562 __getattr__ in the parent package lazily imports the module and reads the mark set inside it (a module cannot be decorated)
4. attribute — the same __getattr__ hook in the owning module, which also holds the mark, since a constant has no namespace of its own
5. keyword argument — a separate variant: a decorator on the owning function that names the parameter; the design is still open

🤖 Generated with Claude Code

[TaiSakuma]

The API design for step 1 of the implementation order proposed above: function, method, and property — one shared wrapper.

The mark is a decorator placed directly above the def:

@experimental
def stupendous_reduce(array):
    ...

Everything user-visible is derived from the decorated object itself: the name from __qualname__, the defining module from __module__. @experimental() works identically; the decorator takes no arguments in the initial release, and the parenthesized form is where configuration options can be added later.

For a method, the mark again sits directly above the def; when @property, @classmethod, or @staticmethod is also present, the mark sits below it — still directly above the def — so in every case it receives a plain function:

class Thing:
    @property
    @experimental
    def stupendous_field(self):
        ...

Each marked function warns once per process, on its first call — for a property, on its first access:

/home/user/analysis.py:12: ExperimentalWarning: stupendous_reduce is experimental.
    It may change or be removed in any release, without a deprecation period.
    Defined in: awkward.operations.ak_stupendous_reduce
  result = ak.stupendous_reduce(events)

The /home/user/analysis.py:12: ExperimentalWarning: prefix and the final echoed source line are added by the warnings machinery; both point at the line in the user's script that triggered the warning. Everything else is the message.

In awkward's own source the mark also has to compose with the decorators already present on the public APIs. The main one is @high_level_function, which wraps the ak.* operations for third-party dispatch; the mark stays directly above the def:

@high_level_function()
@experimental
def stupendous_reduce(array):
    # Dispatch
    yield (array,)

    # Implementation
    return _impl(array)

The composition works, probed against the real dispatcher in src/awkward/_dispatch.py. An operation under @high_level_function is normally a generator function: it yields its array arguments for third-party interception and, when nothing intercepts, resumes into the implementation. The few operations with nothing to intercept — the ak.from_* conversions among others — are plain functions whose result the dispatcher returns directly. In both cases the mark's wrapper warns, calls the decorated function, and returns the result unchanged, so the dispatch protocol proceeds exactly as without the mark. The wrapper applies functools.wraps, which high_level_function depends on: high_level_function reads __qualname__ from the object it wraps — here the mark's wrapper — to build the dispatch name. Because the dispatcher calls the wrapper eagerly to obtain the generator, the warning also fires when a third-party array intercepts the operation through __awkward_function__.

One consequence for the implementation: the warning's stacklevel cannot be a fixed number. Under the dispatcher the user's frame is one call further away than for a directly called function, so the wrapper walks the stack to the first frame outside awkward — the pattern of pandas' find_stack_level — to keep the reported location (/home/user/analysis.py:12 above) on the user's line.

Only one new public name is involved: ExperimentalWarning, in awkward.errors, because users reference it in filters. The decorator itself is internal — the code examples above are awkward's own source, not user code.

🤖 Generated with Claude Code