Closes #12820 and closes #13735. Supersedes #13734.

Partially fixes #13155 by exposing full context to matchers (but not the hooks yet - that would be a breaking change; I can follow up with it).

This is a gentle, backward and forward-compatible rewrite of Matcher API. I attempted to design the API for performance and future extensibility, while retaining compatibility with existing code.

There is quite some legacy logic in this codebase, which I tried to carefully preserve and document by making any such behaviour explicit, but in a few places I encountered behaviour which I don't know how to tackle (no unit tests, nor comments - not sure if existing behaviour was intended or incidental due to accumulated code complexity). I would highly appreciate help on such cases, suggestions on how to handle deprecations (if any are needed) and general feedback on this PR.

User-facing changes

Completion type is returned in the provisional API for most matchers

Before	After

Dictionary completions are restricted to dictionary keys

By default completions in dictionary context will only provide dictionary key matches. This is user-configurable.

The heuristic for this can be improved in the future iterations.

Before	After

Configuration

New configuration options were added:

• IPCompleter.suppress_competing_matchers allows to instruct which matchers should (or should not) short-circuit to return candidate completions ignoring other matchers
• IPCompleter.disable_matchers enables selective disabling of matchers

Existing configuration options are now aliases:

• IPCompleter.merge_completions = False is an alias for IPCompleter.suppress_competing_matchers = True

Matcher API

Briefly, the new Matcher is defined as:

MatcherAPIv1 = Callable[[str], list[str]]
MatcherAPIv2 = Callable[[CompletionContext], MatcherResult]

Matcher = Union[MatcherAPIv1, MatcherAPIv2]

where MatcherAPIv1 is the same public API as exposed via (undocumented) IPCompleter.custom_matchers attribute since v8.0, #12130), MatcherAPIv2 is the new API, and subsequent versions can be added in the future.

Matcher API v2

The MatcherResult of API v2 is a dictionary, as and resolves an old TODO comment:

class MatcherResult(TypedDict):
    #: list of candidate completions
    completions: Union[Sequence[SimpleCompletion], Sequence[_JediCompletionLike]]   # simplified

    #: suffix of the provided ``CompletionContext.token``, if not given defaults to full token.
    matched_fragment: NotRequired[str]

    #: whether to suppress results from all other matchers (True), some
    #: matchers (set of identifiers) or none (False); default is False.
    suppress: NotRequired[Union[bool, Set[str]]]

    #: identifiers of matchers which should NOT be suppressed
    do_not_suppress: NotRequired[Set[str]]

    #: are completions already ordered and should be left as-is? default is False.
    ordered: NotRequired[bool]

This will enable adding additional attributes to the dictionary without breaking compatibility in the future. It also makes the API simple for users, as they only need to provide {'completions': list[SimpleCompletion]}.

SimpleCompletion is a container for completion metadata allowing to add type and in future additional attributes. After profiling initialisation, different forms of attribute/item access and memory use I chose to use a custom class with slots over NamedTuple or TypedDict due to inherent performance advantage. This is consistent with Completion class which also uses __slots__.

class SimpleCompletion:
    __slots__ = ["text", "type"]

    text: str
    type: str

CompletionContext is intended to provide matchers with information they need to generate completion candidates:

class CompletionContext(NamedTuple):
    token: str
    full_text: str
    cursor_position: int
    cursor_line: int

    @cached_property
    def text_until_cursor(self) -> str:
        ...

    @cached_property
    def line_with_cursor(self) -> str:
        ...

Switching between APIs and Matcher configuration

A decorator was added enabling demarking API versions and configuring Matcher-level options. The decorator is optional for v1 API, but required to use v2 API

def completion_matcher(
    *, priority: float = None, identifier: str = None, api_version=1
):
    """Adds attributes describing the matcher.
    Parameters
    ----------
    priority : Optional[float]
        The priority of the matcher, determines the order of execution of matchers.
        Higher priority means that the matcher will be executed first. Defaults to 50.
    identifier : Optional[str]
        identifier of the matcher allowing users to modify the behaviour via traitlets,
        and also used to for debugging (will be passed as ``origin`` with the completions).
        Defaults to matcher function ``__qualname__``.
    api_version: Optional[int]
        version of the Matcher API used by this matcher.
        Currently supported values are 1 and 2.
        Defaults to 1.
    """

    def wrapper(func: Matcher):
        func.matcher_priority = priority
        func.matcher_identifier = identifier or func.__qualname__
        func.matcher_api_version = api_version
        return func

    return wrapper

Maintaining compatibility

Existing methods such as dict_key_matches were not touched; instead wrapper methods converting from API v1 to API v2 and adding desired metadata (when applicable) were added; those methods have a new suffix _matcher. This approach allows users who monkey-patch matcher v1 methods in IPCompleter to continue using their existing code without any changes.

Follow-up tasks

To be discussed in separate issues and addressed in separate PRs:

• benchmark type as a string vs as an enum to inform whether it is worth switching to enum
• investigate how to integrate relevance score of sort (in sortText in LSP terms)