Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add plugin to infer more precise regex match types #7803

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Commits on Oct 27, 2019

  1. Add plugin to infer more precise regex match types

    This pull request adds a plugin to make mypy infer more
    precise types when grabbing regex groups: the plugin will
    when possible analyze original regex to deduce whether a
    given group is required or not.
    
    ```
    from typing_extensions import Final, Literal
    import re
    
    pattern: Final = re.compile("(a)(b)*")
    match: Final = pattern.match("")
    if match:
        reveal_type(match.groups())  # Revealed type is Tuple[str, Optional[str]]
        reveal_type(match.group(0))  # Revealed type is str
        reveal_type(match.group(1))  # Revealed type is str
        reveal_type(match.group(2))  # Revealed type is Optional[str]
    
        index: int
        reveal_type(match.group(index))  # Revealed type is Optional[str]
    
        # Error: Regex has 3 total groups, given group number 5 is too big
        match.group(5)
    ```
    
    To track this information, I added in an optional 'metadata' dict
    field to the Instance class, similar to the metadata dict for
    plugins in TypeInfos. We skip serializing this dict if it does not
    contain any data.
    
    A limitation of this plugin is that both the pattern and the match
    variables must be declared to be final. Otherwise, we just default
    to using whatever types are defined in typeshed.
    
    This is because we set and erase the metadata field in exactly the
    same way we set and erase the `last_known_value` field in Instances:
    both kinds of info are "transient" and are unsafe to keep around if
    the variable reference is mutable.
    
    This limitation *does* end up limiting the usefulness of this plugin
    to some degree: it won't support common patterns like the below, since
    variables aren't allowed to be declared final inside loops:
    
    ```
    for line in file:
        match = pattern.match(line)
        if match:
            ...
    ```
    
    Possibly we can remove this limitation by making mypy less aggressive
    about removing this transient info by tracking the "lifetime" of this
    sort of data in some way?
    
    This pull request should mostly address
    python#7363, though it's unclear if it
    really fully resolves it: we might want to do something about the
    limitation described above and re-tune typeshed first.
    
    The other mostly unrelated change this PR makes is to refactor some of
    the helper functions in checker.py into typeops.py so I could use them
    more cleanly in the plugin.
    Michael0x2a committed Oct 27, 2019
    Configuration menu
    Copy the full SHA
    54dfad8 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9a7097f View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2022

  1. Configuration menu
    Copy the full SHA
    03bd647 View commit details
    Browse the repository at this point in the history
  2. Fix

    97littleleaf11 committed Jan 5, 2022
    Configuration menu
    Copy the full SHA
    45168cd View commit details
    Browse the repository at this point in the history
  3. Fix

    97littleleaf11 committed Jan 5, 2022
    Configuration menu
    Copy the full SHA
    9d468df View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ced8cda View commit details
    Browse the repository at this point in the history
  5. Fix type check

    97littleleaf11 committed Jan 5, 2022
    Configuration menu
    Copy the full SHA
    f54eff2 View commit details
    Browse the repository at this point in the history
  6. Fix test

    97littleleaf11 committed Jan 5, 2022
    Configuration menu
    Copy the full SHA
    3b38dc9 View commit details
    Browse the repository at this point in the history