Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change parsing for exclude field #11746

Closed
wants to merge 2 commits into from

Conversation

mawillcockson
Copy link
Contributor

Description

#11329 added support for nicer exclude lists

TOML has a built-in list syntax, and it would be nice to use that for specifying lists for the exclude option.

This change tries the ini-style first: if exclude is set to a multiline string, it will split that on newlines, otherwise it will assume it's a list.

Examples:

[mypy]
exclude = "pattern1"

produces:

exclude = ["pattern1"]

All of these produce the same result:

exclude = "pattern1\npattern2"
exclude = """
    pattern1
    pattern2
"""
exclude = [
    "pattern1",
    "pattern2",
]

Test Plan

I've only tested on the commandline.

python#11329 added support for nicer `exclude` lists

TOML has a built-in list syntax, and it would be nice to use that for specifying lists for the `exclude` option.

This change tries the ini-style first: if `exclude` is set to a multiline string, it will split that on newlines, otherwise it will assume it's a list.
Copy link
Member

@sobolevn sobolevn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

A couple of questions:

  1. We have files that is similar to exclude. Is it consistent with your solution?
  2. We can surely test this! Take a look at mypy/test/test_find_sources.py

'exclude': lambda s: [
p.strip() for p in
(s.split('\n') if isinstance(s, str) else s)
if p.strip()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can turn this into a separate function, so we can skip doing some work twice: p.strip() for example.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I were to, I'd probably also change the ini exclude function that I stole it from:

'exclude': lambda s: [p.strip() for p in s.split('\n') if p.strip()],

I considered using the try_split() function, but it would not have filtered empty strings, I think:

def try_split(v: Union[str, Sequence[str]], split_regex: str = '[,]') -> List[str]:
"""Split and trim a str or list of str into a list of str"""
if isinstance(v, str):
return [p.strip() for p in re.split(split_regex, v)]
return [p.strip() for p in v]

Would this work better?

    'exclude': list(filter(None, try_split('[\n]')))

@mawillcockson
Copy link
Contributor Author

Thanks!

A couple of questions:

1. We have `files` that is similar to `exclude`. Is it consistent with your solution?

2. We can surely test this! Take a look at `mypy/test/test_find_sources.py`
  1. I think I understand your question: files in a pyproject.toml already takes a list of files, first as a comma-separated list of paths in a string, and second as a TOML list:

'files': lambda s: split_and_match_files_list(try_split(s)),

That's actually what caught me in the first place: trying to use exclude like files.

  1. I ran some of the tests as suggested in the contributing guidelines, which showed no type errors, but I'm not familiar enough with mypy's testing system to feel comfortable adding quality tests.
    That's not to say that tests aren't sorely needed: in the release version of mypy 0.910, providing exclude a TOML list will result in this line calling str() on a list, which is then used as a regex. For example, a pyproject.toml with exclude = ["tests/"] results in excluding any file matching the regex ['tests/'].

I'm worried this means there's a large testing gap that I feel unqualified to fill.

@sobolevn
Copy link
Member

I'm worried this means there's a large testing gap that I feel unqualified to fill.

No worries, I can help you with tests, if all other things are ok! 👍

@JelleZijlstra JelleZijlstra changed the title Patch 2 Change parsing for exclude field Dec 15, 2021
@posita
Copy link
Contributor

posita commented Dec 22, 2021

FYI, I have a competing PR (#11828) in the works to address this (i.e., fix #11825). (Apologies for missing yours.)

Are you sure you want to split the TOML string? While not documented, some folks may have come to rely on treating multi-line strings in TOML as a single regex.

@mawillcockson
Copy link
Contributor Author

Are you sure you want to split the TOML string? While not documented, some folks may have come to rely on treating multi-line strings in TOML as a single regex.

It looked like an easy quality-of-life improvement to provide an additional syntax for specifying multiple regexps for exclude, preferring the ini-style behaviour added in #11329 first.

I agree that breaking expected behaviour, even if undocumented, is always a problem for people using the tools.

I would prefer to not split TOML strings if there's a nicer, more structured syntax available, but I do prefer parity and unity of behaviours and features between all sources of configuration, first and foremost.

Your PR (#11825) does more, and more that I like, so I'll close this one in favour of that one.

@sobolevn It looks like @posita added test cases! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants