Merge branch 'master' into pre-commit-ci-update-config

.github/workflows/test_stubgenc.yml

@@ -1,4 +1,4 @@
-name: Test stubgenc on pybind11-mypy-demo
+name: Test stubgenc on pybind11_fixtures
 
 on:
   workflow_dispatch:
@@ -10,6 +10,7 @@ on:
     - 'misc/test-stubgenc.sh'
     - 'mypy/stubgenc.py'
     - 'mypy/stubdoc.py'
+    - 'mypy/stubutil.py'
     - 'test-data/stubgen/**'
 
 permissions:

.pre-commit-config.yaml

@@ -6,11 +6,12 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
   - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 23.12.1  # must match test-requirements.txt
+    rev: 24.1.1  # must match test-requirements.txt
     hooks:
       - id: black
+        exclude: '^(test-data/)'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.1.9  # must match test-requirements.txt
+    rev: v0.1.15  # must match test-requirements.txt
     hooks:
       - id: ruff
         args: [--exit-non-zero-on-fix]

README.md

@@ -41,8 +41,8 @@ To report a bug or request an enhancement:
 
 To discuss a new type system feature:
 
-- discuss at [typing-sig mailing list](https://mail.python.org/archives/list/typing-sig@python.org/)
-- there is also some historical discussion [here](https://github.com/python/typing/issues)
+- discuss at [discuss.python.org](https://discuss.python.org/c/typing/32)
+- there is also some historical discussion at the [typing-sig mailing list](https://mail.python.org/archives/list/typing-sig@python.org/) and the [python/typing repo](https://github.com/python/typing/issues)
 
 What is mypy?
 -------------

docs/source/cheat_sheet_py3.rst

@@ -209,6 +209,7 @@ Classes
        # This will allow access to any A.x, if x is compatible with the return type
        def __getattr__(self, name: str) -> int: ...
 
+   a = A()
    a.foo = 42  # Works
    a.bar = 'Ex-parrot'  # Fails type checking
 

docs/source/command_line.rst

@@ -415,7 +415,6 @@ None and Optional handling
 **************************
 
 The following flags adjust how mypy handles values of type ``None``.
-For more details, see :ref:`no_strict_optional`.
 
 .. _implicit-optional:
 
@@ -435,16 +434,19 @@ For more details, see :ref:`no_strict_optional`.
 
     **Note:** This was disabled by default starting in mypy 0.980.
 
+.. _no_strict_optional:
+
 .. option:: --no-strict-optional
 
-    This flag disables strict checking of :py:data:`~typing.Optional`
+    This flag effectively disables checking of :py:data:`~typing.Optional`
     types and ``None`` values. With this option, mypy doesn't
-    generally check the use of ``None`` values -- they are valid
-    everywhere. See :ref:`no_strict_optional` for more about this feature.
+    generally check the use of ``None`` values -- it is treated
+    as compatible with every type.
+
+    .. warning::
 
-    **Note:** Strict optional checking was enabled by default starting in
-    mypy 0.600, and in previous versions it had to be explicitly enabled
-    using ``--strict-optional`` (which is still accepted).
+        ``--no-strict-optional`` is evil. Avoid using it and definitely do
+        not use it without understanding what it does.
 
 
 .. _configuring-warnings:

docs/source/common_issues.rst

@@ -119,7 +119,7 @@ and mypy doesn't complain**.
         return None  # No error!
 
 You may have disabled strict optional checking (see
-:ref:`no_strict_optional` for more).
+:ref:`--no-strict-optional <no_strict_optional>` for more).
 
 .. _silencing_checker:
 

docs/source/config_file.rst

@@ -3,6 +3,10 @@
 The mypy configuration file
 ===========================
 
+Mypy is very configurable. This is most useful when introducing typing to
+an existing codebase. See :ref:`existing-code` for concrete advice for
+that situation.
+
 Mypy supports reading configuration settings from a file with the following precedence order:
 
     1. ``./mypy.ini``
@@ -580,10 +584,15 @@ section of the command line docs.
     :type: boolean
     :default: True
 
-    Enables or disables strict Optional checks. If False, mypy treats ``None``
+    Effectively disables checking of :py:data:`~typing.Optional`
+    types and ``None`` values. With this option, mypy doesn't
+    generally check the use of ``None`` values -- it is treated
     as compatible with every type.
 
-    **Note:** This was False by default in mypy versions earlier than 0.600.
+    .. warning::
+
+        ``strict_optional = false`` is evil. Avoid using it and definitely do
+        not use it without understanding what it does.
 
 
 Configuring warnings

docs/source/error_codes.rst

@@ -19,22 +19,6 @@ Most error codes are shared between multiple related error messages.
 Error codes may change in future mypy releases.
 
 
-
-Displaying error codes
-----------------------
-
-Error codes are displayed by default.  Use :option:`--hide-error-codes <mypy --hide-error-codes>`
-or config ``hide_error_codes = True`` to hide error codes. Error codes are shown inside square brackets:
-
-.. code-block:: text
-
-   $ mypy prog.py
-   prog.py:1: error: "str" has no attribute "trim"  [attr-defined]
-
-It's also possible to require error codes for ``type: ignore`` comments.
-See :ref:`ignore-without-code<code-ignore-without-code>` for more information.
-
-
 .. _silence-error-codes:
 
 Silencing errors based on error codes
@@ -121,3 +105,10 @@ Similar logic works for disabling error codes globally. If a given error code
 is a subcode of another one, it will be mentioned in the documentation for the narrower
 code. This hierarchy is not nested: there cannot be subcodes of other
 subcodes.
+
+
+Requiring error codes
+---------------------
+
+It's possible to require error codes be specified in ``type: ignore`` comments.
+See :ref:`ignore-without-code<code-ignore-without-code>` for more information.

docs/source/existing_code.rst

@@ -31,8 +31,8 @@ invocation to your codebase, or adding your mypy invocation to
 existing tools you use to run tests, like ``tox``.
 
 * Make sure everyone runs mypy with the same options. Checking a mypy
-  :ref:`configuration file <config-file>` into your codebase can help
-  with this.
+  :ref:`configuration file <config-file>` into your codebase is the
+  easiest way to do this.
 
 * Make sure everyone type checks the same set of files. See
   :ref:`specifying-code-to-be-checked` for details.
@@ -48,7 +48,7 @@ A simple CI script could look something like this:
 
 .. code-block:: text
 
-    python3 -m pip install mypy==0.971
+    python3 -m pip install mypy==1.8
     # Run your standardised mypy invocation, e.g.
     mypy my_project
     # This could also look like `scripts/run_mypy.sh`, `tox run -e mypy`, `make mypy`, etc
@@ -74,6 +74,11 @@ You could even invert this, by setting ``ignore_errors = True`` in your global
 config section and only enabling error reporting with ``ignore_errors = False``
 for the set of modules you are ready to type check.
 
+The per-module configuration that mypy's configuration file allows can be
+extremely useful. Many configuration options can be enabled or disabled
+only for specific modules. In particular, you can also enable or disable
+various error codes on a per-module basis, see :ref:`error-codes`.
+
 Fixing errors related to imports
 --------------------------------
 
@@ -89,7 +94,7 @@ that it can't find, that don't have types, or don't have stub files:
 Sometimes these can be fixed by installing the relevant packages or
 stub libraries in the environment you're running ``mypy`` in.
 
-See :ref:`ignore-missing-imports` for a complete reference on these errors
+See :ref:`fix-missing-imports` for a complete reference on these errors
 and the ways in which you can fix them.
 
 You'll likely find that you want to suppress all errors from importing
@@ -118,13 +123,15 @@ codebase, use a config like this:
    ignore_missing_imports = True
 
 If you get a large number of errors, you may want to ignore all errors
-about missing imports, for instance by setting :confval:`ignore_missing_imports`
-to true globally. This can hide errors later on, so we recommend avoiding this
+about missing imports, for instance by setting
+:option:`--disable-error-code=import-untyped <mypy --ignore-missing-imports>`.
+or setting :confval:`ignore_missing_imports` to true globally.
+This can hide errors later on, so we recommend avoiding this
 if possible.
 
 Finally, mypy allows fine-grained control over specific import following
 behaviour. It's very easy to silently shoot yourself in the foot when playing
-around with these, so it's mostly recommended as a last resort. For more
+around with these, so this should be a last resort. For more
 details, look :ref:`here <follow-imports>`.
 
 Prioritise annotating widely imported modules

docs/source/kinds_of_types.rst

@@ -429,74 +429,6 @@ the runtime with some limitations (see :ref:`runtime_troubles`).
 
     t2: int | None  # equivalent to Optional[int]
 
-.. _no_strict_optional:
-
-Disabling strict optional checking
-**********************************
-
-Mypy also has an option to treat ``None`` as a valid value for every
-type (in case you know Java, it's useful to think of it as similar to
-the Java ``null``). In this mode ``None`` is also valid for primitive
-types such as ``int`` and ``float``, and :py:data:`~typing.Optional` types are
-not required.
-
-The mode is enabled through the :option:`--no-strict-optional <mypy --no-strict-optional>` command-line
-option. In mypy versions before 0.600 this was the default mode. You
-can enable this option explicitly for backward compatibility with
-earlier mypy versions, in case you don't want to introduce optional
-types to your codebase yet.
-
-It will cause mypy to silently accept some buggy code, such as
-this example -- it's not recommended if you can avoid it:
-
-.. code-block:: python
-
-   def inc(x: int) -> int:
-       return x + 1
-
-   x = inc(None)  # No error reported by mypy if strict optional mode disabled!
-
-However, making code "optional clean" can take some work! You can also use
-:ref:`the mypy configuration file <config-file>` to migrate your code
-to strict optional checking one file at a time, since there exists
-the per-module flag
-:confval:`strict_optional` to control strict optional mode.
-
-Often it's still useful to document whether a variable can be
-``None``. For example, this function accepts a ``None`` argument,
-but it's not obvious from its signature:
-
-.. code-block:: python
-
-    def greeting(name: str) -> str:
-        if name:
-            return f'Hello, {name}'
-        else:
-            return 'Hello, stranger'
-
-    print(greeting('Python'))  # Okay!
-    print(greeting(None))      # Also okay!
-
-You can still use :py:data:`Optional[t] <typing.Optional>` to document that ``None`` is a
-valid argument type, even if strict ``None`` checking is not
-enabled:
-
-.. code-block:: python
-
-    from typing import Optional
-
-    def greeting(name: Optional[str]) -> str:
-        if name:
-            return f'Hello, {name}'
-        else:
-            return 'Hello, stranger'
-
-Mypy treats this as semantically equivalent to the previous example
-if strict optional checking is disabled, since ``None`` is implicitly
-valid for any type, but it's much more
-useful for a programmer who is reading the code. This also makes
-it easier to migrate to strict ``None`` checking in the future.
-
 .. _type-aliases:
 
 Type aliases

docs/source/mypy_daemon.rst

@@ -61,7 +61,7 @@ you have a large codebase.)
 
 .. note::
 
-    The mypy daemon automatically enables ``--local-partial-types`` by default.
+    The mypy daemon requires ``--local-partial-types`` and automatically enables it.
 
 
 Daemon client commands

docs/source/running_mypy.rst

@@ -305,16 +305,20 @@ not catch errors in its use.
     The ``.*`` after ``foobar`` will ignore imports of ``foobar`` modules
     and subpackages in addition to the ``foobar`` top-level package namespace.
 
-3.  To suppress *all* missing import errors for *all* libraries in your codebase,
-    invoke mypy with the :option:`--ignore-missing-imports <mypy --ignore-missing-imports>` command line flag or set
-    the :confval:`ignore_missing_imports`
-    config file option to True
-    in the *global* section of your mypy config file::
+3.  To suppress *all* missing import errors for *all* untyped libraries
+    in your codebase, use :option:`--disable-error-code=import-untyped <mypy --ignore-missing-imports>`.
+    See :ref:`code-import-untyped` for more details on this error code.
+
+    You can also set :confval:`disable_error_code`, like so::
 
         [mypy]
-        ignore_missing_imports = True
+        disable_error_code = import-untyped
+
 
-    We recommend using this approach only as a last resort: it's equivalent
+    You can also set the :option:`--ignore-missing-imports <mypy --ignore-missing-imports>`
+    command line flag or set the :confval:`ignore_missing_imports` config file
+    option to True in the *global* section of your mypy config file. We
+    recommend avoiding ``--ignore-missing-imports`` if possible: it's equivalent
     to adding a ``# type: ignore`` to all unresolved imports in your codebase.
 
 
@@ -387,11 +391,11 @@ this error, try:
     installing into the environment you expect by running pip like
     ``python -m pip ...``.
 
-2.  Reading the :ref:`finding-imports` section below to make sure you
+3.  Reading the :ref:`finding-imports` section below to make sure you
     understand how exactly mypy searches for and finds modules and modify
     how you're invoking mypy accordingly.
 
-3.  Directly specifying the directory containing the module you want to
+4.  Directly specifying the directory containing the module you want to
     type check from the command line, by using the :confval:`mypy_path`
     or :confval:`files` config file options,
     or by using the ``MYPYPATH`` environment variable.

misc/test-stubgenc.sh

@@ -7,7 +7,7 @@ cd "$(dirname "$0")/.."
 
 # Install dependencies, demo project and mypy
 python -m pip install -r test-requirements.txt
-python -m pip install ./test-data/pybind11_mypy_demo
+python -m pip install ./test-data/pybind11_fixtures
 python -m pip install .
 
 EXIT=0
@@ -17,19 +17,29 @@ EXIT=0
 # everything else is passed to stubgen as its arguments
 function stubgenc_test() {
     # Remove expected stubs and generate new inplace
-    STUBGEN_OUTPUT_FOLDER=./test-data/pybind11_mypy_demo/$1
-    rm -rf "${STUBGEN_OUTPUT_FOLDER:?}/*"
+    STUBGEN_OUTPUT_FOLDER=./test-data/pybind11_fixtures/$1
+    rm -rf "${STUBGEN_OUTPUT_FOLDER:?}"
+
     stubgen -o "$STUBGEN_OUTPUT_FOLDER" "${@:2}"
 
+    # Check if generated stubs can actually be type checked by mypy
+    if ! mypy "$STUBGEN_OUTPUT_FOLDER";
+    then
+        echo "Stubgen test failed, because generated stubs failed to type check."
+        EXIT=1
+    fi
+
     # Compare generated stubs to expected ones
     if ! git diff --exit-code "$STUBGEN_OUTPUT_FOLDER";
     then
+        echo "Stubgen test failed, because generated stubs differ from expected outputs."
         EXIT=1
     fi
 }
 
 # create stubs without docstrings
-stubgenc_test stubgen -p pybind11_mypy_demo
+stubgenc_test expected_stubs_no_docs -p pybind11_fixtures
 # create stubs with docstrings
-stubgenc_test stubgen-include-docs -p pybind11_mypy_demo --include-docstrings
+stubgenc_test expected_stubs_with_docs -p pybind11_fixtures --include-docstrings
+
 exit $EXIT

mypy/applytype.py

@@ -93,7 +93,8 @@ def apply_generic_arguments(
     bound or constraints, instead of giving an error.
     """
     tvars = callable.variables
-    assert len(tvars) == len(orig_types)
+    min_arg_count = sum(not tv.has_default() for tv in tvars)
+    assert min_arg_count <= len(orig_types) <= len(tvars)
     # Check that inferred type variable values are compatible with allowed
     # values and bounds.  Also, promote subtype values to allowed values.
     # Create a map from type variable id to target type.