feat: json.dumps, json.dump accuracy improvements

[max-muoto]

This PR addresses #13781 by making json.dumps and json.dump more accurate:

• When default isn't provided, only JSON serializable types are supported (as accurate we can get, we'll go through trade-offs)
• When default is provided, the type must match that of the first call argument for default or what we consider JSON-serializable .

Trade-offs:

• For reasons described here Define a JSON type typing#182 (comment) less accurate Sequence and Mapping typing is needed to support nested types properly but this will obviously still be a big improvement over Any.
• As discussed here, TypedDicts are only compatible with Mapping[str, object] would it would be far preferable to use Mapping[str, _JSON] closing off typed dictionaries is going to lead to a noticeable number of false positives.
• Type-checking custom encoders isn't practical. Realistically to do this we would need to introduce generics on JSONEncoder (defaulting to our custom JSON type as the default) however it would be impractical for users to supply said generic param considering it doesn't exist at runtime. As a result an overload exists to accept object in the case that any custom JSONEncoder is provided.

[max-muoto]

CC: @srittau if you're interested in reviewing.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

spark (https://github.com/apache/spark)
+ python/pyspark/pandas/config.py:409: error: Argument 1 to "dumps" has incompatible type "Any | _NoValueType"; expected "_JSON"  [arg-type]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/io/parquet.py:191: error: Argument 1 to "dumps" has incompatible type "dict[Hashable, Any]"; expected "_JSON"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/cacheprovider.py:191: error: No overload variant of "dumps" matches argument types "object", "bool", "int"  [call-overload]
+ src/_pytest/cacheprovider.py:191: note: Possible overload variants:
+ src/_pytest/cacheprovider.py:191: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/_pytest/cacheprovider.py:191: note:     def [_T] dumps(obj: Union[_JSON, _T], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ src/_pytest/cacheprovider.py:191: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str

spack (https://github.com/spack/spack)
+ lib/spack/spack/util/spack_json.py:26: error: Unused "type: ignore" comment  [unused-ignore]
+ lib/spack/spack/util/spack_json.py:26: error: No overload variant of "dumps" matches argument types "dict[Any, Any]", "dict[str, tuple[str, str] | None]"  [call-overload]
+ lib/spack/spack/util/spack_json.py:26: note: Error code "call-overload" not covered by "type: ignore" comment
+ lib/spack/spack/util/spack_json.py:26: note: Possible overload variants:
+ lib/spack/spack/util/spack_json.py:26: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ lib/spack/spack/util/spack_json.py:26: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ lib/spack/spack/util/spack_json.py:26: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ lib/spack/spack/util/spack_json.py:27: error: Unused "type: ignore" comment  [unused-ignore]
+ lib/spack/spack/util/spack_json.py:27: error: No overload variant of "dump" matches argument types "dict[Any, Any]", "Any", "dict[str, tuple[str, str] | None]"  [call-overload]
+ lib/spack/spack/util/spack_json.py:27: note: Error code "call-overload" not covered by "type: ignore" comment
+ lib/spack/spack/util/spack_json.py:27: note: Possible overload variants:
+ lib/spack/spack/util/spack_json.py:27: note:     def dump(obj: _JSON, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> None
+ lib/spack/spack/util/spack_json.py:27: note:     def [_T] dump(obj: _JSON | _T, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder] | None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> None
+ lib/spack/spack/util/spack_json.py:27: note:     def dump(obj: object, fp: SupportsWrite[str], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> None

strawberry (https://github.com/strawberry-graphql/strawberry)
+ strawberry/http/base.py:51: error: No overload variant of "dumps" matches argument type "object"  [call-overload]
+ strawberry/http/base.py:51: note: Possible overload variants:
+ strawberry/http/base.py:51: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ strawberry/http/base.py:51: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ strawberry/http/base.py:51: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str

scrapy (https://github.com/scrapy/scrapy)
+ scrapy/commands/settings.py:54: error: Argument 1 to "dumps" has incompatible type "dict[bool | float | int | str | None, Any]"; expected "_JSON"  [arg-type]

cki-lib (https://gitlab.com/cki-project/cki-lib)
+ cki_lib/owners.py:249: error: Argument "key" to "sorted" has incompatible type "Callable[[Mapping[str, object] | Sequence[_JSON] | float | bool | None], object | Any]"; expected "Callable[[Mapping[str, object] | Sequence[_JSON] | float | bool | None], SupportsDunderLT[Any] | SupportsDunderGT[Any]]"  [arg-type]
+ cki_lib/owners.py:249: error: No overload variant of "__getitem__" of "Sequence" matches argument type "str"  [call-overload]
+ cki_lib/owners.py:249: note: Possible overload variants:
+ cki_lib/owners.py:249: note:     def __getitem__(self, int, /) -> _JSON
+ cki_lib/owners.py:249: note:     def __getitem__(self, slice[Any, Any, Any], /) -> Sequence[_JSON]
+ cki_lib/owners.py:249: error: Value of type "Mapping[str, object] | Sequence[_JSON] | float | bool | None" is not indexable  [index]
+ cki_lib/owners.py:249: error: Incompatible return value type (got "object | Any", expected "SupportsDunderLT[Any] | SupportsDunderGT[Any]")  [return-value]

poetry (https://github.com/python-poetry/poetry)
+ src/poetry/utils/cache.py:181: error: No overload variant of "dumps" matches argument type "T"  [call-overload]
+ src/poetry/utils/cache.py:181: note: Possible overload variants:
+ src/poetry/utils/cache.py:181: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:181: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:181: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:211: error: No overload variant of "dumps" matches argument types "object", "bool", "tuple[str, str]", "bool"  [call-overload]
+ src/poetry/utils/cache.py:211: note: Possible overload variants:
+ src/poetry/utils/cache.py:211: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:211: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ src/poetry/utils/cache.py:211: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str

mypy (https://github.com/python/mypy)
+ mypy/util.py:935: error: Returning Any from function declared to return "bytes"  [no-any-return]
+ mypy/util.py:935: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-no-any-return for more info
+ mypy/util.py:935: error: No overload variant of "dumps" matches argument types "object", "int", "bool"  [call-overload]
+ mypy/util.py:935: note: Possible overload variants:
+ mypy/util.py:935: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:935: note:     def [_T] dumps(obj: Union[_JSON, _T], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:935: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:935: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-call-overload for more info
+ mypy/util.py:938: error: Returning Any from function declared to return "bytes"  [no-any-return]
+ mypy/util.py:938: error: No overload variant of "dumps" matches argument types "object", "bool", "tuple[str, str]"  [call-overload]
+ mypy/util.py:938: note: Possible overload variants:
+ mypy/util.py:938: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:938: note:     def [_T] dumps(obj: Union[_JSON, _T], *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ mypy/util.py:938: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: Union[int, str, None] = ..., separators: Optional[tuple[str, str]] = ..., default: Optional[Callable[[Any], Any]] = ..., sort_keys: bool = ..., **kwds: Any) -> str

zulip (https://github.com/zulip/zulip)
+ zerver/lib/markdown/api_arguments_table_generator.py:170: error: No overload variant of "dumps" matches argument type "object"  [call-overload]
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note: Possible overload variants:
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/lib/markdown/api_arguments_table_generator.py:170: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:243: error: No overload variant of "dumps" matches argument types "object", "bool"  [call-overload]
+ zerver/openapi/markdown_extension.py:243: note: Possible overload variants:
+ zerver/openapi/markdown_extension.py:243: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:243: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:243: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:258: error: No overload variant of "dumps" matches argument type "object"  [call-overload]
+ zerver/openapi/markdown_extension.py:258: note: Possible overload variants:
+ zerver/openapi/markdown_extension.py:258: note:     def dumps(obj: _JSON, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:258: note:     def [_T] dumps(obj: _JSON | _T, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: None = ..., indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[_T], _JSON], sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:258: note:     def dumps(obj: object, *, skipkeys: bool = ..., ensure_ascii: bool = ..., check_circular: bool = ..., allow_nan: bool = ..., cls: type[JSONEncoder], indent: int | str | None = ..., separators: tuple[str, str] | None = ..., default: Callable[[Any], Any] | None = ..., sort_keys: bool = ..., **kwds: Any) -> str
+ zerver/openapi/markdown_extension.py:264: error: Incompatible return value type (got "object", expected "str")  [return-value]
+ zerver/lib/bot_lib.py:47: error: Incompatible types in assignment (expression has type overloaded function, variable has type "Callable[[object], str]")  [assignment]

[JelleZijlstra]

I don't think we can do this, it will lead to too many false positives.