Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stubtest: verify the contents of __all__ in a stub #12214

Merged
merged 14 commits into from
Aug 1, 2022

Conversation

AlexWaygood
Copy link
Member

Description

Fixes python/typeshed#7284.

Stubtest currently does not check for the presence of __all__ in a stub. Moreover, even if __all__ is defined in the stub, stubtest does not currently check that the stub's __all__ matches the runtime's __all__. This PR fixes that!

Test Plan

I've modified an existing test case and added a new one.

New errors reported by stubtest when checking typeshed, with this patch applied.

There are... 344 new errors reported by stubtest, with this PR applied, so it may not yet be mergeable. I've attached the full list of errors as a .txt file.

all_discrepancies.txt

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Feb 20, 2022

(I'll mark this as draft until I've whittled down the typeshed hits to a more acceptable number.)

Following commit 859df5a, which reduces the number of duplicate errors if a stub is missing __all__, there are now 154 new errors reported by stubtest with this patch applied, when checking the typeshed master branch. (This is also following several typeshed PRs over the last few days.)

@AlexWaygood AlexWaygood marked this pull request as draft February 20, 2022 16:09
@github-actions

This comment has been minimized.

@github-actions
Copy link
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

@AlexWaygood AlexWaygood marked this pull request as ready for review February 28, 2022 22:54
@AlexWaygood
Copy link
Member Author

AlexWaygood commented Feb 28, 2022

After many typeshed PRs, merging this PR would now produce only the following 8 hits when run on my Windows machine (Python 3.10):

+ asyncio.__all__ is not present in stub
+ asyncio.windows_utils module: names exported from the stub do not correspond to the names exported at runtime.
+ Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.
+ distutils.command.__all__ is not present in stub
+ email.__all__ is not present in stub
+ locale module: names exported from the stub do not correspond to the names exported at runtime.
+ Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.
+ os.__all__ is not present in stub
+ socket.__all__ is not present in stub
+ xml.__all__ is not present in stub
Here's the more verbose versions of these errors
error: asyncio.__all__ is not present in stub
Stub:
MISSING
Runtime:
('BaseEventLoop', 'coroutine', 'iscoroutinefunction', 'iscoroutine', 'AbstractEventLoopPolicy', 'AbstractEventLoop', 'AbstractServer', 'Handle', 'TimerHandle', 'get_event_loop_policy', 'set_event_loop_policy', 'get_event_loop', 'set_event_loop', 'new_event_loop', 'get_child_watcher', 'set_child_watcher', '_set_running_loop', 'get_running_loop', '_get_running_loop', 'CancelledError', 'InvalidStateError', 'TimeoutError', 'IncompleteReadError', 'LimitOverrunError', 'SendfileNotAvailableError', 'Future', 'wrap_future', 'isfuture', 'Lock', 'Event', 'Condition', 'Semaphore', 'BoundedSemaphore', 'BaseProtocol', 'Protocol', 'DatagramProtocol', 'SubprocessProtocol', 'BufferedProtocol', 'run', 'Queue', 'PriorityQueue', 'LifoQueue', 'QueueFull', 'QueueEmpty', 'StreamReader', 'StreamWriter', 'StreamReaderProtocol', 'open_connection', 'start_server', 'create_subprocess_exec', 'create_subprocess_shell', 'Task', 'create_task', 'FIRST_COMPLETED', 'FIRST_EXCEPTION', 'ALL_COMPLETED', 'wait', 'wait_for', 'as_completed', 'sleep', 'gather', 'shield', 'ensure_future', 'run_coroutine_threadsafe', 'current_task', 'all_tasks', '_register_task', '_unregister_task', '_enter_task', '_leave_task', 'to_thread', 'BaseTransport', 'ReadTransport', 'WriteTransport', 'Transport', 'DatagramTransport', 'SubprocessTransport', 'SelectorEventLoop', 'ProactorEventLoop', 'IocpProactor', 'DefaultEventLoopPolicy', 'WindowsSelectorEventLoopPolicy', 'WindowsProactorEventLoopPolicy')

error: asyncio.windows_utils module: names exported from the stub do not correspond to the names exported at runtime.
Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.
Stub:
Names exported in the stub but not at runtime: []
Runtime:
Names exported at runtime but not in the stub: ['Popen']

error: distutils.command.__all__ is not present in stub
Stub:
MISSING
Runtime:
['build', 'build_py', 'build_ext', 'build_clib', 'build_scripts', 'clean', 'install', 'install_lib', 'install_headers', 'install_scripts', 'install_data', 'sdist', 'register', 'bdist', 'bdist_dumb', 'bdist_rpm', 'check', 'upload']

error: email.__all__ is not present in stub
Stub:
MISSING
Runtime:
['base64mime', 'charset', 'encoders', 'errors', 'feedparser', 'generator', 'header', 'iterators', 'message', 'message_from_file', 'message_from_binary_file', 'message_from_string', 'message_from_bytes', 'mime', 'parser', 'quoprimime', 'utils']

error: locale module: names exported from the stub do not correspond to the names exported at runtime.
Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.
Stub:
Names exported in the stub but not at runtime: ['LC_MESSAGES']
Runtime:
Names exported at runtime but not in the stub: []

error: os.__all__ is not present in stub
Stub:
MISSING
Runtime:
['altsep', 'curdir', 'pardir', 'sep', 'pathsep', 'linesep', 'defpath', 'name', 'path', 'devnull', 'SEEK_SET', 'SEEK_CUR', 'SEEK_END', 'fsencode', 'fsdecode', 'get_exec_path', 'fdopen', 'extsep', '_exit', 'DirEntry', 'F_OK', 'O_APPEND', 'O_BINARY', 'O_CREAT', 'O_EXCL', 'O_NOINHERIT', 'O_RANDOM', 'O_RDONLY', 'O_RDWR', 'O_SEQUENTIAL', 'O_SHORT_LIVED', 'O_TEMPORARY', 'O_TEXT', 'O_TRUNC', 'O_WRONLY', 'P_DETACH', 'P_NOWAIT', 'P_NOWAITO', 'P_OVERLAY', 'P_WAIT', 'R_OK', 'TMP_MAX', 'W_OK', 'X_OK', 'abort', 'access', 'chdir', 'chmod', 'close', 'closerange', 'cpu_count', 'device_encoding', 'dup', 'dup2', 'environ', 'error', 'execv', 'execve', 'fspath', 'fstat', 'fsync', 'ftruncate', 'get_handle_inheritable', 'get_inheritable', 'get_terminal_size', 'getcwd', 'getcwdb', 'getlogin', 'getpid', 'getppid', 'isatty', 'kill', 'link', 'listdir', 'lseek', 'lstat', 'mkdir', 'open', 'pipe', 'putenv', 'read', 'readlink', 'remove', 'rename', 'replace', 'rmdir', 'scandir', 'set_handle_inheritable', 'set_inheritable', 'spawnv', 'spawnve', 'startfile', 'stat', 'stat_result', 'statvfs_result', 'strerror', 'symlink', 'system', 'terminal_size', 'times', 'times_result', 'truncate', 'umask', 'uname_result', 'unlink', 'unsetenv', 'urandom', 'utime', 'waitpid', 'waitstatus_to_exitcode', 'write', 'makedirs', 'removedirs', 'renames', 'walk', 'execl', 'execle', 'execlp', 'execlpe', 'execvp', 'execvpe', 'getenv', 'supports_bytes_environ', 'spawnl', 'spawnle', 'popen']

error: socket.__all__ is not present in stub
Stub:
MISSING
Runtime:
['fromfd', 'getfqdn', 'create_connection', 'create_server', 'has_dualstack_ipv6', 'AddressFamily', 'SocketKind', 'AF_APPLETALK', 'AF_BLUETOOTH', 'AF_DECnet', 'AF_INET', 'AF_INET6', 'AF_IPX', 'AF_IRDA', 'AF_LINK', 'AF_SNA', 'AF_UNSPEC', 'AI_ADDRCONFIG', 'AI_ALL', 'AI_CANONNAME', 'AI_NUMERICHOST', 'AI_NUMERICSERV', 'AI_PASSIVE', 'AI_V4MAPPED', 'BDADDR_ANY', 'BDADDR_LOCAL', 'BTPROTO_RFCOMM', 'CAPI', 'EAI_AGAIN', 'EAI_BADFLAGS', 'EAI_FAIL', 'EAI_FAMILY', 'EAI_MEMORY', 'EAI_NODATA', 'EAI_NONAME', 'EAI_SERVICE', 'EAI_SOCKTYPE', 'INADDR_ALLHOSTS_GROUP', 'INADDR_ANY', 'INADDR_BROADCAST', 'INADDR_LOOPBACK', 'INADDR_MAX_LOCAL_GROUP', 'INADDR_NONE', 'INADDR_UNSPEC_GROUP', 'IPPORT_RESERVED', 'IPPORT_USERRESERVED', 'IPPROTO_AH', 'IPPROTO_CBT', 'IPPROTO_DSTOPTS', 'IPPROTO_EGP', 'IPPROTO_ESP', 'IPPROTO_FRAGMENT', 'IPPROTO_GGP', 'IPPROTO_HOPOPTS', 'IPPROTO_ICLFXBM', 'IPPROTO_ICMP', 'IPPROTO_ICMPV6', 'IPPROTO_IDP', 'IPPROTO_IGMP', 'IPPROTO_IGP', 'IPPROTO_IP', 'IPPROTO_IPV4', 'IPPROTO_IPV6', 'IPPROTO_L2TP', 'IPPROTO_MAX', 'IPPROTO_ND', 'IPPROTO_NONE', 'IPPROTO_PGM', 'IPPROTO_PIM', 'IPPROTO_PUP', 'IPPROTO_RAW', 'IPPROTO_RDP', 'IPPROTO_ROUTING', 'IPPROTO_SCTP', 'IPPROTO_ST', 'IPPROTO_TCP', 'IPPROTO_UDP', 'IPV6_CHECKSUM', 'IPV6_DONTFRAG', 'IPV6_HOPLIMIT', 'IPV6_HOPOPTS', 'IPV6_JOIN_GROUP', 'IPV6_LEAVE_GROUP', 'IPV6_MULTICAST_HOPS', 'IPV6_MULTICAST_IF', 'IPV6_MULTICAST_LOOP', 'IPV6_PKTINFO', 'IPV6_RECVRTHDR', 'IPV6_RECVTCLASS', 'IPV6_RTHDR', 'IPV6_TCLASS', 'IPV6_UNICAST_HOPS', 'IPV6_V6ONLY', 'IP_ADD_MEMBERSHIP', 'IP_DROP_MEMBERSHIP', 'IP_HDRINCL', 'IP_MULTICAST_IF', 'IP_MULTICAST_LOOP', 'IP_MULTICAST_TTL', 'IP_OPTIONS', 'IP_RECVDSTADDR', 'IP_RECVTOS', 'IP_TOS', 'IP_TTL', 'MSG_BCAST', 'MSG_CTRUNC', 'MSG_DONTROUTE', 'MSG_ERRQUEUE', 'MSG_MCAST', 'MSG_OOB', 'MSG_PEEK', 'MSG_TRUNC', 'MSG_WAITALL', 'NI_DGRAM', 'NI_MAXHOST', 'NI_MAXSERV', 'NI_NAMEREQD', 'NI_NOFQDN', 'NI_NUMERICHOST', 'NI_NUMERICSERV', 'RCVALL_MAX', 'RCVALL_OFF', 'RCVALL_ON', 'RCVALL_SOCKETLEVELONLY', 'SHUT_RD', 'SHUT_RDWR', 'SHUT_WR', 'SIO_KEEPALIVE_VALS', 'SIO_LOOPBACK_FAST_PATH', 'SIO_RCVALL', 'SOCK_DGRAM', 'SOCK_RAW', 'SOCK_RDM', 'SOCK_SEQPACKET', 'SOCK_STREAM', 'SOL_IP', 'SOL_SOCKET', 'SOL_TCP', 'SOL_UDP', 'SOMAXCONN', 'SO_ACCEPTCONN', 'SO_BROADCAST', 'SO_DEBUG', 'SO_DONTROUTE', 'SO_ERROR', 'SO_EXCLUSIVEADDRUSE', 'SO_KEEPALIVE', 'SO_LINGER', 'SO_OOBINLINE', 'SO_RCVBUF', 'SO_RCVLOWAT', 'SO_RCVTIMEO', 'SO_REUSEADDR', 'SO_SNDBUF', 'SO_SNDLOWAT', 'SO_SNDTIMEO', 'SO_TYPE', 'SO_USELOOPBACK', 'SocketType', 'TCP_FASTOPEN', 'TCP_KEEPCNT', 'TCP_KEEPIDLE', 'TCP_KEEPINTVL', 'TCP_MAXSEG', 'TCP_NODELAY', 'close', 'dup', 'error', 'gaierror', 'getaddrinfo', 'getdefaulttimeout', 'gethostbyaddr', 'gethostbyname', 'gethostbyname_ex', 'gethostname', 'getnameinfo', 'getprotobyname', 'getservbyname', 'getservbyport', 'has_ipv6', 'herror', 'htonl', 'htons', 'if_indextoname', 'if_nameindex', 'if_nametoindex', 'inet_aton', 'inet_ntoa', 'inet_ntop', 'inet_pton', 'ntohl', 'ntohs', 'setdefaulttimeout', 'socket', 'timeout', 'errorTab', 'fromshare', 'socketpair']

error: xml.__all__ is not present in stub
Stub:
MISSING
Runtime:
['dom', 'parsers', 'sax', 'etree']

Notes on the remaining hits:

  • asyncio, os, socket: __all__ for these modules is highly dynamic; trying to type these statically is a bit of a fool's errand in my opinion.
  • asyncio.windows_utils: the error here is due to Popen being declared as a public attribute in __all__ at runtime, but being missing from the stub. This will be fixed by Add asyncio.windows_utils.Popen typeshed#7396
  • distutils.command, email, xml: __all__ is deliberately omitted for these modules due to xml package suggests submodules being available ... they're not typeshed#6523
  • locale: The error here is due to LC_MESSAGES apparently not being available on my system, meaning that it is declared in __all__ but not actually exported. It's not entirely clear exactly which systems it is and isn't available for (typeshed currently defines it in the module unconditionally). So, I don't really know what to do about that one.

@AlexWaygood
Copy link
Member Author

@hauntsaninja, could you possibly try running this patch on Linux, to see if there are any unix-only modules that I've missed? :)

@AlexWaygood
Copy link
Member Author

Note that this PR shouldn't have a significant effect on checking third-party stubs in typeshed:

  • The new check that tests whether publicly exported symbols in the stub are the same as at runtime only runs if the stub defines __all__.
  • stubtest won't error for third-party stubs not defining __all__, because it will treat __all__ just like any other symbol mixing from a stub. For third-party stubs, typeshed runs stubtest with the --ignore-missing-stub option.

@Akuli
Copy link
Contributor

Akuli commented Apr 9, 2022

On my linux system, comparing the output of python3 -m mypy.stubtest --concise --check-typeshed --custom-typeshed-dir ~/typeshed on master and this PR (merged with master) gives:

+_pydecimal.__all__ is not present in stub
+asyncio.__all__ is not present in stub
+asyncio.base_events module: names exported from the stub do not correspond to the names exported at runtime.
+(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
+distutils.command.__all__ is not present in stub
+email.__all__ is not present in stub
+os.__all__ is not present in stub
+poplib module: names exported from the stub do not correspond to the names exported at runtime.
+(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
+socket.__all__ is not present in stub
+tty.__all__ is not present in stub
+typing module: names exported from the stub do not correspond to the names exported at runtime.
+(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
+typing_extensions.__all__ is not present in stub
+xml.__all__ is not present in stub

@AlexWaygood
Copy link
Member Author

On my linux system, comparing the output of python3 -m mypy.stubtest --concise --check-typeshed --custom-typeshed-dir ~/typeshed on master and this PR (merged with master) gives:

+_pydecimal.__all__ is not present in stub
+asyncio.__all__ is not present in stub
+asyncio.base_events module: names exported from the stub do not correspond to the names exported at runtime.
+(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
+distutils.command.__all__ is not present in stub
+email.__all__ is not present in stub
+os.__all__ is not present in stub
+poplib module: names exported from the stub do not correspond to the names exported at runtime.
+(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
+socket.__all__ is not present in stub
+tty.__all__ is not present in stub
+typing module: names exported from the stub do not correspond to the names exported at runtime.
+(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
+typing_extensions.__all__ is not present in stub
+xml.__all__ is not present in stub

Thanks! I'm travelling for the next few days, but I'll try to look at fixing some of these when I have the time.

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Apr 13, 2022

@Akuli, could you possibly tell me which Python version you're running on, and give me the full results for asyncio.base_events and typing (by "full results", I mean without the --concise argument)? For all the others, I can either reproduce them locally or can see what the problem is from looking at the source code, but I'm having trouble with those two.

@Akuli
Copy link
Contributor

Akuli commented Apr 13, 2022

Python 3.9.2

The typing error is:

error: typing module: names exported from the stub do not correspond to the names exported at runtime.
(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being declared in `__all__` but not actually defined in the stub.)
Stub:
Names exported in the stub but not at runtime: ['BinaryIO', 'IO', 'Match', 'Pattern', 'TextIO']
Runtime:
Names exported at runtime but not in the stub: []

There's several errors coming from asyncio.base_events because I'm running stubtest without typeshed's allowlists. I didn't bother removing the errors that also happen on mypy's master.

All asyncio.base_events errors
error: asyncio.base_events module: names exported from the stub do not correspond to the names exporte
d at runtime.
(Note: This is probably either due to an inaccurate `__all__` in the stub, or due to a name being decl
ared in `__all__` but not actually defined in the stub.)
Stub:
Names exported in the stub but not at runtime: ['Server']
Runtime:
Names exported at runtime but not in the stub: []

error: asyncio.base_events.BaseEventLoop.subprocess_exec is inconsistent, runtime argument "shell" has
 a default value of type Literal[False], which is incompatible with stub argument type Literal[True]
Stub: at line 362
def [_ProtocolT <: asyncio.protocols.BaseProtocol] (self: asyncio.base_events.BaseEventLoop, protocol_
factory: def () -> _ProtocolT`-1, program: Any, *args: Any, *, stdin: Union[builtins.int, typing.IO[An
y], None] =, stdout: Union[builtins.int, typing.IO[Any], None] =, stderr: Union[builtins.int, typing.I
O[Any], None] =, universal_newlines: Literal[False] =, shell: Literal[True] =, bufsize: Literal[0] =, 
encoding: None =, errors: None =, **kwargs: Any) -> typing.Coroutine[Any, Any, Tuple[asyncio.transport
s.SubprocessTransport, _ProtocolT`-1]]
Runtime: at line 1634 in file /usr/lib/python3.9/asyncio/base_events.py
async def (self, protocol_factory, program, *args, stdin=-1, stdout=-1, stderr=-1, universal_newlines=
False, shell=False, bufsize=0, encoding=None, errors=None, text=None, **kwargs)

error: asyncio.base_events.BaseEventLoop.subprocess_exec is inconsistent, stub does not have argument "text"
Stub: at line 362
def [_ProtocolT <: asyncio.protocols.BaseProtocol] (self: asyncio.base_events.BaseEventLoop, protocol_factory: def () -> _ProtocolT`-1, program: Any, *args: Any, *, stdin: Union[builtins.int, typing.IO[Any], None] =, stdout: Union[builtins.int, typing.IO[Any], None] =, stderr: Union[builtins.int, typing.IO[Any], None] =, universal_newlines: Literal[False] =, shell: Literal[True] =, bufsize: Literal[0] =, encoding: None =, errors: None =, **kwargs: Any) -> typing.Coroutine[Any, Any, Tuple[asyncio.transports.SubprocessTransport, _ProtocolT`-1]]
Runtime: at line 1634 in file /usr/lib/python3.9/asyncio/base_events.py
async def (self, protocol_factory, program, *args, stdin=-1, stdout=-1, stderr=-1, universal_newlines=False, shell=False, bufsize=0, encoding=None, errors=None, text=None, **kwargs)

error: asyncio.base_events.__all__ variable differs from runtime type Tuple[Literal['BaseEventLoop']]
Stub: at line 18
Tuple[builtins.str, builtins.str]
Runtime:
('BaseEventLoop',)

@AlexWaygood
Copy link
Member Author

AlexWaygood commented Apr 13, 2022

The typing errors are because you're using 3.9.2, and those 5 classes were added to __all__ in 3.9.6 (python/cpython@00726e5).

The asyncio.base_events __all__ errors are because you're using 3.9.2, and Server was added to __all__ in 3.9.11 (python/cpython@20e88f7).

So I believe all the linux-only hits should be fixed by python/typeshed#7618.

Thanks very much for your help!

@hauntsaninja hauntsaninja self-assigned this Jun 7, 2022
AlexWaygood added a commit to python/typeshed that referenced this pull request Jun 8, 2022
It doesn't exist at runtime:

```python
>>> import dateparser.date
>>> dateparser.date.Self
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'dateparser.date' has no attribute 'Self'
```

...and it's not used in the stub either.

I'm not sure why stubtest didn't pick it up. My stubtest patch over at python/mypy#12214 _does_ pick it up (and I'm not sure why that is, either... but it's a true positive!)
Copy link
Collaborator

@hauntsaninja hauntsaninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is a great change!

mypy/test/teststubtest.py Outdated Show resolved Hide resolved
mypy/stubtest.py Outdated Show resolved Hide resolved
@AlexWaygood AlexWaygood changed the title Stubtest: verify the presence and contents of __all__ in a stub Stubtest: verify the contents of __all__ in a stub Jul 31, 2022
Copy link
Collaborator

@hauntsaninja hauntsaninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you again, this is super useful functionality

@hauntsaninja
Copy link
Collaborator

Filed #13300 for checking the presence :-)

@AlexWaygood AlexWaygood deleted the stubtest-all branch August 1, 2022 07:04
@AlexWaygood
Copy link
Member Author

Thank you again, this is super useful functionality

Thank you for the review and merge!! Excited to see this go in 😀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

__all__ in a stub will easily go out of date
3 participants