Add stub files for pymongo and bson

[pcorpet]

This is a follow up to #346 that was closed inadvertently.

It implements what was decided in this JIRA ticket: add type annotations in stubs according to PEP 484.

I took the work of @GrierPhillips and tried to answer some of the comments on the previous PR:

• removed internals and private methods and functions
• added type annotations for the bson package as well, as pymongo references it sometimes
• cleaned up the styling to follow typeshed's standards

I did not find a way to automatically test the stubs against real code as this is ongoing work by the mypy team. However I copied the .pyi files in typeshed and ran the tests there that helped me find and fix many issues.

Also I did not update pymongo with updates from version after 3.5 (Phillips' work).

---

This change is 

[ShaneHarvey]

Awesome! It looks like this fixes a lot of mypy errors:

$ git checkout master
$ mypy pymongo | wc -l
      48
$ git checkout pcorpet-typing
$ mypy pymongo | wc -l
      10

The remaining issues I see are:

$ mypy pymongo
pymongo/monotonic.py:25: error: Cannot find module named 'monotime'
pymongo/monotonic.py:31: error: Cannot find module named 'monotonic'
pymongo/saslprep.py:48: error: All conditional function variants must have identical signatures
pymongo/compression_support.py:18: error: Cannot find module named 'snappy'
pymongo/compression_support.py:18: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
pymongo/compression_support.py:31: error: Module 'pymongo.monitoring' has no attribute '_SENSITIVE_COMMANDS'
pymongo/uri_parser.py:21: error: No library stub file for module 'dns'
pymongo/uri_parser.py:21: note: (Stub files are from https://github.com/python/typeshed)
pymongo/uri_parser.py:33: error: Module 'pymongo.common' has no attribute 'URI_OPTIONS_DEPRECATION_MAP'
pymongo/uri_parser.py:33: error: Module 'pymongo.common' has no attribute 'INTERNAL_URI_OPTION_NAME_MAP'

Some of these issues can be fixed by adding a # type: ignore comment to the missing imports: https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports and others are simply because things have changes since pymongo 3.5.

Can you also include the new files in the pypi package so we can become PEP-561 compatible? I think we just need to add this to setup.py:

setup(
...
    package_data={
        'pymongo': ['pymongo/py.typed', 'pymongo/*.pyi'],
        'bson': ['bson/py.typed', 'bson/*.pyi']},

I did not find a way to automatically test the stubs against real code as this is ongoing work by the mypy team. However I copied the .pyi files in typeshed and ran the tests there that helped me find and fix many issues.

I think we can at least test with mypy. I'll open a new issue for this because I don't want to hold up the PR but for now can we try to get mypy to pass locally when run on the pymongo/ and bson/ directories $ mypy bson pymongo?

[pcorpet]

Thanks for the reminder for the package_data, I did want to do that.

About the issues that you've uncovered: running mypy on the folder as such does actually weird stuff we're it's checking the .pyi files when they exist and fall backs at checking the .py files when there are no pyi. I've solved the remaining issues by adding the missing .pyi files. But it's not checking much unfortunately.

I've added a lot of the 3.6 and 3.7 new stuff (array_filters, session, watch). However I do not have a good way of checking what's missing yet. So we might play catchup with some of the library's users that are using both typing and newest features.

[behackett]

Just a quick update to anyone watching this PR, this is on our list to tackle soon. We haven't forgotten and know a lot of users are interested in it.

[Hubro]

@behackett What about now, is it still not forgotten? :)

I'm working on a new project using PyMongo and the lack of type hints or stub files is pretty annoying.

[behackett]

It's at the top of our list post MongoDB 4.4 which is rapidly approaching.

[scharf]

what's the status of this?

[scharf]

I created a gist with the subs, so the can be downloaded and used directly without waiting until this PR is merged.

[ShaneHarvey]

I'd like to provide an update as this project was just completed. While working on this project our team realized that we would be more comfortable following a more conservative approach to adding type hints. Instead of including type annotations or stub files directly in pymongo we decided to ship stub files as a separate package:  pymongo-stubs.

pymongo-stubs' source code is hosted on GitHub (mongodb-labs/pymongo-stubs). The initial release is compatible with PyMongo >=3.11,<4.0 and Python >=3.6. @pcorpet you'll notice that you are already a contributor because I merged these changes as a starting point for the new project. Thank you for your contribution!

Our longer term plan is to get feedback on the type hints and later (in PyMongo 4.0) incorporate them directly into the codebase as inline type annotations (in PYTHON-2432).

For now you are welcome to try out the stubs like this:

python3 -m pip install pymongo-stubs
python3 -m pip install mypy
mypy <codebase that uses pymongo>

Or simply install pymongo-stubs to get a better IDE experience: