[python-package] type hints in python package

[jameslamb]

Request for Comment

#3581 officially ended support for Python 2 in the Python package.

Now that Python 2 support has been ended, I'd like to propose adding type hints to the library (https://docs.python.org/3/library/typing.html).

Benefits of adding type hints:

• serves as additional documentation about the types of arguments and return types of functions
• allows us to enable mypy, a static type checker for Python that will catch issues like "this function call can return None but you're accessing a property of the result as if it isn't None"
• allows other libraries that integrate with LightGBM (like optuna) to catch issues in their own code using mypy

I'm opening this Request for Comment to get thoughts on this narrow proposal:

would you support a PR where I add type hints to the code in the https://github.com/microsoft/LightGBM/blob/master/python-package/lightgbm/dask.py?

Status: Accepted

If maintainers agree to this proposal, we could add a "good first issue" about adding hints for other modules and could talk about mypy separately, at some point in the future.

This was originally opened as a request for comment, but it's been accepted and we're now accepting contributions to add type hints in the Python package!

How to contribute

Anyone is welcome to submit a pull request adding type hints to places in the Python package that are missing them!

Please limit pull requests to 1 function/method per pull request, to make them easier to review.

• .ci/get_workflow_status.py (shouldn't be touched because this file is executed by default Python 2 installed on GitHub Actions workers)
• docs/conf.py ([python] add type hints in docs/conf.py #4526)
• helpers/check_dynamic_dependencies.py ([python] add type hints to check_dynamic_dependencies.py #4382)
• helpers/parameter_generator.py ([python] Add type hints to helpers/parameter_generator.py #4474)
• python-package/lightgbm/basic.py
• python-package/lightgbm/callback.py ([python-package] Add type hints to the callback file #4093)
• python-package/lightgbm/compat.py
• python-package/lightgbm/dask.py ([dask] Add type hints in Dask package #3866)
• python-package/lightgbm/engine.py
• python-package/lightgbm/libpath.py ([python-package] Add type hint to the libpath file #4070)
• python-package/lightgbm/plotting.py ([python] Add type hints to python-package/lightgbm/plotting.py #4367)
• python-package/lightgbm/sklearn.py
• python-package/setup.py ([python] add type hints to python-package/setup.py #4376)

Handling line breaks

If adding hints to a method or function greatly increases the length of its lines, break each argument onto a separate line and have the closing parenthesis begin its own line, vertically aligned with the keyword def. Like this:

# before
def make_ranking(n_samples=100, n_features=20, n_informative=5, gmax=2,
                 group=None, random_gs=False, avg_gs=10, random_state=0):

# after
def make_ranking(
    n_samples: int = 100,
    n_features: int = 20,
    n_informative: int = 5,
    gmax: int = 2,
    group: Optional[List[int]] = None,
    random_gs: bool = False,
    avg_gs: int = 10,
    random_state: int = 0
) -> Tuple[pd.DataFrame, pd.DataFrame]:

references that may help you

• pull request that added type hints in python-package/lightgbm/dask.py: [dask] Add type hints in Dask package #3866
• Python official docs on typing: https://docs.python.org/3/library/typing.html
• related issue looking for help with fixing mypy warnings: [ci] add mypy to linting task #3868

How this improves LightGBM

Type hints in the code serve as additional documentation of the expected input and output types for functions and methods.

Having richer type hints can help static analyzers like mypy catch bugs that otherwise might not be caught by LightGBM's unit tests.

[StrikerRUS]

I'm +1.
This shouldn't be hard to add type hints, because for all public API functions (and even some internal ones) all types are already specified in docstrings.

[trivialfis]

xgboost has been adding type hints gradually and the dask module is completely typed. From my experience it's quite useful in both documentation and type checks.

[jameslamb]

I did the following this morning to make this more friendly for outside contributors:

• replaced "[RFC]" in the title with "[python-package]"
• removed label "question"
• removed assignees
• added more details in the description to help contributors looking to submit PRs that add type hints

Anyone reading this is welcome to contribute! Thanks for helping to improve LightGBM.

[kunjshukla]

I would like to take this issue up if it's not closed

[jameslamb]

@kunjshukla sure, pull requests are welcome. Please make them small and focused... don't try to submit one large PR that adds type hints in all the remaining places missing them.