[python-package] simplify processing of pandas data

[jameslamb]

Contributes to #3756.
Contributes to #3867.

The Python package has an internal utility function, _data_from_pandas(), which takes in pandas DataFrames and extracts from them:

• the raw data in numpy format
• list of feature names
• list of categorical feature names
• mappings from raw values to integer encodings for categorical columns

That function is quite complicated (in my opinion), and some logic hidden in there doesn't actually relate to pandas Dataframes.

This PR proposes the following to simplify the Python package's code:

• only calling _data_from_pandas on pandas DataFrames
• adding return type hints to that function
• resolving some duplication in the function's code
• removing internal code paths where feature_name and categorical_feature could be set to None
  • in the public API, these can only be a list of names, the literal string "auto", or (categorical_feature only) a list of integer column indices

This makes it easier for humans reading the code and for type-checking tools like mypy to understand the flow of data through the package.

Notes for Reviewers

This will be easier to review if you apply the "Hide whitespace" changes in the GitHub diff view.

[jmoralez]

Thanks a lot for this! I agree that function has got quite complicated and it's great that you've found the time to simplify it. Sorry I haven't been able to review this, I'll provide a review in the following days.

[jameslamb]

Thanks @jmoralez , no rush! I have a few other PRs I'm planning like this as well, trying to simplify the flow of the code to make it easier for contributors.

[jmoralez]

There's something wrong with this PR, I can't approve it. I've tried from web, phone and gh CLI and I get the same error on all.

failed to create review: GraphQL: Something went wrong while executing your query.

I'll try later today if that's ok with you.

[jameslamb]

Yep no problem! GitHub is experiencing an outage in some services today.

https://www.githubstatus.com/incidents/smdz34v7j8q0

[jameslamb]

thanks @jmoralez !

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.