[python-package] drop Python 3.6 support, add Python 3.12

[jameslamb]

Inspired by this discussion: #6375 (comment)

Proposes the following:

• bump python-requires to >= 3.7
• run the one "oldest versions of everything" job on Python 3.7 instead of Python 3.6
• run most CI jobs on 1 minor version of Python newer than what it's currently on
  • which includes adding the first Python 3.12 jobs here!
• narrowly pin dependencies in Python 3.8 jobs to a specific, known-working environment

Notes for Reviewers

Why these versions?

I picked the oldest versions where pip install run in the python:3.7 docker image found a wheel. Found that building those old versions scipy and pandas from their sdists failed with a variety of errors.

Summary:

• cffi = unchanged at 1.15.1 (Jun 2022)
• numpy = from 1.16.6 (Dec 2019) to 1.19.1 (Jul 2020)
• pandas = from 0.24.0 (Jan 2019) to 1.1.3 (Oct 2020)
• pyarrow = unchanged at 6.0.1 (Nov 2021)
• scikit-learn = 0.18.2 (Jun 2017) to 0.24.0 (Dec 2020)
• scipy = 0.19.0 (Mar 2017) to 1.6.0 (Dec 2020)

[jameslamb]

proposing swapping this one and the wheel task, so we're always testing newest CUDA together with newest Python

[borchero]

LGTM!

[jameslamb]

Thanks for the review!

For breaking changes like this, I'd like to try to get 2 reviews. @jmoralez I know you'd supported this general idea in #6375 (comment), could you take a look at this PR and let us know what you think about the choices I made here?

[jmoralez]

Looks great. Thanks!

[jameslamb]

hmmmmm all the gcc CUDA jobs are the CUDA 11.8 gcc job is failing with errors like this:

__________________ ERROR collecting tests/c_api_test/test_.py __________________
tests/c_api_test/test_.py:10: in <module>
    from lightgbm.basic import _LIB as LIB
/github/home/.local/lib/python3.10/site-packages/lightgbm/__init__.py:8: in <module>
    from .basic import Booster, Dataset, Sequence, register_logger
/github/home/.local/lib/python3.10/site-packages/lightgbm/basic.py:278: in <module>
    _LIB = _load_lib()
/github/home/.local/lib/python3.10/site-packages/lightgbm/basic.py:262: in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
/tmp/miniforge/envs/test-env/lib/python3.10/ctypes/__init__.py:452: in LoadLibrary
    return self._dlltype(name)
/tmp/miniforge/envs/test-env/lib/python3.10/ctypes/__init__.py:374: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: libomp.so.5: cannot open shared object file: No such file or directory

(example build link)

I see this in the CMake logs:

-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")

It's surprising to see LightGBM linking against libomp, not libgomp, using gcc. I have a feeling this is a result of building inside a conda environment but not using conda's compilers (ref: #5106 (comment)).

I'm gonna push a commit setting LD_DEBUG=libs so we can see more information.

[jameslamb]

I'm moving the testing of the CUDA CI issue to a separate PR, I don't think it's related to this PR's changes: #6465

[jameslamb]

I've merged in #6465, which fixes the CUDA CI issues, and reverted all the debugging stuff I'd pushed here.

Will merge this once CI (hopefully) passes.