[python-package] upgrade to scikit-build-core 0.9.3

[jameslamb]

Proposes upgrading scikit-build-core, 0.4.4 -> 0.9.3 (the latest version).

That new version preserves support for python >= 3.7 and lots of great changes have been made since v0.4.4, including:

• better error messages
• better support for customizing the CMake generator
• support for customizing which targets CMake builds (feat: add build target support scikit-build/scikit-build-core#432)

For full release notes, see https://github.com/scikit-build/scikit-build-core/releases

[jameslamb]

Moving this back to draft and undoing your approval @guolinke . I see some builds failed and this might need to be reworked.

Will re-open it and request reviews when it's ready. Sorry!

[jameslamb]

This ensures that only the shared library is built, and not other targets like the CLI or headers.

scikit-build-core didn't support that back when we did #5759, which is why this exists:

LightGBM/CMakeLists.txt

Lines 54 to 60 in 88cec47

	if(__BUILD_FOR_PYTHON OR __BUILD_FOR_R)
	# the Python and R package don't require the CLI
	set(BUILD_CLI OFF)
	# installing the R and Python package shouldn't place LightGBM's headers
	# outside of where the package is installed
	set(INSTALL_HEADERS OFF)
	endif()

I'd like to leave that in CMakeLists.txt as extra protection for now. It's only a tiny bit of complexity in exchange for protection against some potentially-impactful side effects (like pip install of an sdist placing LightGBM's headers somewhere outside of where pip manages files).

[jameslamb]

See jameslamb/pydistcheck#235 for a full description of this.

Short description: it's great that scikit-build-core strips binaries by default now, but we don't need it in LightGBM. And pydistcheck's check on debug symbols (which runs in CI) gives us confidence that no debug symbols sneak into compiled objects in wheels.

[jameslamb]

Ok I think is working now and is ready for another review whenever you have time, @guolinke . Sorry it took so long to get back to it.

[jameslamb]

Thank you!