Migrate Gradle runner's classpath resolution to stable API

[IgnatBeresnev]

Background

In order to generate documentation for a project, the Gradle runner must first resolve the classpath of that project, which must include dependencies and platform types.

Dokka uses Kotlin Gradle Plugin's API to extract the information about targets, source sets and compilations of any given project. It uses this information to resolve the classpath, which gets passed to the analysis.

The problem

For historical reasons, the Gradle runner relies on some low-level, undocumented, non-stable and/or internal KGP API for classpath and compilation resolution, which leads to compatibility problems almost every Kotlin release.

This means that when a user migrates to a new version of Kotlin (by updating the version of KGP), they might see errors when running Dokka's tasks (examples: #3038, #3068). From the user's perspective, it's a bug/regression of Dokka, but in reality something was changed internally in KGP (either intentionally or not), or the API used by Dokka got deprecated/removed, and it now leads to errors.

This is a major issue for the following reasons:

• it blocks Kotlin version updates in some libraries
• it takes a significant amount of our time to hunt these errors/bugs down and find substitutions
• it affects the Kotlin Multiplatform team as it imposes restrictions on the changes they would like to make (i.e they cannot delete the API which would break Dokka), and it takes their time to help us with debugging and finding solutions to emerging problems

Possible solution

IDEs need similar information as Dokka to perform source code analysis: they need to resolve project files, dependencies and platform types. For this, KGP already provides some high-level and stable API, which is tested and is looked after by the Kotlin Multiplatform team.

Migrating to the mentioned API for IDEs should resolve the majority of the problems outlined in this issue, as Dokka will not have to think about backward compatibility, and the following principle will apply: "If it is resolved by your IDE, it will be resolved by Dokka".

---

This issue is a subtask of Sustainability of dependencies (#3127)

[IgnatBeresnev]

Might resolve: #2599

[IgnatBeresnev]

We've agreed that the new Dokka Gradle plugin will be developed and maintained by the Kotlin Build Tooling team (the same team that develops KGP), so I guess this issue has been resolved in one way or another :)