KDocs start using Doc Preprocessor Gradle plugin

[Jolanrensen]

The final version of this PR consists of:

• the ColumnExpression alias and use of it across the API instead of a specific Selector.
• General concept docs with reusable examples in the :core documentation package
• Docs for update.kt
• Docs for Nulls.kt

The rendered version of the docs can be found at :core/generated-sources and it would probably be useful to review those as well as the source files. I'm curious to hear about how you find the resulting docs and whether the structure I provided for the two API files (and the common docs) is repeatable for the rest of the API or if you have some additions, concerns etc.

Old message:

Exploratory method for being able to reuse KDocs.
After trying, it seems that compiler plugins won't do the trick, since the javadoc and sources.jar are generated from the actual sources before the plugin is run.
So, to have the sources.jar be modified, we need something to run from Gradle itself.
Cue JCP. I've used this in the Kotlin Spark API as well.

The current implementation takes the sources from :core, runs the preprocessor on them, changes the Kotlin compile sources to the files modified by the preprocessor, and after compilation resets them back. With this method, sources.jar will contain the properly preprocessed files.
One downside is that compile errors will thus point you to the preprocessor files.

Since we (probably) will only use JCP to generate KDocs I will experiment and see if there's a way to only change the sources when generating sources.jar and not during compilation.

I've also filed an issue for multiline string variables for JCP since defining reusable comments in the same file now requires them to be a single line... It is possible to define the comments in the Gradle build files, but this doesn't seem right to me.

Check add.kt for an example.

Edit: JCP output is now only used for Jar tasks :D

[Jolanrensen]

The only problem atm is in usability. Due to the lack of multiline strings, we need to do:

import org.jetbrains.kotlinx.dataframe.DataFrame
import org.jetbrains.kotlinx.dataframe.exceptions.DuplicateColumnNamesException
import org.jetbrains.kotlinx.dataframe.exceptions.UnequalColumnSizesException

/** either
//#local ADD = evalFile("add.txt")
*/

/** or
//#local ADD0 = " * Original [DataFrame] is not modified."
//#local ADD1 = " *"
//#local ADD2 = " * @throws [DuplicateColumnNamesException] if columns in expected result have repeated names"
//#local ADD3 = " * @throws [UnequalColumnSizesException] if columns in expected result have different sizes"
//#local ADD4 = " * @return new [DataFrame] with added columns"
//
//#local ADD = ADD0 + "\n" + ADD1 + "\n" + ADD2 + "\n" + ADD3 + "\n" + ADD4
 */


/**
 * Creates new [DataFrame] with given columns added to the end of original [DataFrame.columns] list.
 *
/*$ADD$*/
 * @param columns columns to add
 */
public fun <T> DataFrame<T>.add(vararg columns: AnyBaseCol): DataFrame<T> = addAll(columns.asIterable())

[Jolanrensen]

Using my self-built gradle plugin is probably a better solution https://github.com/Jolanrensen/kdocIncludeGradlePlugin

[Jolanrensen]

Updated to use my own gradle plugin now via JitPack :)

Example can be found in add.kt:

[Jolanrensen]

https://github.com/Jolanrensen/docProcessorGradlePlugin/tree/main was updated to allow other processors as well. Maybe @sample is also interesting for DataFrame :)

[zaleslaw]

What is the future of this PR?

[Jolanrensen]

What is the future of this PR?

Will rebase on master, fix every conflict and then ask for a review :)

[Jolanrensen]

This file I just started with, will probably work on docs after this PR is merged in a separate branch

[Jolanrensen]

ready for review!

[zaleslaw]

Not so many issues in reality, I will try to summarize them

• be accurate with inserted code and probably wrap with the kotlin code insertion
• convert some todos into issues or fix them
• answer some questions
• if possible, extract abstraction required for documentation building to the separate places

Some notes for the future:

• Did we have a chance to use constants instead of interfaces?
• Could we use @see tag instead of direct links with pattern?
• Could we reuse the same snipppets of code used in the documentation to insert them and checking compiling it once

[Jolanrensen]

Not so many issues in reality, I will try to summarize them

• be accurate with inserted code and probably wrap with the kotlin code insertion
• convert some todos into issues or fix them
• answer some questions
• if possible, extract abstraction required for documentation building to the separate places

Some notes for the future:

• Did we have a chance to use constants instead of interfaces?
• Could we use @see tag instead of direct links with pattern?
• Could we reuse the same snipppets of code used in the documentation to insert them and checking compiling it once

I cannot wrap code samples with triple backticks because then links inside won't be clickable anymore. If that was what you meant with be accurate with inserted code and probably wrap with the kotlin code insertion.

if possible, extract abstraction required for documentation building to the separate places
Do you suggest moving general docs from, say update.kt, to a separate file? say updateDocs.kt? I was debating myself whether to do that or not, because it would clutter the internal namespace more as well as moving the information about a file away from the file. I'd rather have them close together, I think. That said, the API is less clear if it's filled with docs.

Did we have a chance to use constants instead of interfaces?
Yes, constants are possible, but unlike interfaces they cannot be nested. Since the interfaces are never instantiated I don't think there's a speed loss. Isn't unreachable/unused code removed from the bytecode?

Could we use @see tag instead of direct links with pattern?
yes, but currently @see tags only show up rendered at the bottom of the doc with no description. That's how I came to the {@include [SomethingLink]} pattern, where interface SomethingLink contains the proper name and destination for what you're pointing to.

Could we reuse the same snipppets of code used in the documentation to insert them and checking compiling it once
We cannot check compilation from the Doc processor. It just processes docs ;). But with @sample you can include the same snippets from the documentation (if they are in the same module, I believe).

[zaleslaw]

Do you suggest moving general docs from, say update.kt, to a separate file? say updateDocs.kt? I was debating myself whether to do that or not, because it would clutter the internal namespace more as well as moving the information about a file away from the file. I'd rather have them close together, I think. That said, the API is less clear if it's filled with docs.

Probably, I'll try to do it in my PR with docs and we could compare two approaches