Problems with the Python scripts toolchain

[NotJustAnna]

After having to deal with Python scripts (See #10), it became clear that some (if not all) scripts should be re-engineered.

I'm opening this issue to discuss about what approach we can go to fix it. I mapped out a few available paths:

1. Rewrite the current Python scripts

This would be the easier, as it's just re-engineering the current Python scripts. But it'd require help from other contributors to do.

2. Write a new toolchain in a JVM scripting language.

This would make the job easier for the Java developers... if we can all agree between Groovy, KotlinScript, JavaScript (Rhino, Nashorn, GraalVM), Scala, Lua (LuaJ, Cobalt), Python (Jython), etc... as the scripting language for the toolchain. It'd require a bit more time to port Python to it.

3. Write a new toolchain as a Gradle plugin.

Hey, people at MinecraftForge, Sponge and FabricMC did it already. We can implement the toolchain as a separate Gradle plugin. It'd be the most effort, since we'd need a separate repository to maintain the Gradle plugin and fixes would most likely take hours instead of minutes, but it'd give out the best user experience.

4. Port it to another different language at all.

The last alternative would be porting it to a different language that most core RethinkDB developers could help, such as JavaScript (for example, running on Node.js). This option could take as much time as option 2 or 3, but fixes would most likely be made by pretty much any RethinkDB maintainer.

[gabor-boros]

While I like the idea of option 2 and 4, I would go with option 1. My only reason is that these scripts sometimes are reused (or should be reused) in other repos too. As I have no strong feelings about option 1 either I can be convinced about option 2 and 4 if we find a common language 😇

[NotJustAnna]

After researching, I come with a proposal:

Kotlin utility scripts

Kotlin released kotlin-main-kts, a tool for running standalone scripts. It resolves all dependencies (which can come from Sonatype or JCenter, by the way) before compilation of the script. The compiled scripts are also cached.

The new setup

The new setup would only require Gradle to download the kotlin command line compiler and kotlin-main-kts.jar from Sonatype. All scripts must have the .main.kts extension and can be run with the following command:

kotlinc -cp kotlin-main-kts.jar -script <script_name.main.kts> -- [args...]

What the tool provides

The kotlin-main-kts tool supports special file annotations, which make the entire setup painless and highly configurable:

@file:Repository("https://jcenter.bintray.com")

@Repository adds the maven repository for resolving dependencies. If none are specified explicitly, the maven central repository is used.

@file:DependsOn("org.jetbrains.kotlinx:kotlinx-html-jvm:0.6.11")

@DependsOn allows specifying library dependencies for the script. Either a direct path to the file or jar, or maven coordinates are accepted as arguments.

@file:Import("common.main.kts")

@Import allows importing another script into a given one. All the actions specified in the imported script run before all the actions specified in the given script and all the properties, functions and classes defined in the imported scripts become visible for the given script.

Maintaining

Maintaining the scripts should be fairly straightforward. IntelliJ IDEA provides code highlighting, analysis and completion, and it's the same language (and language subset) as the Gradle build file.

Converting from Python

Shouldn't be that hard. Mako can be replaced with Apache Velocity. JSON can be handled with either nanojson or Jackson (which is used by the driver). Polyglot tests (YAML) can be parsed with snakeyaml. Polyglot test generation can use javapoet or roaster, libraries that ensure generation of correct, valid Java files.

---

That's my suggestion. What's your thoughts on it?

[gabor-boros]

Umm, to be honest, I’m fine with this since I had lack of time nowadays 😇

[deontologician]

Uh, just stumbled across this. I wrote the original version of the Java driver and came up with this whole protobuf -> json -> python -> Java Rube Goldberg thing. Sorry about that. The initial reason was that most of the developers at rethinkdb were familiar with Python, so it would be the easiest language to maintain and keep the Java driver up to date for a small company at the time.

Obviously, with a real, live actual Java developer maintaining it now, it seems like the best course is to make it familiar and idiomatic Java by yourself and other Java devs. I especially like the idea of ensuring the templates generate valid Java, that's not something the hacky mako scripts allow at all.

I poked around the repo a bit, really great job picking this up. I'm glad someone is finally giving it some love :)

[NotJustAnna]

Oh hi hi @deontologician. First of all, thank you for writing the original version of Java. It was because of this driver that I tried RethinkDB and loved the syntax and everything (back in 2016, time flies).

The whole multi-step conversion thing isn't a problem. My biggest problem is that it's hard to find any external parseable sources. We created the rethinkdb/reql repository to aggregate as much parseable content into a repository as possible. I'd appreciate a lot any parseable files (JSON, etc) laying around.

I'm currently re-doing the scripting with Kotlin on the feature/kotlin-scripts branch but it's being blocked by this PR which is meant to bring more type information to ReQL as a whole.

I'm glad this project managed to stay alive and that I was able to give it an entire cleanup, shapeup and redesign the Java driver. It's a project I really like and I'm more than happy to give this driver some love :)

Thanks for creating the driver.

PS: Any documentation/JSON files/files detailing ReQL/etc regarding either the .json files or ReQL (or information regarding where to get it) would be appreciated. The lack of this and need to reverse-engineer everything makes updating the scripts quite a slow process.

[deontologician]

I don't have much extra context apart from what's been written in the readme's etc. We didn't have like an internal wiki or anything when the company was active, everything I knew is in this repo or on the website. Honestly, you probably know more than me at this point, a lot of this has leaked out of my head in the last 4 years