[NEW EXAMPLE] Spark - Cross-validation

[ainzzorl]

General

• Project name: Spark
• Example name: Cross-validation
• Project home page: https://github.com/apache/spark
• Programming language(s): Scala
• Frameworks, libraries used: N/A

Description

Spark is a unified analytics engine for large-scale data processing. It has support for running machine learning workloads with MLlib. MLlib supports cross-validation: a standard procedure used to evaluate machine learning models on a limited data sample.

Links

CrossValidator.scala

What makes it interesting

• Machine learning workloads use cross-validation very often, but it's almost always delegated to different libraries. It's interesting to see how it's implemented in one of the most popular libraries.
• Spark is known for its performance and scalability.

Related work

N/A

Other

There are so many things to learn from Spark, but it's so big it's hard to get started with the code base. This example, however, seems rather independent from the rest of the code.

[ainzzorl]

I'm struggling to understand how it actually works. While it's an interesting piece of code, I'm moving it to "cooldown" for now.