Spark and Spark History 2.12.0-3.0.1

New features

• Base tech upgrades to 3.0.1
• Adds support of _SPARK_AUTH_SECRET_FILE environment to accept file-based authentication, similar to _SPARK_AUTH_SECRET, which accepts secret directly.
• Adds the flag --executor-auth-secret-path in a Spark CLI to enable File-Based RPC Authentication.
• Adds configurations to support log rotate options stdout_max_size and stderr_max_size.

Updates

• Supports Scala version 2.12 only. No longer supports Scala version 2.11.
• Compiles Spark with Hadoop versions 3.2 and 2.7.

Breaking changes

• Spark CLI flag --executor-auth-secret, no longer accepts filename or secret path.
• No longer publish Spark docker image compiled with Hadoop 2.9.

Spark and Spark History 2.12.0-3.0.1-beta

NOTICE

This is a beta release of the DC/OS Spark framework. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.

Improvements

• Upgraded to Spark 3.0.1
• Added support of _SPARK_AUTH_SECRET_FILE environment to accept file-based authentication, similar to _SPARK_AUTH_SECRET, which accept secret directly.

Updates

• Added the flag --executor-auth-secret-path in a Spark CLI to enable File-Based RPC Authentication
• Updated the flag --executor-auth-secret in a Spark CLI to accept RPC Authentication secret directly. No longer accept filename in this flag.
• Supported Scala version 2.12 only. No longer support Scala version 2.11.
• Compiled Spark with Hadoop versions 3.2 and 2.7

Spark and Spark History 2.11.0-2.4.6

New features

• Base tech upgrades to 2.4.6

Updates

• Removes hardcoded --completed flag from dcos spark log CLI and allows all options supported

Spark and Spark History 2.10.0-2.4.5

New features

• Suppress and Revive support
• Placement Constraints support by Dispatcher

Updates

• Updated libmesos-bundle from 1.14-alpha to 1.14-beta

Spark and Spark History 2.9.0-2.4.3

New features

• Marathon group role enforcement support for DCOS 1.14
• Role propagation from Dispatcher to all submitted Spark applications
• Dispatcher role enforcement for submitted Spark applications
• Node draining support
• Executor memory overhead config property support
• Mesos secrets support by Dispatcher
• Custom StatsD Sink supporting standardized metric names and metric tags for monitoring

Updates

• Switched from Oracle JDK to OpenJDK
• Switched to UCR containerizer as the default
• Scala 2.12 support
• Updated libmesos to 1.14

Bug fixes

• Fixed a bug causing crashes when labels are malformed

Breaking changes

• Standardized metrics rename existing metrics by moving variable parts of metric names to tags

Spark and Spark History 2.8.0-2.4.0

New features

• Mesos checkpointing support for Spark Drivers submitted to Dispatcher

Updates

• Upgraded Hadoop dependency to 2.9.2

Bug fixes

• Fixed bug for Spark Executor ID being different from Mesos Task ID
• Fixed a bug in Spark CLI causing incorrect parsing of jars argument when word "jars" is present in path

Breaking changes

• The default Hadoop dependency is now 2.9 and not 2.7

Spark and Spark History 2.7.0-2.4.0

New features

• Upgraded Spark and Spark History Server to 2.4.0

Updates

• Spark Mesos Dispatcher uses the same user for running Spark jobs as itself and defaults to nobody
• Switched to dcos-commons bootstrap script for IP address detection

Breaking changes

• Removed configuration option use_bootstrap_for_IP_detect since we now use bootstrap by default for Spark container IP detection which works across all containerizers: DOCKER & MESOS (UCR) and networking modes: HOST & CNI Virtual Networks.

Spark and Spark History 2.6.0-2.3.2

New features

• Upgraded Spark and Spark History Server to 2.3.2
• Added DC/OS Spark CLI support for --jars
• Added CNI Support for Dispatcher, Driver, and Executors for Docker and Mesos containerizers
• Added CNI labels support for Mesos containerizer
• Added package configuration for CNI:
  • virtual_network_enabled
  • virtual_network_name
  • virtual_network_plugin_labels
• Spark Dispatcher by default launches Spark Drivers and Executors in the same virtual network it was launched in itself

Updates

• SPARK_HOME environment variable defaults to /opt/spark in Dockerfile and executable scripts
• Switched to Spark's own StatsD Sink instead of 3rd-party dependency
• Updated dcos-commons bootstrap version to 0.55.2

Bug fixes

• Fixed bug for Dispatcher restarting duplicate Spark drivers after agent restart in --supervised mode
• Fixed bug for CLI incorrect --jars parsing resulting in submit failure

Limitations

• Configuration of network plugin labels from DC/OS UI supported only in JSON editing mode
• Network plugin labels are not supported by Docker containerizer
• Currently, DC/OS AdminRouter doesn't support virtual networks so DC/OS Spark endpoints will not be accessible from CLI if Spark installed with virtual network enabled. Spark jobs need to be submitted from a routable network in this case

Spark and Spark History 2.5.0-2.2.1

New features

• Added unique Mesos Task IDs for Spark executors.
• Added trusted Ubuntu 18.04 base Docker image.
• Added nobody user support on RHEL/CentOS (through configuration).

Updates

• Changed the default user for the Docker container from root to nobody.
• Upgraded JRE to 1.8.192.
• Upgraded to Ubuntu 18.04
• Updated Hadoop dependencies from 2.7.3 to 2.7.7 (fixes CVE-2016-6811, CVE-2017-3162, CVE-2017-3166, CVE-2018-8009).
• Updated Jetty dependencies from 9.3.11.v20160721 to 9.3.24.v20180605 (fixes CVE-2017-7658).
• Updated Jackson dependencies from 2.6.5 to 2.9.6 (fixes CVE-2017-15095, CVE-2017-17485, CVE-2017-7525, CVE-2018-7489, CVE-2016-3720).
• Updated ZooKeeper dependencies from 3.4.6 to 3.4.13.

Bug fixes

• dcos task log now works because of unique Mesos Tasks IDs of Spark executors.
• Fixed unstable health checks for Spark dispatcher and history server.
• Spark dispatcher task output now redirects to stdout and is available in logs.

Breaking changes

• Added a new configuration option docker_user to override the user when running Spark using Docker containerizer.
• The default Hadoop dependency is now 2.7.7 and not 2.6.5.

Spark and Spark History 2.4.0-2.2.1-3

New Features

• Added service name to dispatcher’s VIP endpoints (#412)
• Added shell-escape fix to spark-cli (SPARK-21014) (#401)
• Added spark.mesos.executor.gpus (SPARK-21033) (#23)
• Added dispatcher and driver metrics (#24 && #25)
• Added statsd sink for spark metrics (#402)
  • Note: metrics is a beta feature and requires enabling UCR. Production use is not advised.

Updates

• Updated tests, build tools, CLI and vendor packages (#383)
• Updated bootstrap version to 0.50.0 (#372)
• Updated JRE version to 8u172 (#371)

Bug fixes

• Fixed duplicate docker image URLs; use resource.json as default (#385)

Breaking Changes

• VIP endpoints for the dispatcher are no longer spark-dispatcher:<port> and are now dispatcher.{{service.name}}:<port>