0.6.1 - Maintenance release

Features

• #673 - Admin tool: DB migration from v.0.5.5 to 0.6
• #667 - Admin tool: Read passwords from console
• #928 - Consumer API: Filter events by write mode

Bugfixes

• #925 - Producer API: Fix unique constraint violated - in index primary of type primary over '_key'; conflicting key
• #898 - Consumer API: Fix execution plan name is null
• #911 - Persistence: Fix wrong index field "execPlanDetails.executionPlanId"
• #907 - Admin tool: Fix CLI doesn't return on ArangoDB authentication errors

Improvements

• #834 - Persistence: Ensure SSL validation is supported in ArangoDB communication
• #891 - Persistence: Upgrade ArangoDB driver to the upstream version 6.12.2
• #915 - Kafka Gateway: Improve configuration
• #908 - Admin tool help: Improve usage info

0.6.0 - Attribute-level lineage, Kafka and more

Spine 0.6 release comes with a number of new features. improvements and bug fixes.

Features

Improved Attribute-level lineage support

Attributes and expressions are now the first-class citizens in Spline 0.6 domain model. This allows for precise dependency tracking between attributes, directly (attribute-to-attribute) or indirectly (via expressions) depending on what level of details the agent can provide.

• #114 - Attribute lineage

Updated Producer API & Kafka support

We introduced Producer API version mechanism, to improve agents backward compatibility.
In the improved Producer API 1.1 /execution-plans and /execution-events end-points can be called asynchronously.
We have also introduced Kafka Gateway (the feature is still experimental). It can be used side-by-side or as an alternative to the REST Gateway for receiving lineage data from the agents.

• #674 - Producer API versioning
• #671 - Producer API v.1.1
• #814 - Async producer API - Kafka

Consumer API

A few new resources were added to the Consumer API:

• #838 - Consumer REST: /data-sources
• #843 - Consumer REST: Add "name" field for "DataSource" with sorting capabilities
• #793 - Introduce an AffectsDataSrc And DependsDataSrc Endpoint (by @mohanajuhi166)

UI

Starting from this release UI module is not longer a part of the Spline main repo.
A new UI project has been created in a separate repo.

• #796 - Switch to a new UI

Bugfixes

• #869 - ArangoDB cluster mode :: VstConnection :: java.net.SocketException: Socket closed bug
• #850 - VstConnectionAsync ClassCastException in ArangoDB Java Driver
• #883 - 400 JSON parse error when "constant.value=null
• #874 - Admin CLI: ANSI codes aren't interpreted on Windows
• #835 - Producer API fails for some examples run by 0.5 agent
• #837 - Consumer REST: /execution-events: total date range facet doesn't take other filters into account

Improvements

• #878 - Docker image for Admin CLI
• #859 - Consumer REST: More exec plan details on the overview graph component
• #831 - Producer REST API enhancement & documentation
• #789 - Execution plan ID watermark on Operation vertices and edges
• #871 - Admin CLI: Add "-v --version" option
• #677 - Upgrade domain model to accommodate attribute level lineage requirements
• #842 - Build: Store git revision in the build properties
• #823 - POST /producer/execution-plans should not replace execution plan ID
• #790 - Show a warning when ArangoDB version is less than 3.7.3
• #788 - Reduce nesting in the persistent model

Bugfix release

Fixes SSL issue in Admin CLI - #810

Cluster bug fixes

Fixes required to works with ArangoDB on a cluster with replicas

Performance bug fixes

This release addresses a few performance issues.
And it also introduces Spline database schema versioning.

[#721] - ArangoDB schema versioning
[#741] - Spline dashboard takes too long to load component: Consumer scaling / performance
[#738] - lineage-overview - 598 Network read timeout error (spline 0.5.3) component: Consumer scaling / performance

Maintenance release - 0.5.3

Bugfixes

UI

• [#732] Visible URL to detailed lineage + improved navigation

Admin CLI

• [#734] Spline won't initialize on 0.5.2

Maintenance release

In this release we addressed some issues and made a few enhancements:

UI

• More readable and user-friendly visualization of expressions and generic operation properties
• Write and Read operations now displays more metadata.
• Display arrows on graph edges
• Ability to expand lineage overview graphs of depth greater than 10 jobs (minor ArangoDB upgrade is required, see https://absaoss.github.io/spline/0.5.html)

Admin CLI

• #696 Admin CLI: Option '-s` is not working

Gateway

• #697 Added informative index page to monitor the server status

Scala 2.12, Attribute lineage highlighting, MongoDB, ElasticSearch, Cassandra and more

Spline 0.5 comes with a few major improvements and new features:

Spline Agent for Spark (aka Spline Harvester)

1. Agent is now a separate project, located in its own repo
   and loosely coupled with the Spline core via the Producer REST API

2. [#212] Scala 2.12 is supported (for Spark 2.4)

3. Support for some alternative data source types (special thanks to @radford1)

• [#606] ElasticSearch connector support
• [#605] MongoDB connector support
• [#604] Cassandra connector support

Spline Core and UI

• [#112][#559][#451] Attribute lineage and impact highlighting
• [#442] Search by attribute name
• [#616] Support conventional upper snake notation for environment variables
• [#620] ArangoDB SSL connection support
• [#617] ArangoDB fallback config support
• [#615] Connection Performance Improvements (Timeout config, gzip)
• [#561] Docker composite layer for Demo (thanks to @rtyler)

Other improvements and bug fixes

• [#572] Cross-framework end-to-end lineage example
• [#534][#554] Improve logging
• [#635] Client UI WebJar should not have any bytecode dependencies
• [#612] Lineage Timestamp format is misleading
• [#565] Admin CLI doesn't exit normally
• [#535] spark-shell / pyspark --packages not working
• [#629] Missing configuration property spline.consumer.url

...and more

Support for Excel data source

#588 - Add support for capturing lineage from Excel read/write operations
#584 - Producer API: Execution ID is automatically generated if not provided by an agent.

... and a few minor fixes.

Bugfix release

#527 - Arrays of primitives should not show possibility of being unfolded
#522 - Cannot send lineage data
#518 - WebJar isn't published to Maven automatically
#515 - [spark-agent]0.4.0 HttpLineageDispatcher.ensureProducerReady is missing the Success case
#425 - Migrator: check that Producer is accessible before start migration