Neo4j Connector for Apache Spark 5.3.0

What's Changed

⭐ Release Highlights

• Inferred schema type constraints + improved constraints management by @conker84 in #581

This introduces support for type/existence constraints based on the DataFrame definition and improves the handling of existing constraints.

Check out the documentation to learn more!

📜 Change Log

• [Snyk] Security upgrade org.testcontainers:testcontainers from 1.19.0 to 1.19.1 by @fbiville in #554
• Skip POM backups when setting new versions by @fbiville in #552
• Bump com.fasterxml.jackson.core:jackson-annotations from 2.15.2 to 2.15.3 by @dependabot in #557
• Bump com.fasterxml.jackson.core:jackson-annotations from 2.15.3 to 2.16.0 by @dependabot in #562
• Bump testcontainers.version from 1.19.1 to 1.19.2 by @dependabot in #561
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.1.2 to 3.2.1 by @dependabot in #559
• Bump org.neo4j:neo4j-cypher-dsl from 2022.9.0 to 2022.9.1 by @dependabot in #558
• Bump testcontainers.version from 1.19.2 to 1.19.3 by @dependabot in #565
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.1 to 3.2.2 by @dependabot in #564
• Bump org.codehaus.mojo:versions-maven-plugin from 2.16.1 to 2.16.2 by @dependabot in #563
• Remove outdated version checks and rename tests by @fbiville in #567
• Bump actions/setup-java from 3 to 4 by @dependabot in #568
• Bump actions/setup-python from 4 to 5 by @dependabot in #570
• Fix README formatting by @fbiville in #569
• feat: add metrics support to the sink path by @ali-ince in #566
• Remove unneeded action by @fbiville in #571
• Remove unneeded workflow by @fbiville in #574
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.2 to 3.2.3 by @dependabot in #578
• Bump org.neo4j.driver:neo4j-java-driver from 4.4.12 to 4.4.13 by @dependabot in #585
• Add owners by @fbiville in #580
• Bump com.fasterxml.jackson.core:jackson-annotations from 2.16.0 to 2.16.1 by @dependabot in #586
• Bump org.apache.maven.plugins:maven-compiler-plugin from 3.11.0 to 3.12.1 by @dependabot in #587
• Improved user-agent management by @conker84 in #583
• Restrict reviews of CI-related changes by @fbiville in #588
• Improved query error management by @conker84 in #584
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.3 to 3.2.5 by @dependabot in #589
• Followup user agent by @conker84 in #590
• Remove unused bookmarks parameter by @fbiville in #591
• Bump actions/cache from 3 to 4 by @dependabot in #592
• Bump testcontainers.version from 1.19.3 to 1.19.4 by @dependabot in #593
• updated spark version and made it configurable by @conker84 in #595
• Bump testcontainers.version from 1.19.4 to 1.19.5 by @dependabot in #596

Full Changelog: 5.2.0...5.3.0

Neo4j Connector for Apache Spark 5.2.0

Hello Community!
We're happy to announce a new release of the Neo4j Spark Connector!

Spark baseline version

This new release bumps the required Spark version to 3.4, in order to support the new push down Top N optimization.

What's Changed

• Implement Top N pushdown by @fbiville in #526
• Update Maven wrapper by @fbiville in #551
• Fix typo of close utility method name by @fbiville
• Upgrade scala versions by @fbiville in #542
• Bump actions/checkout from 3 to 4 by @dependabot
• Extract documentation away from repository by @ali-ince
  The documentation sources now live in https://github.com/neo4j/docs-spark/
• fixes #527: Add example notebooks by @conker84 in #528
  Interact with the notebooks here: https://neo4j.com/docs/spark/current/playground/

Full Changelog: 5.1.0...5.2.0

Neo4j Connector for Apache Spark 5.1.0

Hello Community!
We're happy to announce a new release of the Neo4j Spark Connector!

Spark baseline version

This new release bumps the required Spark version to 3.3, in order to support the new push down v2 filters and limit optimizations.

What's Changed

• Support push down v2 filters by @fbiville in #492
• Bump neo4j-cypher-dsl from 2020.1.4 to 2022.8.5 by @dependabot in #510
• fixes #505: First-class support to GDS by @conker84 in #506
• Bump org.neo4j:neo4j-cypher-dsl from 2022.8.5 to 2022.9.0 by @dependabot in #533
• [Snyk] Upgrade org.neo4j.driver:neo4j-java-driver from 4.4.11 to 4.4.12 by @fbiville in #532
• Bump testcontainers.version from 1.18.3 to 1.19.0 by @dependabot in #538
• Add support for bearer authentication by @fbiville in #537
• Implement pushdown limit support by @fbiville in #520
• Make build script switch artifact IDs by @fbiville in #517
• fixes #531: Problem with datetime properties with null values by @conker84 in #536
• added fixed bug in version validation by @conker84 in #540
• Update Scala 2.13 version by @fbiville in #539

Full Changelog: 5.0.3...5.1.0

Neo4j Connector for Apache Spark 5.0.3

Hello Community!
We're happy to announce a new release of the Neo4j Spark Connector!
There are two files:

• neo4j-connector-apache-spark_2.12-5.0.3_for_spark_3.jar, md5 46775288407098da460c5668db6fedea
• neo4j-connector-apache-spark_2.13-5.0.3_for_spark_3.jar, md5 72727cc4b399d0fe17926d8a43d91c86

What's Changed

• Bump testcontainers.version from 1.18.0 to 1.18.1 by @dependabot in #495
• Bump testcontainers.version from 1.18.0 to 1.18.2 by @dependabot in #508
• Bump testcontainers.version from 1.18.2 to 1.18.3 by @dependabot in #511
• Bump maven-source-plugin from 3.2.1 to 3.3.0 by @dependabot in #500
• Broken links fix by @lidiazuin in #513
• Bump jackson-annotations from 2.15.0 to 2.15.2 by @dependabot in #509
• Bump maven-surefire-plugin from 3.1.0 to 3.1.2 by @dependabot in #515
• Remove obsolete Spark version condition by @fbiville in #518
• fixes #503: Not able to Insert Neo4j Map Data type using the neo4j-spark connector by @conker84 in #507
• [Snyk] Upgrade @neo4j-antora/antora-add-notes from 0.1.6 to 0.3.0 by @fbiville in #522

Full Changelog: 5.0.2...5.0.3

Neo4j Connector for Apache Spark 5.0.2

Hello Community!
We're happy to announce a new release of the Neo4j Spark Connector!
There are two files:

• neo4j-connector-apache-spark_2.12-5.0.2_for_spark_3.jar, md5 6033893da4ef48be27a9fa2055b681b0
• neo4j-connector-apache-spark_2.13-5.0.2_for_spark_3.jar, md5 c94d6247bb5a6862fefde7d98533c0f0

What's Changed

• Update Neo4j versions used by integration tests by @fbiville in #461
• Remove checks for Scala 2.11 from CI since 2.11 is not supported by @fbiville in #462
• Make the build script optional - use sensible Maven defaults by @fbiville in #463
• Enable Dependabot by @fbiville in #464
• Build with LTS versions of JDK by @fbiville in #465
• Bump testcontainers.version from 1.17.3 to 1.18.0 by @dependabot in #476
• Bump maven-assembly-plugin from 2.5.3 to 3.5.0 by @dependabot in #474
• Bump actions/setup-node from 1 to 3 by @dependabot in #473
• Bump actions/setup-python from 2 to 4 by @dependabot in #467
• Bump actions/setup-java from 1 to 3 by @dependabot in #468
• Bump actions/cache from 1 to 3 by @dependabot in #470
• Bump actions/checkout from 2 to 3 by @dependabot in #471
• Bump avides/actions-project-version-check from 1.2.0 to 1.4.0 by @dependabot in #481
• Bump build-helper-maven-plugin from 3.0.0 to 3.3.0 by @dependabot in #480
• Bump jackson-annotations from 2.12.0 to 2.15.0 by @dependabot in #479
• Bump junit from 4.13.1 to 4.13.2 by @dependabot in #477
• Simplify documentation structure & review quickstart guide by @fbiville in #460
• Bump scala-maven-plugin from 4.5.4 to 4.8.1 by @dependabot in #487
• Remove always-false comparison in Spark version checker by @fbiville in #466
• Bump maven-surefire-plugin from 3.0.0 to 3.1.0 by @dependabot in #488
• Bump maven-source-plugin from 2.2.1 to 3.2.1 by @dependabot in #483
• Bump maven-compiler-plugin from 3.8.1 to 3.11.0 by @dependabot in #485
• Bump hamcrest-library from 1.3 to 2.2 by @dependabot in #486
• Bump maven-assembly-plugin from 3.5.0 to 3.6.0 by @dependabot in #494
• fixes #498: spark version validation fails on EMR / EMR Serverless by @conker84 in #501
• bump version to 5.0.2 by @conker84 in #502

New Contributors

• @fbiville made their first contribution in #461 🎉🎉🎉

Full Changelog: 5.0.1...5.0.2

Neo4j Connector for Apache Spark 5.0.1

Hello Community!
We're happy to announce a new release of the Neo4j Spark Connector!
There are two files:

• neo4j-connector-apache-spark_2.12-5.0.1_for_spark_3.jar, md5 08f1fa9e936eab63aa380ac50f30f2e0
• neo4j-connector-apache-spark_2.13-5.0.1_for_spark_3.jar, md5 692af135dfdf0d188e9c9b2d944996ef

Below are the changes.

What's Changed

• Minor Improvements by @conker84 in #453
• fixes #444: Fixing issue where LocalTime and Time are being written as individual fields instead of single Struct Field (#450) by @conker84 in #454

Full Changelog: 5.0.0...5.0.1

Neo4j Connector for Apache Spark 5.0.0

Hello Community!

We are pleased to announce the 5.0.0 of the Spark Connector which is the first official release that fully supports Neo4j 5.0 and Neo4j 4.4. For earlier versions of <= Neo4j 4.3, please use Neo4j Apache Spark 4.2.

N.B Because of a bug in Spark, the Aggregate Pushdown feature can only be activated with Spark >3.2.2 (please see apache/spark#36727)

There are three files:

• neo4j-connector-apache-spark_2.12-5.0.0_for_spark_3.jar, md5 ce814cd8d0a34f14599302895be13956
• neo4j-connector-apache-spark_2.13-5.0.0_for_spark_3.jar, md5 39e5ebb2b71a3f93d3230db91f25e2db

What's Changed

• Code clean by @conker84 in #445
• fixes #391: Support Spark 3.2 DatasourceV2 Aggregate pushdown by @conker84 in #431
• Support Neo4j 5.0 indexes and constraints syntax changes by @conker84 in #449

Full Changelog: 4.1.5...5.0.0

Neo4j Connector for Apache Spark 4.2.0

Hello Community!
We are pleased to announce the 4.2.0 of the Spark Connector with this version we deprecated the support for Spark 2.4 and support only Spark versions from 3.2.
With this release we added the support for Spark's Aggregate Pushdown which allows doing aggregations directly in Neo4j.

Nota bene Given a Spark bug the Aggregate Pushdown feature can be activated only from Spark 3.2.2 (please see apache/spark#36727)

There are three files:

• neo4j-connector-apache-spark_2.12-4.2.0_for_spark_3.jar, md5 6de1f93ece9f45aa51486317a62620f0
• neo4j-connector-apache-spark_2.13-4.2.0_for_spark_3.jar, md5 a79cf2b9d8d6725587760fa149932be2

What's Changed

• fixes #391: Support Spark 3.2 DatasourceV2 Aggregate pushdown by @conker84 in 1585e97
• Code clean by @conker84 in #445
• Improved Spark version validation by @conker84 in #448

Full Changelog: 4.1.5...4.2.0

Neo4j Connector for Apache Spark 4.1.5

Hello Community!
We are pleased to announce the 4.1.5 of the Spark Connector with this version we added the support for Neo4j Java Driver URL resolvers.

There are three files:

• neo4j-connector-apache-spark_2.11-4.1.5_for_spark_2.4.jar, md5 118dce27b54f16f304880df9c7280615
• neo4j-connector-apache-spark_2.12-4.1.5_for_spark_2.4.jar, md5 705196b006434b659ea6c1601ec0c00a
• neo4j-connector-apache-spark_2.12-4.1.5_for_spark_3.jar, md5 2ed1f9a89bd900b1aac1950fcbc85ee5
• neo4j-connector-apache-spark_2.13-4.1.5_for_spark_3.jar, md5 caed49e951e9b8d4db12345799795481

What's Changed

• fixes #442: Support Neo4j Java Driver URL resolvers by @conker84 in #443

Full Changelog: 4.1.4...4.1.5

Neo4j Connector for Apache Spark 4.1.4

Hello Community!
We are pleased to announce the 4.1.4 of the Spark Connector with this version we added the support for the logging with the Neo4j Java driver.

There are three files:

• neo4j-connector-apache-spark_2.11-4.1.4_for_spark_2.4.jar, md5 cf9baef98e7d9214ef77f8b4bccbd87a
• neo4j-connector-apache-spark_2.12-4.1.4_for_spark_2.4.jar, md5 9a56ec8a7d87c6683e4356063714f37f
• neo4j-connector-apache-spark_2.12-4.1.4_for_spark_3.jar, md5 48f03d73b6df80f3bea254266b327f2f
• neo4j-connector-apache-spark_2.13-4.1.4_for_spark_3.jar, md5 a9c9a01198d7e9f820ae9d120d32472c

What's Changed

• Update antora.yml by @NataliaIvakina in #439
• Update graph projection example by @nvitucci in #441
• Enable driver logging by @conker84 in #440

New Contributors

• @nvitucci made their first contribution in #441 🎉

Full Changelog: 4.1.3...4.1.4