Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Update 1.6.0 release notes with Java 11 tutorial #1403

Merged
merged 2 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/setup/databricks.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,36 @@

## JDK 11+ requirement

Sedona 1.6.0+ requires JDK 11+ to run. Databricks Runtime by default uses JDK 8. You can set up JDK 17 by following the instructions in the [Databricks documentation](https://docs.databricks.com/en/dev-tools/sdk-java.html#create-a-cluster-that-uses-jdk-17).

### on Databricks Runtime versions 13.1 and above

When you create a cluster, specify that the cluster uses JDK 17 for both the driver and executor by adding the following environment variable to `Advanced Options > Spark > Environment Variables`:

```
JNAME=zulu17-ca-amd64
```

If you are using ARM-based clusters (for example, AWS Graviton instances), use the following environment variable instead.

```
JNAME=zulu17-ca-arm64
```

### on Databricks Runtime versions 11.2 - 13.0

When you create a cluster, you can specify that the cluster uses JDK 11 (for both the driver and executor). To do this, add the following environment variable to `Advanced Options > Spark > Environment Variables`:

```
JNAME=zulu11-ca-amd64
```

If you are using ARM-based clusters (for example, AWS Graviton instances), use the following environment variable instead.

```
JNAME=zulu11-ca-arm64
```

## Community edition (free-tier)

You just need to install the Sedona jars and Sedona Python on Databricks using Databricks default web UI. Then everything will work.
Expand Down
37 changes: 37 additions & 0 deletions docs/setup/emr.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,43 @@ This tutorial is tested on EMR on EC2 with EMR Studio (notebooks). EMR on EC2 us
!!!note
If you are using Spark 3.4+ and Scala 2.12, please use `sedona-spark-shaded-3.4_2.12`. Please pay attention to the Spark version postfix and Scala version postfix.

## JDK 11+ requirement

Sedona 1.6.0+ requires JDK 11+ to run. For Amazon EMR 7.x, the default JVM is Java 17. For Amazon EMR 5.x and 6.x, the default JVM is Java 8 but you can configure the cluster to use Java 11 or Java 17. For more information, see [EMR JVM versions](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/configuring-java8.html#configuring-java8-override-spark).

When you use Spark with Amazon EMR releases 6.12 and higher, if you write a driver for submission in cluster mode, the driver uses Java 8, but you can set the environment so that the executors use Java 11 or 17. To override the JVM for Spark, AWS EMR recommends that you set both the Hadoop and Spark classifications.

However, it is unclear that if the following will work on EMR below 6.12.

```
{
"Classification": "hadoop-env",
"Configurations": [
{
"Classification": "export",
"Configurations": [],
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-1.11.0"
}
}
],
"Properties": {}
},
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Configurations": [],
"Properties": {
"JAVA_HOME": "/usr/lib/jvm/java-1.11.0"
}
}
],
"Properties": {}
}
```

## Prepare initialization script

In your S3 bucket, add a script that has the following content:
Expand Down
4 changes: 4 additions & 0 deletions docs/setup/fabric.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
This tutorial will guide you through the process of installing Sedona on Microsoft Fabric Synapse Data Engineering's Spark environment.

## JDK 11+ requirement

Sedona 1.6.0+ requires JDK 11+ to run. Microsoft Fabric Synapse Data Engineering 1.2+ uses JDK 11 by default so we recommend using Microsoft Fabric Synapse Data Engineering 1.2+. For more information, see [Apache Spark Runtimes in Fabric](https://learn.microsoft.com/en-us/fabric/data-engineering/runtime).

## Step 1: Open Microsoft Fabric Synapse Data Engineering

Go to the [Microsoft Fabric portal](https://app.fabric.microsoft.com/) and choose the `Data Engineering` option.
Expand Down
4 changes: 3 additions & 1 deletion docs/setup/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
If you use Sedona < 1.6.0, please use GeoPandas <= `0.11.1` since GeoPandas > 0.11.1 will automatically install Shapely 2.0. If you use Shapely, please use <= `1.8.5`.

!!! warning
Sedona 1.6.0+ requires Java 11+ to compile and run. If you are using Java 8, please use Sedona <= 1.5.2.
Sedona 1.6.0+ requires Java 11+ to compile and run. If you are using Java 8, please use Sedona < 1.6.0. To learn how to set up Java 11+ on different platforms, please refer to the Java 11+ requirement in the corresponding platform setup guide.

## Sedona 1.6.0

Expand Down Expand Up @@ -48,6 +48,8 @@ df_raster.withColumn("mean", expr("mean_udf(rast)")).show()
</li>
<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-543'>SEDONA-543</a>] - RS_Union_aggr gives referenceRaster is null error when run on cluster
</li>
<li>[<a href='https://issues.apache.org/jira/browse/SEDONA-555'>SEDONA-555</a>] - Snowflake Native App should not always create a new role
</li>
</ul>

### New Feature
Expand Down
6 changes: 3 additions & 3 deletions docs/setup/wherobots.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
## WherobotsDB

Wherobots Cloud offers fully-managed and fully provisioned cloud services for WherobotsDB, a comprehensive spatial analytics database system. You can play with it using Wherobots Jupyter Scala and Python kernel. No installation is needed.
Wherobots Cloud offers fully-managed and fully provisioned cloud services for WherobotsDB, a comprehensive spatial analytics database system. You can play with it using in a cloud-hosted Jupyter notebook with Python, Java or Scala kernels; no installation is needed.

WherobotsDB is 100% compatible with Apache Sedona in terms of public APIs but provides more functionalities and better performance.
WherobotsDB is 100% compatible with Apache Sedona in terms of public APIs but provides more functionality and better performance.

It is easy to migrate your existing Sedona workflow to Wherobots Cloud. Please sign up at [Wherobots Cloud](https://www.wherobots.services/).
It is easy to migrate your existing Sedona workflow to [Wherobots Cloud](https://www.wherobots.com). Please sign up [here](https://cloud.wherobots.com/) to create your account.
Loading