Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Microsoft Fabric #1198

Closed
rovin-ms opened this issue Jan 12, 2024 · 32 comments · Fixed by #1350
Closed

Microsoft Fabric #1198

rovin-ms opened this issue Jan 12, 2024 · 32 comments · Fixed by #1350

Comments

@rovin-ms
Copy link

rovin-ms commented Jan 12, 2024 •
edited
Loading

Expected behavior

I'm attempting to setup Sedona in Microsoft Fabric. I've loaded the Python packages and the Jar files.

Actual behavior

When I run this code:

import geopandas as gpd

from sedona.spark import *

SedonaContext.builder().config("spark.sql.autoBroadcastJoinThreshold", "10485760")

config = SedonaContext.builder() .
config('spark.jars.packages',
'org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.5.0,'
'org.datasyslab:geotools-wrapper:1.5.0-28.2').
getOrCreate()

sedona = SedonaContext.create(config)
I receive this error:

TypeError Traceback (most recent call last)
Cell In[19], line 13
5 SedonaContext.builder().config("spark.sql.autoBroadcastJoinThreshold", "10485760")
7 config = SedonaContext.builder() .
8 config('spark.jars.packages',
9 'org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.5.0,'
10 'org.datasyslab:geotools-wrapper:1.5.0-28.2').
11 getOrCreate()
---> 13 sedona = SedonaContext.create(config)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/sedona/spark/SedonaContext.py:38, in SedonaContext.create(cls, spark)
36 spark.sql("SELECT 1 as geom").count()
37 PackageImporter.import_jvm_lib(spark._jvm)
---> 38 spark._jvm.SedonaContext.create(spark._jsparkSession)
39 return spark

TypeError: 'JavaPackage' object is not callable

Steps to reproduce the problem

Load all the Python packages in the Public Libraries of Fabric:

shapely="<=1.8.5"
pandas="<=1.3.5"
geopandas="<=0.10.2"
pyspark=">=2.3.0"
attrs=""
pyarrow=""
keplergl = "==0.3.2"
pydeck = "===0.8.0"

Settings

Sedona version = 1.5

Apache Spark version = 3.3.1.5.2-108696741

Apache Flink version = N/A

API type = Scala, Java, Python? Python

Scala version = 2.11, 2.12, 2.13? 2.12

JRE version = 1.8, 1.11?

Python version = 3.10

Environment = Standalone, AWS EC2, EMR, Azure, Databricks? Microsoft Fabric with Runtime 1.1 and 1.2.
@rovin-ms
Copy link
Author

Found the solution. I placed the Jar files for Sedona in an Azure Blob Storage container, and then set the %%configure magic before running the code above. Here's the doc: https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-configuration-magic-command

And here's close to what was added to the first cell in a Notebook:
%%configure -f
{
"jars": ["https://xxxxxx.blob.core.windows.net/jars/sedona-spark-shaded-3.0_2.12-1.5.0.jar", "https://xxxxxx.blob.core.windows.net/jars/geotools-wrapper-1.5.0-28.2.jar"]

}

I was then able to set the SedonaContext. Note that adding the libraries under the Workspace libraries did not appear to work. https://learn.microsoft.com/en-us/fabric/data-engineering/environment-manage-library

@Sarwat
Copy link

Sarwat commented Jan 18, 2024 via email

@rovin-ms
Copy link
Author

rovin-ms commented Jan 25, 2024 via email

@jiayuasu
Copy link
Member

@rovin-ms You can compile the Sedona doc locally (https://sedona.apache.org/1.5.1/setup/compile/#compile-the-documentation):

Install libs

pip install mkdocs
pip install mkdocs-material
pip install mkdocs-macros-plugin
pip install mkdocs-git-revision-date-localized-plugin
pip install mike

Run:

mkdocs serve

You can add a page here: https://github.com/apache/sedona/tree/master/docs/setup , then add it to mkdocs.yml. Then it will show up here like the tutorial for Databricks, EMR: https://sedona.apache.org/1.5.1/setup/emr/

@robertnagy1
Copy link

I can confirm that this is an issue in Fabric, quite annoying that this has to be set at a session level every time, requiring a restart of the spark pool.

@adild2k
Copy link

adild2k commented Mar 10, 2024

Those could able to resolve the issue of using Apache Sedona in Microsoft Fabric, please could you lay down the steps in some sequential order so that the others can follow it properly.

Thanks in advance,

Regards
Adil

@robertnagy1
Copy link

I guess rovin-ms described that how he made it. I followed that as well and it works. The issue is that it adds about 2-4 minutes worth of time to computations.

@adild2k
Copy link

adild2k commented Mar 11, 2024

i was not clear on the jar files where those will need to be placed

@robertnagy1
Copy link

i have them on a blob storage, and i am accessing them through https just like rovin-ms. But you could might as well host them on github or whatever place cause it would work as well.

@adild2k
Copy link

adild2k commented Mar 11, 2024

thanks for the response. I will definitely host those files on Azure blob storage with the help of my company Azure specialist. i will get back if i come across with the issues.

@adild2k
Copy link

adild2k commented Mar 11, 2024

https://jar-download.com/download-handling.php

is this the right side for the jar files to be get downloaded?

@robertnagy1
Copy link

i think you should use one of the maven repositories and match your scala and spark version : https://mvnrepository.com/search?q=sedona

@adild2k
Copy link

adild2k commented Mar 11, 2024

Sorry, for being novice in this part, i am not able to find the respective jar files.

MavenRepoImg1

Further, is there any folder name along with the rights that needs to be given ?

@jiayuasu
Copy link
Member

Hi @adild2k , below are the Sedona jars you will need:

Spark 3.0 to 3.3 + Scala 2.12: https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/1.5.1/
Spark 3.4 + Scala 2.12: https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.4_2.12/1.5.1/
Spark 3.5 + Scala 2.12:https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/1.5.1/

In addition, you will need geotools-wrapper: https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.5.1-28.2/

For example, sedona-spark-shaded-3.5_2.12-1.5.1.jar

@adild2k
Copy link

adild2k commented Mar 11, 2024

Thanks again for the sweet response.

Further, is there any specific folder name on Azure Blob storage that these files needs to be placed along with the rights that needs to be given?

@robertnagy1
Copy link

It does not have to be stored in AZ blob storage, but if you store it there make sure that the container you are trying to reach and folder are reachable without any kind of Authentication. That config magic command will only work in that form if it is reachable without any kind of Auth.

@adild2k
Copy link

adild2k commented Mar 11, 2024

Thanks for the response. I have save the files in the respective folder on Azure, check the snapshot below. After that i need to follow this step mentioned above. Is my understanding right?

%%configure -f
{
"jars": ["https://xxxxxx.blob.core.windows.net/jars/sedona-spark-shaded-3.0_2.12-1.5.0.jar", "https://xxxxxx.blob.core.windows.net/jars/geotools-wrapper-1.5.0-28.2.jar"]

}

AzureSedonaBS

@robertnagy1
Copy link

run it inline in a notebook and it will take some time to fire up the new spark cluster.

@adild2k
Copy link

adild2k commented Mar 11, 2024

Please check the snapshot below, what should be the next step?

AzureSedonaIL

@robertnagy1
Copy link

Follow the tutorial from the official website on creating the sedona context.

@adild2k
Copy link

adild2k commented Mar 11, 2024

Sorry being a novice in this area, please check the snapshot. i know there is something missing which i am not able to figure it out

AzureSedonaContext

@robertnagy1
Copy link

you have to add the python libraries as well, apache-sedona geopandas, 0.11 i think keplergl and pydeck

@adild2k
Copy link

adild2k commented Mar 11, 2024

i have already install keplergl and pydeck earlier along with geopandas. Still its giving the same error

@robertnagy1
Copy link

what about the apache-sedona python module? have you installed that as well?

@adild2k
Copy link

adild2k commented Mar 11, 2024

Yes, off course. that was the first one that has been installed

@adild2k
Copy link

adild2k commented Mar 12, 2024

still the same error. Any suggestions or pointers?

@jiayuasu jiayuasu linked a pull request Apr 22, 2024 that will close this issue
[DOCS] Add Microsoft Fabric tutorial #1350
Merged
@robertnagy1
Copy link

By the way, I forwarded a ticket to Microsoft regarding this. It is an environment problem which they should fix. We shouldn't do any work arounds.

@jiayuasu
Copy link
Member

Thanks guys. I recently created a tutorial about installing Sedona on Fabric. This will be published to Sedona website soon

https://github.com/apache/sedona/blob/master/docs/setup/fabric.md

@robertnagy1
Copy link

@jiayuasu isn't it better to refer to the jar files from the maven repo? I did something like this, so i don't need to host the jar files on remote storage.
%%configure -f
{
"jars": ["https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.5.1-28.2/geotools-wrapper-1.5.1-28.2.jar", "https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.0_2.12/1.5.1/sedona-spark-shaded-3.0_2.12-1.5.1.jar"]
}

@jiayuasu
Copy link
Member

jiayuasu commented Apr 30, 2024 •
edited
Loading

@robertnagy1 Good point. But does Microsoft Fabric always have internet access? Will some user intentionally shut it down for security purpose?

@robertnagy1
Copy link

I guess having it on a remote abfss requires internet as well, and accessing Fabric requires internet, and the spark clusters require internet. The Lakehouse (as far as i know, but i might be wrong) is an abstraction layer above Microsoft One Lake, which is separate from the Spark Instances. I think it should be safe to assume that Fabric requires internet to work.

@jiayuasu
Copy link
Member

Makes sense. Will update the doc accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[DOCS] Add Microsoft Fabric tutorial apache/sedona
5 participants
@Sarwat @jiayuasu @rovin-ms @adild2k @robertnagy1