Correct process to download and install libraries offline

[grajee-everest]

What is the correct way to download the sparknlp library for offline installation/deployment?

We have a SQLServer Big Data Cluster based on Kubernetes. Unlike Databricks, there is no library management feature and because of this we have to manually download the libraries, copy them to the hdfs location and include them in pyspark using sc.addFile.

So, the command I used to download sparknlp is "pip install spark-nlp -t spark-nlp". Since spark-nlp is not working I tried downloading the files from the link and untarred the files and compared the folders and from the two different methods and they seem to be different.

Nevertheless, I tried both the methods and they both are not working in our environment. Please refer to the Issue - 6506

I just want to be clear as to how/which is the right way to download libraries for offline usage. This is not an issue in Databricks because of its robust library management UI.

Environment:
SQLServer 2019 BDC

All the environment details are here :

Microsoft Spark Runtime 2021.1
Spark: 3.1.2
Delta Lake: 1.0.0
Java: Azul Zulu JRE 1.8.0_275
Scala: 2.12
Python: 3.8 (miniforge 4.9)
R: Microsoft R 3.5.2
Spark SQL Connector: 1.2.0

[grajee-everest]

As a follow up question -

I got it working in Azure Databricks in a breeze without any effort at all. In the SQLServer BDC, I had to download the package from pypi and the FAT JAR.

But from the UI of Databricks, I just chose pypi and it downloaded the libraries and was enough for me to get sparknlp working. Does this mean that the JAR files are not needed for Databricks?

[maziyarpanahi]

Hi,

We have very clear step by step instructions for everything you asked:

• Databricks (you need both PyPI and Maven, the Python is just APIs, you always need the Maven package or the Fat JAR as it’s the core library): https://github.com/JohnSnowLabs/spark-nlp#databricks-cluster
• Total offline install setup: well this one is easy, you download the Whl file from pypi.org and install the PyPI package like other libraries (on all the machines in the cluster - an actual “pip install “ and use the Fat JAR we provide in each release notes to include in SparkSession (no internet ever needed for air-gaped environment, not really worth it for Databricks on AWS/Azure since they have an easy UI to set everything online)
• offline models/pipelines: https://github.com/JohnSnowLabs/spark-nlp#offline

as you tried in other cluster environments, you must have spark-nlp package installed via pip on all machines (python or conda env that is configured for PySpark to look for when it requires a pypi package). It won’t work if you just install it on 1 machine if you are in cluster, it won’t work if you just simply adding some files via addFiles (this will never work, it adds file to Spark session to access during runtime, where you need access to actual python package each executor looms on a local storage (python or conda env) for that package).

I am not familiar with the platform you are using, but it must have a way at least for your Admin or DevOps to install PyPi packages on any cluster that starts. (They all have bootstrap pr scripts to install anything).