MSSQL Spark connector, and readme fixes

samples/features/sql-big-data-cluster/spark/README.md

@@ -1,29 +1,27 @@
 # SQL Server big data clusters
 
-The new built-in notebooks in Azure Data Studio enables data scientists and data engineers to run Python, R, Scala, or Spark SQL code against the cluster.
+SQL Server Big Data cluster bundles Spark and HDFS together with SQL server. Azure Data Studio IDE provides built in notebooks that enables data scientists and data engineers to run Spark notebooks and job in Python, R, or Scala code against the Big Data Cluster. This folder contains spark sample notebook on using Spark in SQL server Big data cluster
 
-## Instructions to open a notebook from Azure Data Studio and execute the commands
+## Folder contents
 
-1. Connect to the SQL Server Master instance in a big data cluster
+[PySpark Hello World](dataloading/hello_PySpark.ipynb)
 
-1. Right-click on the server name, select **Manage**, switch to **SQL Server Big Data Cluster** tab, and use open Notebook.
+[Scala Hello World ](dataloading/hello_Scala.ipynb)
 
-1. Open the notebook in Azure Data Studio, wait for the “Kernel” and the target context (“Attach to”) to be populated.
+[SparkR Hello World ](dataloading/hello_sparkR.ipynb)
 
-1. Run each cell in the Notebook sequentially.
+[DataLoading   - Transforming CSV to Parquet](dataloading/transform-csv-files.ipynb/)
 
-## __[data-loading](data-loading/)__
+[Data Transfer - Spark to SQL using Spark JDBC connector](data-virtualization/spark_to_sql_jdbc.ipynb/)
 
-This folder contains samples that show how to load data using Spark and query them using SQL statements.
+[Data Transfer - Spark to SQL using MSSQL Spark connector](spark_to_sql/mssql_spark_connector.ipynb/)
 
-[data-loading/transform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/)
-
-This samnple notebook shows how to transform CSV files in HDFS to parquet files.
+## Instructions on how to run in Azure Data Studio
 
-[dataloading/spark-sql.ipynb](dataloading/spark-sql.ipynb/)
+[data-loading/transform-csv-files.ipynb](dataloading/transform-csv-files.ipynb/)
 
-This samnple notebook shows how to query hive tables created from Spark.
+2. From Azure Data Studio Connect to the SQL Server Master instance in a big data cluster. 
 
-## __[data-virtualization](data-virtualization/)__
+3. Right-click on the server name, select **Manage**, switch to **SQL Server Big Data Cluster** tab, and open the notebook in Azure Data Studio.  Wait for the “Kernel” and the target context (“Attach to”) to be populated. If required set the relevant “Kernel” ( e.g **PySpark3** )  and **Attach to** needs to be the IP address of your big data cluster endpoint.
 
-This folder contains samples that show how to integrate Spark with other data sources.
+4. Run each cell in the Notebook sequentially.

samples/features/sql-big-data-cluster/spark/data-virtualization/spark_to_sql_jdbc.ipynb

@@ -19,7 +19,7 @@
     "cells": [
         {
             "cell_type": "markdown",
-            "source": "# Read and write from Spark to SQL\r\nA typical big data scenario is large scale ETL in Spark and writing the processed data to SQLServer. The following samples shows \r\n- reading a HDFS file, \r\n- some basic processing on it and \r\n- then processed data to SQL Server table.\r\n\r\nNeed a database precreated in SQL for this sample. Here we are using database name \"MyTestDatabase\" that can be created using SQL statements below.\r\n\r\n``` sql\r\nCreate DATABASE MyTestDatabase\r\nGO \r\n``` \r\n ",
+            "source": "# Read and write from Spark to SQL\r\nA typical big data scenario is large scale ETL in Spark and post processing the data is written out to SQLServer for access to LOB applications. This sample shows how to write to SQLServer from Spark. The main steps in the sample are  \r\n- Reading a HDFS file, \r\n- Basic processing on it and \r\n- Then writing processed data to SQL Server table using JDBC\r\n\r\nPreReq : \r\n- The sample uses a SQL database named \"MyTestDatabase\". Create this before you run this sample. The database can be created as follows\r\n    ``` sql\r\n    Create DATABASE MyTestDatabase\r\n    GO \r\n    ``` \r\n- Download [AdultCensusIncome.csv]( https://amldockerdatasets.azureedge.net/AdultCensusIncome.csv ) to your local machine.  Create a hdfs folder named spark_data and upload the file there.  \r\n\r\n    \r\n ",
             "metadata": {}
         },
         {
@@ -28,51 +28,69 @@
             "metadata": {},
             "outputs": [
                 {
-                    "output_type": "stream",
                     "name": "stdout",
-                    "text": "+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age|       workclass|fnlwgt|education|education-num|    marital-status|       occupation| relationship| race|   sex|capital-gain|capital-loss|hours-per-week|native-country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|\n| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|\n| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|\n| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows"
+                    "text": "Starting Spark application\n",
+                    "output_type": "stream"
+                },
+                {
+                    "data": {
+                        "text/plain": "<IPython.core.display.HTML object>",
+                        "text/html": "<table>\n<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>2</td><td>application_1554755839506_0003</td><td>pyspark3</td><td>idle</td><td><a target=\"_blank\" href=\"https://40.78.42.207:30443/gateway/default/yarn/proxy/application_1554755839506_0003/\">Link</a></td><td><a target=\"_blank\" href=\"http://storage-0-1.storage-0-svc.mssql-cluster.svc.cluster.local:8042/node/containerlogs/container_1554755839506_0003_01_000001/root\">Link</a></td><td>✔</td></tr></table>"
+                    },
+                    "metadata": {},
+                    "output_type": "display_data"
+                },
+                {
+                    "name": "stdout",
+                    "text": "SparkSession available as 'spark'.\n",
+                    "output_type": "stream"
+                },
+                {
+                    "name": "stdout",
+                    "text": "+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age|       workclass|fnlwgt|education|education-num|    marital-status|       occupation| relationship| race|   sex|capital-gain|capital-loss|hours-per-week|native-country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|\n| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|\n| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|\n| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows",
+                    "output_type": "stream"
                 }
             ],
-            "execution_count": 8
+            "execution_count": 3
         },
         {
             "cell_type": "code",
             "source": "\r\n#Process this data. Very simple data cleanup steps. Replacing \"-\" with \"_\" in column names\r\ncolumns_new = [col.replace(\"-\", \"_\") for col in df.columns]\r\ndf = df.toDF(*columns_new)\r\ndf.show(5)\r\n\r\n",
             "metadata": {},
             "outputs": [
                 {
-                    "output_type": "stream",
                     "name": "stdout",
-                    "text": "+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age|       workclass|fnlwgt|education|education_num|    marital_status|       occupation| relationship| race|   sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|\n| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|\n| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|\n| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows"
+                    "text": "+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age|       workclass|fnlwgt|education|education_num|    marital_status|       occupation| relationship| race|   sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|\n| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|\n| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|\n| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows",
+                    "output_type": "stream"
                 }
             ],
-            "execution_count": 9
+            "execution_count": 4
         },
         {
             "cell_type": "code",
-            "source": "#Write from Spark to SQL table using JDBC\r\nprint(\"Use build in JDBC connector to write to SQLServer master instance in Big data \")\r\n\r\nservername = \"jdbc:sqlserver://mssql-master-pool-0.service-master-pool\"\r\ndbname = \"MyTestDatabase\"\r\nurl = servername + \";\" + \"databaseName=\" + dbname + \";\"\r\n\r\nc = \"dbo.AdultCensus\"\r\nuser = \"sa\"\r\npassword = \"****\"\r\n\r\nprint(\"url is \", url)\r\n\r\ntry:\r\n  df.write \\\r\n    .format(\"jdbc\") \\\r\n    .mode(\"overwrite\") \\\r\n    .option(\"url\", url) \\\r\n    .option(\"dbtable\", dbtable) \\\r\n    .option(\"user\", user) \\\r\n    .option(\"password\", password)\\\r\n    .save()\r\nexcept ValueError as error :\r\n    print(\"JDBC Write failed\", error)\r\n\r\nprint(\"JDBC Write done  \")\r\n\r\n\r\n",
+            "source": "#Write from Spark to SQL table using JDBC\r\nprint(\"Use build in JDBC connector to write to SQLServer master instance in Big data \")\r\n\r\nservername = \"jdbc:sqlserver://master-0.master-svc\"\r\ndbname = \"MyTestDatabase\"\r\nurl = servername + \";\" + \"databaseName=\" + dbname + \";\"\r\n\r\ndbtable = \"dbo.AdultCensus\"\r\nuser = \"sa\"\r\npassword = \"Yukon900\"\r\n\r\nprint(\"url is \", url)\r\n\r\ntry:\r\n  df.write \\\r\n    .format(\"jdbc\") \\\r\n    .mode(\"overwrite\") \\\r\n    .option(\"url\", url) \\\r\n    .option(\"dbtable\", dbtable) \\\r\n    .option(\"user\", user) \\\r\n    .option(\"password\", password)\\\r\n    .save()\r\nexcept ValueError as error :\r\n    print(\"JDBC Write failed\", error)\r\n\r\nprint(\"JDBC Write done  \")\r\n\r\n\r\n",
             "metadata": {},
             "outputs": [
                 {
-                    "output_type": "stream",
                     "name": "stdout",
-                    "text": "Use build in JDBC connector to write to SQLServer master instance in Big data \nurl is  jdbc:sqlserver://mssql-master-pool-0.service-master-pool;databaseName=MyTestDatabase;\nJDBC Write done"
+                    "text": "Use build in JDBC connector to write to SQLServer master instance in Big data \nurl is  jdbc:sqlserver://master-0.master-svc;databaseName=MyTestDatabase;\nJDBC Write done",
+                    "output_type": "stream"
                 }
             ],
-            "execution_count": 10
+            "execution_count": 9
         },
         {
             "cell_type": "code",
             "source": "#Read to Spark from SQL table using JDBC\r\nprint(\"read data from SQL server table  \")\r\njdbcDF = spark.read \\\r\n        .format(\"jdbc\") \\\r\n        .option(\"url\", url\r\n        ) \\\r\n        .option(\"dbtable\", dbtable) \\\r\n        .option(\"user\", user) \\\r\n        .option(\"password\", password) \\\r\n        .load()\r\n\r\njdbcDF.show(5)",
             "metadata": {},
             "outputs": [
                 {
-                    "output_type": "stream",
                     "name": "stdout",
-                    "text": "read data from SQL server table  \n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age|       workclass|fnlwgt|education|education_num|    marital_status|       occupation| relationship| race|   sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|\n| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|\n| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|\n| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows"
+                    "text": "read data from SQL server table  \n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n|age|       workclass|fnlwgt|education|education_num|    marital_status|       occupation| relationship| race|   sex|capital_gain|capital_loss|hours_per_week|native_country|income|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\n| 39|       State-gov| 77516|Bachelors|           13|     Never-married|     Adm-clerical|Not-in-family|White|  Male|        2174|           0|            40| United-States| <=50K|\n| 50|Self-emp-not-inc| 83311|Bachelors|           13|Married-civ-spouse|  Exec-managerial|      Husband|White|  Male|           0|           0|            13| United-States| <=50K|\n| 38|         Private|215646|  HS-grad|            9|          Divorced|Handlers-cleaners|Not-in-family|White|  Male|           0|           0|            40| United-States| <=50K|\n| 53|         Private|234721|     11th|            7|Married-civ-spouse|Handlers-cleaners|      Husband|Black|  Male|           0|           0|            40| United-States| <=50K|\n| 28|         Private|338409|Bachelors|           13|Married-civ-spouse|   Prof-specialty|         Wife|Black|Female|           0|           0|            40|          Cuba| <=50K|\n+---+----------------+------+---------+-------------+------------------+-----------------+-------------+-----+------+------------+------------+--------------+--------------+------+\nonly showing top 5 rows",
+                    "output_type": "stream"
                 }
             ],
-            "execution_count": 13
+            "execution_count": 11
         }
     ]
 }