diff --git a/.azuredatastudio/settings.json b/.azuredatastudio/settings.json
new file mode 100644
index 00000000..a7458417
--- /dev/null
+++ b/.azuredatastudio/settings.json
@@ -0,0 +1,8 @@
+{
+ "workbench.colorCustomizations": {
+ "activityBar.background": "#05351D",
+ "titleBar.activeBackground": "#074A29",
+ "titleBar.activeForeground": "#EFFEF6"
+ },
+ "python.linting.pylintEnabled": true
+}
\ No newline at end of file
diff --git a/.vscode/settings.json b/.vscode/settings.json
new file mode 100644
index 00000000..7a73a41b
--- /dev/null
+++ b/.vscode/settings.json
@@ -0,0 +1,2 @@
+{
+}
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/00-Pre-Requisites.md b/ModernizeYourDatabases2019/ModernizeSQL2019/00-Pre-Requisites.md
index 803cd395..77ad1ce5 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/00-Pre-Requisites.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/00-Pre-Requisites.md
@@ -1,42 +1,42 @@
-
-
-# Workshop: Modernize Your Database with SQL Server 2019
-
-#### A Microsoft Course from the SQL Server team
-
-
-
-
00 Pre-Requisites
-
-Pre-Requisites for the "Modernize Your Database with SQL Server 2019" workshop are found in each module. At a later date all pre-requisites will be listed in this module.
-
-*Note that all following activities must be completed prior to class - there will not be time to perform these operations during the workshop.*
-
-
Activity 1: TBD
-
-
Option 1 - TBD
-
-
Activity 2: TBD
-
-
-
Activity 2: TODO: Step within Activity.
-
-
-
TODO: Sub-step
-
-First, ensure all of your updates are current. You can use the following commands to do that in an Administrator-level PowerShell session:
-
-
-
-Next, Continue to Why SQL Server 2019.
+
+
+# Workshop: Modernize Your Database with SQL Server 2019
+
+#### A Microsoft Course from the SQL Server team
+
+
+
+
00 Pre-Requisites
+
+Pre-Requisites for the "Modernize Your Database with SQL Server 2019" workshop are found in each module. At a later date all pre-requisites will be listed in this module.
+
+*Note that all following activities must be completed prior to class - there will not be time to perform these operations during the workshop.*
+
+
Activity 1: TBD
+
+
Option 1 - TBD
+
+
Activity 2: TBD
+
+
+
Activity 2: TODO: Step within Activity.
+
+
+
TODO: Sub-step
+
+First, ensure all of your updates are current. You can use the following commands to do that in an Administrator-level PowerShell session:
+
+
+
+Next, Continue to Why SQL Server 2019.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/01-WhySQL2019.md b/ModernizeYourDatabases2019/ModernizeSQL2019/01-WhySQL2019.md
index dff9edde..04d498d6 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/01-WhySQL2019.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/01-WhySQL2019.md
@@ -1,16 +1,16 @@
-
-
-# Workshop: Modernize Your Database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Why SQL Server 2019?
-
-This module will be completed at a later date.
-
-
+
+Next, Continue to Intelligent Performance.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/02-IntelligentPerformance.md b/ModernizeYourDatabases2019/ModernizeSQL2019/02-IntelligentPerformance.md
index f970e0e1..2e02a00f 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/02-IntelligentPerformance.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/02-IntelligentPerformance.md
@@ -1,60 +1,60 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Intelligent Performance
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [Lightweight Query Profiling Exercise](Module%202%20Activity%20-%20Intelligent%20Performance/lwp)
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to New Security Capabilities.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Intelligent Performance
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [Lightweight Query Profiling Exercise](Module%202%20Activity%20-%20Intelligent%20Performance/lwp)
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to New Security Capabilities.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/03-Security.md b/ModernizeYourDatabases2019/ModernizeSQL2019/03-Security.md
index eb5acebf..e9de4dfa 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/03-Security.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/03-Security.md
@@ -1,39 +1,39 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
New Security Capabilities
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [Static Data Masking Exercise](Module%203%20Activity%20-%20Security/staticmask)
-
-
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to Mission Critical Availability.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
New Security Capabilities
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [Static Data Masking Exercise](Module%203%20Activity%20-%20Security/staticmask)
+
+
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to Mission Critical Availability.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/04-MissionCriticalAvailability.md b/ModernizeYourDatabases2019/ModernizeSQL2019/04-MissionCriticalAvailability.md
index ae74daf6..67ab73e4 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/04-MissionCriticalAvailability.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/04-MissionCriticalAvailability.md
@@ -1,38 +1,38 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Mission Critical Availability
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [Accelerated Database Recovery Exercise](Module%204%20Activity%20-%20MIssion%20Critical%20Availability/adr)
-
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to Modern Development Platform.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Mission Critical Availability
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [Accelerated Database Recovery Exercise](Module%204%20Activity%20-%20MIssion%20Critical%20Availability/adr)
+
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to Modern Development Platform.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/05-ModernDevelopmentPlatform.md b/ModernizeYourDatabases2019/ModernizeSQL2019/05-ModernDevelopmentPlatform.md
index d080d15d..9cfdb68b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/05-ModernDevelopmentPlatform.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/05-ModernDevelopmentPlatform.md
@@ -1,49 +1,49 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Modern Development Platform
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [Java Exercise](Module%205%20Activity%20-%20Modern%20Development%20Platform/java)
-
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to SQL Server on Linux.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Modern Development Platform
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [Java Exercise](Module%205%20Activity%20-%20Modern%20Development%20Platform/java)
+
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to SQL Server on Linux.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/06-SQLLinux.md b/ModernizeYourDatabases2019/ModernizeSQL2019/06-SQLLinux.md
index 3572a226..ed8ec075 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/06-SQLLinux.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/06-SQLLinux.md
@@ -1,49 +1,49 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
SQL Server on Linux
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [Explore SQL Server on Linux Exercise](Module%206%20Activity%20-%20SQL%20Server%20on%20Linux/explore)
-
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to SQL Server Containers and Kubernetes.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
SQL Server on Linux
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [Explore SQL Server on Linux Exercise](Module%206%20Activity%20-%20SQL%20Server%20on%20Linux/explore)
+
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to SQL Server Containers and Kubernetes.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/07-SQLContainers.md b/ModernizeYourDatabases2019/ModernizeSQL2019/07-SQLContainers.md
index 7774d4ff..fd3a8120 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/07-SQLContainers.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/07-SQLContainers.md
@@ -1,49 +1,49 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
SQL Server Containers and Kubernetes
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [SQL Server Containers Fundamentals Exercise](Module%207%20Activity%20-%20SQL%20Server%20Containers/sqlcontainers)
-
-
-
-
-
-Follow the instructions for [Updating and Upgrading with SQL Containers Exercise](Module%207%20Activity%20-%20SQL%20Server%20Containers/sqlcontainerupdate)
-
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to Data Virtualization.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
SQL Server Containers and Kubernetes
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [SQL Server Containers Fundamentals Exercise](Module%207%20Activity%20-%20SQL%20Server%20Containers/sqlcontainers)
+
+
+
+
+
+Follow the instructions for [Updating and Upgrading with SQL Containers Exercise](Module%207%20Activity%20-%20SQL%20Server%20Containers/sqlcontainerupdate)
+
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to Data Virtualization.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/08-DataVirtualization.md b/ModernizeYourDatabases2019/ModernizeSQL2019/08-DataVirtualization.md
index 5ad67255..806f5942 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/08-DataVirtualization.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/08-DataVirtualization.md
@@ -1,49 +1,49 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
SQL Server Data Virtualization
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [SQL Server Data Hub Exercise](Module%208%20Activity%20-%20Data%20Virtualization/sqldatahub)
-
-
-
-
-
For Further Study
-
-
-
-
Next Steps
-
-Next, Continue to What Else is New in SQL Server 2019.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
SQL Server Data Virtualization
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [SQL Server Data Hub Exercise](Module%208%20Activity%20-%20Data%20Virtualization/sqldatahub)
+
+
+
+
+
For Further Study
+
+
+
+
Next Steps
+
+Next, Continue to What Else is New in SQL Server 2019.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/09-WhatElseIsNew.md b/ModernizeYourDatabases2019/ModernizeSQL2019/09-WhatElseIsNew.md
index 26b6107d..3533305a 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/09-WhatElseIsNew.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/09-WhatElseIsNew.md
@@ -1,37 +1,37 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
What else is new in SQL Server 2019
-
-You'll cover the following topics in this Module:
-
-
-
-Next, Continue to Migrating to SQL Server 2019 and Next Steps.
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
What else is new in SQL Server 2019
+
+You'll cover the following topics in this Module:
+
+
+
+Next, Continue to Migrating to SQL Server 2019 and Next Steps.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/10-MigratingAndNextSteps.md b/ModernizeYourDatabases2019/ModernizeSQL2019/10-MigratingAndNextSteps.md
index da753b20..34040a06 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/10-MigratingAndNextSteps.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/10-MigratingAndNextSteps.md
@@ -1,31 +1,31 @@
-
-
-# Workshop: Modernize your database with SQL Server 2019
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Migrating to SQL Server 2019 and Next Steps
-
-You'll cover the following topics in this Module:
-
-
-
-Follow the instructions for [Query Tuning Assistant Exercise](Module%2010%20Activity%20-%20Migrating%20to%20SQL%20Server%202019/qta)
-
-
-
-
For Further Study
+
+
+# Workshop: Modernize your database with SQL Server 2019
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Migrating to SQL Server 2019 and Next Steps
+
+You'll cover the following topics in this Module:
+
+
+
+Follow the instructions for [Query Tuning Assistant Exercise](Module%2010%20Activity%20-%20Migrating%20to%20SQL%20Server%202019/qta)
+
+
+
+
For Further Study
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/qta/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/qta/readme.md
index b6fd30ef..9c542ad3 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/qta/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/qta/readme.md
@@ -1,9 +1,9 @@
-# Using Query Tuning Assistant for Post Migration Optimization
-
-This exercise will show you how to use the new Query Tuning Assistant(QTA) to optimize performance after migrating to SQL Server 2019
-
-To performance this exercise:
-
-1. Download the following zip file to your computer with SQL Server or client computer that can connect to SQL Server at https://github.com/Microsoft/tigertoolbox/blob/master/Sessions/Winter-Ready-2019/Labs/Lab-QTA.zip.
-
-2. Read through the instructions at https://github.com/Microsoft/tigertoolbox/blob/master/Sessions/Winter-Ready-2019/Lab-QTA.md to perform the lab.
+# Using Query Tuning Assistant for Post Migration Optimization
+
+This exercise will show you how to use the new Query Tuning Assistant(QTA) to optimize performance after migrating to SQL Server 2019
+
+To performance this exercise:
+
+1. Download the following zip file to your computer with SQL Server or client computer that can connect to SQL Server at https://github.com/Microsoft/tigertoolbox/blob/master/Sessions/Winter-Ready-2019/Labs/Lab-QTA.zip.
+
+2. Read through the instructions at https://github.com/Microsoft/tigertoolbox/blob/master/Sessions/Winter-Ready-2019/Lab-QTA.md to perform the lab.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/readme.md
index f7c02ad8..e6d14a57 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 10 Activity - Migrating to SQL Server 2019/readme.md
@@ -1,7 +1,7 @@
-# Module 10 Activities - Migrating to SQL Server 2019
-
-These represent demos and examples to show you migration techniques for SQL Server. Right now this Module only includes exercises for post migration optmization using Query Tuning Assistant (QTA)
-
-## qta
-
+# Module 10 Activities - Migrating to SQL Server 2019
+
+These represent demos and examples to show you migration techniques for SQL Server. Right now this Module only includes exercises for post migration optmization using Query Tuning Assistant (QTA)
+
+## qta
+
Learn how to use Query Tuning Assistant (QTA) to assist with post migration optimizations.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/auto_tune.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/auto_tune.sql
index e2e8d19d..c543938a 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/auto_tune.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/auto_tune.sql
@@ -1,4 +1,4 @@
-use WideWorldImporters
-go
-exec auto_tune
+use WideWorldImporters
+go
+exec auto_tune
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/initialize.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/initialize.sql
index 07b8a519..ccfa8ffa 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/initialize.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/initialize.sql
@@ -1,4 +1,4 @@
-use WideWorldImporters
-go
-exec initialize
+use WideWorldImporters
+go
+exec initialize
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/readme.md
index 54ac2b17..a57d6150 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/readme.md
@@ -1,36 +1,36 @@
-# Automatic Tuning with SQL Server
-
-This is a repro package to demonstrate the Automatic Tuning (Auto Plan Correction) in SQL Server 2017. This feature is using telemtry from the Query Store feature we launched with Azure SQL Database and SQL Server 2016 to provide built-in intelligence.
-
-## Requirements
-
-This repro assumes the following:
-
-- SQL Server 2017 installed (pick at minimum Database Engine) on Windows. This feature requires Developer or Enterprise Edition.
-- You have installed SQL Server Management Studio or SQL Operations Studio (https://docs.microsoft.com/en-us/sql/sql-operations-studio/download)
-- You have downloaded the RML Utilities from https://www.microsoft.com/en-us/download/details.aspx?id=4511.
-- These demos use a named instance called SQL2017. You will need to edit the .cmd scripts which connect to SQL Server to change to a default instance or whatever named instance you have installed.
-
-- Install ostress from the package RML_Setup_AMD64.msi. Add C:\Program Files\Microsoft Corporation\RMLUtils to your path.
-
-- Restore the WideWorldImporters database backup to your SQL Server 2017 instance. The WideWorldImporters-Full.bak is provided along with a **restorewwi.sql** script to restore the database. This script assumes the backup is in the C:\sql_sample_databases directory and that all database files will be placed in c:\temp. Change the location for the backup and your files as needed.
-
-## Demo Steps
-
-1. Run **repro_setup.cmd** to customize the WideWorldImporters database for the demo. You will only need to run this one time after restoring the backup.
-
-2. Setup Performance Monitor on Windows to track SQL Statistics/Batch Requests/sec
-
-3. Run **initalize.cmd**to setup the repro for default of recommendations. If you restart the demo from the beginning, you can run this again to "reset" the demo.
-
-4. Run **report.cmd** to start the workload. This will pop-up a command window running the workload. Note the chart showing Batch Requests/Sec as your workload throughput
-
-5. Run **regression.cmd** (you may need to run this a few times for timing reasons). Notice the drop in batch requests/sec which shows a performance regression in your workload.
-
-6. Load **recommendations.sql** into SQL Server Management Studio or SQL Operations Studio and review the results. Notice the time difference under the reason column and value of state_transition_reason which should be AutomaticTuningOptionNotEnabled. This means we found a regression but are recommending it only, not automatically fixing it. The script column shows a query that could be used to fix the problem.
-
-7. Stop the **report.cmd** workload by pressing + in the command window and pressing 'y' to stop. This should close that command window.
-
-8. Now let's see what happens with automatic plan correction. Run **auto_tune.cmd**which sets automatic plan correct ON for WideWorldImporters
-
+# Automatic Tuning with SQL Server
+
+This is a repro package to demonstrate the Automatic Tuning (Auto Plan Correction) in SQL Server 2017. This feature is using telemtry from the Query Store feature we launched with Azure SQL Database and SQL Server 2016 to provide built-in intelligence.
+
+## Requirements
+
+This repro assumes the following:
+
+- SQL Server 2017 installed (pick at minimum Database Engine) on Windows. This feature requires Developer or Enterprise Edition.
+- You have installed SQL Server Management Studio or SQL Operations Studio (https://docs.microsoft.com/en-us/sql/sql-operations-studio/download)
+- You have downloaded the RML Utilities from https://www.microsoft.com/en-us/download/details.aspx?id=4511.
+- These demos use a named instance called SQL2017. You will need to edit the .cmd scripts which connect to SQL Server to change to a default instance or whatever named instance you have installed.
+
+- Install ostress from the package RML_Setup_AMD64.msi. Add C:\Program Files\Microsoft Corporation\RMLUtils to your path.
+
+- Restore the WideWorldImporters database backup to your SQL Server 2017 instance. The WideWorldImporters-Full.bak is provided along with a **restorewwi.sql** script to restore the database. This script assumes the backup is in the C:\sql_sample_databases directory and that all database files will be placed in c:\temp. Change the location for the backup and your files as needed.
+
+## Demo Steps
+
+1. Run **repro_setup.cmd** to customize the WideWorldImporters database for the demo. You will only need to run this one time after restoring the backup.
+
+2. Setup Performance Monitor on Windows to track SQL Statistics/Batch Requests/sec
+
+3. Run **initalize.cmd**to setup the repro for default of recommendations. If you restart the demo from the beginning, you can run this again to "reset" the demo.
+
+4. Run **report.cmd** to start the workload. This will pop-up a command window running the workload. Note the chart showing Batch Requests/Sec as your workload throughput
+
+5. Run **regression.cmd** (you may need to run this a few times for timing reasons). Notice the drop in batch requests/sec which shows a performance regression in your workload.
+
+6. Load **recommendations.sql** into SQL Server Management Studio or SQL Operations Studio and review the results. Notice the time difference under the reason column and value of state_transition_reason which should be AutomaticTuningOptionNotEnabled. This means we found a regression but are recommending it only, not automatically fixing it. The script column shows a query that could be used to fix the problem.
+
+7. Stop the **report.cmd** workload by pressing + in the command window and pressing 'y' to stop. This should close that command window.
+
+8. Now let's see what happens with automatic plan correction. Run **auto_tune.cmd**which sets automatic plan correct ON for WideWorldImporters
+
9. Repeat steps 4-7 as above. In Performance Monitor you will see the batch requests/sec dip but within a second go right back up. This is because SQL Server detected the regression and automatically reverted to "last known good" or the last known good query plan as found in the Query Store. Note in the output of recommendations.sql the state_transition_reason now says LastGoodPlanForced.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/recommendations.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/recommendations.sql
index 75b962e5..a570c530 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/recommendations.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/recommendations.sql
@@ -1,14 +1,14 @@
-use WideWorldImporters
-go
-SELECT reason, score,
- JSON_VALUE(state, '$.currentValue') state,
- JSON_VALUE(state, '$.reason') state_transition_reason,
- JSON_VALUE(details, '$.implementationDetails.script') script,
- planForceDetails.*
-FROM sys.dm_db_tuning_recommendations
- CROSS APPLY OPENJSON (Details, '$.planForceDetails')
- WITH ( [query_id] int '$.queryId',
- [new plan_id] int '$.regressedPlanId',
- [forcedPlanId] int '$.forcedPlanId'
- ) as planForceDetails;
+use WideWorldImporters
+go
+SELECT reason, score,
+ JSON_VALUE(state, '$.currentValue') state,
+ JSON_VALUE(state, '$.reason') state_transition_reason,
+ JSON_VALUE(details, '$.implementationDetails.script') script,
+ planForceDetails.*
+FROM sys.dm_db_tuning_recommendations
+ CROSS APPLY OPENJSON (Details, '$.planForceDetails')
+ WITH ( [query_id] int '$.queryId',
+ [new plan_id] int '$.regressedPlanId',
+ [forcedPlanId] int '$.forcedPlanId'
+ ) as planForceDetails;
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/regression.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/regression.sql
index d5e323de..81b6edb7 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/regression.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/regression.sql
@@ -1,4 +1,4 @@
-use WideWorldImporters
-go
-exec regression
+use WideWorldImporters
+go
+exec regression
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/report.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/report.sql
index 5b5b245c..5c150d3b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/report.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/report.sql
@@ -1,5 +1,5 @@
-use WideWorldImporters
-go
-declare @packagetypeid int = 7;
-exec dbo.report @packagetypeid
+use WideWorldImporters
+go
+declare @packagetypeid int = 7;
+exec dbo.report @packagetypeid
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/restorewwi.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/restorewwi.sql
index 9076a487..5c505aa6 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/restorewwi.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/restorewwi.sql
@@ -1,7 +1,7 @@
-restore database WideWorldImporters from disk = 'C:\sql_sample_databases\WideWorldImporters-Full.bak' with
-move 'WWI_Primary' to 'c:\temp\WideWorldImporters.mdf',
-move 'WWI_UserData' to 'c:\temp\WideWorldImporters_UserData.ndf',
-move 'WWI_Log' to 'c:\temp\WideWorldImporters.ldf',
-move 'WWI_InMemory_Data_1' to 'c:\temp\WideWorldImporters_InMemory_Data_1',
-stats=5
+restore database WideWorldImporters from disk = 'C:\sql_sample_databases\WideWorldImporters-Full.bak' with
+move 'WWI_Primary' to 'c:\temp\WideWorldImporters.mdf',
+move 'WWI_UserData' to 'c:\temp\WideWorldImporters_UserData.ndf',
+move 'WWI_Log' to 'c:\temp\WideWorldImporters.ldf',
+move 'WWI_InMemory_Data_1' to 'c:\temp\WideWorldImporters_InMemory_Data_1',
+stats=5
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/setup.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/setup.sql
index b0f65b48..d5579576 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/setup.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/autotune/setup.sql
@@ -1,47 +1,47 @@
-use wideworldimporters
-go
-DROP procedure IF EXISTS [dbo].[initialize]
-go
-CREATE procedure [dbo].[initialize]
-as begin
-DBCC FREEPROCCACHE;
-ALTER DATABASE current SET QUERY_STORE CLEAR ALL;
-ALTER DATABASE current SET AUTOMATIC_TUNING ( FORCE_LAST_GOOD_PLAN = OFF);
-end
-GO
-
-DROP procedure IF EXISTS [dbo].[auto_tune]
-go
-CREATE procedure [dbo].[auto_tune]
-as begin
-ALTER DATABASE current SET AUTOMATIC_TUNING ( FORCE_LAST_GOOD_PLAN = ON);
-DBCC FREEPROCCACHE;
-ALTER DATABASE current SET QUERY_STORE CLEAR ALL;
-end
-GO
-
-DROP PROCEDURE IF EXISTS [dbo].[report]
-go
-CREATE PROCEDURE [dbo].[report] ( @packagetypeid INT )
-AS
- BEGIN
-
- SELECT AVG([UnitPrice] * [Quantity] - [TaxRate])
- FROM [Sales].[OrderLines]
- WHERE [PackageTypeID] = @packagetypeid;
-
- END;
-GO
-
-DROP PROCEDURE IF EXISTS [dbo].[regression]
-go
-CREATE PROCEDURE [dbo].[regression]
-AS
- BEGIN
- DBCC FREEPROCCACHE;
- BEGIN
- DECLARE @packagetypeid INT = 1;
- EXEC [report] @packagetypeid;
- END;
- END;
+use wideworldimporters
+go
+DROP procedure IF EXISTS [dbo].[initialize]
+go
+CREATE procedure [dbo].[initialize]
+as begin
+DBCC FREEPROCCACHE;
+ALTER DATABASE current SET QUERY_STORE CLEAR ALL;
+ALTER DATABASE current SET AUTOMATIC_TUNING ( FORCE_LAST_GOOD_PLAN = OFF);
+end
+GO
+
+DROP procedure IF EXISTS [dbo].[auto_tune]
+go
+CREATE procedure [dbo].[auto_tune]
+as begin
+ALTER DATABASE current SET AUTOMATIC_TUNING ( FORCE_LAST_GOOD_PLAN = ON);
+DBCC FREEPROCCACHE;
+ALTER DATABASE current SET QUERY_STORE CLEAR ALL;
+end
+GO
+
+DROP PROCEDURE IF EXISTS [dbo].[report]
+go
+CREATE PROCEDURE [dbo].[report] ( @packagetypeid INT )
+AS
+ BEGIN
+
+ SELECT AVG([UnitPrice] * [Quantity] - [TaxRate])
+ FROM [Sales].[OrderLines]
+ WHERE [PackageTypeID] = @packagetypeid;
+
+ END;
+GO
+
+DROP PROCEDURE IF EXISTS [dbo].[regression]
+go
+CREATE PROCEDURE [dbo].[regression]
+AS
+ BEGIN
+ DBCC FREEPROCCACHE;
+ BEGIN
+ DECLARE @packagetypeid INT = 1;
+ EXEC [report] @packagetypeid;
+ END;
+ END;
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/ddl.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/ddl.sql
index 4501e5a6..5ce47c32 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/ddl.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/ddl.sql
@@ -1,40 +1,40 @@
-use WideWorldImporters
-go
-select * from Sales.InvoiceLines
-go
-
-alter database wideworldimporters set compatibility_level = 150
-
-alter database wideworldimporters set compatibility_level = 130
-
-create or alter proc defercompile
-as
-begin
-declare @ilines table
-( [InvoiceLineID] [int] NOT NULL primary key,
- [InvoiceID] [int] NOT NULL,
- [StockItemID] [int] NOT NULL,
- [Description] [nvarchar](100) NOT NULL,
- [PackageTypeID] [int] NOT NULL,
- [Quantity] [int] NOT NULL,
- [UnitPrice] [decimal](18, 2) NULL,
- [TaxRate] [decimal](18, 3) NOT NULL,
- [TaxAmount] [decimal](18, 2) NOT NULL,
- [LineProfit] [decimal](18, 2) NOT NULL,
- [ExtendedPrice] [decimal](18, 2) NOT NULL,
- [LastEditedBy] [int] NOT NULL,
- [LastEditedWhen] [datetime2](7) NOT NULL
-)
-
-insert into @ilines select * from sales.InvoiceLines
-
-select i.CustomerID, sum(il.LineProfit)
-from Sales.Invoices i
-inner join @ilines il
-on i.InvoiceID = il.InvoiceID
-group by i.CustomerID
-
-end
-go
-
-exec defercompile
+use WideWorldImporters
+go
+select * from Sales.InvoiceLines
+go
+
+alter database wideworldimporters set compatibility_level = 150
+
+alter database wideworldimporters set compatibility_level = 130
+
+create or alter proc defercompile
+as
+begin
+declare @ilines table
+( [InvoiceLineID] [int] NOT NULL primary key,
+ [InvoiceID] [int] NOT NULL,
+ [StockItemID] [int] NOT NULL,
+ [Description] [nvarchar](100) NOT NULL,
+ [PackageTypeID] [int] NOT NULL,
+ [Quantity] [int] NOT NULL,
+ [UnitPrice] [decimal](18, 2) NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [TaxAmount] [decimal](18, 2) NOT NULL,
+ [LineProfit] [decimal](18, 2) NOT NULL,
+ [ExtendedPrice] [decimal](18, 2) NOT NULL,
+ [LastEditedBy] [int] NOT NULL,
+ [LastEditedWhen] [datetime2](7) NOT NULL
+)
+
+insert into @ilines select * from sales.InvoiceLines
+
+select i.CustomerID, sum(il.LineProfit)
+from Sales.Invoices i
+inner join @ilines il
+on i.InvoiceID = il.InvoiceID
+group by i.CustomerID
+
+end
+go
+
+exec defercompile
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/proc.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/proc.sql
index 331e3cfc..d5ab3aba 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/proc.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/proc.sql
@@ -1,34 +1,34 @@
-USE WideWorldImporters
-GO
-CREATE or ALTER PROCEDURE defercompile
-AS
-BEGIN
--- Declare the table variable
-DECLARE @ilines TABLE
-( [InvoiceLineID] [int] NOT NULL primary key,
- [InvoiceID] [int] NOT NULL,
- [StockItemID] [int] NOT NULL,
- [Description] [nvarchar](100) NOT NULL,
- [PackageTypeID] [int] NOT NULL,
- [Quantity] [int] NOT NULL,
- [UnitPrice] [decimal](18, 2) NULL,
- [TaxRate] [decimal](18, 3) NOT NULL,
- [TaxAmount] [decimal](18, 2) NOT NULL,
- [LineProfit] [decimal](18, 2) NOT NULL,
- [ExtendedPrice] [decimal](18, 2) NOT NULL,
- [LastEditedBy] [int] NOT NULL,
- [LastEditedWhen] [datetime2](7) NOT NULL
-)
-
--- Insert all the rows from InvoiceLines into the table variable
-INSERT INTO @ilines SELECT * FROM Sales.InvoiceLines
-
--- Find my total profile by customer
-SELECT i.CustomerID, SUM(il.LineProfit)
-FROM Sales.Invoices i
-INNER JOIN @ilines il
-ON i.InvoiceID = il.InvoiceID
-GROUP By i.CustomerID
-
-END
+USE WideWorldImporters
+GO
+CREATE or ALTER PROCEDURE defercompile
+AS
+BEGIN
+-- Declare the table variable
+DECLARE @ilines TABLE
+( [InvoiceLineID] [int] NOT NULL primary key,
+ [InvoiceID] [int] NOT NULL,
+ [StockItemID] [int] NOT NULL,
+ [Description] [nvarchar](100) NOT NULL,
+ [PackageTypeID] [int] NOT NULL,
+ [Quantity] [int] NOT NULL,
+ [UnitPrice] [decimal](18, 2) NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [TaxAmount] [decimal](18, 2) NOT NULL,
+ [LineProfit] [decimal](18, 2) NOT NULL,
+ [ExtendedPrice] [decimal](18, 2) NOT NULL,
+ [LastEditedBy] [int] NOT NULL,
+ [LastEditedWhen] [datetime2](7) NOT NULL
+)
+
+-- Insert all the rows from InvoiceLines into the table variable
+INSERT INTO @ilines SELECT * FROM Sales.InvoiceLines
+
+-- Find my total profile by customer
+SELECT i.CustomerID, SUM(il.LineProfit)
+FROM Sales.Invoices i
+INNER JOIN @ilines il
+ON i.InvoiceID = il.InvoiceID
+GROUP By i.CustomerID
+
+END
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/query_plan_diff.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/query_plan_diff.sql
index 65bbdef8..aa6f721e 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/query_plan_diff.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/query_plan_diff.sql
@@ -1,11 +1,11 @@
-USE WideWorldImporters
-GO
-SELECT qsp.plan_id, qsp.compatibility_level,
-avg(qsrs.avg_duration)/1000 as avg_duration_ms, avg(qsrs.avg_logical_io_reads) as avg_logical_io
-FROM sys.query_store_plan qsp
-INNER JOIN sys.query_store_runtime_stats qsrs
-ON qsp.plan_id = qsrs.plan_id
-AND qsp.query_id = 41998 -- Put in your query_id here
-GROUP BY qsp.plan_id, qsp.compatibility_level
-GO
-
+USE WideWorldImporters
+GO
+SELECT qsp.plan_id, qsp.compatibility_level,
+avg(qsrs.avg_duration)/1000 as avg_duration_ms, avg(qsrs.avg_logical_io_reads) as avg_logical_io
+FROM sys.query_store_plan qsp
+INNER JOIN sys.query_store_runtime_stats qsrs
+ON qsp.plan_id = qsrs.plan_id
+AND qsp.query_id = 41998 -- Put in your query_id here
+GROUP BY qsp.plan_id, qsp.compatibility_level
+GO
+
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/readme.md
index b7f40436..d2768d9a 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/readme.md
@@ -1,27 +1,27 @@
-# SQL Server demo for Intelligent Query Processing - Deferred Table Variable Compilation
-
-## Requirements
-
-You must first install the following for this demo
-
-- SQL Server 2019 CTP 2.0 or greater on Windows Server. While you can run the basics of this demo on SQL Server on Linux, the full affect to the demo requires Windows Performance Monitor so I recommend you use SQL Server on Windows Server.
-- SQL Server client tools (e.g. sqlcmd)
-- SQL Server Management Studio 18.0 (SSMS) installed
-- RML Utilities installed (ostress) from https://www.microsoft.com/en-us/download/details.aspx?id=4511
-- A copy of the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak
-
-## Demo Steps
-
-1. Restore the WWI backup. Use the **restorewwi.sql** as a template.
-
-2. Run **setup_repro.cmd** to install new stored procedure in WideWorldImporters
-
-3. Examine the details of the new procedure from **proc.sql**
-
-4. Run **repro_130.cmd** and observe the total duration time. It should take around 30+ secs
-
-5. Run **repro_150.cm**d and observe the total duration. This is the exact same workload as step 4 except with database compatibility level 150. This sould take around 10secs.
-
-6. Using SSMS, observe the performance of this query and differences in plan and average execution time using Query Store Reports, Top Resource Queries.
-
+# SQL Server demo for Intelligent Query Processing - Deferred Table Variable Compilation
+
+## Requirements
+
+You must first install the following for this demo
+
+- SQL Server 2019 CTP 2.0 or greater on Windows Server. While you can run the basics of this demo on SQL Server on Linux, the full affect to the demo requires Windows Performance Monitor so I recommend you use SQL Server on Windows Server.
+- SQL Server client tools (e.g. sqlcmd)
+- SQL Server Management Studio 18.0 (SSMS) installed
+- RML Utilities installed (ostress) from https://www.microsoft.com/en-us/download/details.aspx?id=4511
+- A copy of the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak
+
+## Demo Steps
+
+1. Restore the WWI backup. Use the **restorewwi.sql** as a template.
+
+2. Run **setup_repro.cmd** to install new stored procedure in WideWorldImporters
+
+3. Examine the details of the new procedure from **proc.sql**
+
+4. Run **repro_130.cmd** and observe the total duration time. It should take around 30+ secs
+
+5. Run **repro_150.cm**d and observe the total duration. This is the exact same workload as step 4 except with database compatibility level 150. This sould take around 10secs.
+
+6. Using SSMS, observe the performance of this query and differences in plan and average execution time using Query Store Reports, Top Resource Queries.
+
7. Optionally use query_plan_diff.sql to observe the differences in Query Store. You need to substitute the proper query_id from the report in SSMS.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro130.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro130.sql
index fd2dd220..bb3414e0 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro130.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro130.sql
@@ -1,8 +1,8 @@
-USE master
-GO
-ALTER DATABASE wideworldimporters SET compatibility_level = 130
-go
-USE WideWorldImporters
-go
-EXEC defercompile
+USE master
+GO
+ALTER DATABASE wideworldimporters SET compatibility_level = 130
+go
+USE WideWorldImporters
+go
+EXEC defercompile
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro150.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro150.sql
index 4678ca44..18ace4e7 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro150.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/repro150.sql
@@ -1,8 +1,8 @@
-USE master
-GO
-ALTER DATABASE wideworldimporters SET compatibility_level = 150
-go
-USE WideWorldImporters
-go
-EXEC defercompile
+USE master
+GO
+ALTER DATABASE wideworldimporters SET compatibility_level = 150
+go
+USE WideWorldImporters
+go
+EXEC defercompile
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/restorewwi.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/restorewwi.sql
index d57411cf..f1f26580 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/restorewwi.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/restorewwi.sql
@@ -1,7 +1,7 @@
-restore database WideWorldImporters from disk = 'd:\sql_sample_databases\WideWorldImporters-Full.bak' with
-move 'WWI_Primary' to 'd:\sql_sample_databases\WideWorldImporters.mdf',
-move 'WWI_UserData' to 'd:\sql_sample_databases\WideWorldImporters_UserData.ndf',
-move 'WWI_Log' to 'd:\sql_sample_databases\WideWorldImporters.ldf',
-move 'WWI_InMemory_Data_1' to 'd:\sql_sample_databases\WideWorldImporters_InMemory_Data_1',
-stats=5
+restore database WideWorldImporters from disk = 'd:\sql_sample_databases\WideWorldImporters-Full.bak' with
+move 'WWI_Primary' to 'd:\sql_sample_databases\WideWorldImporters.mdf',
+move 'WWI_UserData' to 'd:\sql_sample_databases\WideWorldImporters_UserData.ndf',
+move 'WWI_Log' to 'd:\sql_sample_databases\WideWorldImporters.ldf',
+move 'WWI_InMemory_Data_1' to 'd:\sql_sample_databases\WideWorldImporters_InMemory_Data_1',
+stats=5
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set130.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set130.sql
index 843b9347..ce272699 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set130.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set130.sql
@@ -1,2 +1,2 @@
-alter database wideworldimporters set compatibility_level = 130
+alter database wideworldimporters set compatibility_level = 130
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set150.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set150.sql
index 9c0ab483..d039188f 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set150.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/deferredtablevariable/set150.sql
@@ -1,2 +1,2 @@
-alter database wideworldimporters set compatibility_level = 150
+alter database wideworldimporters set compatibility_level = 150
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/readme.md
index 141512f8..701e2d9d 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/iqp/readme.md
@@ -1,7 +1,7 @@
-# SQL Server Demos for Intelligent Query Processing in SQL Server
-
-These are demos you can use to show the Intelligent Query Processing capabilities in SQL Server 2017 and 2019
-
-## deferredtablevariable
-
+# SQL Server Demos for Intelligent Query Processing in SQL Server
+
+These are demos you can use to show the Intelligent Query Processing capabilities in SQL Server 2017 and 2019
+
+## deferredtablevariable
+
This is a demo to show the new cardinality estimation for table variables called deferred compilation.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/dm_exec_query_profiles.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/dm_exec_query_profiles.sql
index a832727b..f35baca6 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/dm_exec_query_profiles.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/dm_exec_query_profiles.sql
@@ -1,9 +1,9 @@
---Run this in a different session than the session in which your query is running.
---Note that you may need to change session id 54 below with the session id you want to monitor.
-SELECT node_id,physical_operator_name, SUM(row_count) row_count,
- SUM(estimate_row_count) AS estimate_row_count,
- CAST(SUM(row_count)*100 AS float)/SUM(estimate_row_count) as percent_complete
-FROM sys.dm_exec_query_profiles
-WHERE session_id=91
-GROUP BY node_id,physical_operator_name
+--Run this in a different session than the session in which your query is running.
+--Note that you may need to change session id 54 below with the session id you want to monitor.
+SELECT node_id,physical_operator_name, SUM(row_count) row_count,
+ SUM(estimate_row_count) AS estimate_row_count,
+ CAST(SUM(row_count)*100 AS float)/SUM(estimate_row_count) as percent_complete
+FROM sys.dm_exec_query_profiles
+WHERE session_id=91
+GROUP BY node_id,physical_operator_name
ORDER BY node_id;
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/mysmartsqlquery.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/mysmartsqlquery.sql
index f1599e8e..e5f1d430 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/mysmartsqlquery.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/mysmartsqlquery.sql
@@ -1,10 +1,10 @@
---alter database wideworldimporters set compatibility_level = 150
---go
-use wideworldimporters
-go
-select si.CustomerID, sil.LineProfit
-from Sales.Invoices si
-join Sales.InvoiceLines sil
-on si.InvoiceID = si.InvoiceID
-option (maxdop 1)
+--alter database wideworldimporters set compatibility_level = 150
+--go
+use wideworldimporters
+go
+select si.CustomerID, sil.LineProfit
+from Sales.Invoices si
+join Sales.InvoiceLines sil
+on si.InvoiceID = si.InvoiceID
+option (maxdop 1)
go
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/readme.md
index b1e926ed..aacf1c38 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/lwp/readme.md
@@ -1,17 +1,17 @@
-# SQL Server Demo Lightweight Query Profiling
-
-This is a demo to show the feature of Lightweight Query Profiling which is on by default in SQL Server 2019
-
-## Requirements
-
-- Install SQL Server 2019 CTP 2.0 or higher
-- Restore the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak
-- Install SQL Server Management Studio 18.0 or higher
-
-## Demo Steps
-
-1. Open up SSMS and Activity Monitor
-2. Run the script **mysmartquery.cmd**
-3. Choose the process for this query in Activity Monitor in the Process section and right-click. Choose Show Live Execution Plan
-4. You should see a live view of the plan in execution at the operator level. Note the query has a syntax bug which causes the crazy behavior.
+# SQL Server Demo Lightweight Query Profiling
+
+This is a demo to show the feature of Lightweight Query Profiling which is on by default in SQL Server 2019
+
+## Requirements
+
+- Install SQL Server 2019 CTP 2.0 or higher
+- Restore the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak
+- Install SQL Server Management Studio 18.0 or higher
+
+## Demo Steps
+
+1. Open up SSMS and Activity Monitor
+2. Run the script **mysmartquery.cmd**
+3. Choose the process for this query in Activity Monitor in the Process section and right-click. Choose Show Live Execution Plan
+4. You should see a live view of the plan in execution at the operator level. Note the query has a syntax bug which causes the crazy behavior.
5. Load **dm_exec_query_profiles.sql** and fill in the correct session_id. Note you can see the same info through a DMV.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/readme.md
index 4018a2ae..a8ecb8ea 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 2 Activity - Intelligent Performance/readme.md
@@ -1,15 +1,15 @@
-# Module 2 Activities - Intelligent Performance
-
-These represent demos and examples you can run to see the behavior of Intelligent Performance in SQL Server 2019. Note the autotune demo will work on SQL Server 2017.
-
-## autotune
-
-Show the capabilities of Automatic Tuning (Automatic Plan Correction)
-
-## iqp
-
-Show the capabilities of Intelligent Query Processing including deferred table variable compilation.
-
-## lwp
-
+# Module 2 Activities - Intelligent Performance
+
+These represent demos and examples you can run to see the behavior of Intelligent Performance in SQL Server 2019. Note the autotune demo will work on SQL Server 2017.
+
+## autotune
+
+Show the capabilities of Automatic Tuning (Automatic Plan Correction)
+
+## iqp
+
+Show the capabilities of Intelligent Query Processing including deferred table variable compilation.
+
+## lwp
+
Show the capabilities of "on by default" Lightweight Query Profiling
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/readme.md
index 997a5156..76c3c129 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/readme.md
@@ -1,7 +1,7 @@
-# Module 2 Activities - Security
-
-These represent demos and examples you can run to see new capabilities for security in SQL Servrer 2019.
-
-## staticmask
-
+# Module 2 Activities - Security
+
+These represent demos and examples you can run to see new capabilities for security in SQL Servrer 2019.
+
+## staticmask
+
Show the capabilities of Static Data Masking using SSMS 18.0 against SQL Server 2019
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/createhr.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/createhr.sql
index 1fb2e46b..a1d946da 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/createhr.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/createhr.sql
@@ -1,24 +1,24 @@
-USE master
-GO
-DROP DATABASE IF EXISTS HumanResources
-GO
-CREATE DATABASE HumanResources
-GO
-USE HumanResources
-GO
-DROP TABLE IF EXISTS Employees
-GO
-CREATE TABLE Employees
-(EmployeeID int primary key clustered,
-EmployeeName nvarchar(100) not null,
-EmployeeSSN nvarchar(20) not null,
-EmployeeEmail nvarchar(100) not null,
-EmployeePhone nvarchar(20) not null,
-EmployeeHireDate datetime not null
-)
-INSERT INTO Employees VALUES (1, 'Bob Ward', '123-457-6891', 'bward@microsoft.com', '817-455-0111', '10/27/1993')
-INSERT INTO Employees VALUES (2, 'Dak Prescott', '256-908-1234', 'dakprescott@dallascowboys.com', '214-123-9999', '08/01/2016')
-INSERT INTO Employees VALUES (3, 'Ryan Ward', '569-28-9123', 'ryan.ward@baylor.edu', '817-623-2391', '03/27/1996')
-INSERT INTO Employees VALUES (4, 'Ginger Ward', '971-11-2378', 'ginger.ward@outlook.com', '817-455-9872', '01/01/2000')
-INSERT INTO Employees VALUES (5, 'Troy Ward', '567-12-9291', 'troy.ward@tulane.edu', '682-111-2391', '08/30/1993')
+USE master
+GO
+DROP DATABASE IF EXISTS HumanResources
+GO
+CREATE DATABASE HumanResources
+GO
+USE HumanResources
+GO
+DROP TABLE IF EXISTS Employees
+GO
+CREATE TABLE Employees
+(EmployeeID int primary key clustered,
+EmployeeName nvarchar(100) not null,
+EmployeeSSN nvarchar(20) not null,
+EmployeeEmail nvarchar(100) not null,
+EmployeePhone nvarchar(20) not null,
+EmployeeHireDate datetime not null
+)
+INSERT INTO Employees VALUES (1, 'Bob Ward', '123-457-6891', 'bward@microsoft.com', '817-455-0111', '10/27/1993')
+INSERT INTO Employees VALUES (2, 'Dak Prescott', '256-908-1234', 'dakprescott@dallascowboys.com', '214-123-9999', '08/01/2016')
+INSERT INTO Employees VALUES (3, 'Ryan Ward', '569-28-9123', 'ryan.ward@baylor.edu', '817-623-2391', '03/27/1996')
+INSERT INTO Employees VALUES (4, 'Ginger Ward', '971-11-2378', 'ginger.ward@outlook.com', '817-455-9872', '01/01/2000')
+INSERT INTO Employees VALUES (5, 'Troy Ward', '567-12-9291', 'troy.ward@tulane.edu', '682-111-2391', '08/30/1993')
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/hrmasking.xml b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/hrmasking.xml
index 049819c5..2ac83fe9 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/hrmasking.xml
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/hrmasking.xml
@@ -1,14 +1,14 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/readme.md
index c0893110..b863deec 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 3 Activity - Security/staticmask/readme.md
@@ -1,24 +1,24 @@
-# Static Data Masking with SQL Server
-
-This demo is to show the Static Data Masking feature with SQL Server which is documented at https://docs.microsoft.com/en-us/sql/relational-databases/security/static-data-masking?view=sql-server-2017
-
-## Requirements
-
-- Install SQL Server Management Studio (SSMS) 18.0 Preview 6 or higher
-- Install SQL Server 2019 (TODO: We are checking on whether this will work with previous versions)
-
-## Demo Steps
-
-1. Run the script **createhr.sql** to create the database and populate it with data.
-
-2. In SSMS, right click on the HumanResources database and select Tasks/Mask Database (Preview)
-
-3. In the dialog box that appears, select Load Config and choose the **hrmasking.xml** file provided.
-
-4. Select the Configure options to see how the data will be masked.
-
-5. Type in Step 3 the Masked Database name as HRMasked
-
-6. Click OK. This will take a few minutes to run. The tool is taking a backup of the current database, restoring it, and modifying the data to be masked.
-
+# Static Data Masking with SQL Server
+
+This demo is to show the Static Data Masking feature with SQL Server which is documented at https://docs.microsoft.com/en-us/sql/relational-databases/security/static-data-masking?view=sql-server-2017
+
+## Requirements
+
+- Install SQL Server Management Studio (SSMS) 18.0 Preview 6 or higher
+- Install SQL Server 2019 (TODO: We are checking on whether this will work with previous versions)
+
+## Demo Steps
+
+1. Run the script **createhr.sql** to create the database and populate it with data.
+
+2. In SSMS, right click on the HumanResources database and select Tasks/Mask Database (Preview)
+
+3. In the dialog box that appears, select Load Config and choose the **hrmasking.xml** file provided.
+
+4. Select the Configure options to see how the data will be masked.
+
+5. Type in Step 3 the Masked Database name as HRMasked
+
+6. Click OK. This will take a few minutes to run. The tool is taking a backup of the current database, restoring it, and modifying the data to be masked.
+
7. Compare the data in the original database to the new masked database.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/checkpoint_and_shutdown.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/checkpoint_and_shutdown.sql
index f86cac5a..6251492e 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/checkpoint_and_shutdown.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/checkpoint_and_shutdown.sql
@@ -1,12 +1,12 @@
--- Let's cause rollback with server startup
--- First CHECKPOINT the database
-USE gocowboys
-GO
-CHECKPOINT
-GO
-
-USE master
-GO
--- SHUTDOWN WITH NO WAIT
-SHUTDOWN WITH NOWAIT
-GO
+-- Let's cause rollback with server startup
+-- First CHECKPOINT the database
+USE gocowboys
+GO
+CHECKPOINT
+GO
+
+USE master
+GO
+-- SHUTDOWN WITH NO WAIT
+SHUTDOWN WITH NOWAIT
+GO
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_rollback.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_rollback.sql
index 3e5bf3f2..a8d02618 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_rollback.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_rollback.sql
@@ -1,75 +1,75 @@
--- Make sure ADR is OFF
---
-USE master
-GO
-ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = OFF
-GO
--- Try to delete a bunch of rows
--- Should take about 30 secs
---
-USE gocowboys
-GO
-BEGIN TRAN
-DELETE from howboutthemcowboys
-GO
-
--- What is the log space usage
-SELECT * FROM sys.dm_db_log_space_usage
-go
-
--- Does checkpoint truncate the log?
---
-CHECKPOINT
-GO
-SELECT * FROM sys.dm_db_log_space_usage
-go
-
--- Try to roll it back and measure the time
-ROLLBACK TRAN
-GO
-
--- What is the log space usage
-SELECT * FROM sys.dm_db_log_space_usage
-go
-
--- Does checkpoint truncate the log?
---
-CHECKPOINT
-GO
-SELECT * FROM sys.dm_db_log_space_usage
-go
-
--- Now try it with ADR
---
-USE master
-GO
-ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = ON
-GO
-
--- Try to delete a bunch of rows and roll it back
---
-USE gocowboys
-GO
-BEGIN TRAN
-DELETE from howboutthemcowboys
-GO
-
--- What is the log space usage
-SELECT * FROM sys.dm_db_log_space_usage
-go
-
--- Try to roll it back and measure the time
--- 0 secs!
-ROLLBACK TRAN
-GO
-
--- What is the log space usage
-SELECT * FROM sys.dm_db_log_space_usage
-go
-
--- Clear ADR
---
-USE master
-GO
-ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = OFF
+-- Make sure ADR is OFF
+--
+USE master
+GO
+ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = OFF
+GO
+-- Try to delete a bunch of rows
+-- Should take about 30 secs
+--
+USE gocowboys
+GO
+BEGIN TRAN
+DELETE from howboutthemcowboys
+GO
+
+-- What is the log space usage
+SELECT * FROM sys.dm_db_log_space_usage
+go
+
+-- Does checkpoint truncate the log?
+--
+CHECKPOINT
+GO
+SELECT * FROM sys.dm_db_log_space_usage
+go
+
+-- Try to roll it back and measure the time
+ROLLBACK TRAN
+GO
+
+-- What is the log space usage
+SELECT * FROM sys.dm_db_log_space_usage
+go
+
+-- Does checkpoint truncate the log?
+--
+CHECKPOINT
+GO
+SELECT * FROM sys.dm_db_log_space_usage
+go
+
+-- Now try it with ADR
+--
+USE master
+GO
+ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = ON
+GO
+
+-- Try to delete a bunch of rows and roll it back
+--
+USE gocowboys
+GO
+BEGIN TRAN
+DELETE from howboutthemcowboys
+GO
+
+-- What is the log space usage
+SELECT * FROM sys.dm_db_log_space_usage
+go
+
+-- Try to roll it back and measure the time
+-- 0 secs!
+ROLLBACK TRAN
+GO
+
+-- What is the log space usage
+SELECT * FROM sys.dm_db_log_space_usage
+go
+
+-- Clear ADR
+--
+USE master
+GO
+ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = OFF
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_undo_recovery.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_undo_recovery.sql
index 77919caa..375546ed 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_undo_recovery.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/delete_undo_recovery.sql
@@ -1,47 +1,47 @@
--- Try to delete a bunch of rows
--- Should take about 30 secs
---
-USE gocowboys
-GO
-BEGIN TRAN
-DELETE from howboutthemcowboys
-GO
-
--- Let's cause rollback with server startup
--- First CHECKPOINT the database
-CHECKPOINT
-GO
-
--- SHUTDOWN WITH NO WAIT
-SHUTDOWN WITH NOWAIT
-GO
-
-
-
-
-
--- Now try it with ADR
---
-USE master
-GO
-ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = ON
-GO
-
--- Try to delete a bunch of rows and roll it back
---
-USE gocowboys
-GO
-BEGIN TRAN
-DELETE from howboutthemcowboys
-GO
--- Try to roll it back and measure the time
--- 0 secs!
-ROLLBACK TRAN
-GO
-
--- Clear ADR
---
-USE master
-GO
-ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = OFF
+-- Try to delete a bunch of rows
+-- Should take about 30 secs
+--
+USE gocowboys
+GO
+BEGIN TRAN
+DELETE from howboutthemcowboys
+GO
+
+-- Let's cause rollback with server startup
+-- First CHECKPOINT the database
+CHECKPOINT
+GO
+
+-- SHUTDOWN WITH NO WAIT
+SHUTDOWN WITH NOWAIT
+GO
+
+
+
+
+
+-- Now try it with ADR
+--
+USE master
+GO
+ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = ON
+GO
+
+-- Try to delete a bunch of rows and roll it back
+--
+USE gocowboys
+GO
+BEGIN TRAN
+DELETE from howboutthemcowboys
+GO
+-- Try to roll it back and measure the time
+-- 0 secs!
+ROLLBACK TRAN
+GO
+
+-- Clear ADR
+--
+USE master
+GO
+ALTER DATABASE gocowboys SET ACCELERATED_DATABASE_RECOVERY = OFF
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/readme.md
index 55eb3526..aa45fb8c 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/readme.md
@@ -1,15 +1,15 @@
-# Accelerated Data Recovery in SQL Server
-
-This demo is to show Accelerated Data Recovery feature in SQL Server 2019
-
-## Requirements
-
-- Install SQL Server 2019 CTP 2.3 or higher
-
-## Demo Steps
-
-1. Run the statements in **setup_database.sql**
-
-2. Follow the statements in **delete_rollback.sql**
-
+# Accelerated Data Recovery in SQL Server
+
+This demo is to show Accelerated Data Recovery feature in SQL Server 2019
+
+## Requirements
+
+- Install SQL Server 2019 CTP 2.3 or higher
+
+## Demo Steps
+
+1. Run the statements in **setup_database.sql**
+
+2. Follow the statements in **delete_rollback.sql**
+
3. Follow the steps in **delete_undo_recovery.sql**. Use the **checkpoint_and_shutdown.sql** script in another session to checkpoing and shutdown SQL Server.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/setup_database.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/setup_database.sql
index 7dcf78f9..677ba5ad 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/setup_database.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/adr/setup_database.sql
@@ -1,34 +1,34 @@
-USE master
-GO
-DROP DATABASE IF EXISTS gocowboys
-GO
-CREATE DATABASE gocowboys
-ON PRIMARY
-(NAME = N'gocowboys_primary', FILENAME = 'c:\data\gocowboys.mdf', SIZE = 10Gb , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB)
-LOG ON
-(NAME = N'gocowboys_Log', FILENAME = 'c:\data\gocowboys_log.ldf', SIZE = 10Gb , MAXSIZE = UNLIMITED , FILEGROWTH = 65536KB)
-GO
-ALTER DATABASE gocowboys SET RECOVERY SIMPLE
-GO
-USE gocowboys
-GO
-DROP TABLE IF EXISTS howboutthemcowboys
-GO
-CREATE TABLE howboutthemcowboys (playerid int primary key clustered, playername char(7000) not null)
-GO
-SET NOCOUNT ON
-GO
-BEGIN TRAN
-DECLARE @x int
-SET @x = 0
-WHILE (@x < 100000)
-BEGIN
- INSERT INTO howboutthemcowboys VALUES (@x, 'All players are great')
- SET @x = @x + 1
-END
-COMMIT TRAN
-GO
-SET NOCOUNT OFF
-GO
-USE master
+USE master
+GO
+DROP DATABASE IF EXISTS gocowboys
+GO
+CREATE DATABASE gocowboys
+ON PRIMARY
+(NAME = N'gocowboys_primary', FILENAME = 'c:\data\gocowboys.mdf', SIZE = 10Gb , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB)
+LOG ON
+(NAME = N'gocowboys_Log', FILENAME = 'c:\data\gocowboys_log.ldf', SIZE = 10Gb , MAXSIZE = UNLIMITED , FILEGROWTH = 65536KB)
+GO
+ALTER DATABASE gocowboys SET RECOVERY SIMPLE
+GO
+USE gocowboys
+GO
+DROP TABLE IF EXISTS howboutthemcowboys
+GO
+CREATE TABLE howboutthemcowboys (playerid int primary key clustered, playername char(7000) not null)
+GO
+SET NOCOUNT ON
+GO
+BEGIN TRAN
+DECLARE @x int
+SET @x = 0
+WHILE (@x < 100000)
+BEGIN
+ INSERT INTO howboutthemcowboys VALUES (@x, 'All players are great')
+ SET @x = @x + 1
+END
+COMMIT TRAN
+GO
+SET NOCOUNT OFF
+GO
+USE master
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/readme.md
index 37e443fb..2b6987fd 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 4 Activity - MIssion Critical Availability/readme.md
@@ -1,7 +1,7 @@
-# Module 4 Activities - Mission Critical Availability
-
-These represent demos and examples you can run to see new capabilities for mission critical availability in SQL Server 2019.
-
-## adr
-
+# Module 4 Activities - Mission Critical Availability
+
+These represent demos and examples you can run to see new capabilities for mission critical availability in SQL Server 2019.
+
+## adr
+
Show the capabilities of Accelerated Data Recovery in SQL Server 2019
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/InputRow.java b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/InputRow.java
index 08b7b781..b7789a1d 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/InputRow.java
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/InputRow.java
@@ -1,12 +1,12 @@
-package pkg;
-
-//This object represents one input row
-public class InputRow {
- public final int id;
- public final String text;
-
- public InputRow(final int id, final String text) {
- this.id = id;
- this.text = text;
- }
+package pkg;
+
+//This object represents one input row
+public class InputRow {
+ public final int id;
+ public final String text;
+
+ public InputRow(final int id, final String text) {
+ this.id = id;
+ this.text = text;
+ }
}
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Javangramdemo.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Javangramdemo.sql
index 73415854..d08b2647 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Javangramdemo.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Javangramdemo.sql
@@ -1,19 +1,19 @@
-USE JavaTest
-GO
-DECLARE @myClassPath nvarchar(50)
-DECLARE @n int
---This is where you store your classes or jars.
---Update this to your own classpath
-SET @myClassPath = N'C:\java'
---This is the size of the ngram
-SET @n = 3
-EXEC sp_execute_external_script
- @language = N'Java'
-, @script = N'pkg.Ngram.getNGrams'
-, @input_data_1 = N'SELECT id, text FROM reviews'
-, @parallel = 0
-, @params = N'@CLASSPATH nvarchar(50), @param1 INT'
-, @CLASSPATH = @myClassPath
-, @param1 = @n
-with result sets ((ID int, ngram varchar(20)))
+USE JavaTest
+GO
+DECLARE @myClassPath nvarchar(50)
+DECLARE @n int
+--This is where you store your classes or jars.
+--Update this to your own classpath
+SET @myClassPath = N'C:\java'
+--This is the size of the ngram
+SET @n = 3
+EXEC sp_execute_external_script
+ @language = N'Java'
+, @script = N'pkg.Ngram.getNGrams'
+, @input_data_1 = N'SELECT id, text FROM reviews'
+, @parallel = 0
+, @params = N'@CLASSPATH nvarchar(50), @param1 INT'
+, @CLASSPATH = @myClassPath
+, @param1 = @n
+with result sets ((ID int, ngram varchar(20)))
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Ngram.java b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Ngram.java
index f82bbab5..6f77b881 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Ngram.java
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/Ngram.java
@@ -1,91 +1,91 @@
-//We will package our classes in a package called pkg
-//Packages are option in Java-SQL, but required for this sample.
-package pkg;
-
-import java.util.List;
-import java.util.stream.Collectors;
-import java.util.stream.IntStream;
-
-public class Ngram {
-
- //Required: This is only required if you are passing data in @input_data_1
- //from SQL Server in sp_execute_external_script
- public static int[] inputDataCol1 = new int[1];
- public static String[] inputDataCol2 = new String[1];
-
- //Required: Input null map. Size just needs to be set to "1"
- public static boolean[][] inputNullMap = new boolean[1][1];
-
- //Required: Output data columns returned back to SQL Server
- public static int[] outputDataCol1;
- public static String[] outputDataCol2;
-
- //Required: Output null map. Is populated with true or false values
- //to indicate nulls
- public static boolean[][] outputNullMap;
-
- //Optional: This is only required if parameters are passed with @params
- // from SQL Server in sp_execute_external_script
- // n is giving us the size of ngram substrings
- public static int param1;
-
- //Optional: The number of rows we will be returning
- public static int numberOfRows;
-
- //Required: Number of output columns returned
- public static short numberOfOutputCols;
-
- /*Java main method - Only for testing purposes outside of SQL Server
- public static void main(String... args) {
- //getNGrams();
- }*/
-
- //This is the method we will be calling from SQL Server
- public static void getNGrams() {
-
- System.out.println("inputDataCol1.length= "+ inputDataCol1.length);
- if (inputDataCol1.length == 0 ) {
- // TODO: Set empty return
- return;
- }
- //Using a stream to "loop" over the input data inputDataCol1.length. You can also use a for loop for this.
- final List inputDataSet = IntStream.range(0, inputDataCol1.length)
- .mapToObj(i -> new InputRow(inputDataCol1[i], inputDataCol2[i]))
- .collect(Collectors.toList());
-
-
- //Again, we are using a stream to loop over data
- final List outputDataSet = inputDataSet.stream()
- // Generate ngrams of size n for each incoming string
- // Each invocation of ngrams returns a list. flatMap flattens
- // the resulting list-of-lists to a flat list.
- .flatMap(inputRow -> ngrams(param1, inputRow.text).stream().map(s -> new OutputRow(inputRow.id, s)))
- .collect(Collectors.toList());
-
- //Print the outputDataSet
- System.out.println(outputDataSet);
-
- //Set the number of rows and columns we will be returning
- numberOfOutputCols = 2;
- numberOfRows = outputDataSet.size();
- outputDataCol1 = new int[numberOfRows]; // ID column
- outputDataCol2 = new String[numberOfRows]; //The ngram column
- outputNullMap = new boolean[2][numberOfRows];// output null map
-
- //Since we don't have any null values, we will populate all values in the outputNullMap to false
- IntStream.range(0, numberOfRows).forEach(i -> {
- final OutputRow outputRow = outputDataSet.get(i);
- outputDataCol1[i] = outputRow.id;
- outputDataCol2[i] = outputRow.ngram;
- outputNullMap[0][i] = false;
- outputNullMap[1][i] = false;
- });
- }
-
- // Example: ngrams(3, "abcde") = ["abc", "bcd", "cde"].
- private static List ngrams(int n, String text) {
- return IntStream.range(0, text.length() - n + 1)
- .mapToObj(i -> text.substring(i, i + n))
- .collect(Collectors.toList());
- }
-}
+//We will package our classes in a package called pkg
+//Packages are option in Java-SQL, but required for this sample.
+package pkg;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+public class Ngram {
+
+ //Required: This is only required if you are passing data in @input_data_1
+ //from SQL Server in sp_execute_external_script
+ public static int[] inputDataCol1 = new int[1];
+ public static String[] inputDataCol2 = new String[1];
+
+ //Required: Input null map. Size just needs to be set to "1"
+ public static boolean[][] inputNullMap = new boolean[1][1];
+
+ //Required: Output data columns returned back to SQL Server
+ public static int[] outputDataCol1;
+ public static String[] outputDataCol2;
+
+ //Required: Output null map. Is populated with true or false values
+ //to indicate nulls
+ public static boolean[][] outputNullMap;
+
+ //Optional: This is only required if parameters are passed with @params
+ // from SQL Server in sp_execute_external_script
+ // n is giving us the size of ngram substrings
+ public static int param1;
+
+ //Optional: The number of rows we will be returning
+ public static int numberOfRows;
+
+ //Required: Number of output columns returned
+ public static short numberOfOutputCols;
+
+ /*Java main method - Only for testing purposes outside of SQL Server
+ public static void main(String... args) {
+ //getNGrams();
+ }*/
+
+ //This is the method we will be calling from SQL Server
+ public static void getNGrams() {
+
+ System.out.println("inputDataCol1.length= "+ inputDataCol1.length);
+ if (inputDataCol1.length == 0 ) {
+ // TODO: Set empty return
+ return;
+ }
+ //Using a stream to "loop" over the input data inputDataCol1.length. You can also use a for loop for this.
+ final List inputDataSet = IntStream.range(0, inputDataCol1.length)
+ .mapToObj(i -> new InputRow(inputDataCol1[i], inputDataCol2[i]))
+ .collect(Collectors.toList());
+
+
+ //Again, we are using a stream to loop over data
+ final List outputDataSet = inputDataSet.stream()
+ // Generate ngrams of size n for each incoming string
+ // Each invocation of ngrams returns a list. flatMap flattens
+ // the resulting list-of-lists to a flat list.
+ .flatMap(inputRow -> ngrams(param1, inputRow.text).stream().map(s -> new OutputRow(inputRow.id, s)))
+ .collect(Collectors.toList());
+
+ //Print the outputDataSet
+ System.out.println(outputDataSet);
+
+ //Set the number of rows and columns we will be returning
+ numberOfOutputCols = 2;
+ numberOfRows = outputDataSet.size();
+ outputDataCol1 = new int[numberOfRows]; // ID column
+ outputDataCol2 = new String[numberOfRows]; //The ngram column
+ outputNullMap = new boolean[2][numberOfRows];// output null map
+
+ //Since we don't have any null values, we will populate all values in the outputNullMap to false
+ IntStream.range(0, numberOfRows).forEach(i -> {
+ final OutputRow outputRow = outputDataSet.get(i);
+ outputDataCol1[i] = outputRow.id;
+ outputDataCol2[i] = outputRow.ngram;
+ outputNullMap[0][i] = false;
+ outputNullMap[1][i] = false;
+ });
+ }
+
+ // Example: ngrams(3, "abcde") = ["abc", "bcd", "cde"].
+ private static List ngrams(int n, String text) {
+ return IntStream.range(0, text.length() - n + 1)
+ .mapToObj(i -> text.substring(i, i + n))
+ .collect(Collectors.toList());
+ }
+}
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/OutputRow.java b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/OutputRow.java
index c33af8a2..d450904d 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/OutputRow.java
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/OutputRow.java
@@ -1,15 +1,15 @@
-package pkg;
-
-//This object represents one output row
-public class OutputRow {
- public final int id;
- public final String ngram;
-
- public OutputRow(final int id, final String ngram) {
- this.id = id;
- this.ngram = ngram;
- }
-
- @Override
- public String toString() { return id + ":" + ngram; }
+package pkg;
+
+//This object represents one output row
+public class OutputRow {
+ public final int id;
+ public final String ngram;
+
+ public OutputRow(final int id, final String ngram) {
+ this.id = id;
+ this.ngram = ngram;
+ }
+
+ @Override
+ public String toString() { return id + ":" + ngram; }
}
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/buildjava8.cmd b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/buildjava8.cmd
index 2edf120b..66cc775b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/buildjava8.cmd
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/buildjava8.cmd
@@ -1,4 +1,4 @@
-rmdir /s /q c:\java\pkg
-mkdir c:\java\pkg
-"C:\Program Files\Java\jdk1.8.0_181\bin\javac" Ngram.java InputRow.java OutputRow.java
+rmdir /s /q c:\java\pkg
+mkdir c:\java\pkg
+"C:\Program Files\Java\jdk1.8.0_181\bin\javac" Ngram.java InputRow.java OutputRow.java
copy *.class c:\java\pkg
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/javasetup.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/javasetup.sql
index f9c746f0..ecfad278 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/javasetup.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/javasetup.sql
@@ -1,27 +1,27 @@
--- Enable external scripts.
--- No restart is required in SQL Server 2019!
-EXEC sp_configure 'show advanced options', 1
-GO
-RECONFIGURE
-EXEC sp_configure 'external scripts enabled', 1
-GO
-RECONFIGURE
-GO
--- Create a database and populate some data
---
-DROP DATABASE IF EXISTS JavaTest
-GO
-CREATE DATABASE JavaTest
-GO
-USE JavaTest
-GO
-DROP TABLE IF exists reviews;
-GO
-CREATE TABLE reviews(
- id int NOT NULL,
- "text" nvarchar(30) NOT NULL)
-
-INSERT INTO reviews(id, "text") VALUES (1, 'AAA BBB CCC DDD EEE FFF')
-INSERT INTO reviews(id, "text") VALUES (2, 'GGG HHH III JJJ KKK LLL')
-INSERT INTO reviews(id, "text") VALUES (3, 'MMM NNN OOO PPP QQQ RRR')
-GO
+-- Enable external scripts.
+-- No restart is required in SQL Server 2019!
+EXEC sp_configure 'ahow advanced options', 1
+GO
+RECONFIGURE
+EXEC sp_configure 'external scripts enabled', 1
+GO
+RECONFIGURE
+GO
+-- Create a database and populate some data
+--
+DROP DATABASE IF EXISTS JavaTest
+GO
+CREATE DATABASE JavaTest
+GO
+USE JavaTest
+GO
+DROP TABLE IF exists reviews;
+GO
+CREATE TABLE reviews(
+ id int NOT NULL,
+ "text" nvarchar(30) NOT NULL)
+
+INSERT INTO reviews(id, "text") VALUES (1, 'AAA BBB CCC DDD EEE FFF')
+INSERT INTO reviews(id, "text") VALUES (2, 'GGG HHH III JJJ KKK LLL')
+INSERT INTO reviews(id, "text") VALUES (3, 'MMM NNN OOO PPP QQQ RRR')
+GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/readme.md
index fae152c8..50a3163f 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/java/readme.md
@@ -1,25 +1,25 @@
-# SQL Server Demo for the Java Extension
-
-This is a demo to show the Java Extension capability in SQL Server 2019 and later. This demo shows how to run Java code on SQL Server on Windows. The same code will work with SQL Server Java on Linux. Demo for this scenario coming soon.
-
-## Requirements
-
-1. Install SQL Server 2019 CTP 2.0 on Windows or later following the guidance on this page for Java.
-
-https://docs.microsoft.com/en-us/sql/advanced-analytics/java/extension-java?view=sqlallproducts-allversions#install-on-windows
-
-2. Follow the rest of the steps on this page to install all dependencies
-
-https://docs.microsoft.com/en-us/sql/advanced-analytics/java/extension-java?view=sqlallproducts-allversions#install-on-windows
-
-## Demo Steps
-
-1. I used the sample code from https://docs.microsoft.com/en-us/sql/advanced-analytics/java/java-first-sample?view=sqlallproducts-allversions and included the 3 Java class source: **InputRow.java**, **OutputRow.java**, **Ngram.java**
-
-2. I compiled the Java classes using the script **buildjava8.cmd** with Java 8 SDK (so it will be compatible with Linux). The compiled code is placed into C:\java\pkg. You can adjust this to whatever directory you need but you will need to make edits in the following steps.
-
-3. You need to setup permissions for the folder for your compiled classes. Use the instructions per the documentation at https://docs.microsoft.com/en-us/sql/advanced-analytics/java/java-first-sample?view=sqlallproducts-allversions#6---set-permissions
-
-4. Run the **javasetup.sql** script to create the database and objects.
-
+# SQL Server Demo for the Java Extension
+
+This is a demo to show the Java Extension capability in SQL Server 2019 and later. This demo shows how to run Java code on SQL Server on Windows. The same code will work with SQL Server Java on Linux. Demo for this scenario coming soon.
+
+## Requirements
+
+1. Install SQL Server 2019 CTP 2.0 on Windows or later following the guidance on this page for Java.
+
+https://docs.microsoft.com/en-us/sql/advanced-analytics/java/extension-java?view=sqlallproducts-allversions#install-on-windows
+
+2. Follow the rest of the steps on this page to install all dependencies
+
+https://docs.microsoft.com/en-us/sql/advanced-analytics/java/extension-java?view=sqlallproducts-allversions#install-on-windows
+
+## Demo Steps
+
+1. I used the sample code from https://docs.microsoft.com/en-us/sql/advanced-analytics/java/java-first-sample?view=sqlallproducts-allversions and included the 3 Java class source: **InputRow.java**, **OutputRow.java**, **Ngram.java**
+
+2. I compiled the Java classes using the script **buildjava8.cmd** with Java 8 SDK (so it will be compatible with Linux). The compiled code is placed into C:\java\pkg. You can adjust this to whatever directory you need but you will need to make edits in the following steps.
+
+3. You need to setup permissions for the folder for your compiled classes. Use the instructions per the documentation at https://docs.microsoft.com/en-us/sql/advanced-analytics/java/java-first-sample?view=sqlallproducts-allversions#6---set-permissions
+
+4. Run the **javasetup.sql** script to create the database and objects.
+
5. Run the **javangramdemo.sql** to execute the Java code. This script assumes the class files are in c:\java\pkg. I recommend you keep the pkg subdirectory. If you need to have the classes in a different parent folder, modify this part of the script SET @myClassPath = N'C:\java'
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/native_scoring.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/native_scoring.sql
index 31dce92e..af1c22d9 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/native_scoring.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/native_scoring.sql
@@ -1,77 +1,77 @@
-USE TutorialDB;
-
---STEP 1 - Setup model table for storing the model
-DROP TABLE IF EXISTS rental_models;
-GO
-CREATE TABLE rental_models (
- model_name VARCHAR(30) NOT NULL DEFAULT('default model'),
- lang VARCHAR(30),
- model VARBINARY(MAX),
- native_model VARBINARY(MAX),
- PRIMARY KEY (model_name, lang)
-
-);
-GO
-
---STEP 2 - Train model using revoscalepy rx_dtree or rxlinmod
-DROP PROCEDURE IF EXISTS generate_rental_py_native_model;
-go
-CREATE PROCEDURE generate_rental_py_native_model (@model_type varchar(30), @trained_model varbinary(max) OUTPUT)
-AS
-BEGIN
- EXECUTE sp_execute_external_script
- @language = N'Python'
- , @script = N'
-from revoscalepy import rx_lin_mod, rx_serialize_model, rx_dtree
-from pandas import Categorical
-import pickle
-
-rental_train_data["Holiday"] = rental_train_data["Holiday"].astype("category")
-rental_train_data["Snow"] = rental_train_data["Snow"].astype("category")
-rental_train_data["WeekDay"] = rental_train_data["WeekDay"].astype("category")
-
-if model_type == "linear":
- linmod_model = rx_lin_mod("RentalCount ~ Month + Day + WeekDay + Snow + Holiday", data = rental_train_data)
- trained_model = rx_serialize_model(linmod_model, realtime_scoring_only = True);
-if model_type == "dtree":
- dtree_model = rx_dtree("RentalCount ~ Month + Day + WeekDay + Snow + Holiday", data = rental_train_data)
- trained_model = rx_serialize_model(dtree_model, realtime_scoring_only = True);
-'
-
- , @input_data_1 = N'select "RentalCount", "Year", "Month", "Day", "WeekDay", "Snow", "Holiday" from dbo.rental_data where Year < 2015'
- , @input_data_1_name = N'rental_train_data'
- , @params = N'@trained_model varbinary(max) OUTPUT, @model_type varchar(30)'
- , @model_type = @model_type
- , @trained_model = @trained_model OUTPUT;
-END;
-GO
-
---STEP 3 - Save model to table
-
---Line of code to empty table with models
---TRUNCATE TABLE rental_models;
-
---Save Linear model to table
-DECLARE @model VARBINARY(MAX);
-EXEC generate_rental_py_native_model "linear", @model OUTPUT;
-INSERT INTO rental_models (model_name, native_model, lang) VALUES('linear_model', @model, 'Python');
-
---Save DTree model to table
-DECLARE @model2 VARBINARY(MAX);
-EXEC generate_rental_py_native_model "dtree", @model2 OUTPUT;
-INSERT INTO rental_models (model_name, native_model, lang) VALUES('dtree_model', @model2, 'Python');
-
--- Look at the models in the table
-SELECT * FROM rental_models;
-
-GO
-
---STEP 4 - Use the native PREDICT (native scoring) to predict number of rentals for both models
-DECLARE @model VARBINARY(MAX) = (SELECT TOP(1) native_model FROM dbo.rental_models WHERE model_name = 'linear_model' AND lang = 'Python');
-SELECT d.*, p.* FROM PREDICT(MODEL = @model, DATA = dbo.rental_data AS d) WITH(RentalCount_Pred float) AS p;
-GO
-
---Native scoring with dtree model
-DECLARE @model VARBINARY(MAX) = (SELECT TOP(1) native_model FROM dbo.rental_models WHERE model_name = 'dtree_model' AND lang = 'Python');
-SELECT d.*, p.* FROM PREDICT(MODEL = @model, DATA = dbo.rental_data AS d) WITH(RentalCount_Pred float) AS p;
+USE TutorialDB;
+
+--STEP 1 - Setup model table for storing the model
+DROP TABLE IF EXISTS rental_models;
+GO
+CREATE TABLE rental_models (
+ model_name VARCHAR(30) NOT NULL DEFAULT('default model'),
+ lang VARCHAR(30),
+ model VARBINARY(MAX),
+ native_model VARBINARY(MAX),
+ PRIMARY KEY (model_name, lang)
+
+);
+GO
+
+--STEP 2 - Train model using revoscalepy rx_dtree or rxlinmod
+DROP PROCEDURE IF EXISTS generate_rental_py_native_model;
+go
+CREATE PROCEDURE generate_rental_py_native_model (@model_type varchar(30), @trained_model varbinary(max) OUTPUT)
+AS
+BEGIN
+ EXECUTE sp_execute_external_script
+ @language = N'Python'
+ , @script = N'
+from revoscalepy import rx_lin_mod, rx_serialize_model, rx_dtree
+from pandas import Categorical
+import pickle
+
+rental_train_data["Holiday"] = rental_train_data["Holiday"].astype("category")
+rental_train_data["Snow"] = rental_train_data["Snow"].astype("category")
+rental_train_data["WeekDay"] = rental_train_data["WeekDay"].astype("category")
+
+if model_type == "linear":
+ linmod_model = rx_lin_mod("RentalCount ~ Month + Day + WeekDay + Snow + Holiday", data = rental_train_data)
+ trained_model = rx_serialize_model(linmod_model, realtime_scoring_only = True);
+if model_type == "dtree":
+ dtree_model = rx_dtree("RentalCount ~ Month + Day + WeekDay + Snow + Holiday", data = rental_train_data)
+ trained_model = rx_serialize_model(dtree_model, realtime_scoring_only = True);
+'
+
+ , @input_data_1 = N'select "RentalCount", "Year", "Month", "Day", "WeekDay", "Snow", "Holiday" from dbo.rental_data where Year < 2015'
+ , @input_data_1_name = N'rental_train_data'
+ , @params = N'@trained_model varbinary(max) OUTPUT, @model_type varchar(30)'
+ , @model_type = @model_type
+ , @trained_model = @trained_model OUTPUT;
+END;
+GO
+
+--STEP 3 - Save model to table
+
+--Line of code to empty table with models
+--TRUNCATE TABLE rental_models;
+
+--Save Linear model to table
+DECLARE @model VARBINARY(MAX);
+EXEC generate_rental_py_native_model "linear", @model OUTPUT;
+INSERT INTO rental_models (model_name, native_model, lang) VALUES('linear_model', @model, 'Python');
+
+--Save DTree model to table
+DECLARE @model2 VARBINARY(MAX);
+EXEC generate_rental_py_native_model "dtree", @model2 OUTPUT;
+INSERT INTO rental_models (model_name, native_model, lang) VALUES('dtree_model', @model2, 'Python');
+
+-- Look at the models in the table
+SELECT * FROM rental_models;
+
+GO
+
+--STEP 4 - Use the native PREDICT (native scoring) to predict number of rentals for both models
+DECLARE @model VARBINARY(MAX) = (SELECT TOP(1) native_model FROM dbo.rental_models WHERE model_name = 'linear_model' AND lang = 'Python');
+SELECT d.*, p.* FROM PREDICT(MODEL = @model, DATA = dbo.rental_data AS d) WITH(RentalCount_Pred float) AS p;
+GO
+
+--Native scoring with dtree model
+DECLARE @model VARBINARY(MAX) = (SELECT TOP(1) native_model FROM dbo.rental_models WHERE model_name = 'dtree_model' AND lang = 'Python');
+SELECT d.*, p.* FROM PREDICT(MODEL = @model, DATA = dbo.rental_data AS d) WITH(RentalCount_Pred float) AS p;
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/readme.md
index fda6fb4d..e67000b0 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/readme.md
@@ -1,26 +1,26 @@
-# SQL Server demo using Python and Native Scoring
-
-This demo shows the capability of running Python code in SQL Server 2017 and later and the Native scoring feature.
-
-This demo pulls SQL commands from the main site for a rental prediction tutorial using Python which you can find at https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/. Use this site for the entire tutorial in case any changes or additions are made there. If you want to use Native Scoring as a feature you do not have to install the Machine Learning Services feature but this demo will require it because we train the model using Python with SQL Server.
-
-## Requirements
-
-- SQL Server 2017 or later for Windows installed (Developer Edition will work just fine). You must choose the Machine Learning Services feature during installation (or add this feature if you have already installed)
-- You need to download the TutorialDB database backup from https://sqlchoice.blob.core.windows.net/sqlchoice/static/TutorialDB.bak
-- Enable this configuration option and restart SQL Server
-
-EXEC sp_configure 'external scripts enabled', 1;
-RECONFIGURE WITH OVERRIDE
-GO
-
-**Note:** With SQL Server 2019, python support exists for SQL Server on Linux. A linux version of this same demo can be achieved by installing Machine Learning services for Linux as documented at https://docs.microsoft.com/en-us/machine-learning-server/install/machine-learning-server-linux-install and follow the demo steps below.
-
-## Demo Steps
-
-1. Run **setup.sql** to restore the TutorialDB database backup
-2. Run the statements and examine the output from **rental_prediction.sql** to see an example of a machine learning model with Python and SQL Server.
-3. Run the statements and examine the output from **native_scoring.sql** to see an example of a machine learning model trained with Python but executed with native scoring in T-SQL.
-
-
-
+# SQL Server demo using Python and Native Scoring
+
+This demo shows the capability of running Python code in SQL Server 2017 and later and the Native scoring feature.
+
+This demo pulls SQL commands from the main site for a rental prediction tutorial using Python which you can find at https://microsoft.github.io/sql-ml-tutorials/python/rentalprediction/. Use this site for the entire tutorial in case any changes or additions are made there. If you want to use Native Scoring as a feature you do not have to install the Machine Learning Services feature but this demo will require it because we train the model using Python with SQL Server.
+
+## Requirements
+
+- SQL Server 2017 or later for Windows installed (Developer Edition will work just fine). You must choose the Machine Learning Services feature during installation (or add this feature if you have already installed)
+- You need to download the TutorialDB database backup from https://sqlchoice.blob.core.windows.net/sqlchoice/static/TutorialDB.bak
+- Enable this configuration option and restart SQL Server
+
+EXEC sp_configure 'external scripts enabled', 1;
+RECONFIGURE WITH OVERRIDE
+GO
+
+**Note:** With SQL Server 2019, python support exists for SQL Server on Linux. A linux version of this same demo can be achieved by installing Machine Learning services for Linux as documented at https://docs.microsoft.com/en-us/machine-learning-server/install/machine-learning-server-linux-install and follow the demo steps below.
+
+## Demo Steps
+
+1. Run **setup.sql** to restore the TutorialDB database backup
+2. Run the statements and examine the output from **rental_prediction.sql** to see an example of a machine learning model with Python and SQL Server.
+3. Run the statements and examine the output from **native_scoring.sql** to see an example of a machine learning model trained with Python but executed with native scoring in T-SQL.
+
+
+
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/rental_prediction.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/rental_prediction.sql
index ee1826d7..d7c9295c 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/rental_prediction.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/rental_prediction.sql
@@ -1,113 +1,113 @@
-USE TutorialDB;
--- Table containing ski rental data
-SELECT * FROM [dbo].[rental_data];
-
--------------------------- STEP 1 - Setup model table ----------------------------------------
-
-DROP TABLE IF EXISTS rental_py_models;
-GO
-CREATE TABLE rental_py_models (
- model_name VARCHAR(30) NOT NULL DEFAULT('default model') PRIMARY KEY,
- model VARBINARY(MAX) NOT NULL
-);
-GO
-
--------------------------- STEP 2 - Train model ----------------------------------------
--- Stored procedure that trains and generates an R model using the rental_data and a decision tree algorithm
-DROP PROCEDURE IF EXISTS generate_rental_py_model;
-go
-CREATE PROCEDURE generate_rental_py_model (@trained_model varbinary(max) OUTPUT)
-AS
-BEGIN
- EXECUTE sp_execute_external_script
- @language = N'Python'
- , @script = N'
-
-df = rental_train_data
-# Get all the columns from the dataframe.
-columns = df.columns.tolist()
-# Store the variable well be predicting on.
-target = "RentalCount"
-from sklearn.linear_model import LinearRegression
-# Initialize the model class.
-lin_model = LinearRegression()
-# Fit the model to the training data.
-lin_model.fit(df[columns], df[target])
-import pickle
-#Before saving the model to the DB table, we need to convert it to a binary object
-trained_model = pickle.dumps(lin_model)
-'
- , @input_data_1 = N'select "RentalCount", "Year", "Month", "Day", "WeekDay", "Snow", "Holiday" from dbo.rental_data where Year < 2015'
- , @input_data_1_name = N'rental_train_data'
- , @params = N'@trained_model varbinary(max) OUTPUT'
- , @trained_model = @trained_model OUTPUT;
-END;
-GO
-------------------- STEP 3 - Save model to table -------------------------------------
-TRUNCATE TABLE rental_py_models;
-DECLARE @model VARBINARY(MAX);
-EXEC generate_rental_py_model @model OUTPUT;
-INSERT INTO rental_py_models (model_name, model) VALUES('linear_model', @model);
-SELECT * FROM rental_py_models;
-
------------------- STEP 4 - Use the model to predict number of rentals --------------------------
-DROP PROCEDURE IF EXISTS py_predict_rentalcount;
-GO
-CREATE PROCEDURE py_predict_rentalcount (@model varchar(100))
-AS
-BEGIN
- DECLARE @py_model varbinary(max) = (select model from rental_py_models where model_name = @model);
- EXEC sp_execute_external_script
- @language = N'Python'
- , @script = N'import pickle
-rental_model = pickle.loads(py_model)
-df = rental_score_data
-#print(df)
-# Get all the columns from the dataframe.
-columns = df.columns.tolist()
-# Filter the columns to remove ones we dont want.
-# columns = [c for c in columns if c not in ["Year"]]
-# Store the variable well be predicting on.
-target = "RentalCount"
-# Generate our predictions for the test set.
-lin_predictions = rental_model.predict(df[columns])
-print(lin_predictions)
-# Import the scikit-learn function to compute error.
-from sklearn.metrics import mean_squared_error
-# Compute error between our test predictions and the actual values.
-lin_mse = mean_squared_error(lin_predictions, df[target])
-#print(lin_mse)
-import pandas as pd
-predictions_df = pd.DataFrame(lin_predictions)
-OutputDataSet = pd.concat([predictions_df, df["RentalCount"], df["Month"], df["Day"], df["WeekDay"], df["Snow"], df["Holiday"], df["Year"]], axis=1)
-'
- , @input_data_1 = N'Select "RentalCount", "Year" ,"Month", "Day", "WeekDay", "Snow", "Holiday" from rental_data where Year = 2015'
- , @input_data_1_name = N'rental_score_data'
- , @params = N'@py_model varbinary(max)'
- , @py_model = @py_model
- with result sets (("RentalCount_Predicted" float, "RentalCount" float, "Month" float,"Day" float,"WeekDay" float,"Snow" float,"Holiday" float, "Year" float));
-END;
-GO
-
----------------- STEP 5 - Create DB table to store predictions -----------------------
-DROP TABLE IF EXISTS [dbo].[py_rental_predictions];
-GO
---Create a table to store the predictions in
-CREATE TABLE [dbo].[py_rental_predictions](
- [RentalCount_Predicted] [int] NULL,
- [RentalCount_Actual] [int] NULL,
- [Month] [int] NULL,
- [Day] [int] NULL,
- [WeekDay] [int] NULL,
- [Snow] [int] NULL,
- [Holiday] [int] NULL,
- [Year] [int] NULL
-) ON [PRIMARY]
-GO
----------------- STEP 6 - Save the predictions in a DB table -----------------------
-TRUNCATE TABLE py_rental_predictions;
---Insert the results of the predictions for test set into a table
-INSERT INTO py_rental_predictions
-EXEC py_predict_rentalcount 'linear_model';
--- Select contents of the table
-SELECT * FROM py_rental_predictions;
+USE TutorialDB;
+-- Table containing ski rental data
+SELECT * FROM [dbo].[rental_data];
+
+-------------------------- STEP 1 - Setup model table ----------------------------------------
+
+DROP TABLE IF EXISTS rental_py_models;
+GO
+CREATE TABLE rental_py_models (
+ model_name VARCHAR(30) NOT NULL DEFAULT('default model') PRIMARY KEY,
+ model VARBINARY(MAX) NOT NULL
+);
+GO
+
+-------------------------- STEP 2 - Train model ----------------------------------------
+-- Stored procedure that trains and generates an R model using the rental_data and a decision tree algorithm
+DROP PROCEDURE IF EXISTS generate_rental_py_model;
+go
+CREATE PROCEDURE generate_rental_py_model (@trained_model varbinary(max) OUTPUT)
+AS
+BEGIN
+ EXECUTE sp_execute_external_script
+ @language = N'Python'
+ , @script = N'
+
+df = rental_train_data
+# Get all the columns from the dataframe.
+columns = df.columns.tolist()
+# Store the variable well be predicting on.
+target = "RentalCount"
+from sklearn.linear_model import LinearRegression
+# Initialize the model class.
+lin_model = LinearRegression()
+# Fit the model to the training data.
+lin_model.fit(df[columns], df[target])
+import pickle
+#Before saving the model to the DB table, we need to convert it to a binary object
+trained_model = pickle.dumps(lin_model)
+'
+ , @input_data_1 = N'select "RentalCount", "Year", "Month", "Day", "WeekDay", "Snow", "Holiday" from dbo.rental_data where Year < 2015'
+ , @input_data_1_name = N'rental_train_data'
+ , @params = N'@trained_model varbinary(max) OUTPUT'
+ , @trained_model = @trained_model OUTPUT;
+END;
+GO
+------------------- STEP 3 - Save model to table -------------------------------------
+TRUNCATE TABLE rental_py_models;
+DECLARE @model VARBINARY(MAX);
+EXEC generate_rental_py_model @model OUTPUT;
+INSERT INTO rental_py_models (model_name, model) VALUES('linear_model', @model);
+SELECT * FROM rental_py_models;
+
+------------------ STEP 4 - Use the model to predict number of rentals --------------------------
+DROP PROCEDURE IF EXISTS py_predict_rentalcount;
+GO
+CREATE PROCEDURE py_predict_rentalcount (@model varchar(100))
+AS
+BEGIN
+ DECLARE @py_model varbinary(max) = (select model from rental_py_models where model_name = @model);
+ EXEC sp_execute_external_script
+ @language = N'Python'
+ , @script = N'import pickle
+rental_model = pickle.loads(py_model)
+df = rental_score_data
+#print(df)
+# Get all the columns from the dataframe.
+columns = df.columns.tolist()
+# Filter the columns to remove ones we dont want.
+# columns = [c for c in columns if c not in ["Year"]]
+# Store the variable well be predicting on.
+target = "RentalCount"
+# Generate our predictions for the test set.
+lin_predictions = rental_model.predict(df[columns])
+print(lin_predictions)
+# Import the scikit-learn function to compute error.
+from sklearn.metrics import mean_squared_error
+# Compute error between our test predictions and the actual values.
+lin_mse = mean_squared_error(lin_predictions, df[target])
+#print(lin_mse)
+import pandas as pd
+predictions_df = pd.DataFrame(lin_predictions)
+OutputDataSet = pd.concat([predictions_df, df["RentalCount"], df["Month"], df["Day"], df["WeekDay"], df["Snow"], df["Holiday"], df["Year"]], axis=1)
+'
+ , @input_data_1 = N'Select "RentalCount", "Year" ,"Month", "Day", "WeekDay", "Snow", "Holiday" from rental_data where Year = 2015'
+ , @input_data_1_name = N'rental_score_data'
+ , @params = N'@py_model varbinary(max)'
+ , @py_model = @py_model
+ with result sets (("RentalCount_Predicted" float, "RentalCount" float, "Month" float,"Day" float,"WeekDay" float,"Snow" float,"Holiday" float, "Year" float));
+END;
+GO
+
+---------------- STEP 5 - Create DB table to store predictions -----------------------
+DROP TABLE IF EXISTS [dbo].[py_rental_predictions];
+GO
+--Create a table to store the predictions in
+CREATE TABLE [dbo].[py_rental_predictions](
+ [RentalCount_Predicted] [int] NULL,
+ [RentalCount_Actual] [int] NULL,
+ [Month] [int] NULL,
+ [Day] [int] NULL,
+ [WeekDay] [int] NULL,
+ [Snow] [int] NULL,
+ [Holiday] [int] NULL,
+ [Year] [int] NULL
+) ON [PRIMARY]
+GO
+---------------- STEP 6 - Save the predictions in a DB table -----------------------
+TRUNCATE TABLE py_rental_predictions;
+--Insert the results of the predictions for test set into a table
+INSERT INTO py_rental_predictions
+EXEC py_predict_rentalcount 'linear_model';
+-- Select contents of the table
+SELECT * FROM py_rental_predictions;
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/setup.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/setup.sql
index dc9d02a5..f84cb9b9 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/setup.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/python/setup.sql
@@ -1,8 +1,8 @@
-USE master
-GO
-RESTORE DATABASE TutorialDB
-FROM DISK = 'c:\demos\sql2019\python\TutorialDB.bak'
-WITH
-MOVE 'TutorialDB' TO 'c:\demos\sql2019\python\TutorialDB.mdf',
-MOVE 'TutorialDB_log' TO 'c:\demos\sql2019\python\TutorialDB.ldf'
+USE master
+GO
+RESTORE DATABASE TutorialDB
+FROM DISK = 'c:\demos\sql2019\python\TutorialDB.bak'
+WITH
+MOVE 'TutorialDB' TO 'c:\demos\sql2019\python\TutorialDB.mdf',
+MOVE 'TutorialDB_log' TO 'c:\demos\sql2019\python\TutorialDB.ldf'
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/readme.md
index fa4e5ad1..ea6a5b84 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 5 Activity - Modern Development Platform/readme.md
@@ -1,11 +1,11 @@
-# Module 5 Activities - Modern Development Platform
-
-These represent demos and examples you can run to see new capabilities for the modern development platform integrated with SQL Server 2019
-
-## python
-
-Learn how to execute a machine learning model with Python and built-in native scoring with SQL Server.
-
-## java
-
+# Module 5 Activities - Modern Development Platform
+
+These represent demos and examples you can run to see new capabilities for the modern development platform integrated with SQL Server 2019
+
+## python
+
+Learn how to execute a machine learning model with Python and built-in native scoring with SQL Server.
+
+## java
+
Learn how to execute a java program with the extensibility architecture of SQL Server 2019 using T-SQL.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/deploy/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/deploy/readme.md
index f2ba2803..369d09a9 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/deploy/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/deploy/readme.md
@@ -1,48 +1,48 @@
-# Deploy SQL Server on Linux
-
-This demo is used to show the basic steps to deploy SQL Server on Linux using RedHat Enterprise Linux. While this exercise shows you how to deploy SQL Server on RedHat, Ubuntu and SUSE are also supported. See our installation gudiance for SQL Server 2017 https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup?view=sql-server-2017 and SQL Server 2019 https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup?view=sql-server-2017#sqlvnext for more information on how to install SQL Server on those Linux distributions.
-
-## Requirements
-
-- A RedHat Linux installation. This can be either in a VM or bare metal machine. Look at the the Prerequisites in our documentation page at https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-red-hat?view=sql-server-2017#prerequisites
-- Ensure your machine or VM meets the Systems requirements at https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup?view=sql-server-2017#system.
-- Use either a built-in ssh from your client (Ex. Linux, macOS, or bash shell for Windows at https://docs.microsoft.com/en-us/windows/wsl/install-win10). As a windows usser, I like to use Mobaxterm (https://mobaxterm.mobatek.net/).
-- You also need to establish connectivity with your Linux server or VM by having the IP address available and port 22 for ssh open for connectivity.
-
-## How to deploy SQL Server on Linux
-
-Run all of the following commands from your ssh session with the bash shell. This lab assumes your Linux Server is connected to the internet. You can do an offline installation of SQL Server on Linux. See the documentation at
-
-For this exercise we will be using SQL Server 2019 Preview but you can also do the same set of instructions for SQL Server 2017 as documented at https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-red-hat?view=sql-server-2017.
-
-1. Copy the repository configuration file using the following command
-
- `sudo curl -o /etc/yum.repos.d/mssql-server.repo https://packages.microsoft.com/config/rhel/7/mssql-server-preview.repo`
-
- The repository configuration file is a text file that contains the location of the SQL Server package for RHEL. This repo file will point to the latest preview build of SQL Server 2019 on Linux. See our documentation for how to use a repository file for other branches
-
- With a good internet connection this should take a few seconds.
-
-2. Use the yum package manager to kick off the installation with the following command (-y means don't prompt)
-
- `sudo yum install -y mssql-server`
-
- This involves download an ~220Mb file and performing the first phase of install. With a good internet connection this should only take a few minutes.
-
-3. Now you must complete the installation by executing a bash shell script we install called **mssql-conf** (which uses a series of python scripts). We will also use mssql-conf later to perform a configuration task for SQL Server. Execute the following command
-
- `sudo /opt/mssql/bin/mssql-conf setup`
-
- Go through the prompts to pick Edition (choose Developer or Enterprise Core for these labs), accept the EULA, and put in the sa password (must meet strong password requirements like SQL Server on Windows). Remember the sa password as you will use it often in the labs.
-
-4. Open up the firewall on Linux for the SQL Server port by running the following two commands. This is required if you plan to connect to SQL Server on a remote client.
-
- `sudo firewall-cmd --zone=public --add-port=1433/tcp --permanent`
-
- `sudo firewall-cmd --reload`
-
- Believe it or not, that's it! You have now installed SQL Server on Linux which includes the core database engine and SQL Server Agent
-
-5. Install the command line tools as documented at https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-red-hat?view=sql-server-2017#tools
-
+# Deploy SQL Server on Linux
+
+This demo is used to show the basic steps to deploy SQL Server on Linux using RedHat Enterprise Linux. While this exercise shows you how to deploy SQL Server on RedHat, Ubuntu and SUSE are also supported. See our installation gudiance for SQL Server 2017 https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup?view=sql-server-2017 and SQL Server 2019 https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup?view=sql-server-2017#sqlvnext for more information on how to install SQL Server on those Linux distributions.
+
+## Requirements
+
+- A RedHat Linux installation. This can be either in a VM or bare metal machine. Look at the the Prerequisites in our documentation page at https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-red-hat?view=sql-server-2017#prerequisites
+- Ensure your machine or VM meets the Systems requirements at https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup?view=sql-server-2017#system.
+- Use either a built-in ssh from your client (Ex. Linux, macOS, or bash shell for Windows at https://docs.microsoft.com/en-us/windows/wsl/install-win10). As a windows usser, I like to use Mobaxterm (https://mobaxterm.mobatek.net/).
+- You also need to establish connectivity with your Linux server or VM by having the IP address available and port 22 for ssh open for connectivity.
+
+## How to deploy SQL Server on Linux
+
+Run all of the following commands from your ssh session with the bash shell. This lab assumes your Linux Server is connected to the internet. You can do an offline installation of SQL Server on Linux. See the documentation at
+
+For this exercise we will be using SQL Server 2019 Preview but you can also do the same set of instructions for SQL Server 2017 as documented at https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-red-hat?view=sql-server-2017.
+
+1. Copy the repository configuration file using the following command
+
+ `sudo curl -o /etc/yum.repos.d/mssql-server.repo https://packages.microsoft.com/config/rhel/7/mssql-server-preview.repo`
+
+ The repository configuration file is a text file that contains the location of the SQL Server package for RHEL. This repo file will point to the latest preview build of SQL Server 2019 on Linux. See our documentation for how to use a repository file for other branches
+
+ With a good internet connection this should take a few seconds.
+
+2. Use the yum package manager to kick off the installation with the following command (-y means don't prompt)
+
+ `sudo yum install -y mssql-server`
+
+ This involves download an ~220Mb file and performing the first phase of install. With a good internet connection this should only take a few minutes.
+
+3. Now you must complete the installation by executing a bash shell script we install called **mssql-conf** (which uses a series of python scripts). We will also use mssql-conf later to perform a configuration task for SQL Server. Execute the following command
+
+ `sudo /opt/mssql/bin/mssql-conf setup`
+
+ Go through the prompts to pick Edition (choose Developer or Enterprise Core for these labs), accept the EULA, and put in the sa password (must meet strong password requirements like SQL Server on Windows). Remember the sa password as you will use it often in the labs.
+
+4. Open up the firewall on Linux for the SQL Server port by running the following two commands. This is required if you plan to connect to SQL Server on a remote client.
+
+ `sudo firewall-cmd --zone=public --add-port=1433/tcp --permanent`
+
+ `sudo firewall-cmd --reload`
+
+ Believe it or not, that's it! You have now installed SQL Server on Linux which includes the core database engine and SQL Server Agent
+
+5. Install the command line tools as documented at https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-red-hat?view=sql-server-2017#tools
+
6. Install the mssql-cli tool as documented at https://github.com/dbcli/mssql-cli/blob/master/doc/installation/linux.md#red-hat-enterprise-linux-rhel-7. This is a new command line tool which you will use in other exercises.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/cpwwi.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/cpwwi.sh
index 9d8e1516..f4504c7b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/cpwwi.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/cpwwi.sh
@@ -1,2 +1,2 @@
-sudo cp WideWorldImporters-Full.bak /var/opt/mssql
-sudo chown mssql:mssql /var/opt/mssql/WideWorldImporters-Full.bak
+sudo cp WideWorldImporters-Full.bak /var/opt/mssql
+sudo chown mssql:mssql /var/opt/mssql/WideWorldImporters-Full.bak
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/readme.md
index 63276280..42e0e7f8 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/readme.md
@@ -1,290 +1,290 @@
-# Explore SQL Server on Linux
-
-In this exercise you will learn how to explore SQL Server on Linux after you have deployed by learning
-
-- Explore the SQL Server installation
-- Common Linux commands
-- How to connect to SQL Server
-- How to restore a backup and run queries
-- How to configure SQL Server
-
-## Requirements
-
-Many of these requirements are met if you follow the **deploy** exercises in this Module.
-
-- SQL Server deployed on Linux
-- SQL Server command line tools deployed for Linux
-- SQL Server Managed Studio installed on a Windows client that can connect to your Linux Server or VM.
-- Azure Data Studio deployed on Windows, Linux, or macOS on a client that can connect to your Linux Server of VM. You can read more about how to install Azure Data Studio at https://docs.microsoft.com/en-us/sql/azure-data-studio/download?view=sql-server-2017.
-
-**Tip**: For many of my demos, I setup a resource group in Azure and put all of my VMs that I use for connectivity in the same resource group. This puts them all in the same virtual network. Now I can put the private IP addresses of all my VMs in the /etc/hosts file on either Windows or Linux and use those names to connect between VMs.
-
-## Explore the SQL Server installation
-
-In this exercise, you will explore the SQL Server installation post deployment.
-
-1. Run the following command to see the state of SQL Server
-
- `sudo systemctl status mssql-server`
-
- Using sudo allows you to see the tail of the SQL Server ERRORLOG file.
-
-2. Run the following commands to stop, start, and restart SQL Server.
-
- `sudo systemctl stop mssql-server`
-
- `sudo systemctl status mssql-server`
-
- `sudo systemctl start mssql-server`
-
- `sudo systemctl status mssql-server`
-
- `sudo systemctl restart mssql-server`
-
- `sudo systemctl status mssql-server`
-
- Note that there are no return values when starting, stopping, or restarting. You must run systemctl status to check on the status of SQL Server. With each start of SQL Server, you should see different PID values (for new processes).
-
-3. Let's see where everything is installed. Run the following command to see where the binaries are installed
-
- `sudo ls -l /opt/mssql/bin`
-
- This directory contains the sqlservr executable, mssql-conf script, and other files to support crash dumps. There is no method today to change the location of these files.
-
-4. Run these commands to see where the default directories for databases and ERRORLOG log (and other log files) are stored
-
- `sudo ls -l /var/opt/mssql/data`
-
- `sudo ls -l /var/opt/mssql/log`
-
- Note from the results that the owner of these files is mssql and mssql. This is a group and non-interactive user called mssql which is the context under which sqlservr executes. Any time sqlservr needs to read or write a file, that file or directory must have mssql:mssql permissions. There is no method to change this today. You can change the default locations of database files, backups, transaction log files, and the ERRORLOG directory using the mssql-conf script.
-
-5. Let's dump out the current ERRORLOG file using a command on Linux called **cat** (and another variation using **more** so you can page the results)
-
- `sudo cat /var/opt/mssql/log/errorlog`
-
- `sudo more /var/opt/mssql/log/errorlog`
-
-## Common Linux commands
-
-Now that you have deployed SQL Server on Linux here are a few common commands you may need for Linux while working with SQL Server.
-
-1. Find out information about the computer running Linux by running the following command
-
- `sudo dmidecode -t 1`
-
-2. Find out information about the Linux distribution by running the following command
-
- `cat /etc/*-release`
-
-3. Find out information about memory configured on the Linux Server by running the following command
-
- `cat /proc/meminfo`
-
- The **MemTotal** is the total amount of physical memory on the Linux Server
-
- The /proc directory is known as the *proc filesystem* and there is other interesting information exposed in files in this directory.
-
-4. Find out about the number of cores, sockets, NUMA nodes, and chip architecture by running the following command
-
- `lscpu`
-
-5. The **ps** command is used to view all processes on the Linux Server. Use this command to scroll through all processes including parent/child process relationships
-
- `ps axjf | more`
-
-6. Run the following command to see a list of disks and mounted file systems on these disks including disk sizes
-
- `df -H`
-
- The disk starting with /dev are the true disks for the server.
-
-7. To see basic performance information by process run the following command
-
- `top`
-
- **top** will sort the results with the process using the most CPU at the top which since nothing else is running is sqlservr
-
- The **KiB Mem** values show physical total, free, and used memory.
- The **RES** column is the amount of physical memory used by a process like sqlservr.
-
- **top** is interactive so type in "q" to quit the program
-
-8. **iotop** is a utility to monitor I/O statistics per process. However, it is not installed by default. Run the following command to first install iotop
-
- `sudo yum install -y iotop`
-
- Now run the following command to execute iotop
-
- `sudo iotop`
-
- This shows the overall I/O on the system plus I/O per process. Type in "q" to exit the program. Run this version of the command to only view I/O for processes actually using I/O. This program is interactive and refreshes itself every few seconds
-
- `sudo iotop -o`
-
- There are many other options with iotop. Execute the command `man iotop` to experiment with all iotop options.
-
-9. **htop** is an interactive program to see process utilization information across processors and processes. However, it is not installed by default so run the following commands first to install htop.
-
- `sudo wget dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm`
-
- `sudo rpm -ihv epel-release-7-11.noarch.rpm`
-
- `sudo yum install -y htop`
-
- Now run the interactive htop command to observe its display
-
- `htop`
-
- Type "q" to exit the tool
-
-10. You will likely need a good editor while using Linux. While the editor vi is installed by default, I recommend you use the **nano** editor. It may be already installed but if not run the following command to install it
-
- `sudo yum install -y nano`
-
- Let's use nano to create a shell script to run on Linux
-
- `nano dumperrorlog.sh`
-
- nano is a full screen editor. Type in the following in the editor window
-
- `sudo cat /var/opt/mssql/log/errorlog`
-
- Type Ctrl+X to exit. You will get prompted to save the file
-
- Run the following command to make the script executable
-
- `chmod u+x dumperrorlog.sh`
-
- Now execute the script
-
- `./dumperrorlog.sh`
-
-## How to connect to SQL Server
-
-Here are a few examples of how to do basic connectivity to SQL Server on Linux. These exercises assume you have installed the command line tools for Linux, mssql-cli, and have a machine or VM running Windows that can connect to the Linux server or VM.
-
-1. Run a quick test to connect with sqlcmd by executing the following
-
- `sqlcmd -Usa -Slocalhost`
-
- Put in your sa password. At the sqlcmd prompt, run the following T-SQL statement
-
- ```sql
- SELECT @@VERSION
- GO
- ```
- Type in "exit" to quit sqlcmd
-
- Note: You can connect with any sqlcmd tool from Windows, Linux, or macOS in the same was provided you have connectivity to your Linux Server or VM. The -S parameter would be the hostname or IP address of the Linux Server or VM.
-
-2. Now test mssql-cli like we did for sqlcmd by running the following command
-
- `mssql-cli -Usa -Slocalhost`
-
- You should get a new prompt like sqlcmd. At this prompt type in the following T-SQL command and hit Enter
-
- ```sql
- SELECT @@VERSION
- ```
-
- Notice as you started typing you see Intellisense functionality kick-in which is one of the differences from sqlcmd.
-
- If you are not put back into the mssql-cli prompt, type "q" to get back to the prompt.
-
- mssql-cli does not recognize the "GO" keyword as sqlcmd does. Use a ";" to separate batches. You can also hit F3 to type statements in multiple lines but they will all be in one batch.
-
- Type in "exit" to quit mssql-cli
-
- Note: You can connect with any mssql-cli tool from Windows, Linux, or macOS in the same was provided you have connectivity to your Linux Server or VM. The -S parameter would be the hostname or IP address of the Linux Server or VM.
-
-3. Connect with SQL Server Management Studio (SSMS) using SQL Authentication with the sa account and the server name or IP address:port for your Linux Server. Notice how SSMS works "as is" against the Linux Server and looks almost like a SQL Server on Windows deployment.
-
- Use Object Explorer and the Query Editor just like you would a normal SQL Server instance. Go through some of the steps in the SSMS tutorial in our documentation at
-
-4. Go through the the quickstart tutorial for connecting to SQL Server from Azure Data Studio with the SQL Server on Linux deployment at https://docs.microsoft.com/en-us/sql/azure-data-studio/quickstart-sql-server?view=sql-server-2017.
-
-## How to restore a backup and run queries
-
-In this exercise, you will learn how to restore a backup of a database to SQL Server on Linux, and run queries against the database.
-
-Now you will learn the great compatibility story of SQL Server on Linux by restoring a backup from SQL Server on Windows to SQL Server on Linux. And you will interact with this database using sqlcmd and mssql-cli. This section of the lab assumes your Linux Server is connected to the internet. If you are not connected to the internet, you can download the database to restore from and then copy it to your Linux Server (MobaXterm drag and drop is really nice for this)
-
-1. From your Linux ssh session, run the following command from the bash shell
-
- `wget https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak`
-
- Depending on your network speed this should take no more than a few minutes
-
-2. Copy and restore the WideWorldImporters database. Copy the **cpwwi.sh**, **restorewwi.sh**, and **restorewwi_linux.sql** files from the downloaded zip of the gitHub repo into your home directory on Linux. MobaXterm provides drag and drop capabilities to do this. Copy these files and drop them into the "explorer" pane in MobaXterm on the left hand side from your ssh session.
-
- Note: You can skip this step if you have already cloned the git repo in the prelab. If you have done this, the scripts in this lab are in the **sqllinuxlab** subdirectory. You can copy them into your home directory or edit them to ensure you have the right path for the WideWorldImporters backup file.
-
-3. Run the following commands from the bash shell to make the scripts executable (supply the root password if prompted)
-
- `sudo chmod u+x cpwwi.sh`
-
- `sudo chmod u+x restorewwi.sh`
-
-4. Copy the backup file to the SQL Server directory so it can access the file and change permissions on the backup file by executing the following command in the bash shell
-
- `./cpwwi.sh`
-
-5. Now restore the database by executing the following command from the bash shell
-
- `./restorewwi.sh`
-
-6. Connect with sa to run a query against this database. Run sqlcmd first to connect. Type in the sa password when prompted
-
- `sqlcmd -Usa -Slocalhost`
-
-7. From the sqlcmd prompt run these commands
-
- ```sql
- USE WideWorldImporters
- GO
- SELECT * FROM [Sales].[Customers]
- GO
- ```
-
- Type in "exit" to quit sqlcmd
-
-9. Now run the same set of commands using mssql-cli. Connect to SQL Server with mssql-cli. Type in the sa password when prompted
-
- `mssql-cli -Usa -Slocalhost`
-
-10. Run the following T-SQL commands from the msql-cli prompt (BONUS: Use Intellisense to complete these queries)
-
- `USE WideWorldImporters;SELECT * FROM Sales.Customers;`
-
- See how mssql-cli by default will present rows in a vertical record format. Hit Enter or Space to keep paging as many rows as you like.
-
- Type in "q" at any time to get back to the prompt and "exit" to quit mssql-cli
-
-## How to configure SQL Server
-
-In this exercise, you will learn how to configure SQL Server on Linux with the mssql-conf tool.
-
-There may be situations where you need to enable a traceflag as global and at SQL Server startup time. For Windows, this is done through the SQL Server Configuration Manager. For SQL Server on Linux, you will use the mssql-conf script. A list of all documented traceflags can be found at .
-
-Let's say you wanted to enable trace flag 1222 for deadlock details to be reported in the ERRORLOG.
-
-1. Run the following command from an ssh session with the bash shell
-
- `sudo /opt/mssql/bin/mssql-conf traceflag 1222 on`
-
-2. Per these instructions, restart SQL Server with the following command:
-
- `sudo systemctl restart mssql-server`
-
- Note: If this is successful, the command just returns to the shell prompt
-
-3. Verify the trace flag was properly set by looking at the ERRORLOG with the following command
-
- `sudo more /var/opt/mssql/log/errorlog`
-
-4. Use sqlcmd or mssql-cli to verify this trace flag is set by running the following T-SQL statement
-
- ```sql
- DBCC TRACESTATUS(-1)
+# Explore SQL Server on Linux
+
+In this exercise you will learn how to explore SQL Server on Linux after you have deployed by learning
+
+- Explore the SQL Server installation
+- Common Linux commands
+- How to connect to SQL Server
+- How to restore a backup and run queries
+- How to configure SQL Server
+
+## Requirements
+
+Many of these requirements are met if you follow the **deploy** exercises in this Module.
+
+- SQL Server deployed on Linux
+- SQL Server command line tools deployed for Linux
+- SQL Server Managed Studio installed on a Windows client that can connect to your Linux Server or VM.
+- Azure Data Studio deployed on Windows, Linux, or macOS on a client that can connect to your Linux Server of VM. You can read more about how to install Azure Data Studio at https://docs.microsoft.com/en-us/sql/azure-data-studio/download?view=sql-server-2017.
+
+**Tip**: For many of my demos, I setup a resource group in Azure and put all of my VMs that I use for connectivity in the same resource group. This puts them all in the same virtual network. Now I can put the private IP addresses of all my VMs in the /etc/hosts file on either Windows or Linux and use those names to connect between VMs.
+
+## Explore the SQL Server installation
+
+In this exercise, you will explore the SQL Server installation post deployment.
+
+1. Run the following command to see the state of SQL Server
+
+ `sudo systemctl status mssql-server`
+
+ Using sudo allows you to see the tail of the SQL Server ERRORLOG file.
+
+2. Run the following commands to stop, start, and restart SQL Server.
+
+ `sudo systemctl stop mssql-server`
+
+ `sudo systemctl status mssql-server`
+
+ `sudo systemctl start mssql-server`
+
+ `sudo systemctl status mssql-server`
+
+ `sudo systemctl restart mssql-server`
+
+ `sudo systemctl status mssql-server`
+
+ Note that there are no return values when starting, stopping, or restarting. You must run systemctl status to check on the status of SQL Server. With each start of SQL Server, you should see different PID values (for new processes).
+
+3. Let's see where everything is installed. Run the following command to see where the binaries are installed
+
+ `sudo ls -l /opt/mssql/bin`
+
+ This directory contains the sqlservr executable, mssql-conf script, and other files to support crash dumps. There is no method today to change the location of these files.
+
+4. Run these commands to see where the default directories for databases and ERRORLOG log (and other log files) are stored
+
+ `sudo ls -l /var/opt/mssql/data`
+
+ `sudo ls -l /var/opt/mssql/log`
+
+ Note from the results that the owner of these files is mssql and mssql. This is a group and non-interactive user called mssql which is the context under which sqlservr executes. Any time sqlservr needs to read or write a file, that file or directory must have mssql:mssql permissions. There is no method to change this today. You can change the default locations of database files, backups, transaction log files, and the ERRORLOG directory using the mssql-conf script.
+
+5. Let's dump out the current ERRORLOG file using a command on Linux called **cat** (and another variation using **more** so you can page the results)
+
+ `sudo cat /var/opt/mssql/log/errorlog`
+
+ `sudo more /var/opt/mssql/log/errorlog`
+
+## Common Linux commands
+
+Now that you have deployed SQL Server on Linux here are a few common commands you may need for Linux while working with SQL Server.
+
+1. Find out information about the computer running Linux by running the following command
+
+ `sudo dmidecode -t 1`
+
+2. Find out information about the Linux distribution by running the following command
+
+ `cat /etc/*-release`
+
+3. Find out information about memory configured on the Linux Server by running the following command
+
+ `cat /proc/meminfo`
+
+ The **MemTotal** is the total amount of physical memory on the Linux Server
+
+ The /proc directory is known as the *proc filesystem* and there is other interesting information exposed in files in this directory.
+
+4. Find out about the number of cores, sockets, NUMA nodes, and chip architecture by running the following command
+
+ `lscpu`
+
+5. The **ps** command is used to view all processes on the Linux Server. Use this command to scroll through all processes including parent/child process relationships
+
+ `ps axjf | more`
+
+6. Run the following command to see a list of disks and mounted file systems on these disks including disk sizes
+
+ `df -H`
+
+ The disk starting with /dev are the true disks for the server.
+
+7. To see basic performance information by process run the following command
+
+ `top`
+
+ **top** will sort the results with the process using the most CPU at the top which since nothing else is running is sqlservr
+
+ The **KiB Mem** values show physical total, free, and used memory.
+ The **RES** column is the amount of physical memory used by a process like sqlservr.
+
+ **top** is interactive so type in "q" to quit the program
+
+8. **iotop** is a utility to monitor I/O statistics per process. However, it is not installed by default. Run the following command to first install iotop
+
+ `sudo yum install -y iotop`
+
+ Now run the following command to execute iotop
+
+ `sudo iotop`
+
+ This shows the overall I/O on the system plus I/O per process. Type in "q" to exit the program. Run this version of the command to only view I/O for processes actually using I/O. This program is interactive and refreshes itself every few seconds
+
+ `sudo iotop -o`
+
+ There are many other options with iotop. Execute the command `man iotop` to experiment with all iotop options.
+
+9. **htop** is an interactive program to see process utilization information across processors and processes. However, it is not installed by default so run the following commands first to install htop.
+
+ `sudo wget dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm`
+
+ `sudo rpm -ihv epel-release-7-11.noarch.rpm`
+
+ `sudo yum install -y htop`
+
+ Now run the interactive htop command to observe its display
+
+ `htop`
+
+ Type "q" to exit the tool
+
+10. You will likely need a good editor while using Linux. While the editor vi is installed by default, I recommend you use the **nano** editor. It may be already installed but if not run the following command to install it
+
+ `sudo yum install -y nano`
+
+ Let's use nano to create a shell script to run on Linux
+
+ `nano dumperrorlog.sh`
+
+ nano is a full screen editor. Type in the following in the editor window
+
+ `sudo cat /var/opt/mssql/log/errorlog`
+
+ Type Ctrl+X to exit. You will get prompted to save the file
+
+ Run the following command to make the script executable
+
+ `chmod u+x dumperrorlog.sh`
+
+ Now execute the script
+
+ `./dumperrorlog.sh`
+
+## How to connect to SQL Server
+
+Here are a few examples of how to do basic connectivity to SQL Server on Linux. These exercises assume you have installed the command line tools for Linux, mssql-cli, and have a machine or VM running Windows that can connect to the Linux server or VM.
+
+1. Run a quick test to connect with sqlcmd by executing the following
+
+ `sqlcmd -Usa -Slocalhost`
+
+ Put in your sa password. At the sqlcmd prompt, run the following T-SQL statement
+
+ ```sql
+ SELECT @@VERSION
+ GO
+ ```
+ Type in "exit" to quit sqlcmd
+
+ Note: You can connect with any sqlcmd tool from Windows, Linux, or macOS in the same was provided you have connectivity to your Linux Server or VM. The -S parameter would be the hostname or IP address of the Linux Server or VM.
+
+2. Now test mssql-cli like we did for sqlcmd by running the following command
+
+ `mssql-cli -Usa -Slocalhost`
+
+ You should get a new prompt like sqlcmd. At this prompt type in the following T-SQL command and hit Enter
+
+ ```sql
+ SELECT @@VERSION
+ ```
+
+ Notice as you started typing you see Intellisense functionality kick-in which is one of the differences from sqlcmd.
+
+ If you are not put back into the mssql-cli prompt, type "q" to get back to the prompt.
+
+ mssql-cli does not recognize the "GO" keyword as sqlcmd does. Use a ";" to separate batches. You can also hit F3 to type statements in multiple lines but they will all be in one batch.
+
+ Type in "exit" to quit mssql-cli
+
+ Note: You can connect with any mssql-cli tool from Windows, Linux, or macOS in the same was provided you have connectivity to your Linux Server or VM. The -S parameter would be the hostname or IP address of the Linux Server or VM.
+
+3. Connect with SQL Server Management Studio (SSMS) using SQL Authentication with the sa account and the server name or IP address:port for your Linux Server. Notice how SSMS works "as is" against the Linux Server and looks almost like a SQL Server on Windows deployment.
+
+ Use Object Explorer and the Query Editor just like you would a normal SQL Server instance. Go through some of the steps in the SSMS tutorial in our documentation at
+
+4. Go through the the quickstart tutorial for connecting to SQL Server from Azure Data Studio with the SQL Server on Linux deployment at https://docs.microsoft.com/en-us/sql/azure-data-studio/quickstart-sql-server?view=sql-server-2017.
+
+## How to restore a backup and run queries
+
+In this exercise, you will learn how to restore a backup of a database to SQL Server on Linux, and run queries against the database.
+
+Now you will learn the great compatibility story of SQL Server on Linux by restoring a backup from SQL Server on Windows to SQL Server on Linux. And you will interact with this database using sqlcmd and mssql-cli. This section of the lab assumes your Linux Server is connected to the internet. If you are not connected to the internet, you can download the database to restore from and then copy it to your Linux Server (MobaXterm drag and drop is really nice for this)
+
+1. From your Linux ssh session, run the following command from the bash shell
+
+ `wget https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak`
+
+ Depending on your network speed this should take no more than a few minutes
+
+2. Copy and restore the WideWorldImporters database. Copy the **cpwwi.sh**, **restorewwi.sh**, and **restorewwi_linux.sql** files from the downloaded zip of the gitHub repo into your home directory on Linux. MobaXterm provides drag and drop capabilities to do this. Copy these files and drop them into the "explorer" pane in MobaXterm on the left hand side from your ssh session.
+
+ Note: You can skip this step if you have already cloned the git repo in the prelab. If you have done this, the scripts in this lab are in the **sqllinuxlab** subdirectory. You can copy them into your home directory or edit them to ensure you have the right path for the WideWorldImporters backup file.
+
+3. Run the following commands from the bash shell to make the scripts executable (supply the root password if prompted)
+
+ `sudo chmod u+x cpwwi.sh`
+
+ `sudo chmod u+x restorewwi.sh`
+
+4. Copy the backup file to the SQL Server directory so it can access the file and change permissions on the backup file by executing the following command in the bash shell
+
+ `./cpwwi.sh`
+
+5. Now restore the database by executing the following command from the bash shell
+
+ `./restorewwi.sh`
+
+6. Connect with sa to run a query against this database. Run sqlcmd first to connect. Type in the sa password when prompted
+
+ `sqlcmd -Usa -Slocalhost`
+
+7. From the sqlcmd prompt run these commands
+
+ ```sql
+ USE WideWorldImporters
+ GO
+ SELECT * FROM [Sales].[Customers]
+ GO
+ ```
+
+ Type in "exit" to quit sqlcmd
+
+9. Now run the same set of commands using mssql-cli. Connect to SQL Server with mssql-cli. Type in the sa password when prompted
+
+ `mssql-cli -Usa -Slocalhost`
+
+10. Run the following T-SQL commands from the msql-cli prompt (BONUS: Use Intellisense to complete these queries)
+
+ `USE WideWorldImporters;SELECT * FROM Sales.Customers;`
+
+ See how mssql-cli by default will present rows in a vertical record format. Hit Enter or Space to keep paging as many rows as you like.
+
+ Type in "q" at any time to get back to the prompt and "exit" to quit mssql-cli
+
+## How to configure SQL Server
+
+In this exercise, you will learn how to configure SQL Server on Linux with the mssql-conf tool.
+
+There may be situations where you need to enable a traceflag as global and at SQL Server startup time. For Windows, this is done through the SQL Server Configuration Manager. For SQL Server on Linux, you will use the mssql-conf script. A list of all documented traceflags can be found at .
+
+Let's say you wanted to enable trace flag 1222 for deadlock details to be reported in the ERRORLOG.
+
+1. Run the following command from an ssh session with the bash shell
+
+ `sudo /opt/mssql/bin/mssql-conf traceflag 1222 on`
+
+2. Per these instructions, restart SQL Server with the following command:
+
+ `sudo systemctl restart mssql-server`
+
+ Note: If this is successful, the command just returns to the shell prompt
+
+3. Verify the trace flag was properly set by looking at the ERRORLOG with the following command
+
+ `sudo more /var/opt/mssql/log/errorlog`
+
+4. Use sqlcmd or mssql-cli to verify this trace flag is set by running the following T-SQL statement
+
+ ```sql
+ DBCC TRACESTATUS(-1)
```
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi.sh
index 34e4328c..bcbc1b7b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi.sh
@@ -1 +1 @@
-/opt/mssql-tools/bin/sqlcmd -Usa -irestorewwi_linux.sql
+/opt/mssql-tools/bin/sqlcmd -Usa -irestorewwi_linux.sql
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi_linux.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi_linux.sql
index d22e5251..14b02fc8 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi_linux.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/explore/restorewwi_linux.sql
@@ -1,6 +1,6 @@
-restore database WideWorldImporters from disk = '/var/opt/mssql/WideWorldImporters-Full.bak' with
-move 'WWI_Primary' to '/var/opt/mssql/data/WideWorldImporters.mdf',
-move 'WWI_UserData' to '/var/opt/mssql/data/WideWorldImporters_UserData.ndf',
-move 'WWI_Log' to '/var/opt/mssql/data/WideWorldImporters.ldf',
-move 'WWI_InMemory_Data_1' to '/var/opt/mssql/data/WideWorldImporters_InMemory_Data_1'
-go
+restore database WideWorldImporters from disk = '/var/opt/mssql/WideWorldImporters-Full.bak' with
+move 'WWI_Primary' to '/var/opt/mssql/data/WideWorldImporters.mdf',
+move 'WWI_UserData' to '/var/opt/mssql/data/WideWorldImporters_UserData.ndf',
+move 'WWI_Log' to '/var/opt/mssql/data/WideWorldImporters.ldf',
+move 'WWI_InMemory_Data_1' to '/var/opt/mssql/data/WideWorldImporters_InMemory_Data_1'
+go
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/readme.md
index 97e3c80c..ed5eddd9 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 6 Activity - SQL Server on Linux/readme.md
@@ -1,11 +1,11 @@
-# Module 6 Activities - SQL Server on Linux
-
-These represent demos and examples you can run to deploy and explore and use SQL Server on Linux
-
-## deploy
-
-Learn the basics of how to deploy SQL Server on Linux.
-
-## explore
-
+# Module 6 Activities - SQL Server on Linux
+
+These represent demos and examples you can run to deploy and explore and use SQL Server on Linux
+
+## deploy
+
+Learn the basics of how to deploy SQL Server on Linux.
+
+## explore
+
Learn how to explore SQL Server on Linux including common Linux commands, connecting to SQL Server, restoring a backup, querying, and configuring SQL Server.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/readme.md
index 9fdb6161..81529299 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/readme.md
@@ -1,11 +1,11 @@
-# Module 7 Activities - SQL Server Containers
-
-These represent demos and examples to show you the basics of SQL Server containers and how to update and upgrade SQL Server using containers.
-
-## sqlcontainers
-
-Learn the basics of SQL Server with containers
-
-## sqlcontainerupdate
-
+# Module 7 Activities - SQL Server Containers
+
+These represent demos and examples to show you the basics of SQL Server containers and how to update and upgrade SQL Server using containers.
+
+## sqlcontainers
+
+Learn the basics of SQL Server with containers
+
+## sqlcontainerupdate
+
Learn more about storage volumes and updating and upgrading SQL Server using containers.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/cleanup.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/cleanup.sh
index 864a7ad2..400cc928 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/cleanup.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/cleanup.sh
@@ -1,5 +1,5 @@
-sudo docker stop sql2017cu10
-sudo docker stop sql2
-sudo docker rm sql2017cu10
-sudo docker rm sql2
-sudo docker volume rm sqlvolume sqlvolume2
+sudo docker stop sql2017cu10
+sudo docker stop sql2
+sudo docker rm sql2017cu10
+sudo docker rm sql2
+sudo docker volume rm sqlvolume sqlvolume2
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/readme.md
index 917f1563..7b210f82 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/readme.md
@@ -1,56 +1,56 @@
-## SQL Server Containers Fundamentals
-
-In this exercise you will be exploring the fundamentals for SQL Server containers.
-
-## Requirements
-
-- These exercises were built to run using Docker for Linux on RedHat Enterprise Linux. However, you can review all the details of the scripts provided and use them with Docker for Windows or Docker for macOS.
-- If using a RedHat Linux Server or VM and Docker is not installed, you can use these steps to install the Docker Community Edition for centOS
-
- `sudo yum install -y yum-utils device-mapper-persistent-data lvm2`
-
- `sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo`
-
- `sudo yum install http://mirror.centos.org/centos/7/extras/x86_64/Packages/pigz-2.3.3-1.el7.centos.x86_64.rpm`
-
- `sudo yum install docker-ce`
-
- Then make sure the Docker engine is running with these commands
-
- `sudo systemctl status docker`
-
- `sudo systemctl start docker`
-
-- Make sure all scripts are executable by running the following command
-
- `chmod u+x *.sh`
-
-- Restore the WideWorldImporters backup by using the **pullwwi.sh** script. This requires internet connectivity. You can also manually copy the backup into the current directory where you run the exercises.
-
-## Learning the basics of SQL Server containers
-
-1. Run **step1_dockerruncu10.sh** to start a container with SQL Server 2017 CU8. This container is called sql2017cu10
-
-2. Run **step2_dockercopy.sh** to copy the WWI backyup into the container.
-
-3. Run **step3_docker_restorewwi.sh** to restore the backup. This uses docker exec to **run sqlcmd inside the container**. Since this takes a few minutes it will run in the background using the -d paramater for docker exec.
-
-4. Run **step4_dockerrun2.sh** to start another SQL container with the latest SQL 2017 update. This container is called sql2. Notice a different volume is used along with port 1402 instead of 1401.
-
-5. Run **step5_containers.sh** to see both containers running. You now have two SQL Servers running on the same Linux machine using containers.
-
-6. Run **step6_procs.sh** to see the process for the Linux host which include the docker daemon. Note the sqlservr processes as children underneath that process.
-
-7. Run **step7_namespaces.sh** to see the different namespaces for the SQL Server containers
-
-8. The restore should be finished from Step 3. Run **step8_dockerquery.sh** to run a query for the database by connecting **using sqlcmd outside of the container**.
-
-9. Use docker exec to interact with the sql2017cu10 container through a shell by executing **step9_dockerexec.sh**. Notice the shell has the hostname used to run the container at the prompt.
-
-- Run ps -axf to see the isolation of containers and that sqlservr is just about the only process running.
-
-- Go look at the ERRORLOG file in /var/opt/mssql/log.
-
-- Exit the shell inside the container by typing **exit**.
-
+## SQL Server Containers Fundamentals
+
+In this exercise you will be exploring the fundamentals for SQL Server containers.
+
+## Requirements
+
+- These exercises were built to run using Docker for Linux on RedHat Enterprise Linux. However, you can review all the details of the scripts provided and use them with Docker for Windows or Docker for macOS.
+- If using a RedHat Linux Server or VM and Docker is not installed, you can use these steps to install the Docker Community Edition for centOS
+
+ `sudo yum install -y yum-utils device-mapper-persistent-data lvm2`
+
+ `sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo`
+
+ `sudo yum install http://mirror.centos.org/centos/7/extras/x86_64/Packages/pigz-2.3.3-1.el7.centos.x86_64.rpm`
+
+ `sudo yum install docker-ce`
+
+ Then make sure the Docker engine is running with these commands
+
+ `sudo systemctl status docker`
+
+ `sudo systemctl start docker`
+
+- Make sure all scripts are executable by running the following command
+
+ `chmod u+x *.sh`
+
+- Restore the WideWorldImporters backup by using the **pullwwi.sh** script. This requires internet connectivity. You can also manually copy the backup into the current directory where you run the exercises.
+
+## Learning the basics of SQL Server containers
+
+1. Run **step1_dockerruncu10.sh** to start a container with SQL Server 2017 CU8. This container is called sql2017cu10
+
+2. Run **step2_dockercopy.sh** to copy the WWI backyup into the container.
+
+3. Run **step3_docker_restorewwi.sh** to restore the backup. This uses docker exec to **run sqlcmd inside the container**. Since this takes a few minutes it will run in the background using the -d paramater for docker exec.
+
+4. Run **step4_dockerrun2.sh** to start another SQL container with the latest SQL 2017 update. This container is called sql2. Notice a different volume is used along with port 1402 instead of 1401.
+
+5. Run **step5_containers.sh** to see both containers running. You now have two SQL Servers running on the same Linux machine using containers.
+
+6. Run **step6_procs.sh** to see the process for the Linux host which include the docker daemon. Note the sqlservr processes as children underneath that process.
+
+7. Run **step7_namespaces.sh** to see the different namespaces for the SQL Server containers
+
+8. The restore should be finished from Step 3. Run **step8_dockerquery.sh** to run a query for the database by connecting **using sqlcmd outside of the container**.
+
+9. Use docker exec to interact with the sql2017cu10 container through a shell by executing **step9_dockerexec.sh**. Notice the shell has the hostname used to run the container at the prompt.
+
+- Run ps -axf to see the isolation of containers and that sqlservr is just about the only process running.
+
+- Go look at the ERRORLOG file in /var/opt/mssql/log.
+
+- Exit the shell inside the container by typing **exit**.
+
10. Leave all containers running as they will be used in the next set of exercises in the **sqlcontainerupdate** folder.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step1_dockerruncu10.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step1_dockerruncu10.sh
index 488d3761..c3bfd30f 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step1_dockerruncu10.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step1_dockerruncu10.sh
@@ -1,8 +1,8 @@
-sudo docker run -e\
- 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
- --hostname sql2017cu10\
- -p 1401:1433\
- -v sqlvolume:/var/opt/mssql\
- --name sql2017cu10\
- -d\
- mcr.microsoft.com/mssql/server:2017-CU10-ubuntu
+sudo docker run -e\
+ 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
+ --hostname sql2017cu10\
+ -p 1401:1433\
+ -v sqlvolume:/var/opt/mssql\
+ --name sql2017cu10\
+ -d\
+ mcr.microsoft.com/mssql/server:2017-CU10-ubuntu
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step2_dockercopy.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step2_dockercopy.sh
index 6eb2b881..4709a825 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step2_dockercopy.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step2_dockercopy.sh
@@ -1 +1 @@
-sudo docker cp WideWorldImporters-Full.bak sql2017cu10:/var/opt/mssql
+sudo docker cp WideWorldImporters-Full.bak sql2017cu10:/var/opt/mssql
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step3_docker_restorewwi.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step3_docker_restorewwi.sh
index 1e9aabd6..b63149a9 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step3_docker_restorewwi.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step3_docker_restorewwi.sh
@@ -1,2 +1,2 @@
-sudo docker exec -d sql2017cu10\
+sudo docker exec -d sql2017cu10\
/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P 'Sql2017isfast' -Q 'RESTORE DATABASE WideWorldImp$
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step4_dockerrun2.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step4_dockerrun2.sh
index d4c66502..f53e385e 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step4_dockerrun2.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step4_dockerrun2.sh
@@ -1,8 +1,8 @@
-sudo docker run\
- -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
- --hostname sql2\
- -p 1402:1433\
- -v sqlvolume2:/var/opt/mssql\
- --name sql2\
- -d\
+sudo docker run\
+ -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
+ --hostname sql2\
+ -p 1402:1433\
+ -v sqlvolume2:/var/opt/mssql\
+ --name sql2\
+ -d\
mcr.microsoft.com/mssql/server:2017-latest-ubuntu
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step6_procs.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step6_procs.sh
index 25a036e4..fa76445b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step6_procs.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step6_procs.sh
@@ -1 +1 @@
-sudo ps -axf
+sudo ps -axf
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step7_namespaces.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step7_namespaces.sh
index e4a0d05c..a8958882 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step7_namespaces.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step7_namespaces.sh
@@ -1 +1 @@
-sudo lsns
+sudo lsns
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step8_dockerquery.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step8_dockerquery.sh
index 8c5fd6e5..1c144842 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step8_dockerquery.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step8_dockerquery.sh
@@ -1 +1 @@
-sqlcmd -Usa -Slocalhost,1401 -Q'USE WideWorldImporters;SELECT * FROM [Application].[People];' -PSql2017isfast
+sqlcmd -Usa -Slocalhost,1401 -Q'USE WideWorldImporters;SELECT * FROM [Application].[People];' -PSql2017isfast
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step9_dockerexec.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step9_dockerexec.sh
index 3382533e..f73af388 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step9_dockerexec.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainers/step9_dockerexec.sh
@@ -1 +1 @@
-sudo docker exec -it sql2017cu10 bash
+sudo docker exec -it sql2017cu10 bash
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/cleanup.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/cleanup.sh
index ae112902..f4706544 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/cleanup.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/cleanup.sh
@@ -1,6 +1,6 @@
-sudo docker stop sql2019
-sudo docker rm sql2019
-sudo docker stop sql2017latest
-sudo docker rm sql2017latest
-
-
+sudo docker stop sql2019
+sudo docker rm sql2019
+sudo docker stop sql2017latest
+sudo docker rm sql2017latest
+
+
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/execintodocker.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/execintodocker.sh
index 077fae17..5744dd80 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/execintodocker.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/execintodocker.sh
@@ -1,2 +1,2 @@
-
-sudo docker exec -it sql2017latest bash
+
+sudo docker exec -it sql2017latest bash
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/readme.md
index b298adf6..f6388452 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/readme.md
@@ -1,26 +1,26 @@
-# SQL Containers exercises showing volume storage and update/upgrade of containers
-
-This is a set of exercies to learn more about storage volumes for containers and see how to update, rollback, and upgrade SQL Server using containers.
-
-## Requirements
-
-- Complete the exercises in the sqlcontainers folder in this Module.
-- Make sure all scripts are executable by running the following command
-
- `chmod u+x *.sh`
-
-## Looking at volume storage, updating, rolling back, and upgrading SQL containers
-
-1. Let's update the sql2017cu10 container with the latest CU by running **step1_dockerupdate.sh**. This has to run a few upgrade scripts so takes a few minutes. While this is running, let's look at volume storage.
-
-2. See details of the volumes used by the containers from the sqlcontainers exercise volumes by running **step2_inspectvols.sh**.
-
-3. See what files are stored in the host folders used to provide volume storage by running **step3_volstorage.sh**.
-
-4. Let's see if the container is updated by running **step4_dockerquery.sh**. If the query cannot run because script upgrades are still running use **execintodocker.sh** to see the status in the ERRORLOG file of the container.
-
-5. If time permits, you can execute **step5_dockerrollback.sh** to go back to the SQL 2017 CU10 build of that container.
-
-6. Upgrade the container to SQL Server 2019 preview by execute **step6_dockerupgrade.sh**.
-
+# SQL Containers exercises showing volume storage and update/upgrade of containers
+
+This is a set of exercies to learn more about storage volumes for containers and see how to update, rollback, and upgrade SQL Server using containers.
+
+## Requirements
+
+- Complete the exercises in the sqlcontainers folder in this Module.
+- Make sure all scripts are executable by running the following command
+
+ `chmod u+x *.sh`
+
+## Looking at volume storage, updating, rolling back, and upgrading SQL containers
+
+1. Let's update the sql2017cu10 container with the latest CU by running **step1_dockerupdate.sh**. This has to run a few upgrade scripts so takes a few minutes. While this is running, let's look at volume storage.
+
+2. See details of the volumes used by the containers from the sqlcontainers exercise volumes by running **step2_inspectvols.sh**.
+
+3. See what files are stored in the host folders used to provide volume storage by running **step3_volstorage.sh**.
+
+4. Let's see if the container is updated by running **step4_dockerquery.sh**. If the query cannot run because script upgrades are still running use **execintodocker.sh** to see the status in the ERRORLOG file of the container.
+
+5. If time permits, you can execute **step5_dockerrollback.sh** to go back to the SQL 2017 CU10 build of that container.
+
+6. Upgrade the container to SQL Server 2019 preview by execute **step6_dockerupgrade.sh**.
+
7. Run **cleanup.sh** to stop and remove containers from this exercise. There is a cleanup.sh script in the **sqlcontainers** folder to cleanup containers created for that exercise.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step1_dockerupdate.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step1_dockerupdate.sh
index 0e0b027b..dce519dd 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step1_dockerupdate.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step1_dockerupdate.sh
@@ -1,10 +1,10 @@
-sudo docker stop sql2017cu10
-sudo docker run\
- -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
- -p 1401:1433\
- -v sqlvolume:/var/opt/mssql\
- --hostname sql2017latest\
- --name\
- sql2017latest\
- -d\
+sudo docker stop sql2017cu10
+sudo docker run\
+ -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
+ -p 1401:1433\
+ -v sqlvolume:/var/opt/mssql\
+ --hostname sql2017latest\
+ --name\
+ sql2017latest\
+ -d\
mcr.microsoft.com/mssql/server:2017-latest
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step3_volstorage.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step3_volstorage.sh
index 6624b7c1..6e656cd1 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step3_volstorage.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step3_volstorage.sh
@@ -1,2 +1,2 @@
-sudo ls /var/lib/docker/volumes/sqlvolume/_data
+sudo ls /var/lib/docker/volumes/sqlvolume/_data
sudo ls /var/lib/docker/volumes/sqlvolume2/_data
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step4_dockerquery.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step4_dockerquery.sh
index 8c5fd6e5..1c144842 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step4_dockerquery.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step4_dockerquery.sh
@@ -1 +1 @@
-sqlcmd -Usa -Slocalhost,1401 -Q'USE WideWorldImporters;SELECT * FROM [Application].[People];' -PSql2017isfast
+sqlcmd -Usa -Slocalhost,1401 -Q'USE WideWorldImporters;SELECT * FROM [Application].[People];' -PSql2017isfast
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step5_dockerrollback.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step5_dockerrollback.sh
index 4c7fc0f3..9eadab05 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step5_dockerrollback.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step5_dockerrollback.sh
@@ -1,2 +1,2 @@
-sudo docker stop sql2017latest
-sudo docker start sql2017cu10
+sudo docker stop sql2017latest
+sudo docker start sql2017cu10
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step6_dockerupgrade.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step6_dockerupgrade.sh
index 6cb27c21..e3e952ef 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step6_dockerupgrade.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 7 Activity - SQL Server Containers/sqlcontainerupdate/step6_dockerupgrade.sh
@@ -1,9 +1,9 @@
-sudo docker stop sql2017cu10
-sudo docker stop sql2017latest
-sudo docker run\
- -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
- -p 1401:1433\
- -v sqlvolume:/var/opt/mssql\
- --name sql2019\
- -d\
+sudo docker stop sql2017cu10
+sudo docker stop sql2017latest
+sudo docker run\
+ -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Sql2017isfast'\
+ -p 1401:1433\
+ -v sqlvolume:/var/opt/mssql\
+ --name sql2019\
+ -d\
mcr.microsoft.com/mssql/rhel/server:2019-CTP2.2
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/hdfs_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/hdfs_external_table.sql
index e6d910a6..330dbfed 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/hdfs_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/hdfs_external_table.sql
@@ -1,97 +1,97 @@
-USE [master]
-GO
--- Enabled PB connectivity to a Hadoop HDFS source which in this case is just Azure Blob Storage
---
-sp_configure @configname = 'hadoop connectivity', @configvalue = 7;
-GO
-RECONFIGURE
-GO
-
--- Enable PB export to be able to ingest data into the HDFS target
---
-sp_configure 'allow polybase export', 1
-GO
-RECONFIGURE
-GO
-
--- STOP: SQL Server must be restarted for this to take effect
---
-USE [WideWorldImporters]
-GO
--- Only run this if you have not already created a master key in the db
---
---CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
---GO
--- IDENTITY: any string (this is not used for authentication to Azure storage).
--- SECRET: your Azure storage account key.
-DROP DATABASE SCOPED CREDENTIAL AzureStorageCredential
-GO
-CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
-WITH IDENTITY = 'user', Secret = 'C5aFpK587sIDFIMSEqXwA08xlhDM34/rfOz2g+sVq/hcKReo6agvT9JZcWGe9NtEyHEypK095WZtDdE/gkKZNQ=='
-GO
--- LOCATION: Azure account storage account name and blob container name.
--- CREDENTIAL: The database scoped credential created above.
-DROP EXTERNAL DATA SOURCE bwdatalake
-GO
-CREATE EXTERNAL DATA SOURCE bwdatalake with (
- TYPE = HADOOP,
- LOCATION ='wasbs://wwi@bwdatalake.blob.core.windows.net',
- CREDENTIAL = AzureStorageCredential
-)
-GO
--- FORMAT TYPE: Type of format in Hadoop (DELIMITEDTEXT, RCFILE, ORC, PARQUET).
-CREATE EXTERNAL FILE FORMAT TextFileFormat WITH (
- FORMAT_TYPE = DELIMITEDTEXT,
- FORMAT_OPTIONS (FIELD_TERMINATOR ='|',
- USE_TYPE_DEFAULT = TRUE))
-GO
--- Create a schema called hdfs
---
-DROP SCHEMA hdfs
-GO
-CREATE SCHEMA hdfs
-GO
--- LOCATION: path to file or directory that contains the data (relative to HDFS root).
-DROP EXTERNAL TABLE [hdfs].[WWI_Order_Reviews]
-GO
-CREATE EXTERNAL TABLE [hdfs].[WWI_Order_Reviews] (
- [OrderID] int NOT NULL,
- [CustomerID] int NOT NULL,
- [Rating] int NULL,
- [Review_Comments] nvarchar(1000) NOT NULL
-)
-WITH (LOCATION='/WWI/',
- DATA_SOURCE = bwdatalake,
- FILE_FORMAT = TextFileFormat
-)
-GO
-
--- Ingest some data
---
-INSERT INTO [hdfs].[WWI_Order_Reviews] VALUES (1, 832, 10, 'I had a great experience with my order')
-GO
-CREATE STATISTICS StatsforReviews on [hdfs].[WWI_Order_Reviews](OrderID, CustomerID)
-GO
-
--- Now query the external table
---
-SELECT * FROM [hdfs].[WWI_Order_Reviews]
-GO
-
--- Let's do a filter to enable pushdown
---
-SELECT * FROM [hdfs].[WWI_Order_Reviews]
-WHERE OrderID = 1
-GO
-
--- Let's join the review with our order and customer data
---
-SELECT o.OrderDate, c.CustomerName, p.FullName as SalesPerson, wor.Rating, wor.Review_Comments
-FROM [Sales].[Orders] o
-JOIN [hdfs].[WWI_Order_Reviews] wor
-ON o.OrderID = wor.OrderID
-JOIN [Application].[People] p
-ON p.PersonID = o.SalespersonPersonID
-JOIN [Sales].[Customers] c
-ON c.CustomerID = wor.CustomerID
+USE [master]
+GO
+-- Enabled PB connectivity to a Hadoop HDFS source which in this case is just Azure Blob Storage
+--
+sp_configure @configname = 'hadoop connectivity', @configvalue = 7;
+GO
+RECONFIGURE
+GO
+
+-- Enable PB export to be able to ingest data into the HDFS target
+--
+sp_configure 'allow polybase export', 1
+GO
+RECONFIGURE
+GO
+
+-- STOP: SQL Server must be restarted for this to take effect
+--
+USE [WideWorldImporters]
+GO
+-- Only run this if you have not already created a master key in the db
+--
+--CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+--GO
+-- IDENTITY: any string (this is not used for authentication to Azure storage).
+-- SECRET: your Azure storage account key.
+DROP DATABASE SCOPED CREDENTIAL AzureStorageCredential
+GO
+CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
+WITH IDENTITY = 'user', Secret = 'C5aFpK587sIDFIMSEqXwA08xlhDM34/rfOz2g+sVq/hcKReo6agvT9JZcWGe9NtEyHEypK095WZtDdE/gkKZNQ=='
+GO
+-- LOCATION: Azure account storage account name and blob container name.
+-- CREDENTIAL: The database scoped credential created above.
+DROP EXTERNAL DATA SOURCE bwdatalake
+GO
+CREATE EXTERNAL DATA SOURCE bwdatalake with (
+ TYPE = HADOOP,
+ LOCATION ='wasbs://wwi@bwdatalake.blob.core.windows.net',
+ CREDENTIAL = AzureStorageCredential
+)
+GO
+-- FORMAT TYPE: Type of format in Hadoop (DELIMITEDTEXT, RCFILE, ORC, PARQUET).
+CREATE EXTERNAL FILE FORMAT TextFileFormat WITH (
+ FORMAT_TYPE = DELIMITEDTEXT,
+ FORMAT_OPTIONS (FIELD_TERMINATOR ='|',
+ USE_TYPE_DEFAULT = TRUE))
+GO
+-- Create a schema called hdfs
+--
+DROP SCHEMA hdfs
+GO
+CREATE SCHEMA hdfs
+GO
+-- LOCATION: path to file or directory that contains the data (relative to HDFS root).
+DROP EXTERNAL TABLE [hdfs].[WWI_Order_Reviews]
+GO
+CREATE EXTERNAL TABLE [hdfs].[WWI_Order_Reviews] (
+ [OrderID] int NOT NULL,
+ [CustomerID] int NOT NULL,
+ [Rating] int NULL,
+ [Review_Comments] nvarchar(1000) NOT NULL
+)
+WITH (LOCATION='/WWI/',
+ DATA_SOURCE = bwdatalake,
+ FILE_FORMAT = TextFileFormat
+)
+GO
+
+-- Ingest some data
+--
+INSERT INTO [hdfs].[WWI_Order_Reviews] VALUES (1, 832, 10, 'I had a great experience with my order')
+GO
+CREATE STATISTICS StatsforReviews on [hdfs].[WWI_Order_Reviews](OrderID, CustomerID)
+GO
+
+-- Now query the external table
+--
+SELECT * FROM [hdfs].[WWI_Order_Reviews]
+GO
+
+-- Let's do a filter to enable pushdown
+--
+SELECT * FROM [hdfs].[WWI_Order_Reviews]
+WHERE OrderID = 1
+GO
+
+-- Let's join the review with our order and customer data
+--
+SELECT o.OrderDate, c.CustomerName, p.FullName as SalesPerson, wor.Rating, wor.Review_Comments
+FROM [Sales].[Orders] o
+JOIN [hdfs].[WWI_Order_Reviews] wor
+ON o.OrderID = wor.OrderID
+JOIN [Application].[People] p
+ON p.PersonID = o.SalespersonPersonID
+JOIN [Sales].[Customers] c
+ON c.CustomerID = wor.CustomerID
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/polybase_status.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/polybase_status.sql
index 94ca9eb7..c1c745c0 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/polybase_status.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/polybase_status.sql
@@ -1,12 +1,12 @@
--- List out the nodes in the scale out group
---
-SELECT * FROM sys.dm_exec_compute_nodes
-GO
--- Get more details about the status of the nodes
---
-SELECT * FROM sys.dm_exec_compute_node_status
-GO
--- List out detailed errors from the nodes
---
-SELECT * FROM sys.dm_exec_compute_node_errors
+-- List out the nodes in the scale out group
+--
+SELECT * FROM sys.dm_exec_compute_nodes
+GO
+-- Get more details about the status of the nodes
+--
+SELECT * FROM sys.dm_exec_compute_node_status
+GO
+-- List out detailed errors from the nodes
+--
+SELECT * FROM sys.dm_exec_compute_node_errors
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/readme.md
index 36fb1c13..454608ee 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/readme.md
@@ -1,51 +1,51 @@
-# SQL Server 2019 Polybase Fundamentals
-
-This folder contains demo scripts to show the basic funcionality of Polybase by examining the configuration of nodes through DMV, creating an external table over HDFS, and monitoring execution details through DMVs.
-
-## Requirements - Install and Configure Polybase
-
-These demos require that you install SQL Server 2019 on Windows Server and configure a head node and at least one compute node (i.e. a scale out group). This demo currently requires SQL Server 2019 CTP 2.3 or higher.
-
-I used the installation instructions in the documentation from:
-
-https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-installation?view=sql-server-ver15
-
-and this to setup the scale out group
-
-https://docs.microsoft.com/en-us/sql/relational-databases/polybase/configure-scale-out-groups-windows?view=sql-server-ver15
-
-For my demos, I used the following deployment to setup a head node and 2 compute nodes using Azure (Note: I used the same resource group for all of these servers so they were part of the same virtual network)
-
-1 Windows Server 2019 Server which I configured using Server Manager as a domain controller (with a domain name of bobsql.com)
-
-1 Windows Server 2019 Server (bwpolybase) which I joined to the bobsql.com domain. I installed SQL Server 2019 and chose the Polybase feature (including Java which required me to stop the install and install JRE 8 from the web). I chose the option for a Scale out group which required me to use the domain admin from bobsql.com during the install option for services.
-
-2 other Windows Server 2019 Servers (bwpolybase2 and bwpolybase3)) with the same process to join the domain bobsql.com and install SQL server 2019 with Polybase.
-
-I had to enable Polybase on all 3 SQL Servers per the documentation using sp_configure 'polybase enabled' 1 and this required a restart of SQL SErver. You must first do this step before setting up the scale out group.
-
-I also first ensured that the Windows Firewall was configured for SQL Server and Polybase to open up firewall ports. Rules are already installed you just have to make sure they are enabled
-
-- SQL Server PolyBase - Database Engine - (TCP-In)
-- SQL Server PolyBase - PolyBase Services - (TCP-In)
-- SQL Server PolyBase - SQL Browser - (UDP-In)
-
-I then used the sp_polybase_join_group procedure per the documentation on bwpolybase2 and bwpolybase3 to join the scale out group. This required restarting the Polybase services on each machine.
-
-## Demo Steps
-
-### Check the Polybase configuration
-
-1. Run the T-SQL commands in the script **polybase_status.sql** to see configuration of the scale out group and details of the head and compute nodes
-
-2. Use SSMS to browse tables in the DWConfiguration, DWDiagnostics, and DWQueue databases which are installed on all nodes.
-
-### Create an external table and track query and polybase execution
-
-1. Download and restore the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers
-
-2. For my demo, I simply setup an Azure storage container using the instructions as found at https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal (note I did not create a blob just the storage account and container which in my case I named **wwi**).
-
-3. Run all the T-SQL commands in **hdfs_external_table.sql**. You will need to edit the appropriate details to point to your Azure storage container including the credential and location for the data source
-
+# SQL Server 2019 Polybase Fundamentals
+
+This folder contains demo scripts to show the basic funcionality of Polybase by examining the configuration of nodes through DMV, creating an external table over HDFS, and monitoring execution details through DMVs.
+
+## Requirements - Install and Configure Polybase
+
+These demos require that you install SQL Server 2019 on Windows Server and configure a head node and at least one compute node (i.e. a scale out group). This demo currently requires SQL Server 2019 CTP 2.3 or higher.
+
+I used the installation instructions in the documentation from:
+
+https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-installation?view=sql-server-ver15
+
+and this to setup the scale out group
+
+https://docs.microsoft.com/en-us/sql/relational-databases/polybase/configure-scale-out-groups-windows?view=sql-server-ver15
+
+For my demos, I used the following deployment to setup a head node and 2 compute nodes using Azure (Note: I used the same resource group for all of these servers so they were part of the same virtual network)
+
+1 Windows Server 2019 Server which I configured using Server Manager as a domain controller (with a domain name of bobsql.com)
+
+1 Windows Server 2019 Server (bwpolybase) which I joined to the bobsql.com domain. I installed SQL Server 2019 and chose the Polybase feature (including Java which required me to stop the install and install JRE 8 from the web). I chose the option for a Scale out group which required me to use the domain admin from bobsql.com during the install option for services.
+
+2 other Windows Server 2019 Servers (bwpolybase2 and bwpolybase3)) with the same process to join the domain bobsql.com and install SQL server 2019 with Polybase.
+
+I had to enable Polybase on all 3 SQL Servers per the documentation using sp_configure 'polybase enabled' 1 and this required a restart of SQL SErver. You must first do this step before setting up the scale out group.
+
+I also first ensured that the Windows Firewall was configured for SQL Server and Polybase to open up firewall ports. Rules are already installed you just have to make sure they are enabled
+
+- SQL Server PolyBase - Database Engine - (TCP-In)
+- SQL Server PolyBase - PolyBase Services - (TCP-In)
+- SQL Server PolyBase - SQL Browser - (UDP-In)
+
+I then used the sp_polybase_join_group procedure per the documentation on bwpolybase2 and bwpolybase3 to join the scale out group. This required restarting the Polybase services on each machine.
+
+## Demo Steps
+
+### Check the Polybase configuration
+
+1. Run the T-SQL commands in the script **polybase_status.sql** to see configuration of the scale out group and details of the head and compute nodes
+
+2. Use SSMS to browse tables in the DWConfiguration, DWDiagnostics, and DWQueue databases which are installed on all nodes.
+
+### Create an external table and track query and polybase execution
+
+1. Download and restore the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers
+
+2. For my demo, I simply setup an Azure storage container using the instructions as found at https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal (note I did not create a blob just the storage account and container which in my case I named **wwi**).
+
+3. Run all the T-SQL commands in **hdfs_external_table.sql**. You will need to edit the appropriate details to point to your Azure storage container including the credential and location for the data source
+
4. The the T-SQL commands in **trace_pb_query_execution.sql** to trace the execution of a query in Polybase.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/trace_pb_query_execution.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/trace_pb_query_execution.sql
index c96739a9..2a59498b 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/trace_pb_query_execution.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/polybase/trace_pb_query_execution.sql
@@ -1,37 +1,37 @@
--- Find out queries against external tables
---
-SELECT er.execution_id, t.*, er.*
-FROM sys.dm_exec_distributed_requests er
-CROSS APPLY sys.dm_exec_sql_text(er.sql_handle) AS t
-ORDER BY end_time DESC
-go
-
--- Find your execution_id and use this for the next query
---
-SELECT execution_id, step_index, operation_type, distribution_type, location_type, status, total_elapsed_time, command
-FROM sys.dm_exec_distributed_request_steps
-WHERE execution_id = 'QID1285'
-GO
-
--- Get more details on each step
---
-SELECT execution_id, compute_node_id, spid, step_index, distribution_id, status, total_elapsed_time, row_count
-FROM sys.dm_exec_distributed_sql_requests
-WHERE execution_id = 'QID1285'
-GO
-
--- Get more details from the compute nodes
---
-SELECT *
-FROM sys.dm_exec_dms_workers
-WHERE execution_id = 'QID1285'
-ORDER BY step_index, dms_step_index, distribution_id
-go
-
--- Look more at external operations
---
-SELECT *
-FROM sys.dm_exec_external_work
-WHERE execution_id = 'QID1285'
-GO
-
+-- Find out queries against external tables
+--
+SELECT er.execution_id, t.*, er.*
+FROM sys.dm_exec_distributed_requests er
+CROSS APPLY sys.dm_exec_sql_text(er.sql_handle) AS t
+ORDER BY end_time DESC
+go
+
+-- Find your execution_id and use this for the next query
+--
+SELECT execution_id, step_index, operation_type, distribution_type, location_type, status, total_elapsed_time, command
+FROM sys.dm_exec_distributed_request_steps
+WHERE execution_id = 'QID1285'
+GO
+
+-- Get more details on each step
+--
+SELECT execution_id, compute_node_id, spid, step_index, distribution_id, status, total_elapsed_time, row_count
+FROM sys.dm_exec_distributed_sql_requests
+WHERE execution_id = 'QID1285'
+GO
+
+-- Get more details from the compute nodes
+--
+SELECT *
+FROM sys.dm_exec_dms_workers
+WHERE execution_id = 'QID1285'
+ORDER BY step_index, dms_step_index, distribution_id
+go
+
+-- Look more at external operations
+--
+SELECT *
+FROM sys.dm_exec_external_work
+WHERE execution_id = 'QID1285'
+GO
+
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/readme.md
index 193babcb..b15af8f0 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/readme.md
@@ -1,11 +1,11 @@
-# Module 8 Activities - Data Virtualization
-
-These represent demos and examples to show you the basics of SQL Server containers and how to update and upgrade SQL Server using containers.
-
-## polybase
-
-Learn the basics of Polybase with SQL Server 2019
-
-## sqldatahub
-
+# Module 8 Activities - Data Virtualization
+
+These represent demos and examples to show you the basics of SQL Server containers and how to update and upgrade SQL Server using containers.
+
+## polybase
+
+Learn the basics of Polybase with SQL Server 2019
+
+## sqldatahub
+
Learn how to connect to many different data sources with SQL Server 2019 to create your own data hub connecting to SQL Server 2008R2, Azure SQL Database, CosmosDB, Oracle, HDFS, and SAP HANA.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/azuredb_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/azuredb_external_table.sql
index c40707a6..48328ef4 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/azuredb_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/azuredb_external_table.sql
@@ -1,94 +1,94 @@
-USE [WideWorldImporters]
-GO
-CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
-GO
-/* specify credentials to external data source
-* IDENTITY: user name for external source.
-* SECRET: password for external source.
-*/
-DROP DATABASE SCOPED CREDENTIAL AzureSQLDatabaseCredentials
-GO
-CREATE DATABASE SCOPED CREDENTIAL AzureSQLDatabaseCredentials
-WITH IDENTITY = 'thewandog', Secret = '$cprsqlserver2019'
-GO
-/* LOCATION: Location string should be of format '://[:]'.
-* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
-* CREDENTIAL: the database scoped credential, created above.
-*/
-DROP EXTERNAL DATA SOURCE AzureSQLDatabase
-GO
-CREATE EXTERNAL DATA SOURCE AzureSQLDatabase
-WITH (
-LOCATION = 'sqlserver://bwazuredb.database.windows.net',
-PUSHDOWN = ON,
-CREDENTIAL = AzureSQLDatabaseCredentials
-)
-GO
-DROP SCHEMA azuresqldb
-go
-CREATE SCHEMA azuresqldb
-GO
--- WWI was created with Latin1_General_100_CI_AS collation so I need to make my columns that
--- if I want to support UNION.
---
-DROP EXTERNAL TABLE azuresqldb.ModernStockItems
-GO
-CREATE EXTERNAL TABLE azuresqldb.ModernStockItems
-(
- [StockItemID] [int] NOT NULL,
- [StockItemName] [nvarchar](100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [SupplierID] [int] NOT NULL,
- [ColorID] [int] NULL,
- [UnitPackageID] [int] NOT NULL,
- [OuterPackageID] [int] NOT NULL,
- [Brand] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
- [Size] [nvarchar](20) COLLATE Latin1_General_100_CI_AS NULL,
- [LeadTimeDays] [int] NOT NULL,
- [QuantityPerOuter] [int] NOT NULL,
- [IsChillerStock] [bit] NOT NULL,
- [Barcode] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
- [TaxRate] [decimal](18, 3) NOT NULL,
- [UnitPrice] [decimal](18, 2) NOT NULL,
- [RecommendedRetailPrice] [decimal](18, 2) NULL,
- [TypicalWeightPerUnit] [decimal](18, 3) NOT NULL,
- --[MarketingComments] [nvarchar](max) NULL,
- --[InternalComments] [nvarchar](max) NULL,
- --[Photo] [varbinary](max) NULL,
- --[CustomFields] [nvarchar](max) NULL,
- --[Tags] AS (json_query([CustomFields],N'$.Tags')),
- --[SearchDetails] AS (concat([StockItemName],N' ',[MarketingComments])),
- [LastEditedBy] [int] NOT NULL
-)
- WITH (
- LOCATION='wwiazure.dbo.ModernStockItems',
- DATA_SOURCE=AzureSQLDatabase
-)
-GO
-CREATE STATISTICS ModernStockItemsStats ON azuresqldb.ModernStockItems ([StockItemID]) WITH FULLSCAN
-GO
--- Let's scan the table first to make sure it works
---
-SELECT * FROM azuresqldb.ModernStockItems
-GO
--- Now try to filter on just the stockitemid
---
-SELECT * FROM azuresqldb.ModernStockItems WHERE StockItemID = 100000
-GO
--- Find all stockitems from the Graphic Design Institute supplier
---
-SELECT msi.StockItemName, msi.Brand, c.ColorName
-FROM azuresqldb.ModernStockItems msi
-JOIN [Purchasing].[Suppliers] s
-ON msi.SupplierID = s.SupplierID
-and s.SupplierName = 'Graphic Design Institute'
-JOIN [Warehouse].[Colors] c
-ON msi.ColorID = c.ColorID
-UNION
-SELECT si.StockItemName, si.Brand, c.ColorName
-FROM [Warehouse].[StockItems] si
-JOIN [Purchasing].[Suppliers] s
-ON si.SupplierID = s.SupplierID
-and s.SupplierName = 'Graphic Design Institute'
-JOIN [Warehouse].[Colors] c
-ON si.ColorID = c.ColorID
+USE [WideWorldImporters]
+GO
+CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+GO
+/* specify credentials to external data source
+* IDENTITY: user name for external source.
+* SECRET: password for external source.
+*/
+DROP DATABASE SCOPED CREDENTIAL AzureSQLDatabaseCredentials
+GO
+CREATE DATABASE SCOPED CREDENTIAL AzureSQLDatabaseCredentials
+WITH IDENTITY = 'thewandog', Secret = '$cprsqlserver2019'
+GO
+/* LOCATION: Location string should be of format '://[:]'.
+* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
+* CREDENTIAL: the database scoped credential, created above.
+*/
+DROP EXTERNAL DATA SOURCE AzureSQLDatabase
+GO
+CREATE EXTERNAL DATA SOURCE AzureSQLDatabase
+WITH (
+LOCATION = 'sqlserver://bwazuredb.database.windows.net',
+PUSHDOWN = ON,
+CREDENTIAL = AzureSQLDatabaseCredentials
+)
+GO
+DROP SCHEMA azuresqldb
+go
+CREATE SCHEMA azuresqldb
+GO
+-- WWI was created with Latin1_General_100_CI_AS collation so I need to make my columns that
+-- if I want to support UNION.
+--
+DROP EXTERNAL TABLE azuresqldb.ModernStockItems
+GO
+CREATE EXTERNAL TABLE azuresqldb.ModernStockItems
+(
+ [StockItemID] [int] NOT NULL,
+ [StockItemName] [nvarchar](100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [SupplierID] [int] NOT NULL,
+ [ColorID] [int] NULL,
+ [UnitPackageID] [int] NOT NULL,
+ [OuterPackageID] [int] NOT NULL,
+ [Brand] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
+ [Size] [nvarchar](20) COLLATE Latin1_General_100_CI_AS NULL,
+ [LeadTimeDays] [int] NOT NULL,
+ [QuantityPerOuter] [int] NOT NULL,
+ [IsChillerStock] [bit] NOT NULL,
+ [Barcode] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [UnitPrice] [decimal](18, 2) NOT NULL,
+ [RecommendedRetailPrice] [decimal](18, 2) NULL,
+ [TypicalWeightPerUnit] [decimal](18, 3) NOT NULL,
+ --[MarketingComments] [nvarchar](max) NULL,
+ --[InternalComments] [nvarchar](max) NULL,
+ --[Photo] [varbinary](max) NULL,
+ --[CustomFields] [nvarchar](max) NULL,
+ --[Tags] AS (json_query([CustomFields],N'$.Tags')),
+ --[SearchDetails] AS (concat([StockItemName],N' ',[MarketingComments])),
+ [LastEditedBy] [int] NOT NULL
+)
+ WITH (
+ LOCATION='wwiazure.dbo.ModernStockItems',
+ DATA_SOURCE=AzureSQLDatabase
+)
+GO
+CREATE STATISTICS ModernStockItemsStats ON azuresqldb.ModernStockItems ([StockItemID]) WITH FULLSCAN
+GO
+-- Let's scan the table first to make sure it works
+--
+SELECT * FROM azuresqldb.ModernStockItems
+GO
+-- Now try to filter on just the stockitemid
+--
+SELECT * FROM azuresqldb.ModernStockItems WHERE StockItemID = 100000
+GO
+-- Find all stockitems from the Graphic Design Institute supplier
+--
+SELECT msi.StockItemName, msi.Brand, c.ColorName
+FROM azuresqldb.ModernStockItems msi
+JOIN [Purchasing].[Suppliers] s
+ON msi.SupplierID = s.SupplierID
+and s.SupplierName = 'Graphic Design Institute'
+JOIN [Warehouse].[Colors] c
+ON msi.ColorID = c.ColorID
+UNION
+SELECT si.StockItemName, si.Brand, c.ColorName
+FROM [Warehouse].[StockItems] si
+JOIN [Purchasing].[Suppliers] s
+ON si.SupplierID = s.SupplierID
+and s.SupplierName = 'Graphic Design Institute'
+JOIN [Warehouse].[Colors] c
+ON si.ColorID = c.ColorID
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/createazuredbtable.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/createazuredbtable.sql
index d5521292..eaf7ff04 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/createazuredbtable.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/createazuredbtable.sql
@@ -1,61 +1,61 @@
--- Database created in Azure is called wwiazure
--- This is not managed instance so you can't execute a USE database
--- Create a new database called wwiazure (server tier doesn't matter for this demo)
---
--- This table is supposed to mimic the [Warehouse].[StockItems] table in the WWI database
--- in SQL Server. I need to use Latin1_General_100_CI_AS collation for the columns because that
--- is how WWI was created so if I want to UNION data together with WWI I must use that collation
-DROP TABLE IF EXISTS [ModernStockItems]
-GO
-CREATE TABLE [ModernStockItems](
- [StockItemID] [int] NOT NULL,
- [StockItemName] [nvarchar](100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [SupplierID] [int] NOT NULL,
- [ColorID] [int] NULL,
- [UnitPackageID] [int] NOT NULL,
- [OuterPackageID] [int] NOT NULL,
- [Brand] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
- [Size] [nvarchar](20) COLLATE Latin1_General_100_CI_AS NULL,
- [LeadTimeDays] [int] NOT NULL,
- [QuantityPerOuter] [int] NOT NULL,
- [IsChillerStock] [bit] NOT NULL,
- [Barcode] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
- [TaxRate] [decimal](18, 3) NOT NULL,
- [UnitPrice] [decimal](18, 2) NOT NULL,
- [RecommendedRetailPrice] [decimal](18, 2) NULL,
- [TypicalWeightPerUnit] [decimal](18, 3) NOT NULL,
- --[MarketingComments] [nvarchar](max) NULL, -- Not allowed for an external table
- --[InternalComments] [nvarchar](max) NULL, -- Not allowed for an external table
- --[Photo] [varbinary](max) NULL, -- Not allowed for an external table
- --[CustomFields] [nvarchar](max) NULL, -- Not allowed for an external table
- --[Tags] AS (json_query([CustomFields],N'$.Tags')), -- Not allowed for an external table
- --[SearchDetails] AS (concat([StockItemName],N' ',[MarketingComments])), -- Not allowed for an external table
- [LastEditedBy] [int] NOT NULL,
-CONSTRAINT [PK_Warehouse_StockItems] PRIMARY KEY CLUSTERED
-(
- [StockItemID] ASC
-)
-)
-GO
--- Now insert some data. We don't coordinate with unique keys in WWI on SQL Server
--- so pick numbers way larger than exist in the current StockItems in WWI which is only 227
-INSERT INTO ModernStockItems VALUES
-(100000,
-'Dallas Cowboys Jersey',
-5,
-4, -- Blue
-4, -- Box
-4, -- Bob
-'Under Armour',
-'L',
-30,
-1,
-0,
-'123456789',
-2.0,
-50,
-75,
-2.0,
-1
-)
-GO
+-- Database created in Azure is called wwiazure
+-- This is not managed instance so you can't execute a USE database
+-- Create a new database called wwiazure (server tier doesn't matter for this demo)
+--
+-- This table is supposed to mimic the [Warehouse].[StockItems] table in the WWI database
+-- in SQL Server. I need to use Latin1_General_100_CI_AS collation for the columns because that
+-- is how WWI was created so if I want to UNION data together with WWI I must use that collation
+DROP TABLE IF EXISTS [ModernStockItems]
+GO
+CREATE TABLE [ModernStockItems](
+ [StockItemID] [int] NOT NULL,
+ [StockItemName] [nvarchar](100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [SupplierID] [int] NOT NULL,
+ [ColorID] [int] NULL,
+ [UnitPackageID] [int] NOT NULL,
+ [OuterPackageID] [int] NOT NULL,
+ [Brand] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
+ [Size] [nvarchar](20) COLLATE Latin1_General_100_CI_AS NULL,
+ [LeadTimeDays] [int] NOT NULL,
+ [QuantityPerOuter] [int] NOT NULL,
+ [IsChillerStock] [bit] NOT NULL,
+ [Barcode] [nvarchar](50) COLLATE Latin1_General_100_CI_AS NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [UnitPrice] [decimal](18, 2) NOT NULL,
+ [RecommendedRetailPrice] [decimal](18, 2) NULL,
+ [TypicalWeightPerUnit] [decimal](18, 3) NOT NULL,
+ --[MarketingComments] [nvarchar](max) NULL, -- Not allowed for an external table
+ --[InternalComments] [nvarchar](max) NULL, -- Not allowed for an external table
+ --[Photo] [varbinary](max) NULL, -- Not allowed for an external table
+ --[CustomFields] [nvarchar](max) NULL, -- Not allowed for an external table
+ --[Tags] AS (json_query([CustomFields],N'$.Tags')), -- Not allowed for an external table
+ --[SearchDetails] AS (concat([StockItemName],N' ',[MarketingComments])), -- Not allowed for an external table
+ [LastEditedBy] [int] NOT NULL,
+CONSTRAINT [PK_Warehouse_StockItems] PRIMARY KEY CLUSTERED
+(
+ [StockItemID] ASC
+)
+)
+GO
+-- Now insert some data. We don't coordinate with unique keys in WWI on SQL Server
+-- so pick numbers way larger than exist in the current StockItems in WWI which is only 227
+INSERT INTO ModernStockItems VALUES
+(100000,
+'Dallas Cowboys Jersey',
+5,
+4, -- Blue
+4, -- Box
+4, -- Bob
+'Under Armour',
+'L',
+30,
+1,
+0,
+'123456789',
+2.0,
+50,
+75,
+2.0,
+1
+)
+GO
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/readme.md
index 49e87cf5..3d3c4d01 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/azuredb/readme.md
@@ -1,13 +1,13 @@
-# SQL Server 2019 Polybase example connecting to Azure SQL Database
-
-This demo shows you how to setup an Azure SQL Database external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
-
-## Requirements
-
-1. Create a new database in Azure. For purposes of this demo it doesn't matter whether the database is a Managed Instance or any tier of Azure DB. For purposes of this demo I called my database **wwiazure**. To make connectivity easier, I created a new virtual network for my Azure SQL Server and included the polybase head node server, bwpolybase, in the same virtual network as Azure SQL Server. You can read more about how to do this at https://docs.microsoft.com/en-us/azure/sql-database/sql-database-vnet-service-endpoint-rule-overview
-
-2. Connecting to the azure SQL Server hosting your database, I ran the script **createazuredbtable.sql** to create the table and insert some data. Notice the COLLATE clauses I needed to use to match what WWI uses from the example database. The table created for this demo mimics the **Warehouse.StockItem** table in the WideWorldImporters database.
-
-## Demo Steps
-
+# SQL Server 2019 Polybase example connecting to Azure SQL Database
+
+This demo shows you how to setup an Azure SQL Database external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
+
+## Requirements
+
+1. Create a new database in Azure. For purposes of this demo it doesn't matter whether the database is a Managed Instance or any tier of Azure DB. For purposes of this demo I called my database **wwiazure**. To make connectivity easier, I created a new virtual network for my Azure SQL Server and included the polybase head node server, bwpolybase, in the same virtual network as Azure SQL Server. You can read more about how to do this at https://docs.microsoft.com/en-us/azure/sql-database/sql-database-vnet-service-endpoint-rule-overview
+
+2. Connecting to the azure SQL Server hosting your database, I ran the script **createazuredbtable.sql** to create the table and insert some data. Notice the COLLATE clauses I needed to use to match what WWI uses from the example database. The table created for this demo mimics the **Warehouse.StockItem** table in the WideWorldImporters database.
+
+## Demo Steps
+
1. On my SQL Server 2019 head node (bwpolybase), I used the **azuredb_external_table.sql** script to create the database scoped credential, externaL data source, external table, and sample SELECT statements to query the external table and join it with local SQL Server 2019 tables in the WideWorldImporters database. Take note of the COLLATE required to match WWI and the syntax for the external data source to point to the azure SQL Server.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/cosmosdb_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/cosmosdb_external_table.sql
index 51eac544..60c0a6be 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/cosmosdb_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/cosmosdb_external_table.sql
@@ -1,69 +1,69 @@
-USE [WideWorldImporters]
-GO
-CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
-GO
-/* specify credentials to external data source
-* IDENTITY: user name for external source.
-* SECRET: password for external source.
-*/
-DROP DATABASE SCOPED CREDENTIAL CosmosDBCredentials
-GO
--- You can get the IDENTITY (user) and secret (password) from the Connection String option in the
--- Azure portal
-CREATE DATABASE SCOPED CREDENTIAL CosmosDBCredentials
-WITH IDENTITY = 'wwi', Secret = 'hSoxMUeEgNjeeWh4FTz5jmGRlSN4Ko6HoYqiJsbleFzewe86EEXJrvwkAqBgitypJdjUbeJqnTVNBO6NUa0DZQ=='
-GO
-DROP EXTERNAL DATA SOURCE CosmosDB
-GO
--- The LOCATION is built from : from the Connection String in the Azure Portal
-CREATE EXTERNAL DATA SOURCE CosmosDB
-WITH (
-LOCATION = 'mongodb://wwi.documents.azure.com:10255',
-PUSHDOWN = ON,
-CREDENTIAL = CosmosDBCredentials
-)
-GO
-DROP SCHEMA cosmosdb
-go
-CREATE SCHEMA cosmosdb
-GO
-/* LOCATION: sql server table/view in 'database_name.schema_name.object_name' format
-* DATA_SOURCE: the external data source, created above.
-*/
-DROP EXTERNAL TABLE cosmosdb.Orders
-GO
-CREATE EXTERNAL TABLE cosmosdb.Orders
-(
- [_id] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [id] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [OrderID] int NOT NULL,
- [SalesPersonPersonID] int NOT NULL,
- [CustomerName] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [CustomerContact] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [OrderDate] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
- [CustomerPO] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NULL,
- [ExpectedDeliverDate] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL
-)
- WITH (
- LOCATION='WideWorldImporters.Orders',
- DATA_SOURCE=CosmosDB
-)
-GO
-CREATE STATISTICS CosmosDBOrderSalesPersonStats ON cosmosdb.Orders ([SalesPersonPersonID]) WITH FULLSCAN
-GO
-
--- Scan the external table just to make sure it works
---
-SELECT * FROM cosmosdb.Orders
-GO
--- Filter on a specific SalesPersonPersonID
---
-SELECT * FROM cosmosdb.Orders WHERE SalesPersonPersonID = 2
-GO
--- Find out the name of the salesperson and which customer they worked with
--- to test out the new mobile app experience.
-SELECT FullName, o.CustomerName, o.CustomerContact, o.OrderDate
-FROM cosmosdb.Orders o
-JOIN [Application].[People] p
-ON o.SalesPersonPersonID = p.PersonID
-GO
+USE [WideWorldImporters]
+GO
+CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+GO
+/* specify credentials to external data source
+* IDENTITY: user name for external source.
+* SECRET: password for external source.
+*/
+DROP DATABASE SCOPED CREDENTIAL CosmosDBCredentials
+GO
+-- You can get the IDENTITY (user) and secret (password) from the Connection String option in the
+-- Azure portal
+CREATE DATABASE SCOPED CREDENTIAL CosmosDBCredentials
+WITH IDENTITY = 'wwi', Secret = 'hSoxMUeEgNjeeWh4FTz5jmGRlSN4Ko6HoYqiJsbleFzewe86EEXJrvwkAqBgitypJdjUbeJqnTVNBO6NUa0DZQ=='
+GO
+DROP EXTERNAL DATA SOURCE CosmosDB
+GO
+-- The LOCATION is built from : from the Connection String in the Azure Portal
+CREATE EXTERNAL DATA SOURCE CosmosDB
+WITH (
+LOCATION = 'mongodb://wwi.documents.azure.com:10255',
+PUSHDOWN = ON,
+CREDENTIAL = CosmosDBCredentials
+)
+GO
+DROP SCHEMA cosmosdb
+go
+CREATE SCHEMA cosmosdb
+GO
+/* LOCATION: sql server table/view in 'database_name.schema_name.object_name' format
+* DATA_SOURCE: the external data source, created above.
+*/
+DROP EXTERNAL TABLE cosmosdb.Orders
+GO
+CREATE EXTERNAL TABLE cosmosdb.Orders
+(
+ [_id] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [id] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [OrderID] int NOT NULL,
+ [SalesPersonPersonID] int NOT NULL,
+ [CustomerName] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [CustomerContact] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [OrderDate] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL,
+ [CustomerPO] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NULL,
+ [ExpectedDeliverDate] NVARCHAR(100) COLLATE Latin1_General_100_CI_AS NOT NULL
+)
+ WITH (
+ LOCATION='WideWorldImporters.Orders',
+ DATA_SOURCE=CosmosDB
+)
+GO
+CREATE STATISTICS CosmosDBOrderSalesPersonStats ON cosmosdb.Orders ([SalesPersonPersonID]) WITH FULLSCAN
+GO
+
+-- Scan the external table just to make sure it works
+--
+SELECT * FROM cosmosdb.Orders
+GO
+-- Filter on a specific SalesPersonPersonID
+--
+SELECT * FROM cosmosdb.Orders WHERE SalesPersonPersonID = 2
+GO
+-- Find out the name of the salesperson and which customer they worked with
+-- to test out the new mobile app experience.
+SELECT FullName, o.CustomerName, o.CustomerContact, o.OrderDate
+FROM cosmosdb.Orders o
+JOIN [Application].[People] p
+ON o.SalesPersonPersonID = p.PersonID
+GO
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/readme.md
index 9aca38d7..eda7799c 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/cosmosdb/readme.md
@@ -1,23 +1,23 @@
-# SQL Server 2019 Polybase example connecting to CosmosDB
-
-This demo shows you how to setup a CosmosDB external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
-
-## Requirements
-
-1. Create a new database, collection, and document with CosmosDB in Azure. I used the Azure portal to create a new cosmosdb database in the same resource group as my polybase head node (bwpolybase). When I used the portal to create a new cosmosdb instance, I chose the Azure CosmosDB Database for Mongo API for the API selection. I used the Data Explorer tool from the portal to create my database called WideWorldImporters with a collection called Orders. Then I created a new document with field names and values like the following (Note: the _id field was created by Data Explorer and the id field was a default value already provided by the tool)
-
-{
- "_id" : ObjectId("5c54aa72dd13c70f445745bf"),
- "id" : "1",
- "OrderID" : 1,
- "SalesPersonPersonID" : 2,
- "CustomerName" : "Vandelay Industries",
- "CustomerContact" : "Art Vandelay",
- "OrderDate" : "2018-05-14",
- "CustomerPO" : "20180514",
- "ExpectedDeliveryDate" : "2018-05-21"
-}
-
-## Demo Steps
-
+# SQL Server 2019 Polybase example connecting to CosmosDB
+
+This demo shows you how to setup a CosmosDB external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
+
+## Requirements
+
+1. Create a new database, collection, and document with CosmosDB in Azure. I used the Azure portal to create a new cosmosdb database in the same resource group as my polybase head node (bwpolybase). When I used the portal to create a new cosmosdb instance, I chose the Azure CosmosDB Database for Mongo API for the API selection. I used the Data Explorer tool from the portal to create my database called WideWorldImporters with a collection called Orders. Then I created a new document with field names and values like the following (Note: the _id field was created by Data Explorer and the id field was a default value already provided by the tool)
+
+{
+ "_id" : ObjectId("5c54aa72dd13c70f445745bf"),
+ "id" : "1",
+ "OrderID" : 1,
+ "SalesPersonPersonID" : 2,
+ "CustomerName" : "Vandelay Industries",
+ "CustomerContact" : "Art Vandelay",
+ "OrderDate" : "2018-05-14",
+ "CustomerPO" : "20180514",
+ "ExpectedDeliveryDate" : "2018-05-21"
+}
+
+## Demo Steps
+
1. On my SQL Server 2019 head node (bwpolybase), I used the **cosmosdb_external_table.sql** script to create the database scoped credential, externaL data source, external table, and sample SELECT statements to query the external table and join it with local SQL Server 2019 tables in the WideWorldImporters database. The **Connection String** option from the portal of the instance shows you the username and password to use. It also has HOST and PORT fields which are used to build the LOCATION sytnax for the data source.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/hdfs_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/hdfs_external_table.sql
index e6d910a6..330dbfed 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/hdfs_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/hdfs_external_table.sql
@@ -1,97 +1,97 @@
-USE [master]
-GO
--- Enabled PB connectivity to a Hadoop HDFS source which in this case is just Azure Blob Storage
---
-sp_configure @configname = 'hadoop connectivity', @configvalue = 7;
-GO
-RECONFIGURE
-GO
-
--- Enable PB export to be able to ingest data into the HDFS target
---
-sp_configure 'allow polybase export', 1
-GO
-RECONFIGURE
-GO
-
--- STOP: SQL Server must be restarted for this to take effect
---
-USE [WideWorldImporters]
-GO
--- Only run this if you have not already created a master key in the db
---
---CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
---GO
--- IDENTITY: any string (this is not used for authentication to Azure storage).
--- SECRET: your Azure storage account key.
-DROP DATABASE SCOPED CREDENTIAL AzureStorageCredential
-GO
-CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
-WITH IDENTITY = 'user', Secret = 'C5aFpK587sIDFIMSEqXwA08xlhDM34/rfOz2g+sVq/hcKReo6agvT9JZcWGe9NtEyHEypK095WZtDdE/gkKZNQ=='
-GO
--- LOCATION: Azure account storage account name and blob container name.
--- CREDENTIAL: The database scoped credential created above.
-DROP EXTERNAL DATA SOURCE bwdatalake
-GO
-CREATE EXTERNAL DATA SOURCE bwdatalake with (
- TYPE = HADOOP,
- LOCATION ='wasbs://wwi@bwdatalake.blob.core.windows.net',
- CREDENTIAL = AzureStorageCredential
-)
-GO
--- FORMAT TYPE: Type of format in Hadoop (DELIMITEDTEXT, RCFILE, ORC, PARQUET).
-CREATE EXTERNAL FILE FORMAT TextFileFormat WITH (
- FORMAT_TYPE = DELIMITEDTEXT,
- FORMAT_OPTIONS (FIELD_TERMINATOR ='|',
- USE_TYPE_DEFAULT = TRUE))
-GO
--- Create a schema called hdfs
---
-DROP SCHEMA hdfs
-GO
-CREATE SCHEMA hdfs
-GO
--- LOCATION: path to file or directory that contains the data (relative to HDFS root).
-DROP EXTERNAL TABLE [hdfs].[WWI_Order_Reviews]
-GO
-CREATE EXTERNAL TABLE [hdfs].[WWI_Order_Reviews] (
- [OrderID] int NOT NULL,
- [CustomerID] int NOT NULL,
- [Rating] int NULL,
- [Review_Comments] nvarchar(1000) NOT NULL
-)
-WITH (LOCATION='/WWI/',
- DATA_SOURCE = bwdatalake,
- FILE_FORMAT = TextFileFormat
-)
-GO
-
--- Ingest some data
---
-INSERT INTO [hdfs].[WWI_Order_Reviews] VALUES (1, 832, 10, 'I had a great experience with my order')
-GO
-CREATE STATISTICS StatsforReviews on [hdfs].[WWI_Order_Reviews](OrderID, CustomerID)
-GO
-
--- Now query the external table
---
-SELECT * FROM [hdfs].[WWI_Order_Reviews]
-GO
-
--- Let's do a filter to enable pushdown
---
-SELECT * FROM [hdfs].[WWI_Order_Reviews]
-WHERE OrderID = 1
-GO
-
--- Let's join the review with our order and customer data
---
-SELECT o.OrderDate, c.CustomerName, p.FullName as SalesPerson, wor.Rating, wor.Review_Comments
-FROM [Sales].[Orders] o
-JOIN [hdfs].[WWI_Order_Reviews] wor
-ON o.OrderID = wor.OrderID
-JOIN [Application].[People] p
-ON p.PersonID = o.SalespersonPersonID
-JOIN [Sales].[Customers] c
-ON c.CustomerID = wor.CustomerID
+USE [master]
+GO
+-- Enabled PB connectivity to a Hadoop HDFS source which in this case is just Azure Blob Storage
+--
+sp_configure @configname = 'hadoop connectivity', @configvalue = 7;
+GO
+RECONFIGURE
+GO
+
+-- Enable PB export to be able to ingest data into the HDFS target
+--
+sp_configure 'allow polybase export', 1
+GO
+RECONFIGURE
+GO
+
+-- STOP: SQL Server must be restarted for this to take effect
+--
+USE [WideWorldImporters]
+GO
+-- Only run this if you have not already created a master key in the db
+--
+--CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+--GO
+-- IDENTITY: any string (this is not used for authentication to Azure storage).
+-- SECRET: your Azure storage account key.
+DROP DATABASE SCOPED CREDENTIAL AzureStorageCredential
+GO
+CREATE DATABASE SCOPED CREDENTIAL AzureStorageCredential
+WITH IDENTITY = 'user', Secret = 'C5aFpK587sIDFIMSEqXwA08xlhDM34/rfOz2g+sVq/hcKReo6agvT9JZcWGe9NtEyHEypK095WZtDdE/gkKZNQ=='
+GO
+-- LOCATION: Azure account storage account name and blob container name.
+-- CREDENTIAL: The database scoped credential created above.
+DROP EXTERNAL DATA SOURCE bwdatalake
+GO
+CREATE EXTERNAL DATA SOURCE bwdatalake with (
+ TYPE = HADOOP,
+ LOCATION ='wasbs://wwi@bwdatalake.blob.core.windows.net',
+ CREDENTIAL = AzureStorageCredential
+)
+GO
+-- FORMAT TYPE: Type of format in Hadoop (DELIMITEDTEXT, RCFILE, ORC, PARQUET).
+CREATE EXTERNAL FILE FORMAT TextFileFormat WITH (
+ FORMAT_TYPE = DELIMITEDTEXT,
+ FORMAT_OPTIONS (FIELD_TERMINATOR ='|',
+ USE_TYPE_DEFAULT = TRUE))
+GO
+-- Create a schema called hdfs
+--
+DROP SCHEMA hdfs
+GO
+CREATE SCHEMA hdfs
+GO
+-- LOCATION: path to file or directory that contains the data (relative to HDFS root).
+DROP EXTERNAL TABLE [hdfs].[WWI_Order_Reviews]
+GO
+CREATE EXTERNAL TABLE [hdfs].[WWI_Order_Reviews] (
+ [OrderID] int NOT NULL,
+ [CustomerID] int NOT NULL,
+ [Rating] int NULL,
+ [Review_Comments] nvarchar(1000) NOT NULL
+)
+WITH (LOCATION='/WWI/',
+ DATA_SOURCE = bwdatalake,
+ FILE_FORMAT = TextFileFormat
+)
+GO
+
+-- Ingest some data
+--
+INSERT INTO [hdfs].[WWI_Order_Reviews] VALUES (1, 832, 10, 'I had a great experience with my order')
+GO
+CREATE STATISTICS StatsforReviews on [hdfs].[WWI_Order_Reviews](OrderID, CustomerID)
+GO
+
+-- Now query the external table
+--
+SELECT * FROM [hdfs].[WWI_Order_Reviews]
+GO
+
+-- Let's do a filter to enable pushdown
+--
+SELECT * FROM [hdfs].[WWI_Order_Reviews]
+WHERE OrderID = 1
+GO
+
+-- Let's join the review with our order and customer data
+--
+SELECT o.OrderDate, c.CustomerName, p.FullName as SalesPerson, wor.Rating, wor.Review_Comments
+FROM [Sales].[Orders] o
+JOIN [hdfs].[WWI_Order_Reviews] wor
+ON o.OrderID = wor.OrderID
+JOIN [Application].[People] p
+ON p.PersonID = o.SalespersonPersonID
+JOIN [Sales].[Customers] c
+ON c.CustomerID = wor.CustomerID
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/readme.md
index 767aa494..1a0a27b3 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/hdfs/readme.md
@@ -1,9 +1,9 @@
-# SQL Server demo with Polybase for HDFS
-
-## Requirements
-
-Follow all the instructions in the fundamentals folder which is at the same level as the sqldatahub folder.
-
-## Demos Steps
-
+# SQL Server demo with Polybase for HDFS
+
+## Requirements
+
+Follow all the instructions in the fundamentals folder which is at the same level as the sqldatahub folder.
+
+## Demos Steps
+
1. Run all the T-SQL commands in **hdfs_external_table.sql**. You will need to edit the appropriate details to point to your Azure storage container including the credential and location for the data source.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createtab.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createtab.sql
index eca1f0d8..da9e90f7 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createtab.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createtab.sql
@@ -1,7 +1,7 @@
-CREATE TABLE gl.accountsreceivable (
-arid int primary key,
-ardate date,
-ardesc varchar2(100),
-arref int,
-aramt number(10,2)
-);
+CREATE TABLE gl.accountsreceivable (
+arid int primary key,
+ardate date,
+ardesc varchar2(100),
+arref int,
+aramt number(10,2)
+);
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createuser.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createuser.sql
index 5ad3af1d..0d81b36f 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createuser.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/createuser.sql
@@ -1,8 +1,8 @@
-CREATE USER gl IDENTIFIED BY glpwd DEFAULT TABLESPACE users TEMPORARY TABLESPACE temp QUOTA UNLIMITED ON users;
-GRANT CREATE SESSION TO gl;
-GRANT CREATE TABLE TO gl;
-GRANT CREATE VIEW TO gl;
-GRANT CREATE ANY TRIGGER TO gl;
-GRANT CREATE ANY PROCEDURE TO gl;
-GRANT CREATE SEQUENCE TO gl;
-GRANT CREATE SYNONYM TO gl;
+CREATE USER gl IDENTIFIED BY glpwd DEFAULT TABLESPACE users TEMPORARY TABLESPACE temp QUOTA UNLIMITED ON users;
+GRANT CREATE SESSION TO gl;
+GRANT CREATE TABLE TO gl;
+GRANT CREATE VIEW TO gl;
+GRANT CREATE ANY TRIGGER TO gl;
+GRANT CREATE ANY PROCEDURE TO gl;
+GRANT CREATE SEQUENCE TO gl;
+GRANT CREATE SYNONYM TO gl;
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oracle_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oracle_external_table.sql
index 87c67a47..2e4a9a1e 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oracle_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oracle_external_table.sql
@@ -1,67 +1,67 @@
-USE [WideWorldImporters]
-GO
-CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
-GO
-/* specify credentials to external data source
-* IDENTITY: user name for external source.
-* SECRET: password for external source.
-*/
-DROP DATABASE SCOPED CREDENTIAL OracleCredentials
-GO
-CREATE DATABASE SCOPED CREDENTIAL OracleCredentials
-WITH IDENTITY = 'gl', Secret = 'glpwd'
-GO
-/* LOCATION: Location string should be of format '://[:]'.
-* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
-* CREDENTIAL: the database scoped credential, created above.
-*/
-DROP EXTERNAL DATA SOURCE OracleServer
-GO
-CREATE EXTERNAL DATA SOURCE OracleServer
-WITH (
-LOCATION = 'oracle://bworacle:49161',
-PUSHDOWN = ON,
-CREDENTIAL = OracleCredentials
-)
-GO
-DROP SCHEMA oracle
-go
-CREATE SCHEMA oracle
-GO
-/* LOCATION: oracle table/view in 'database_name.schema_name.object_name' format
-* DATA_SOURCE: the external data source, created above.
-*/
-DROP EXTERNAL TABLE oracle.accountsreceivable
-GO
-CREATE EXTERNAL TABLE oracle.accountsreceivable
-(
-arid int,
-ardate date,
-ardesc nvarchar(100) COLLATE Latin1_General_100_CI_AS,
-arref int,
-aramt decimal(10,2)
-
-)
- WITH (
- LOCATION='[XE].[GL].[ACCOUNTSRECEIVABLE]',
- DATA_SOURCE=OracleServer
-)
-GO
-CREATE STATISTICS arrefstats ON oracle.accountsreceivable ([arref]) WITH FULLSCAN
-GO
--- Let's scan the table to make sure it works
-SELECT * FROM oracle.accountsreceivable
-GO
-
--- Try a simple filter
-SELECT * FROM oracle.accountsreceivable
-WHERE arref = 336252
-GO
-
--- Join with a local table
---
-SELECT ct.*, oa.arid, oa.ardesc
-FROM oracle.accountsreceivable oa
-JOIN [Sales].[CustomerTransactions] ct
-ON oa.arref = ct.CustomerTransactionID
+USE [WideWorldImporters]
+GO
+CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+GO
+/* specify credentials to external data source
+* IDENTITY: user name for external source.
+* SECRET: password for external source.
+*/
+DROP DATABASE SCOPED CREDENTIAL OracleCredentials
+GO
+CREATE DATABASE SCOPED CREDENTIAL OracleCredentials
+WITH IDENTITY = 'gl', Secret = 'glpwd'
+GO
+/* LOCATION: Location string should be of format '://[:]'.
+* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
+* CREDENTIAL: the database scoped credential, created above.
+*/
+DROP EXTERNAL DATA SOURCE OracleServer
+GO
+CREATE EXTERNAL DATA SOURCE OracleServer
+WITH (
+LOCATION = 'oracle://bworacle:49161',
+PUSHDOWN = ON,
+CREDENTIAL = OracleCredentials
+)
+GO
+DROP SCHEMA oracle
+go
+CREATE SCHEMA oracle
+GO
+/* LOCATION: oracle table/view in 'database_name.schema_name.object_name' format
+* DATA_SOURCE: the external data source, created above.
+*/
+DROP EXTERNAL TABLE oracle.accountsreceivable
+GO
+CREATE EXTERNAL TABLE oracle.accountsreceivable
+(
+arid int,
+ardate date,
+ardesc nvarchar(100) COLLATE Latin1_General_100_CI_AS,
+arref int,
+aramt decimal(10,2)
+
+)
+ WITH (
+ LOCATION='[XE].[GL].[ACCOUNTSRECEIVABLE]',
+ DATA_SOURCE=OracleServer
+)
+GO
+CREATE STATISTICS arrefstats ON oracle.accountsreceivable ([arref]) WITH FULLSCAN
+GO
+-- Let's scan the table to make sure it works
+SELECT * FROM oracle.accountsreceivable
+GO
+
+-- Try a simple filter
+SELECT * FROM oracle.accountsreceivable
+WHERE arref = 336252
+GO
+
+-- Join with a local table
+--
+SELECT ct.*, oa.arid, oa.ardesc
+FROM oracle.accountsreceivable oa
+JOIN [Sales].[CustomerTransactions] ct
+ON oa.arref = ct.CustomerTransactionID
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oraconnect.sh b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oraconnect.sh
index 30896950..d4d5ea93 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oraconnect.sh
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/oraconnect.sh
@@ -1,2 +1,2 @@
-sqlplus64 system/oracle@localhost:49161/xe
-
+sqlplus64 system/oracle@localhost:49161/xe
+
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/readme.md
index fdff0d49..53c24f0a 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/oracle/readme.md
@@ -1,45 +1,45 @@
-# SQL Server 2019 Polybase example connecting to Oracle
-
-This demo shows you how to setup a Oracle external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
-
-## Requirements - Installing and setting up Oracle
-
-SQL Server external tables should work with most current Oracle versions (11g+) so for this demo you can choose any Oracle installation or platform you like. For my demo, I used Oracle Express 11g using docker containers on Red Hat Enterpise Linux in an Azure Virtual Machine. The following are the steps and scripts I used to install and Oracle instance using a docker container and create a table to be used for the demo. I created my RHEL VM in Azure in the same resource group (bwsql2019demos) as the polybase head node running SQL Server 2019 on Windows Server. I then on this head node server added an entry in the /etc/hosts file for the RHEL Azure VM private IP address with a name of bworacle so I can use this name when creating an external data source.
-
-1. Install Docker CE for CentOS using these instructions at https://docs.docker.com/install/linux/docker-ce/centos/
-
-2. Used these instructions to pull a docker container image for Oracle at https://github.com/wnameless/docker-oracle-xe-11g. This site has instructions for running the container, instance ID, and password for SYSTEM.
-
-3. Installed OCI client and SQLPlus RPM packages from http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64/index.htm
-
-4. I had to configure SQLPLUS (sqlplus64 is actually the program to use) by setting the following environment variables:
-- ORACLE_SID=xe
-- LD_LIBRARY_PATH=/usr/lib/oracle/18.3/client64/lib
-- ORACLE_HOME=/usr/lib/oracle/18.3/client64
-
-5. I was then able to connect to ORACLE XE on this machine using syntax like
-
-sqlplus64 system/oracle@localhost:49161/xe
-
-49161 is the port number from running the docker image for XE
-oracle is the password for SYSTEM
-
-I've included a script called **oraconnect.sh** as an example.
-
-6. I wanted a user other than SYSTEM so used the script **createuser.sql** to create a new user called g1.
-
-7. Using this new login, I ran the script **createtab.sql** to create a new table with the instance. You can run this script using sqlplus64 like the following:
-
-sqlplus64 gl/glpwd@localhost:49161/xe @createtab.sql
-
-8. I then executed the **insertdata.sql** script finding a valid CustomerTransactionID from the Sales.CustomerTransactions table in the WideWorldImporters database. This ID becomes the arref fields in the accounts receivable table.
-
-## Demo Steps
-
-1. With everything in hand on my Oracle server, I now can use the oracle_external_table.sql script to create the data source and external table.
-
-Note the syntax for the LOCATION string for the external table which I was required to use UPPERCASE even though I didn't create these objects in uppercase using sqlplus64.
-
-LOCATION='[XE].[GL].[ACCOUNTSRECEIVABLE]'
-
-This script also includes examples to query the table and join together with the [Sales].[CustomerTransactions] table.
+# SQL Server 2019 Polybase example connecting to Oracle
+
+This demo shows you how to setup a Oracle external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
+
+## Requirements - Installing and setting up Oracle
+
+SQL Server external tables should work with most current Oracle versions (11g+) so for this demo you can choose any Oracle installation or platform you like. For my demo, I used Oracle Express 11g using docker containers on Red Hat Enterpise Linux in an Azure Virtual Machine. The following are the steps and scripts I used to install and Oracle instance using a docker container and create a table to be used for the demo. I created my RHEL VM in Azure in the same resource group (bwsql2019demos) as the polybase head node running SQL Server 2019 on Windows Server. I then on this head node server added an entry in the /etc/hosts file for the RHEL Azure VM private IP address with a name of bworacle so I can use this name when creating an external data source.
+
+1. Install Docker CE for CentOS using these instructions at https://docs.docker.com/install/linux/docker-ce/centos/
+
+2. Used these instructions to pull a docker container image for Oracle at https://github.com/wnameless/docker-oracle-xe-11g. This site has instructions for running the container, instance ID, and password for SYSTEM.
+
+3. Installed OCI client and SQLPlus RPM packages from http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64/index.htm
+
+4. I had to configure SQLPLUS (sqlplus64 is actually the program to use) by setting the following environment variables:
+- ORACLE_SID=xe
+- LD_LIBRARY_PATH=/usr/lib/oracle/18.3/client64/lib
+- ORACLE_HOME=/usr/lib/oracle/18.3/client64
+
+5. I was then able to connect to ORACLE XE on this machine using syntax like
+
+sqlplus64 system/oracle@localhost:49161/xe
+
+49161 is the port number from running the docker image for XE
+oracle is the password for SYSTEM
+
+I've included a script called **oraconnect.sh** as an example.
+
+6. I wanted a user other than SYSTEM so used the script **createuser.sql** to create a new user called g1.
+
+7. Using this new login, I ran the script **createtab.sql** to create a new table with the instance. You can run this script using sqlplus64 like the following:
+
+sqlplus64 gl/glpwd@localhost:49161/xe @createtab.sql
+
+8. I then executed the **insertdata.sql** script finding a valid CustomerTransactionID from the Sales.CustomerTransactions table in the WideWorldImporters database. This ID becomes the arref fields in the accounts receivable table.
+
+## Demo Steps
+
+1. With everything in hand on my Oracle server, I now can use the oracle_external_table.sql script to create the data source and external table.
+
+Note the syntax for the LOCATION string for the external table which I was required to use UPPERCASE even though I didn't create these objects in uppercase using sqlplus64.
+
+LOCATION='[XE].[GL].[ACCOUNTSRECEIVABLE]'
+
+This script also includes examples to query the table and join together with the [Sales].[CustomerTransactions] table.
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/readme.md
index fae0f9a5..e4925384 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/readme.md
@@ -1,28 +1,28 @@
-# SQL Server Data Hub Polybase demos
-
-In this demo I show you how to use SQL Server as a hub for data virtualization. Consider the example company WideWorldImporters (read more at https://docs.microsoft.com/en-us/sql/samples/wide-world-importers-what-is?view=sql-server-2017 )
-
-This demo will cover scenarios where this company has data in other sources but would like to avoid building complex and expensive ETL programs to move the data into SQL Server 2019. In some cases, they are going to migrate their data but they would first like access to the data so applications and reports can seamlessly run while just connecting to SQL Server 2019.
-
-They have identified the following data sources and business scenarios:
-
-**SQL Server 2008R2** - This is the legacy SQL Server which contains a list of Suppliers the company no longer uses but they want to access for historical reasons.
-
-**Azure SQL Database** - A new cloud based application is prototyping data for StockItems in Azure
-
-**CosmosDB** - A research team is experimenting with a mobile based application to take orders from customers using a noSQL data store like Azure CosmosDB
-
-**Oracle** - The company's accounting system is in Oracle but will be migrated soon to SQL Server. For now, the company wants to be able to access accounts receivable data which lines up with transactions in WideWorldImporters database.
-
-**Hadoop** - The company's website ordering system now has a new feature for customers to review the ordering process. The developers find it very convenient to stream a large amount of data for these reviews in the form of files in Hadoop. The system today just streams this in Azure Blog Storage.
-
-**SAPHana** - The company just acquired a new company and would like to start reviewing the customer profiles the new acquired company brings. The new company has a data warehouse stored in SAPHana that can be queried.
-
-All of these data sources will become external data sources and tables. For the purposes of this demo, all of the examples will use resources in Azure. The hub for SQL Server 2019 will be based on the Polybase configuration installed and configured in the **Fundamentals** folder.
-
-
-1. If you have not restored the backup already, download and restore the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers
-
-2. First, run the scenario for SQL2008r2 by going to the **SQL2008r2** folder and following the instructions in the readme.md file.
-
+# SQL Server Data Hub Polybase demos
+
+In this demo I show you how to use SQL Server as a hub for data virtualization. Consider the example company WideWorldImporters (read more at https://docs.microsoft.com/en-us/sql/samples/wide-world-importers-what-is?view=sql-server-2017 )
+
+This demo will cover scenarios where this company has data in other sources but would like to avoid building complex and expensive ETL programs to move the data into SQL Server 2019. In some cases, they are going to migrate their data but they would first like access to the data so applications and reports can seamlessly run while just connecting to SQL Server 2019.
+
+They have identified the following data sources and business scenarios:
+
+**SQL Server 2008R2** - This is the legacy SQL Server which contains a list of Suppliers the company no longer uses but they want to access for historical reasons.
+
+**Azure SQL Database** - A new cloud based application is prototyping data for StockItems in Azure
+
+**CosmosDB** - A research team is experimenting with a mobile based application to take orders from customers using a noSQL data store like Azure CosmosDB
+
+**Oracle** - The company's accounting system is in Oracle but will be migrated soon to SQL Server. For now, the company wants to be able to access accounts receivable data which lines up with transactions in WideWorldImporters database.
+
+**Hadoop** - The company's website ordering system now has a new feature for customers to review the ordering process. The developers find it very convenient to stream a large amount of data for these reviews in the form of files in Hadoop. The system today just streams this in Azure Blog Storage.
+
+**SAPHana** - The company just acquired a new company and would like to start reviewing the customer profiles the new acquired company brings. The new company has a data warehouse stored in SAPHana that can be queried.
+
+All of these data sources will become external data sources and tables. For the purposes of this demo, all of the examples will use resources in Azure. The hub for SQL Server 2019 will be based on the Polybase configuration installed and configured in the **Fundamentals** folder.
+
+
+1. If you have not restored the backup already, download and restore the WideWorldImporters backup from https://github.com/Microsoft/sql-server-samples/tree/master/samples/databases/wide-world-importers
+
+2. First, run the scenario for SQL2008r2 by going to the **SQL2008r2** folder and following the instructions in the readme.md file.
+
3.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/createuser.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/createuser.sql
index 30945199..bf2dd339 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/createuser.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/createuser.sql
@@ -1,4 +1,4 @@
-CREATE USER bwsaphana PASSWORD Cprsql2019 NO FORCE_FIRST_PASSWORD_CHANGE;
-GRANT AFLPM_CREATOR_ERASER_EXECUTE TO bwsaphana;
-GRANT CONTENT_ADMIN TO bwsaphana;
+CREATE USER bwsaphana PASSWORD Cprsql2019 NO FORCE_FIRST_PASSWORD_CHANGE;
+GRANT AFLPM_CREATOR_ERASER_EXECUTE TO bwsaphana;
+GRANT CONTENT_ADMIN TO bwsaphana;
GRANT MODELING to bwsaphana;
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/readme.md
index 83026ef8..69a81f37 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/readme.md
@@ -1,60 +1,60 @@
-# SQL Server Polybase demo for SAP HANA using the odbc connector
-
-This is a demo to show how to connect to a SAP HANA database using the odbc connector shipped with SQL Server 2019.
-
-This demo assumes you have setup a Polybase scale out group as shown in the fundamentals folder above this folder. NOTE: You must be running SQL Server 2019 CTP 2.3 or higher in order to use this SAP HANA demo.
-
-## Requirements - SAP HANA setup
-
-### SAP HANA Server Setup
-
-You can use an existing SAP HANA system. For purposes of this demo I created an SAP HANA server using the SAP HANA Express Edition template from Azure which you can read more at https://azuremarketplace.microsoft.com/ja-/marketplace/apps/sap.hanaexpress?tab=Overview. This is a SUSE Linux VM with SAP HANA installed. I created this VM in the resource group of the polybase head node, bwsql2019demos, so I would be on the same virtual net. I also took the private IP address of the SAP VM and put this in c:\windows\system32\drivers\etc\hosts as bwsaphana so I could just refer to that hostname on my polybase head node.
-
-1. Go through the steps to get the SAP HANA server running on Azure with this guide
-https://developers.sap.com/tutorials/hxe-ms-azure-marketplace-getting-started.html
-
-2. After following these steps, I disconnected by current ssh session and started a new one but this time logged in with hxeadm. The docs say the default password is HXEHana1. I was prompted to change my password which I did to Cprsql2019. Note that the instance "number" is 90 using this Azure template.
-
-3. The client command line tool **hdbsql** is installed already through this process. So to create a database, a user, and a table with data, use the next set of scripts
-
-4. Execute **createdb.sh** to create a database. This will connect as the SYSTEM user and create a database called VANDELAY.
-
-5. Execute **createuser.sh** which will create a new user, bwsaphana, in the VANDELAY database and grant permissions.
-
-6. Execute **createtab.sh** which will create a new table, Customers, in the VANDELAY database connected as the bwsaphana user. I found out that the default schema for this table is called BWSAPHANA after querying the tables view. That is important as we will need it for the external table script.
-
-7. Execute **insertdata.sh** which will insert a row into the Customers table. I used a CustomerID of 100000 because this is far bigger than what the WideWorldImporters database would have in its Sales.Customer table set of ID values.
-
-### Install the SAP HANA Driver
-
-In order to use the general odbc connector built into SQL Server 2019 you must install the driver to connect to your ODBC data source. Since I am going to show you how to connect to SAP HANA, I found the official 64bit ODBC Driver for SAP HANA called HDBODBC. In order to support a scale out group you need to install this on each of the Polybase nodes. For purposes of this demo, I will only install this on the head node, bwpolybase.
-
-I found the driver to download from https://developers.sap.com/trials-downloads.html
-
-and found these docs on how to setup the driver on Windows
-
-https://help.sap.com/viewer/e9146b36040844d0b1f309bc8c1ba6ab/2.5.0.0/en-US/321ff67a31f54654af30dad9f82347dc.html
-
-**Note: You must install the driver on all nodes for the Polybase scale out group and configure them all with the same server and port. If you don't install this on the compute nodes you may get intermittent errors when querying the external table because Polybase may redirect your query through a compute node where the driver doesn't exist.**
-
-The experience was interesting and here our some tips:
-
-- Run the installer from a powershell or cmd window as Administrator
-- Depending on what version of Windows Server, you may get an error on the VC++ runtime install like "Program terminated with exit code 1638". This driver depends on installing the VC++ runtime for VS 2015. I had a the VS 2017 runtime so it failed. The solution is to first uninstall the VS 2017 VC++ runtime, install the ODBC driver, and the reinstall the VS 2017 VC++ runtime.
-- You want your System DSN to use the right port for SAP HANA. The port is based on the instance number and tenant (database) of the SAP HANA Server. The port will always be 3XXYY where XX = instance number and YY = number for the database. The instance number was 90 from the default Azure template install for me. But what about the VANDELAY database? Turns out there is a view called sys_databases.m_services in the SYSTEMDB database which tells you the SQL port for each database. When I ran a query against this view logged in as SYSTEM I found out the port for VANDELAY was 39041. So I used this port in the DSN configuration.
-- Since bwsaphana is imy hosts file to point to the private IP of the SAP VM, I can use that as the host server name.
-
-## Demo Steps
-
-1. All the instructions are in the **sap_external_table.sql** script to create the credential, data source, external table, and query against the remote table.
-
-Take note of the syntax for the external data source. Note these strings are required even though the DSN has port specifics
-
-CREATE EXTERNAL DATA SOURCE SAPHANAServer
-WITH (
-LOCATION = 'odbc://bwsaphana',
-CONNECTION_OPTIONS = 'Driver={HDBODBC};ServerNode=bwsaphana:39041',
-PUSHDOWN = ON,
-CREDENTIAL = SAPHANACredentials
-)
+# SQL Server Polybase demo for SAP HANA using the odbc connector
+
+This is a demo to show how to connect to a SAP HANA database using the odbc connector shipped with SQL Server 2019.
+
+This demo assumes you have setup a Polybase scale out group as shown in the fundamentals folder above this folder. NOTE: You must be running SQL Server 2019 CTP 2.3 or higher in order to use this SAP HANA demo.
+
+## Requirements - SAP HANA setup
+
+### SAP HANA Server Setup
+
+You can use an existing SAP HANA system. For purposes of this demo I created an SAP HANA server using the SAP HANA Express Edition template from Azure which you can read more at https://azuremarketplace.microsoft.com/ja-/marketplace/apps/sap.hanaexpress?tab=Overview. This is a SUSE Linux VM with SAP HANA installed. I created this VM in the resource group of the polybase head node, bwsql2019demos, so I would be on the same virtual net. I also took the private IP address of the SAP VM and put this in c:\windows\system32\drivers\etc\hosts as bwsaphana so I could just refer to that hostname on my polybase head node.
+
+1. Go through the steps to get the SAP HANA server running on Azure with this guide
+https://developers.sap.com/tutorials/hxe-ms-azure-marketplace-getting-started.html
+
+2. After following these steps, I disconnected by current ssh session and started a new one but this time logged in with hxeadm. The docs say the default password is HXEHana1. I was prompted to change my password which I did to Cprsql2019. Note that the instance "number" is 90 using this Azure template.
+
+3. The client command line tool **hdbsql** is installed already through this process. So to create a database, a user, and a table with data, use the next set of scripts
+
+4. Execute **createdb.sh** to create a database. This will connect as the SYSTEM user and create a database called VANDELAY.
+
+5. Execute **createuser.sh** which will create a new user, bwsaphana, in the VANDELAY database and grant permissions.
+
+6. Execute **createtab.sh** which will create a new table, Customers, in the VANDELAY database connected as the bwsaphana user. I found out that the default schema for this table is called BWSAPHANA after querying the tables view. That is important as we will need it for the external table script.
+
+7. Execute **insertdata.sh** which will insert a row into the Customers table. I used a CustomerID of 100000 because this is far bigger than what the WideWorldImporters database would have in its Sales.Customer table set of ID values.
+
+### Install the SAP HANA Driver
+
+In order to use the general odbc connector built into SQL Server 2019 you must install the driver to connect to your ODBC data source. Since I am going to show you how to connect to SAP HANA, I found the official 64bit ODBC Driver for SAP HANA called HDBODBC. In order to support a scale out group you need to install this on each of the Polybase nodes. For purposes of this demo, I will only install this on the head node, bwpolybase.
+
+I found the driver to download from https://developers.sap.com/trials-downloads.html
+
+and found these docs on how to setup the driver on Windows
+
+https://help.sap.com/viewer/e9146b36040844d0b1f309bc8c1ba6ab/2.5.0.0/en-US/321ff67a31f54654af30dad9f82347dc.html
+
+**Note: You must install the driver on all nodes for the Polybase scale out group and configure them all with the same server and port. If you don't install this on the compute nodes you may get intermittent errors when querying the external table because Polybase may redirect your query through a compute node where the driver doesn't exist.**
+
+The experience was interesting and here our some tips:
+
+- Run the installer from a powershell or cmd window as Administrator
+- Depending on what version of Windows Server, you may get an error on the VC++ runtime install like "Program terminated with exit code 1638". This driver depends on installing the VC++ runtime for VS 2015. I had a the VS 2017 runtime so it failed. The solution is to first uninstall the VS 2017 VC++ runtime, install the ODBC driver, and the reinstall the VS 2017 VC++ runtime.
+- You want your System DSN to use the right port for SAP HANA. The port is based on the instance number and tenant (database) of the SAP HANA Server. The port will always be 3XXYY where XX = instance number and YY = number for the database. The instance number was 90 from the default Azure template install for me. But what about the VANDELAY database? Turns out there is a view called sys_databases.m_services in the SYSTEMDB database which tells you the SQL port for each database. When I ran a query against this view logged in as SYSTEM I found out the port for VANDELAY was 39041. So I used this port in the DSN configuration.
+- Since bwsaphana is imy hosts file to point to the private IP of the SAP VM, I can use that as the host server name.
+
+## Demo Steps
+
+1. All the instructions are in the **sap_external_table.sql** script to create the credential, data source, external table, and query against the remote table.
+
+Take note of the syntax for the external data source. Note these strings are required even though the DSN has port specifics
+
+CREATE EXTERNAL DATA SOURCE SAPHANAServer
+WITH (
+LOCATION = 'odbc://bwsaphana',
+CONNECTION_OPTIONS = 'Driver={HDBODBC};ServerNode=bwsaphana:39041',
+PUSHDOWN = ON,
+CREDENTIAL = SAPHANACredentials
+)
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/sap_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/sap_external_table.sql
index 2a91c9e8..aaa22b01 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/sap_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/saphana/sap_external_table.sql
@@ -1,62 +1,62 @@
-USE [WideWorldImporters]
-GO
-CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
-GO
-/* specify credentials to external data source
-* IDENTITY: user name for external source.
-* SECRET: password for external source.
-*/
--- Need to change this a different user than SYSTEM
-DROP DATABASE SCOPED CREDENTIAL SAPHANACredentials
-GO
-CREATE DATABASE SCOPED CREDENTIAL SAPHANACredentials
-WITH IDENTITY = 'bwsaphana', Secret = 'Cprsql2019'
-GO
-/* LOCATION: Location string should be of format '://[:]'.
-* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
-* CREDENTIAL: the database scoped credential, created above.
-*/
-DROP EXTERNAL DATA SOURCE SAPHANAServer
-GO
-CREATE EXTERNAL DATA SOURCE SAPHANAServer
-WITH (
-LOCATION = 'odbc://bwsaphana',
-CONNECTION_OPTIONS = 'Driver={HDBODBC};ServerNode=bwsaphana:39041',
-PUSHDOWN = ON,
-CREDENTIAL = SAPHANACredentials
-)
-GO
-DROP SCHEMA saphana
-go
-CREATE SCHEMA saphana
-GO
-/* LOCATION: oracle table/view in 'database_name.schema_name.object_name' format
-* DATA_SOURCE: the external data source, created above.
-*/
-DROP EXTERNAL TABLE saphana.customers
-GO
-CREATE EXTERNAL TABLE saphana.customers
-(
-CustomerID int,
-CustomerName nvarchar(100) COLLATE Latin1_General_100_CI_AS,
-AccountOpenedDate date,
-CustomerWebSite nvarchar(256) COLLATE Latin1_General_100_CI_AS
-)
- WITH (
- LOCATION='[BWSAPHANA].[CUSTOMERS]',
- DATA_SOURCE=SAPHANAServer
-)
-GO
-CREATE STATISTICS customerstats ON saphana.customrs ([CustomerName]) WITH FULLSCAN
-GO
--- Let's scan the table to make sure it works
-SELECT * FROM saphana.customers
-GO
--- Union with our current customers
---
-SELECT CustomerID, CustomerName, AccountOpenedDate, CustomerWebSite
-FROM saphana.customers
-UNION
-SELECT CustomerID, CustomerName, AccountOpenedDate, WebsiteURL
-FROM Sales.Customers
+USE [WideWorldImporters]
+GO
+CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+GO
+/* specify credentials to external data source
+* IDENTITY: user name for external source.
+* SECRET: password for external source.
+*/
+-- Need to change this a different user than SYSTEM
+DROP DATABASE SCOPED CREDENTIAL SAPHANACredentials
+GO
+CREATE DATABASE SCOPED CREDENTIAL SAPHANACredentials
+WITH IDENTITY = 'bwsaphana', Secret = 'Cprsql2019'
+GO
+/* LOCATION: Location string should be of format '://[:]'.
+* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
+* CREDENTIAL: the database scoped credential, created above.
+*/
+DROP EXTERNAL DATA SOURCE SAPHANAServer
+GO
+CREATE EXTERNAL DATA SOURCE SAPHANAServer
+WITH (
+LOCATION = 'odbc://bwsaphana',
+CONNECTION_OPTIONS = 'Driver={HDBODBC};ServerNode=bwsaphana:39041',
+PUSHDOWN = ON,
+CREDENTIAL = SAPHANACredentials
+)
+GO
+DROP SCHEMA saphana
+go
+CREATE SCHEMA saphana
+GO
+/* LOCATION: oracle table/view in 'database_name.schema_name.object_name' format
+* DATA_SOURCE: the external data source, created above.
+*/
+DROP EXTERNAL TABLE saphana.customers
+GO
+CREATE EXTERNAL TABLE saphana.customers
+(
+CustomerID int,
+CustomerName nvarchar(100) COLLATE Latin1_General_100_CI_AS,
+AccountOpenedDate date,
+CustomerWebSite nvarchar(256) COLLATE Latin1_General_100_CI_AS
+)
+ WITH (
+ LOCATION='[BWSAPHANA].[CUSTOMERS]',
+ DATA_SOURCE=SAPHANAServer
+)
+GO
+CREATE STATISTICS customerstats ON saphana.customrs ([CustomerName]) WITH FULLSCAN
+GO
+-- Let's scan the table to make sure it works
+SELECT * FROM saphana.customers
+GO
+-- Union with our current customers
+--
+SELECT CustomerID, CustomerName, AccountOpenedDate, CustomerWebSite
+FROM saphana.customers
+UNION
+SELECT CustomerID, CustomerName, AccountOpenedDate, WebsiteURL
+FROM Sales.Customers
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/justwwi_suppliers.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/justwwi_suppliers.sql
index 36d11100..254abcec 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/justwwi_suppliers.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/justwwi_suppliers.sql
@@ -1,83 +1,83 @@
-USE [JustWorldImporters]
-GO
--- Create a partition function
---
-CREATE PARTITION FUNCTION [PF_Supplier_Names](nvarchar(100))
-AS RANGE RIGHT FOR VALUES (N'Brooks Brothers', N'Old Suppliers -1', N'Old Suppliers -250000',
-N'Old Suppliers -500000', N'Old Suppliers -750000')
-GO
--- Create the partition scheme
---
-CREATE PARTITION SCHEME [PS_Supplier_Names]
-AS PARTITION [PF_Supplier_Names]
-TO ([PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY])
-GO
--- Create the table
---
-DROP TABLE [Suppliers]
-GO
-CREATE TABLE [Suppliers](
- [SupplierID] [int] NOT NULL,
- [SupplierName] [nvarchar](100) NOT NULL,
- [SupplierCategoryID] [int] NOT NULL,
- [PrimaryContactPersonID] [int] NOT NULL,
- [AlternateContactPersonID] [int] NOT NULL,
- [DeliveryMethodID] [int] NULL,
- [DeliveryCityID] [int] NOT NULL,
- [PostalCityID] [int] NOT NULL,
- [SupplierReference] [nvarchar](20) NULL,
- [BankAccountName] [nvarchar](50) NULL,
- [BankAccountBranch] [nvarchar](50) NULL,
- [BankAccountCode] [nvarchar](20) NULL,
- [BankAccountNumber] [nvarchar](20) NULL,
- [BankInternationalCode] [nvarchar](20) NULL,
- [PaymentDays] [int] NOT NULL,
--- [InternalComments] [nvarchar](max) NULL,
- [PhoneNumber] [nvarchar](20) NOT NULL,
- [FaxNumber] [nvarchar](20) NOT NULL,
- [WebsiteURL] [nvarchar](256) NOT NULL,
- [DeliveryAddressLine1] [nvarchar](60) NOT NULL,
- [DeliveryAddressLine2] [nvarchar](60) NULL,
- [DeliveryPostalCode] [nvarchar](10) NOT NULL,
--- [DeliveryLocation] [geography] NULL,
- [PostalAddressLine1] [nvarchar](60) NOT NULL,
- [PostalAddressLine2] [nvarchar](60) NULL,
- [PostalPostalCode] [nvarchar](10) NOT NULL,
- [LastEditedBy] [int] NOT NULL
- CONSTRAINT [PK_Purchasing_Suppliers] PRIMARY KEY CLUSTERED
-(
- [SupplierName] ASC
-) ON [PS_Supplier_names]([SupplierName]),
--- CONSTRAINT [UQ_Purchasing_Suppliers_ID] UNIQUE NONCLUSTERED
---(
--- [SupplierID] ASC
---)
-)
--- Insert some data
---
-TRUNCATE TABLE [Suppliers]
-GO
-INSERT INTO [Suppliers]
-VALUES (-1, 'Brooks Brothers', 4, -1, -2, 1, 24161, 24161, 'First US Clothing', 'Bank of New York Mellon', 'New York', NULL, '123456789', NULL, 30, '2121111111', '2121112222', 'brooksbrothers.com', '1 Broadway', NULL, '10004', '1 Broadway', NULL, '10004', 1)
-GO
-
--- Let's go insert 1M fake suppliers
---
-SET NOCOUNT ON
-GO
-BEGIN TRAN
-GO
-DECLARE @x int
-DECLARE @y nvarchar(100)
-SET @x = -2
-WHILE @x > -1000001
-BEGIN
- SET @y = 'Old Supplier'+CAST(@X as nvarchar(10))
- INSERT INTO Suppliers VALUES (@x, @Y, 4, -1, -2, 1, 24161, 24161, 'Unknown', 'Unknown', 'Unknown', NULL, '123456789', NULL, 0, '2121111111', '2121112222', 'Unknown', 'Unknown', NULL, '00000', 'Unknown', NULL, '00000', 1)
- SET @x = @x - 1
-END
-GO
-COMMIT TRAN
-GO
-SET NOCOUNT OFF
+USE [JustWorldImporters]
+GO
+-- Create a partition function
+--
+CREATE PARTITION FUNCTION [PF_Supplier_Names](nvarchar(100))
+AS RANGE RIGHT FOR VALUES (N'Brooks Brothers', N'Old Suppliers -1', N'Old Suppliers -250000',
+N'Old Suppliers -500000', N'Old Suppliers -750000')
+GO
+-- Create the partition scheme
+--
+CREATE PARTITION SCHEME [PS_Supplier_Names]
+AS PARTITION [PF_Supplier_Names]
+TO ([PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY])
+GO
+-- Create the table
+--
+DROP TABLE [Suppliers]
+GO
+CREATE TABLE [Suppliers](
+ [SupplierID] [int] NOT NULL,
+ [SupplierName] [nvarchar](100) NOT NULL,
+ [SupplierCategoryID] [int] NOT NULL,
+ [PrimaryContactPersonID] [int] NOT NULL,
+ [AlternateContactPersonID] [int] NOT NULL,
+ [DeliveryMethodID] [int] NULL,
+ [DeliveryCityID] [int] NOT NULL,
+ [PostalCityID] [int] NOT NULL,
+ [SupplierReference] [nvarchar](20) NULL,
+ [BankAccountName] [nvarchar](50) NULL,
+ [BankAccountBranch] [nvarchar](50) NULL,
+ [BankAccountCode] [nvarchar](20) NULL,
+ [BankAccountNumber] [nvarchar](20) NULL,
+ [BankInternationalCode] [nvarchar](20) NULL,
+ [PaymentDays] [int] NOT NULL,
+-- [InternalComments] [nvarchar](max) NULL,
+ [PhoneNumber] [nvarchar](20) NOT NULL,
+ [FaxNumber] [nvarchar](20) NOT NULL,
+ [WebsiteURL] [nvarchar](256) NOT NULL,
+ [DeliveryAddressLine1] [nvarchar](60) NOT NULL,
+ [DeliveryAddressLine2] [nvarchar](60) NULL,
+ [DeliveryPostalCode] [nvarchar](10) NOT NULL,
+-- [DeliveryLocation] [geography] NULL,
+ [PostalAddressLine1] [nvarchar](60) NOT NULL,
+ [PostalAddressLine2] [nvarchar](60) NULL,
+ [PostalPostalCode] [nvarchar](10) NOT NULL,
+ [LastEditedBy] [int] NOT NULL
+ CONSTRAINT [PK_Purchasing_Suppliers] PRIMARY KEY CLUSTERED
+(
+ [SupplierName] ASC
+) ON [PS_Supplier_names]([SupplierName]),
+-- CONSTRAINT [UQ_Purchasing_Suppliers_ID] UNIQUE NONCLUSTERED
+--(
+-- [SupplierID] ASC
+--)
+)
+-- Insert some data
+--
+TRUNCATE TABLE [Suppliers]
+GO
+INSERT INTO [Suppliers]
+VALUES (-1, 'Brooks Brothers', 4, -1, -2, 1, 24161, 24161, 'First US Clothing', 'Bank of New York Mellon', 'New York', NULL, '123456789', NULL, 30, '2121111111', '2121112222', 'brooksbrothers.com', '1 Broadway', NULL, '10004', '1 Broadway', NULL, '10004', 1)
+GO
+
+-- Let's go insert 1M fake suppliers
+--
+SET NOCOUNT ON
+GO
+BEGIN TRAN
+GO
+DECLARE @x int
+DECLARE @y nvarchar(100)
+SET @x = -2
+WHILE @x > -1000001
+BEGIN
+ SET @y = 'Old Supplier'+CAST(@X as nvarchar(10))
+ INSERT INTO Suppliers VALUES (@x, @Y, 4, -1, -2, 1, 24161, 24161, 'Unknown', 'Unknown', 'Unknown', NULL, '123456789', NULL, 0, '2121111111', '2121112222', 'Unknown', 'Unknown', NULL, '00000', 'Unknown', NULL, '00000', 1)
+ SET @x = @x - 1
+END
+GO
+COMMIT TRAN
+GO
+SET NOCOUNT OFF
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/readme.md b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/readme.md
index 6722e0cd..e59352b8 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/readme.md
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/readme.md
@@ -1,15 +1,15 @@
-# SQL Server 2019 Polybase example connecting to SQL Server 2008R2
-
-This demo shows you how to setup a SQL Server 2008R2 external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
-
-## Requirements
-
-Install SQL Server 2008R2. For my environment, I installed SQL Server 2008R2 on Windows Server 2008R2 using Azure with the gallery template provided on Azure. I put this VM in the same resource group of my SQL Server 2019 Windows Server head node so I would be on the same virtual network. I then added the IP address of my SQL Server 2008R2 server in the /etc/hosts file of my SQL Server 2019 head node (bwpolybase) with the convenient name of bwsql2008r2. I also created a new user called sqluser (password is in the scripts to create the external data source), created a database (defaults) called JustWorldImporters, and made sqluser a dbo of that database.
-
-## Demo Steps
-
-1. Connecting to the bwsql2008r2 server, I ran the script **justwwi_suppliers.sql** to create the table and insert some data. Notice in this script I used partitions so that when scanning the remote table from Polybase, each compute node will query a specific set of partitions.
-
-2. On my SQL Server 2019 head node (bwpolybase), I used the **sql2008r2_external_table.sql** script to create the database scoped credential, externaL data source, external table, and sample SELECT statements to query the external table and join it with local SQL Server 2019 tables in the WideWorldImporters database.
-
+# SQL Server 2019 Polybase example connecting to SQL Server 2008R2
+
+This demo shows you how to setup a SQL Server 2008R2 external data source and table with examples of how to query the data source and join data to local SQL Server 2019 data. The demo assumes you have installed a SQL Server 2019 Polybase scale out group as documented in the **fundamentals** folder of this overall demo.
+
+## Requirements
+
+Install SQL Server 2008R2. For my environment, I installed SQL Server 2008R2 on Windows Server 2008R2 using Azure with the gallery template provided on Azure. I put this VM in the same resource group of my SQL Server 2019 Windows Server head node so I would be on the same virtual network. I then added the IP address of my SQL Server 2008R2 server in the /etc/hosts file of my SQL Server 2019 head node (bwpolybase) with the convenient name of bwsql2008r2. I also created a new user called sqluser (password is in the scripts to create the external data source), created a database (defaults) called JustWorldImporters, and made sqluser a dbo of that database.
+
+## Demo Steps
+
+1. Connecting to the bwsql2008r2 server, I ran the script **justwwi_suppliers.sql** to create the table and insert some data. Notice in this script I used partitions so that when scanning the remote table from Polybase, each compute node will query a specific set of partitions.
+
+2. On my SQL Server 2019 head node (bwpolybase), I used the **sql2008r2_external_table.sql** script to create the database scoped credential, externaL data source, external table, and sample SELECT statements to query the external table and join it with local SQL Server 2019 tables in the WideWorldImporters database.
+
**BONUS**: Connect to your SQL Server 2008R2 instance and run SQL Profiler. Include the Hostname column in the trace. Notice when you scan the entire remote table all 3 nodes query individual partitions.
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/sql2008r2_external_table.sql b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/sql2008r2_external_table.sql
index 43c17df5..291a06f7 100644
--- a/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/sql2008r2_external_table.sql
+++ b/ModernizeYourDatabases2019/ModernizeSQL2019/Module 8 Activity - Data Virtualization/sqldatahub/sql2008r2/sql2008r2_external_table.sql
@@ -1,90 +1,90 @@
-USE [WideWorldImporters]
-GO
-CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
-GO
-/* specify credentials to external data source
-* IDENTITY: user name for external source.
-* SECRET: password for external source.
-*/
-DROP DATABASE SCOPED CREDENTIAL SqlServerCredentials
-GO
-CREATE DATABASE SCOPED CREDENTIAL SqlServerCredentials
-WITH IDENTITY = 'sqluser', Secret = '$cprsqlserver2019'
-GO
-/* LOCATION: Location string should be of format '://[:]'.
-* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
-* CREDENTIAL: the database scoped credential, created above.
-*/
-DROP EXTERNAL DATA SOURCE SQLServerInstance
-GO
-CREATE EXTERNAL DATA SOURCE SQLServerInstance
-WITH (
-LOCATION = 'sqlserver://bwsql2008r2',
-PUSHDOWN = ON,
-CREDENTIAL = SQLServerCredentials,
-CONNECTION_OPTIONS = 'UseDefaultEncryptionOptions=false' -- This is a workaround for a bug in SQL 2019 CTP 2.3 under investigation.
-)
-GO
-DROP SCHEMA sqlserver
-go
-CREATE SCHEMA sqlserver
-GO
-/* LOCATION: sql server table/view in 'database_name.schema_name.object_name' format
-* DATA_SOURCE: the external data source, created above.
-*/
-DROP EXTERNAL TABLE sqlserver.suppliers
-GO
-CREATE EXTERNAL TABLE sqlserver.suppliers
-(
- [SupplierID] [int] NOT NULL,
- [SupplierName] [nvarchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [SupplierCategoryID] [int] NOT NULL,
- [PrimaryContactPersonID] [int] NOT NULL,
- [AlternateContactPersonID] [int] NOT NULL,
- [DeliveryMethodID] [int] NULL,
- [DeliveryCityID] [int] NOT NULL,
- [PostalCityID] [int] NOT NULL,
- [SupplierReference] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [BankAccountName] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [BankAccountBranch] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [BankAccountCode] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [BankAccountNumber] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [BankInternationalCode] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [PaymentDays] [int] NOT NULL,
- [PhoneNumber] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [FaxNumber] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [WebsiteURL] [nvarchar](256) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [DeliveryAddressLine1] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [DeliveryAddressLine2] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [DeliveryPostalCode] [nvarchar](10) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [PostalAddressLine1] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [PostalAddressLine2] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
- [PostalPostalCode] [nvarchar](10) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
- [LastEditedBy] [int] NOT NULL
-)
- WITH (
- LOCATION='JustWorldImporters.dbo.Suppliers',
- DATA_SOURCE=SqlServerInstance
-)
-GO
-CREATE STATISTICS SupplierNameStatistics ON sqlserver.suppliers ([SupplierName]) WITH FULLSCAN
-GO
--- Scan the table to make sure it works
---
-SELECT * FROM sqlserver.suppliers
-GO
-
--- Find a specific supplier
---
-SELECT * FROM sqlserver.suppliers where SupplierName = 'Brooks Brothers'
-GO
---
--- Find all former clothing suppliers and their city
-SELECT s.SupplierName, s.SupplierReference, c.cityname
-FROM sqlserver.suppliers s
-JOIN [Purchasing].[SupplierCategories] sc
-on s.SupplierCategoryID = sc.SupplierCategoryID
-and sc.SupplierCategoryName = 'Clothing Supplier'
-JOIN [Application].[Cities] c
-ON s.DeliveryCityID = c.CityID
+USE [WideWorldImporters]
+GO
+CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo'
+GO
+/* specify credentials to external data source
+* IDENTITY: user name for external source.
+* SECRET: password for external source.
+*/
+DROP DATABASE SCOPED CREDENTIAL SqlServerCredentials
+GO
+CREATE DATABASE SCOPED CREDENTIAL SqlServerCredentials
+WITH IDENTITY = 'sqluser', Secret = '$cprsqlserver2019'
+GO
+/* LOCATION: Location string should be of format '://[:]'.
+* PUSHDOWN: specify whether computation should be pushed down to the source. ON by default.
+* CREDENTIAL: the database scoped credential, created above.
+*/
+DROP EXTERNAL DATA SOURCE SQLServerInstance
+GO
+CREATE EXTERNAL DATA SOURCE SQLServerInstance
+WITH (
+LOCATION = 'sqlserver://bwsql2008r2',
+PUSHDOWN = ON,
+CREDENTIAL = SQLServerCredentials,
+CONNECTION_OPTIONS = 'UseDefaultEncryptionOptions=false' -- This is a workaround for a bug in SQL 2019 CTP 2.3 under investigation.
+)
+GO
+DROP SCHEMA sqlserver
+go
+CREATE SCHEMA sqlserver
+GO
+/* LOCATION: sql server table/view in 'database_name.schema_name.object_name' format
+* DATA_SOURCE: the external data source, created above.
+*/
+DROP EXTERNAL TABLE sqlserver.suppliers
+GO
+CREATE EXTERNAL TABLE sqlserver.suppliers
+(
+ [SupplierID] [int] NOT NULL,
+ [SupplierName] [nvarchar](100) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [SupplierCategoryID] [int] NOT NULL,
+ [PrimaryContactPersonID] [int] NOT NULL,
+ [AlternateContactPersonID] [int] NOT NULL,
+ [DeliveryMethodID] [int] NULL,
+ [DeliveryCityID] [int] NOT NULL,
+ [PostalCityID] [int] NOT NULL,
+ [SupplierReference] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [BankAccountName] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [BankAccountBranch] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [BankAccountCode] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [BankAccountNumber] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [BankInternationalCode] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [PaymentDays] [int] NOT NULL,
+ [PhoneNumber] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [FaxNumber] [nvarchar](20) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [WebsiteURL] [nvarchar](256) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [DeliveryAddressLine1] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [DeliveryAddressLine2] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [DeliveryPostalCode] [nvarchar](10) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [PostalAddressLine1] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [PostalAddressLine2] [nvarchar](60) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
+ [PostalPostalCode] [nvarchar](10) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
+ [LastEditedBy] [int] NOT NULL
+)
+ WITH (
+ LOCATION='JustWorldImporters.dbo.Suppliers',
+ DATA_SOURCE=SqlServerInstance
+)
+GO
+CREATE STATISTICS SupplierNameStatistics ON sqlserver.suppliers ([SupplierName]) WITH FULLSCAN
+GO
+-- Scan the table to make sure it works
+--
+SELECT * FROM sqlserver.suppliers
+GO
+
+-- Find a specific supplier
+--
+SELECT * FROM sqlserver.suppliers where SupplierName = 'Brooks Brothers'
+GO
+--
+-- Find all former clothing suppliers and their city
+SELECT s.SupplierName, s.SupplierReference, c.cityname
+FROM sqlserver.suppliers s
+JOIN [Purchasing].[SupplierCategories] sc
+on s.SupplierCategoryID = sc.SupplierCategoryID
+and sc.SupplierCategoryName = 'Clothing Supplier'
+JOIN [Application].[Cities] c
+ON s.DeliveryCityID = c.CityID
GO
\ No newline at end of file
diff --git a/ModernizeYourDatabases2019/README.md b/ModernizeYourDatabases2019/README.md
index f06a3615..829db69e 100644
--- a/ModernizeYourDatabases2019/README.md
+++ b/ModernizeYourDatabases2019/README.md
@@ -1,69 +1,69 @@
-
-
-# Workshop: Modernize Your Database with SQL Server 2019
-
-#### A Microsoft Course from the SQL Server team
-
-
-
-
About this Workshop
-
-Welcome to this Microsoft solutions workshop on *Modernizing Your Database with SQL Server 2019*. In this workshop, you'll learn how SQL Server 2017 and SQL Server 2019 can help you to modernizing your database workloads.
-
-This README.MD file explains how the workshop is laid out, what you will learn, and the technologies you will use in this solution.
-
-
-
-
Learning Objectives
-
-
-
Business Applications of this Workshop
-
-
-
-
Technologies used in this Workshop
-
-
-
-
Before Taking this Workshop
-
-
-
-
Setup
-
-
-
-
Workshop Details
-
-
-
-
Related Workshops
-
-
-
-
-
Workshop Modules
-
-This is a modular workshop, and in each section, you'll learn concepts, technologies, and processes to help you complete the solution.
-
-
Learn how SQL Server 2019 meets the needs of the modern developer incuding graph database, machine learning built into the database, and extending the T-SQL language with Java
Learn how to migrate to SQL Server 2019 and your next steps
-
-
-
-
-
-
Next Steps
-
+
+
+# Workshop: Modernize Your Database with SQL Server 2019
+
+#### A Microsoft Course from the SQL Server team
+
+
+
+
About this Workshop
+
+Welcome to this Microsoft solutions workshop on *Modernizing Your Database with SQL Server 2019*. In this workshop, you'll learn how SQL Server 2017 and SQL Server 2019 can help you to modernizing your database workloads.
+
+This README.MD file explains how the workshop is laid out, what you will learn, and the technologies you will use in this solution.
+
+
+
+
Learning Objectives
+
+
+
Business Applications of this Workshop
+
+
+
+
Technologies used in this Workshop
+
+
+
+
Before Taking this Workshop
+
+
+
+
Setup
+
+
+
+
Workshop Details
+
+
+
+
Related Workshops
+
+
+
+
+
Workshop Modules
+
+This is a modular workshop, and in each section, you'll learn concepts, technologies, and processes to help you complete the solution.
+
+
Learn how SQL Server 2019 meets the needs of the modern developer incuding graph database, machine learning built into the database, and extending the T-SQL language with Java
Learn how to migrate to SQL Server 2019 and your next steps
+
+
+
+
+
+
Next Steps
+
Next, Continue to Pre-Requisites
\ No newline at end of file
diff --git a/README.md b/README.md
index 985b29ca..78fd73af 100644
--- a/README.md
+++ b/README.md
@@ -1,10 +1,11 @@
# SQL Server Workshops
+## (https://aka.ms/sqlworkshops)
This site is a map of learning content produced by and curated by the SQL Server team in Microsoft Engineering. These materials are meant to be instructor-led, but you can work through the materials on a test system on your own if desired. You can view all materials directly in this interface, or you can [view the raw github site for this content here](https://github.com/Microsoft/sqlworkshops).
-See the license information at the bottom of this README.md file.
+*See the license information at the bottom of this README.md file*
Find a problem? Spot a bug? [Post an issue here](https://github.com/Microsoft/sqlworkshops/issues) and we'll try and fix it.
@@ -29,12 +30,11 @@ Find a problem? Spot a bug? [Post an issue here](https://github.com/Microsoft/sq
- [In Development](https://github.com/Microsoft/sqlworkshops)
-## Resources, Source Slide Decks and Code Examples
-
-- [PowerPoint Decks from SQL Server Team Sessions](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#decks)
-- [Sample Code](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#code)
-- [References and Resources](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#links)
+## Presentation Materials, Code, and References
+- [Presentation Materials - *(PowerPoint Decks, etc.)*](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#decks)
+- [Example Code](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#code)
+- [References and Tools from the Microsoft SQL Server Team](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#links)
## Learning how to self-learn
@@ -66,13 +66,13 @@ You can follow the steps below to clone individual files from a git repo using a
Example:
-```
+
For more information about `sparse checkout please` visit [this](https://stackoverflow.com/questions/23289006/on-windows-git-error-sparse-checkout-leaves-no-entry-on-the-working-directory) stackoverflow thread.
diff --git a/References/README.MD b/References/README.MD
index 57a40ec6..39bf0aac 100644
--- a/References/README.MD
+++ b/References/README.MD
@@ -1,6 +1,6 @@
-# References: Microsoft SQL Server and Machine Learning in SQL Server (On-prem and Azure)
+# Resources from the Microsoft SQL Server and Azure Data Services Teams
#### Microsoft References from the Azure Data Services Team
@@ -13,7 +13,7 @@ The Microsoft SQL Server organization (along with its parent, the Microsoft Azur
*(Note: These scripts, tools and information site are a not guarenteed to work in every situation, or to be placed into production without thoroughly testing on your own. Microsoft assumes no risk associated with your use of these tools. These sites have adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.)*
-
These PowerPoint Decks deal with SQL Server in general, for many different versions and editions. If you find a dead link, hit up the Issues link above to report it.
@@ -25,7 +25,6 @@ These PowerPoint Decks deal with SQL Server in general, for many different versi
-
SQL Server Samples Repository that demonstrate how to use Microsoft's SQL products including SQL Server, Azure SQL Database, and Azure SQL Data Warehouse
Report all you need to know about in-flight queries and any blocking in SQL Server. Also available as SQL Notebook
diff --git a/SQLonOpenShift/README.md b/SQLonOpenShift/README.md
index e0c05bec..4b99f72f 100644
--- a/SQLonOpenShift/README.md
+++ b/SQLonOpenShift/README.md
@@ -1,69 +1,128 @@
-
-
-# Workshop: SQL Server on OpenShift
-
-#### A Microsoft Course from the SQL Server team
-
-
-
-
About this Workshop
-
-Welcome to this Microsoft solutions workshop on *SQL Server on OpenShift*. In this workshop, you'll learn how deploy and use SQL Server on an OpenShift cluster.
-
-This README.MD file explains how the workshop is laid out, what you will learn, and the technologies you will use in this solution.
-
-
-
-
Learning Objectives
-
-
-
Business Applications of this Workshop
-
-
-
-
Technologies used in this Workshop
-
-
-
-
Before Taking this Workshop
-
-
-
-
Setup
-
-
-
-
Workshop Details
-
-
-
-
Related Workshops
-
-
-
-
-
Workshop Modules
-
-This is a modular workshop, and in each section, you'll learn concepts, technologies, and processes to help you complete the solution.
-
-
-
-Next, Continue to Pre-Requisites
\ No newline at end of file
+
+
+
+# Workshop: SQL Server on OpenShift
+
+#### A Microsoft Course from the SQL Server team
+
+
+
+
About this Workshop
+
+Welcome to this Microsoft solutions workshop on *SQL Server on OpenShift*. In this workshop, you'll learn how to deploy and use SQL Server containers on an OpenShift cluster.
+
+You will learn the basics of deployment, connection, query execution, performance, high availability, operators, and Always On Availability Groups.
+
+This README.MD file explains how the workshop is laid out, what you will learn, and the technologies you will use in this solution.
+
+The workshop is currently built to support SQL Server on OpenShift 3.11. A future version of the course will support OpenShift 4.0
+
+
+
+
Learning Objectives
+
+- Learn the basics of deploying SQL Server on an OpenShift cluster.
+- Learn how to connect and run queries against SQL Server deployed on OpenShift.
+- Learn performance capabilities of SQL Server deployed on OpenShift
+- Learn basic High Availability capabilities of SQL Server deployed on OpenShift
+- Learn how to use an operator to deploy and manage an Always On Availability Group on OpenShift.
+
+
+
Business Applications of this Workshop
+
+
+
+- Developers looking to deploy a database container for their applications on OpenShift
+- Database Administrators looking to understand how to deploy database platforms like SQL Server in a Kubernetes cluster using OpenShift.
+
+
+
Technologies used in this Workshop
+
+
+
+
+
+
Technology
Description
+
+
SQL Server
Database Platform produced by Microsoft
+
SQL Server 2019
Most current release of SQL Server currently in preview
+
Intelligent Query Processing
Automated query processing enhancements in SQL Server 2019
+
Always On Availability Group
SQL Server High Availability Disaster Recovery capabilities based on replicas
+
Linux
Operating system used in Containers and Container Orchestration
+
Docker
Engine for running and manage containers
+
Kubernetes
Orchestration and Management platform for Containers
+
OpenShift
Family of containerization software developed by Red Hat
+
OpenShift Platform
Kubernetes based platform for containers
+
oc
OpenShift CLI program. Similar to kubectl
+
project
Equivalent to Kubernetes namespace
+
Azure Data Studio
Graphical User Interface to execute T-SQL queries and manage SQL Server
+
SQL Server Command Line Tools
Utilities to connect and query SQL Server such as sqlcmd
+
+
+
+
+
+
Before Taking this Workshop
+
+
+
+To complete this workshop you will need the following:
+
+- A client computer connected to the Internet that has can run a Linux shell and can run SQL Server command line tools.
+- Access to a OpenShift 3.11 cluster
+- Access to all the scripts provided from this workshop from the GitHub repo.
+
+You might be taking this workshop from an instructor who will provide access to an OpenShift cluster.
+
+
+
Setup
+
+A complete Prerequisites [document](sqlonopenshift/00_Prereqs.md) exists as part of this workshop. You will be guided to go through Prerequisites in the Next Steps at the bottom of this page.
+
+
+
+
Workshop Details
+
+This workshop uses OpenShift, SQL Server 2019, Azure Data Studio, SQL Command Line Tools, and the OpenShift CLI (oc).
+
+
+
+
Primary Audience:
Administrators looking to learn how to deploy, use, and manage SQL Server on OpenShift
+
Secondary Audience:
Developers, Architects, and IT Pros
+
Level:
300
+
Type:
Self-Paced or In-Person
+
Length:
2 hours (self-paced) 4 hours (In-Person)
+
+
+
+
+
+
Related Workshops
+
+- [Modernize your Database with SQL Server 2019](https://github.com/Microsoft/sqlworkshops/tree/rgward/ModernizeYourDatabases2019)
+
+
+
+
Workshop Modules
+
+This is a modular workshop, and in each section, you'll learn concepts, technologies, and processes to help you complete the solution.
+
+**Proceed to Next Steps below to start the workhop.**
+
+
+In this module you will learn what the prerequisites are for this workshop.
+
+The workshop does not assume a deep working knowledge of SQL Server or OpenShift. It does have an assumption to know basics of using a Linux bash shell but all commands are provided to run the activities including scripts.
+
+In order to go through the activities of this workshop you will need the following:
+
+**Note**: It is possible your instructor will provide you with a client environment and full access to an OpenShift cluster including login credentials.
+
+- Access to an OpenShift 3.11 cluster. The workshop is currently designed for OpenShift 3.11 and has not been tested for OpenShift 4.0
+- Modules 1 through 4 require user privileges for the OpenShift cluster for **anyuid** at minimum. Module 5 currently requires cluster admin rights.
+- A client computer that has access to connect to the OpenShift cluster and has the following software installed
+
+1. A Linux bash shell
+2. The OpenShift CLI (oc.exe)
+3. Azure Data Studio - Minimum version is 1.5.2. Install from https://docs.microsoft.com/en-us/sql/azure-data-studio/download
+4. SQL Command Line Tools (sqlcmd). Check the **For Further Study** section for links to install these tools.
+5. git client (only needed if you do not have the latest version of the workshop provided to you by the instructor)
+6. In addition, the client computer must be able to connect to the Internet to download a sample file or your instructor must provide it for you (WideWorldImporters-Full.bak)
+
+The workshop currently supports a single node OpenShift cluster but can be run on a multiple cluster environment. Each user will need ~8Gb of RAM to run the containers in the workshop.
+
+Proceed to the Activity to go through the prerequisites.
+
+
+
+
+
+1. Download the latest version of the workshop from https://aka.ms/sqlworkshops. If you have used git clone to pull down the repo of the workshops, run **git pull** to get the latest version.
+
+2. Login to your OpenShift cluster by using the URL provided to you for the **openshiftConsoleUrl** in a web browser.
+
+3. You may get warnings from the web page saying "This site is not secure". Click Details and then "Go on to the webpage"
+
+4. You will be presented with a login screen like the following
+
+ 
+
+5. Type in the user name and password provided to you for OpenShift cluster access. Your instructor may call this **openshiftAdminUsername** and **openshiftPassword**
+
+6. You will now see a new web page like the following
+
+ 
+
+7. In the upper right hand corner, click on your user name and select Copy Login Command like the following
+
+ 
+
+8. You now have on your clipboard a complete oc login sytnax with a token. Open up a shell and paste in the copy (right click your mouse)
+
+ The command should look something like this
+
+ `oc login https://[masterconsoleaddress]:443 --token=[tokenstring]`
+
+ Hit enter
+
+ You should see results like the following and then placed back at the command prompt
+
+
Logged into "https://[masterconsoleurl]:443" as "ocpadmin" using the token provided.
+
+You have access to the following projects and can switch between them with 'oc project projectname':
+
+ * default
+ kube-public
+ kube-service-catalog
+ kube-system
+ management-infra
+ openshift
+ openshift-ansible-service-broker
+ openshift-console
+ openshift-infra
+ openshift-logging
+ openshift-monitoring
+ openshift-node
+ openshift-sdn
+ openshift-template-service-broker
+ openshift-web-console
+
+Using project "default".
+
+
+You have now successfully logged into the OpenShift Cluster and can proceed with Next Steps below.
+
+
+
+
For Further Study
+
+- [Red Hat OpenShift](https://www.openshift.com/)
+- [oc CLI downloads](https://www.okd.io/download.html)
+- [Azure Data Studio](https://docs.microsoft.com/en-us/sql/azure-data-studio/what-is)
+- [SQL Command Line Tools for Linux](https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-tools)
+- [SQL Command Line Tools for MacOS](https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-tools?view=sql-server-2017#macos)
+- [SQL Command Line Tools for Windows](https://www.microsoft.com/en-us/download/details.aspx?id=53591)
+
+
+
+
Next Steps
+
+Next, Continue to
+Deploy SQL Server on OpenShift.
diff --git a/SQLonOpenShift/sqlonopenshift/01_Deploy.md b/SQLonOpenShift/sqlonopenshift/01_Deploy.md
new file mode 100644
index 00000000..f41d6905
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/01_Deploy.md
@@ -0,0 +1,137 @@
+
+
+# Workshop: SQL Server on OpenShift
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Deploy SQL Server on OpenShift
+
+You'll cover the following topics in this Module:
+
+
+
+In this module you will learn the fundamentals of how to deploy SQL Server on OpenShift. The SQL Server engine can be run in a container and on a Kubernetes cluster. In this module, you will learn the basic steps for how to deploy a SQL Server container in an OpenShift cluster. This module is required to complete Modules 2, 3, and 4 of the workshop.
+
+In the deployment of SQL Server on OpenShift, you will:
+
+- Create a new project to deploy SQL Server.
+- Create a secret for the password to login to SQL Server
+- Create a PersistentVolumeClaim where SQL Server databases and files will be stored
+- Deploy a SQL Server container using a declarative .yaml file which will include a LoadBalancer service to connect to SQL Server.
+
+ Kubernetes and OpenShift are *declarative* systems. You program how to run and manage objects in OpenShift using a command line tool like oc. yaml files are used to declare how to build and manage objects through the Kubernetes API Server.
+
+In the activity for this module and others in the workshop you will often complete exercises by *executing* yaml files using a command like
+
+`oc apply -f .yaml`
+
+Inside each yaml file is declarations of objects to deploy or commands to execute.
+
+Proceed to the Activity to learn these deployment steps.
+
+
+
+
+
+Follow these steps to deploy SQL Server on OpenShift:
+
+1. Open a shell prompt and change directories to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/01_deploy** folder.
+
+2. You should have already logged into the OpenShift cluster using instructions from the Prerequisites.
+
+3. If you are running this workshop as a cluster admin and the instructor did not create a new project, then create a new project called **mssql** with the following command or execute the **step1_create_project.sh** script.
+
+ `oc new-project mssql`
+
+ When this completes, you should see the following messages and be placed back at the shell prompt
+
+
Now using project "mssql" on server "https://[servername]".
+ You can add applications to this project with the 'new-app' command. For example, try:
+ oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git
+ to build a new example application in Ruby.
+
+3. Create a secret to store the System Administrator (sa) password. For this workshop, you will be connecting as the sa login. In production SQL Server environments, you would not normally use the sa login. Use the following command or execute the **step2_create_secret.sh** script:
+
+ `oc create secret generic mssql --from-literal=SA_PASSWORD="Sql2019isfast"`
+
+ When this completes you should see the following message and be placed back at the shell prompt
+
+
secret/mssql created
+
+ **IMPORTANT: Take note of the value for SA_PASSWORD (without the quotes). The scripts in all modules use this password but you may need it to interactively work with SQL Server.**
+
+4. Create a PersistentVolumeClaim to store SQL Server databases and files. Use the following command or execute the **step3_storage.sh** script:
+
+ **Note**: The PersistentVolumeClaim created assumes the default StorageClass of the OpenShift cluster.
+
+ `oc apply -f storage.yaml`
+
+ When this completes you should see the following message and be placed back at the shell prompt
+
+
persistentvolumeclaim/mssql-data created
+
+5. Deploy SQL Server using the following command or the **step4_deploy_sql.sh** script:
+
+ `oc apply -f sqldeployment.yaml`
+
+ When this completes, you should see the following messages and be placed back at the shell prompt
+
+
deployment.apps/mssql-deployment created
+ service/mssql-service created
+
+ Take a minute to browse the **sqldeployment.yaml** file to see key pieces of how SQL Server was deployed including details of the container image, arguments, label to "tag" the deployment, what PersistentVolumeClaim to use (from the previous step) and a LoadBalancer service attached to this pod.
+
+ At this time, you have submitted a deployment, which is a logical collection of objects including a pod, container, and LoadBalancer service. OpenShift will schedule a SQL Server container in a pod on a node on the cluster. Proceed to the next step to check on whether the deployment was successful.
+
+6. Verify the SQL Server deployment has succeeded by running the following command:
+
+ `oc get deployment mssql-deployment`
+
+ When the value of **AVAILABLE** becomes 1, the deployment is successful including a Running Container. Depending on the load of your cluster and whether the container image of SQL Server is already present, the deployment could take several minutes.
+
+ You can run the following command to check on the status of the pod and LoadBalancer service:
+
+ `oc get all`
+
+ It is possible for the deployment to be successful but the LoadBalancer is not created. When everything about this deployment is successful, the STATUS of the pod is **Running** and the LoadBalancer service has a valid IP address for EXTERNAL-IP.
+
+7. The SQL Server database engine produces a file called the ERRORLOG file when it starts and can be used to gather interesting information about SQL Server or be used for troubleshooting. Since the output of the ERRORLOG is sent to stdout as part of running SQL Server as a container you can view these logs using OpenShift commands. Run the following commands to view the ERRORLOG or execute the script **step5_get_errorlog.sh**:
+
+ `POD=$(oc get pods | grep mssql | awk {'print $1'})`
+`oc logs $POD`
+
+ The ERRORLOG will scroll across the screen and you can scroll up in your shell to see all the output.
+
+A pod with a SQL Server container is now deployed and a LoadBalancer service is attached to the pod. The LoadBalancer will be a key component to connecting to SQL Server no matter where the pod is running in the cluster.
+
+
Do not proceed to the next Module until you have a valid IP address for the EXTERNAL-IP value for the LoadBalancer service. The value will say pending while it is being created. One some OpenShift cluster systems this process can take a few minutes.
+
+You can now proceed to Next Steps to Connect and Query SQL Server on OpenShift.
+
+
+
+
+
+
For Further Study
+
+- [Quickstart: Run SQL Server container images with Docker](https://docs.microsoft.com/en-us/sql/linux/quickstart-install-connect-docker?view=sql-server-linux-ver15&pivots=cs1-bash)
+- [Deploy a SQL Server container in Kubernetes with Azure Kubernetes Services (AKS)](https://docs.microsoft.com/en-us/sql/linux/tutorial-sql-server-containers-kubernetes?view=sql-server-2017)
+- [How Deployments Work in OpenShift](https://docs.openshift.com/online/dev_guide/deployments/how_deployments_work.html)
+- [Using a LoadBalancer in OpenShift](https://docs.openshift.com/container-platform/3.11/dev_guide/expose_service/expose_internal_ip_load_balancer.html)
+- [Using PersistentVolumeClaims in OpenShift](https://docs.openshift.com/container-platform/3.11/dev_guide/persistent_volumes.html)
+
+
+
+
+
+SQL Server provides several tools to connect and execute queries. Applications can use a variety of languages including C++, .Net, node.js, and Java. To see examples of how to write applications to connect to SQL Server, visit https://aka.ms/sqldev.
+
+The simplest method to connect to SQL Server deployed on OpenShift is to use the command line tool **sqlcmd**, which is natively built for Windows, Linux, and MacOS systems. The Prerequisites for this workshop provided instructions for how to install the SQL Command Line tools including sqlcmd. In some deliveries of this workshop, sqlcmd may already be installed.
+
+To connect to SQL Server, you need first know:
+
+- The name of the server or IP address hosting SQL Server
+- The port number (if SQL was configured to not use the default port of 1433)
+- A login name
+- A password
+
+In order to complete the Activities of this Module, you must first complete the Activity in **Module 1 Deploy SQL Server on OpenShift**. In Module 1, you deployed a pod with a SQL Server Container. The Activities in this module will help you determine the above information on how to connect to your SQL Server deployment.
+
+Proceed to the Activity to learn the fundamentals of connecting to SQL Server deployed on OpenShift.
+
+
+
+Follow these steps to connect to SQL Server deployed on OpenShift:
+
+1. Open a shell prompt and change to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/02_query** directory.
+
+2. The most fundamental method to connect to SQL Server is to use sqlcmd and execute the T-SQL query SELECT @@version to determine the version of SQL Server installed. Think of this as the *Hello World* test of SQL Server. Execute the following commands from your shell prompt or use the script **step1_test_sql.sh**:
+`SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+`PORT=31433`
+`sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT @@version"`
+
+ In this example, the IP address and the port of the Load Balancer service is used to connect to SQL Server as the IP address of the pod may change if OpenShift has to restart or move the pod. You used the -Q parameter of sqlcmd which allows you to specify a T-SQL statement(s) directly from the command line.
+
+ The output of this command will look something like this should the connection and query be successful
+
+
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Microsoft SQL Server 2019 (CTP2.4) - 15.0.1400.75 (X64)
+ Mar 16 2019 11:53:26
+ Copyright (C) 2019 Microsoft Corporation
+ Developer Edition (64-bit) on Linux (Red Hat Enterprise Linux Server 7.6 (Maipo)) X64
+ (1 rows affected)
+
+Your version information may be different depending if this workshop is updated to use newer versions of SQL Server.
+
+Proceed to the next activity to work with databases with SQL Server by restoring a backup.
+
+
+
+
+
+SQL Server is all about data so one of the first things any user wants to do is create a database, populate some data into it, and run queries. For the purposes of this workshop, a simpler method to get going quickly is to restore a backup of an existing database and run queries. For this workshop, you will use the sample database provided by Microsoft called **WideWorldImporters**.
+
+Proceed to the Activity to learn how to restore a database backup to SQL Server deployed on OpenShift.
+
+
+
+Follow these steps to restore a database backup to SQL Server deployed on OpenShift:
+
+1. Open a shell prompt and change to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/02_query** directory.
+
+2. If your workshop does not already include a copy of the backup of the WideWorldImporters database (a file called WideWorldImporters-Full.bak) execute the script **getwwi.sh** to download the backup.
+
+3. In Module 1, you deployed SQL Server on OpenShift which is running in a container. So to restore a backup of WideWorldImporters, you must copy the backup you downloaded into the filesystem of the container. In this step, you will copy the backup into the container folder /var/opt/mssql. The SQL Server database engine by default has permissions to read backup files in this folder to restore the database. Execute the following command to copy the database backup file into the container or execute the script **step2_copy_backup_into_container.sh**
+
+ `POD=$(oc get pods | grep mssql-deployment | awk {'print $1'})`
+ `oc cp ./WideWorldImporters-Full.bak $POD:/var/opt/mssql/WideWorldImporters-Full.bak`
+
+ Depending on the speed of the connectivity of your computer, the process to copy the file into the container could take several minutes.
+
+ When this command completes, there is no output. You are placed back a the shell prompt.
+
+4. Now you will use the T-SQL **RESTORE DATABASE** command to restore the database backup. Execute the following commands using the sqlcmd tool or execute the script **step3_restore_backup.sh**:
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -irestorewwi.sql`
+
+ In this example, you used the -i parameter for sqlcmd to execute a *script* with the RESTORE DATABASE command. You can examine the contents of the restorewwi.sql T-SQL script to see the example syntax using `cat restorewwi.sql` from the shell.
+
+ The WideWorldImporters backup you downladed was created on SQL Server 2016 on Windows (one of the great stories about SQL Server is compatibility. A backup on Windows can be restored on Linux and vice versa). SQL Server 2019 will automatically detect the older version and upgrade the database. This is why the RESTORE command can take a few minutes to execute. When the command completes the output to the shell prompt will scroll across several lines but end with something like the following:
+
+
Database 'WideWorldImporters' running the upgrade step from version 895 to version 896 Database 'WideWorldImporters' running the upgrade step from version 896 to version 897 RESTORE DATABASE successfully processed 58455 pages in 30.797 seconds (14.828 MB/sec).
+
+ Notice the end of the restore command displays how many database pages were restored (SQL Server stores data in 8K pages) and the duration it took to restore the database. The database has been restored, brought online, and is available to run queries.
+
+
+
+
+
+
+The T-SQL language allows all types of queries to be executed against your data including basic CRUD operations (Create, Read, Update, and Delete) and a host of other functionality. You can find the complete T-SQL reference at https://docs.microsoft.com/en-us/sql/t-sql/language-reference.
+
+If you are given a database backup to restore, one of the first things you want to do is explore what is in the database. SQL Server provides a rich set of metadata about the database through *catalog views*. This allows you to find out what tables, columns, and other objects exist in a database.
+
+In addition, to find out what data exists within tables in the database, you will use the most often used T-SQL command SELECT against tables you have permissions to query.
+
+SQL Server also provides a robust set of *dynamic management views* (DMV) through SELECT statements to query the state of the database engine.
+
+Proceed to the Activity to learn how to run queries against a SQL Server deployed on OpenShift including catalog views, data within the database, and Dynamic Management Views.
+
+
+
+Follow these steps to execute example queries against a SQL Server deployed on OpenShift:
+
+ **Note**: These steps assume you have followed the previous sections of this module.
+
+1. Open a shell prompt and change to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/02_query** directory.
+
+2. Run the following set of commands to find out what user tables are in the WideWorldImporters database backup you restored. You can also use the script **step4_find_tables.sh**:
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"USE WideWorldImporters;SELECT name, SCHEMA_NAME(schema_id) as schema_name FROM sys.objects WHERE type = 'U' ORDER BY name" -Y30`
+
+ When the command completes, the output should scroll across your screen like this
+
+
+
+ You can see from the bottom of this output that there are 48 tables in this database. The output includes two columns, one is for the name of the table, and other is for the *schema* of the table. A schema allows you to organize objects in a group for applications, provide isolation of objects at a group (e.g. there can be the same table name in two different schemas), and security permissions at a group level. In order to query data from a table you need to know the name of the schema and have permissions for that schema.
+
+ **Note**: In the above use of sqlcmd, the -Y30 parameter is used to ensure results are displayed as fixed width characters no longer than 30 characters for readability.
+
+3. Run the following commands to query data from the **People** table in the **Application** schema. In this database, the Application schema is used for tables that are used across the application and the People table holds data for any persons used across the Application for the WideWorldImporters company. You can also execute the script **step5_find_people.sh** to run these commands:
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"USE WideWorldImporters;SELECT TOP 10 FullName, PhoneNumber, EmailAddress FROM [Application].[People] ORDER BY FullName;" -Y30`
+
+ Your results should look like the following
+
+
+
+ In this example, you used the TOP 10 option of a SELECT statement to only retrieve the first 10 rows in the People table and the ORDER BY clause to sort the results by name (default ascending)).
+
+ These results contain privacy information. You can review a feature of SQL Server called Dynamic Data Masking to mask privacy information from application users. See more at https://docs.microsoft.com/en-us/sql/relational-databases/security/dynamic-data-masking.
+
+4. Everything in SQL Server is a query even to gather dynamic information about the running state of the SQL Server database engine. Run the following commands to see insights into running sessions, queries, and memory consumption. You can also run the script **step6_dmv.sh** to execute these commands.
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -idmv.sql`
+
+ This an example of running a T-SQL script with three *batches*. A batch is a series of T-SQL statements and with a script you can submit several batches from a single file. The contents of **dmv.sql** look like this:
+
+
+ ```sql
+ SELECT session_id, login_time, host_name, program_name, reads, writes, cpu_time
+ FROM sys.dm_exec_sessions WHERE is_user_process = 1
+ GO
+ SELECT dr.session_id, dr.start_time, dr.status, dr.command
+ FROM sys.dm_exec_requests dr
+ JOIN sys.dm_exec_sessions de
+ ON dr.session_id = de.session_id
+ AND de.is_user_process = 1
+ GO
+ SELECT cpu_count, committed_kb from sys.dm_os_sys_info
+ GO
+ ```
+ The output should look something similar to this
+
+
+
+The first T-SQL batch provides information about what sessions are connected to SQL Server with information about the session. The second T-SQL batch provides information about active queries against SQL Server. And the third batch provides information about how many CPUs SQL Server detected and how much memory the database engine has consumed. There are many Dynamic Management Views and more columns available than in the examples you used. You can read about all DMVs at https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/system-dynamic-management-views.
+
+5. As an optional exercise, you can connect with sqlcmd and run ad-hoc queries against SQL Server. I recommend you only run SELECT statements to read from SQL Server so you will not have issues with other Modules. Run the following commands to get a prompt to interactively use sqlcmd:
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -dWideWorldImporters`
+
+ You should be presented with a prompt like this
+
+
1>
+
+ You can now run T-SQL statement interactively with sqlcmd. By typing in a query and hitting Enter, you can type in the keyword **GO** and hit Enter to execute a query. Type in the keyword **exit** to leave sqlcmd.
+
+You can now proceed to Next Steps to learn more about SQL Server Performance.
+
+
+
+
For Further Study
+
+- [The sqlcmd utility](https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility)
+- [Query Data with SQL Server](https://docs.microsoft.com/en-us/sql/lp/sql-server/query-data)
+- [Backup and Restore of SQL Server Databases](https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/back-up-and-restore-of-sql-server-databases)
+- [SQL Server Catalog Views](https://docs.microsoft.com/en-us/sql/relational-databases/system-catalog-views/catalog-views-transact-sql)
+- [SQL Server Dynamic Management Views](https://docs.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/system-dynamic-management-views)
+- [The WideWorldImporters sample database](https://docs.microsoft.com/en-us/sql/samples/wide-world-importers-what-is)
+
+
+
+
Next Steps
+
+Next, Continue to SQL Server Performance.
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/dmv.sql b/SQLonOpenShift/sqlonopenshift/02_query/dmv.sql
new file mode 100644
index 00000000..aaca3cdf
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/dmv.sql
@@ -0,0 +1,11 @@
+SELECT session_id, login_time, host_name, program_name, reads, writes, cpu_time
+FROM sys.dm_exec_sessions WHERE is_user_process = 1
+GO
+SELECT dr.session_id, dr.start_time, dr.status, dr.command
+FROM sys.dm_exec_requests dr
+JOIN sys.dm_exec_sessions de
+ON dr.session_id = de.session_id
+AND de.is_user_process = 1
+GO
+SELECT cpu_count, committed_kb from sys.dm_os_sys_info
+GO
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/getwwi.sh b/SQLonOpenShift/sqlonopenshift/02_query/getwwi.sh
new file mode 100644
index 00000000..9dfd98fa
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/getwwi.sh
@@ -0,0 +1 @@
+wget --no-clobber --no-hsts https://github.com/Microsoft/sql-server-samples/releases/download/wide-world-importers-v1.0/WideWorldImporters-Full.bak
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/restorewwi.sql b/SQLonOpenShift/sqlonopenshift/02_query/restorewwi.sql
new file mode 100644
index 00000000..9bd6920c
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/restorewwi.sql
@@ -0,0 +1,7 @@
+restore database WideWorldImporters from disk = '/var/opt/mssql/WideWorldImporters-Full.bak' with
+move 'WWI_Primary' to '/var/opt/mssql/data/WideWorldImporters.mdf',
+move 'WWI_UserData' to '/var/opt/mssql/data/WideWorldImporters_UserData.ndf',
+move 'WWI_Log' to '/var/opt/mssql/data/WideWorldImporters.ldf',
+move 'WWI_InMemory_Data_1' to '/var/opt/mssql/data/WideWorldImporters_InMemory_Data_1'
+go
+
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/step1_test_sql.sh b/SQLonOpenShift/sqlonopenshift/02_query/step1_test_sql.sh
new file mode 100644
index 00000000..26de9969
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/step1_test_sql.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT @@version"
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/step2_copy_backup_into_container.sh b/SQLonOpenShift/sqlonopenshift/02_query/step2_copy_backup_into_container.sh
new file mode 100644
index 00000000..3ab7865f
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/step2_copy_backup_into_container.sh
@@ -0,0 +1,2 @@
+POD=$(oc get pods | grep mssql-deployment | awk {'print $1'})
+oc cp ./WideWorldImporters-Full.bak $POD:/var/opt/mssql/WideWorldImporters-Full.bak
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/step3_restore_backup.sh b/SQLonOpenShift/sqlonopenshift/02_query/step3_restore_backup.sh
new file mode 100644
index 00000000..8dc039ce
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/step3_restore_backup.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -irestorewwi.sql
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/step4_find_tables.sh b/SQLonOpenShift/sqlonopenshift/02_query/step4_find_tables.sh
new file mode 100644
index 00000000..90b780f2
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/step4_find_tables.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"USE WideWorldImporters;SELECT name, SCHEMA_NAME(schema_id) as schema_name FROM sys.objects WHERE type = 'U' ORDER BY name" -Y30
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/step5_find_people.sh b/SQLonOpenShift/sqlonopenshift/02_query/step5_find_people.sh
new file mode 100644
index 00000000..d8d3c868
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/step5_find_people.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"USE WideWorldImporters;SELECT TOP 10 FullName, PhoneNumber, EmailAddress FROM [Application].[People] ORDER BY FullName;" -Y30
diff --git a/SQLonOpenShift/sqlonopenshift/02_query/step6_dmv.sh b/SQLonOpenShift/sqlonopenshift/02_query/step6_dmv.sh
new file mode 100644
index 00000000..ea8fdc86
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/02_query/step6_dmv.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -idmv.sql
diff --git a/SQLonOpenShift/sqlonopenshift/03_Performance.md b/SQLonOpenShift/sqlonopenshift/03_Performance.md
new file mode 100644
index 00000000..45130575
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/03_Performance.md
@@ -0,0 +1,297 @@
+
+
+# Workshop: SQL Server on OpenShift
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
SQL Server Performance
+
+You'll cover the following topics in this Module:
+
+
+
+In this module you will learn about the Intelligent Query processing capabilities new to SQL Server 2019. You will use activities to learn these concepts against the SQL Server container you have deployed in OpenShift. This demonstrates the compatibility of the SQL Server engine across multiple platforms as this entire module could be used against SQL Server 2019 running on Windows, Linux, and containers.
+
+Intelligent Query processing is a suite of features built into the query processor for SQL Server 2019 allowing developers and data professionals to accelerate database performance automatically without any application changes. T-SQL queries simply need to be run with a database compatibility level of 150 to take advantage of these enhancements.
+
+You can read more about database compatibility at https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-database-transact-sql-compatibility-level?view=sql-server-2017#compatibility-levels-and-sql-server-upgrades.
+
+The following is a diagram showing the features of Intelligent Query Processing including capabilities from SQL Server 2017
+
+
+
+You can read the documentation for a description and example of all of these features at https://docs.microsoft.com/en-us/sql/relational-databases/performance/intelligent-query-processing.
+
+**Note**: One of the features of Intelligent Query Processing, approximate count distinct, does not require database compatibility of 150.
+
+Proceed to the Activity to learn an example of how Intelligent Query Processing can accelerate query performance automatically with no application changes.
+
+
+
+
+
+Follow these steps to learn more about SQL Server Intelligent Query Processing. **Note**: this module assumes you have completed all the steps in Module 01 and 02.
+
+In this activity, you will learn how to use the built-in capabilities of Intelligent Query Processing in SQL Server 2019 simply by changing the database compatibility of WideWorldImporters to version 150 with no application changes.
+
+You have been provided a stored procedure called **CustomerProfits** which you will deploy in the **Facts** schema of the WideWorldImporters database. The stored procedure uses a concept called a table variable to store interim results from a user table and then use that table variable to join with other data in the WideWorldImporters database. In past releases of SQL Server, this design pattern can be a problem as SQL Server would always estimate the table variable only contains 1 row of data. This can cause issues with building the optimal query plan for fast performance.
+
+SQL Server 2019 Intelligent Query Processing includes a capability called *deferred table variable compilation* to improve the performance of stored procedures like these by ensuring the stored procedure is created in a database with a compatibility level of 150, which is the default for SQL Server 2019.
+
+The WideWorldImporters database example was created with SQL Server 2016 which had a default database compatibility level of 130. When a database is restored from a previous version oF SQL Server, the compatibility level of the database is preserved to help reduce the risk of upgrades.
+
+You will observe the performance of the CustomerProfits stored procedure with database compatibility level of 130 on SQL Server 2019. You will then compare the performance of the same procedure with no changes with a database compatibility of 150 which will enable the query processor to use deferred table variable compilation.
+
+You will be running a series of commands and SQL Server T-SQL statements to observe performance.
+
+1. To connect to the SQL Server deployed on OpenShift, run the following command to get the IP address and port of the LoadBalancer service associated with the SQL Server container.
+
+ `oc get service mssql-service`
+
+ You should see results like the following
+
+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+ mssql-service LoadBalancer 172.30.166.56 168.61.45.217 31433:30738/TCP 1h
+
+ Take note of the EXTERNAL-IP and PORT (the number before the ":"). You will use these to connect to SQL Server throughout this module.
+
+2. Launch the Azure Data Studio application. Look for the icon
+
+
+
+3. The first time you launch Azure Data Studio, you may see the following choices. For the purposes of this workshop, select No to not load the preview feature and use x to close out the 2nd choice to collect usage data
+
+
+
+4. You will now be presented with the following screen to enter in your connection details for SQL Server. For Server, put in the values for EXTERNAL IP, PORT from step 1 above. Change the Authentication type to SQL Login, Put in a User name of sa with the Password you used for the secret in Module 01 when you deployed SQL Server. I also recommend you click the checkbox for Remember Password so you will not have to do it again for future connections. Click the Connect button to connect. An example of a connection looks like the following
+
+
+
+5. A successful connection should look like the following
+
+ 
+
+6. Open the script **proc.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder
+
+ The stored procedure looks like the following
+
+ ```sql
+ USE WideWorldImporters
+ GO
+ CREATE or ALTER PROCEDURE [Sales].[CustomerProfits]
+ AS
+ BEGIN
+ -- Declare the table variable
+ DECLARE @ilines TABLE
+ ([InvoiceLineID] [int] NOT NULL primary key,
+ [InvoiceID] [int] NOT NULL,
+ [StockItemID] [int] NOT NULL,
+ [Description] [nvarchar](100) NOT NULL,
+ [PackageTypeID] [int] NOT NULL,
+ [Quantity] [int] NOT NULL,
+ [UnitPrice] [decimal](18, 2) NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [TaxAmount] [decimal](18, 2) NOT NULL,
+ [LineProfit] [decimal](18, 2) NOT NULL,
+ [ExtendedPrice] [decimal](18, 2) NOT NULL,
+ [LastEditedBy] [int] NOT NULL,
+ [LastEditedWhen] [datetime2](7) NOT NULL
+ )
+
+ -- Insert all the rows from InvoiceLines into the table variable
+ INSERT INTO @ilines SELECT * FROM Sales.InvoiceLines
+ -- Find my total profile by customer
+ SELECT TOP 1 COUNT(i.CustomerID) as customer_count, SUM(il.LineProfit) as total_profit
+ FROM Sales.Invoices i
+ INNER JOIN @ilines il
+ ON i.InvoiceID = il.InvoiceID
+ GROUP By i.CustomerID
+ END
+ GO
+ ```
+
+ This procedure uses a table variable populated from a user table and then joins it with a user table to provide output. T-SQL functions like COUNT and SUM are often seen in analytic queries that benefit from Intelligent Query Processing. **Note**: In this example the TOP 1 T-SQL syntax is used so that the procedure only produces 1 row. This is only done to make the output easier to read using this workshop and demo since this procedure will be executed multiple times. Normal execution of this procedure may not include TOP.
+
+ Click the Run button to execute the script. You will be prompted to pick the connection to execute the script. Select the connection you created in Step 4.
+
+ When you execute this script the results should look like the following
+
+ 
+
+7. You have been told this procedure executes fairly quickly with a single execution in a few seconds but over several iterations, the total duration is over 20 seconds, which is not acceptable to the application.
+
+ Open the script **repro130.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder.
+
+ The script looks like the following
+
+ ```sql
+ USE master
+ GO
+ ALTER DATABASE wideworldimporters SET compatibility_level = 130
+ GO
+ USE WideWorldImporters
+ GO
+ SET NOCOUNT ON
+ GO
+ EXEC [Sales].[CustomerProfits]
+ GO 25
+ SET NOCOUNT OFF
+ GO
+ ```
+ The script will ensure the database is in a compatibility mode that is less than 150 so Intelligent Query Processing will NOT be enabled. The script also turns off rowcount messages to be returned to the client to reduce network traffic for this test. Then the script executes the stored procedure. Notice the syntax of **GO 25**. This is a client tool tip that says to run the batch 25 times (avoids having to construct a loop)
+
+ Click the Run button to execute the script to observe the results. Choose the connection by clicking on the IP,PORT you created for the SQL Server container and click Connect.
+
+ You will see while the query is executing in the bottom status bar the current elapsed execution time, the server connection details, a status of **Executing Query**, and number of rows being returned to the client.
+
+ 
+
+ The query should complete in well over 30 seconds. Given result sets are being sent back to the client, it can take even longer depending on connectivity speeds of your client workstation.
+
+ The final results of Azure Data Studio should look like this
+
+ 
+
+ You can scroll in the RESULTS or MESSAGES pane. If you scroll down to the bottom of the MESSAGES pane, you can see the total execution time (this includes time to execute the stored procedure 25 times on the server but also return results to the client). Your results should look something like this (30 seconds or greater)
+
+ 
+
+8. Now let's run the same exact test but with database compatibility of 150. You will not make any changes to the stored procedure.
+
+ Open the script **repro150.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder.
+
+ Use the SQL container connection as you have done in previous steps. The script should look like this
+
+
+ ```sql
+ USE master
+ GO
+ ALTER DATABASE wideworldimporters SET compatibility_level = 150
+ GO
+ USE WideWorldImporters
+ GO
+ SET NOCOUNT ON
+ GO
+ EXEC [Sales].[CustomerProfits]
+ GO 25
+ SET NOCOUNT OFF
+ GO
+ ```
+ Notice this is the same script except database compatibility of 150 is used. This time, the query processor in SQL Server will enable table variable deferred compilation so a better query plan can be chosen
+
+9. Run the script and choose the SQL Server container connection. Go through the same steps as in Step 8 to analyze the results. The script should execute far faster than before. Your speeds can vary but should be 15 seconds or less.
+
+As a **post workshop exercise** you can go through this activity in a SQL notebook. For SQL notebooks,use the File menu of Azure Data Studio (Open File option) to open the **03_IQP_Table_Variable.ipynb** notebook in the **sqlworkshops/SQLOnOpenShift/sqlonopenshift/iqp/03_performance** folder. Follow the steps provided in the notebook to complete the activity.
+
+The SQL notebook experience looks like the following:
+
+
+
+Read each step in the notebook and use the Play button to execute each T-SQL script. Use the scrollbar to scroll through the notebook.
+
+The rest of the steps in this section of the Module is for the experience without using a notebook. Proceed to section 3.1.
+
+
+
+In this module you will learn how to use the Query Store, a built-in performance analysis feature of SQL Server, to analyze the performance differences of the queries run in Section 3.0 of this Module.
+
+The Query Store is built into the query processing engine, enabled using an option for each database in SQL Server. Once enabled, performance statistics for queries are cached and stored in the SQL user database so they are persisted across server restarts.
+
+In addition, the Query Store comes with a series of catalog and dynamic management views to gain insight into recorded query performance.
+
+You can read more about the Query Store at https://docs.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store.
+
+
+
+
+
+The WideWorldImporters sample database that you restored in Module 02 has the Query Store enabled. Therefore, if you performed the activity in Section 3.0 of this module, the Query Store was recording performance information about each query execution.
+
+Go through the following steps to use the Query Store to examine the query performance differences for the CustomerProfits stored procedure when executed with database compatibility 130 vs 150.
+
+Open the script **querystore.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder.
+
+Use the SQL container connection as you have done in previous steps. The T-SQL statement to use the Query Store for this activity looks like the following
+
+
+```sql
+USE WideWorldImporters
+GO
+SELECT qsp.query_id, qsp.plan_id, qsp.compatibility_level, AVG(qsrs.avg_duration)/1000 as avg_duration_ms, AVG(qsrs.avg_logical_io_reads) as avg_logical_io,CAST (qsp.query_plan as XML),qsqt.query_sql_text
+FROM sys.query_store_plan qsp
+INNER JOIN sys.query_store_runtime_stats qsrs
+ON qsp.plan_id = qsrs.plan_id
+INNER JOIN sys.query_store_runtime_stats_interval qsrsi
+ON qsrs.runtime_stats_interval_id = qsrsi.runtime_stats_interval_id
+AND qsrsi.start_time between DATEADD(HOUR, -1, GETDATE()) and GETDATE()
+INNER JOIN sys.query_store_query qsq
+ON qsp.query_id = qsq. query_id
+INNER JOIN sys.query_store_query_text qsqt
+ON qsq.query_text_id = qsqt.query_text_id
+AND query_sql_text LIKE '%Invoices%'
+GROUP BY qsp.query_id, qsp.plan_id, qsrs.avg_duration, qsp.compatibility_level, qsp.query_plan, query_sql_text, qsrs.last_execution_time
+ORDER BY qsrs.last_execution_time DESC, query_id DESC, qsp.plan_id DESC
+GO
+```
+ To even the seasoned SQL user this query looks daunting. This activity is not intended for you to understand every details of how the query works but as an example of how to use the Query Store Dynamic Management Views (DMV) to analyze the performance differences for the stored procedure you used in Section 3.0.
+
+ The query is designed to focus on the most recent execution of the query in the stored procedure CustomerProfits that has run faster with Intelligent Query Processing.
+
+ The results of the query should look something like the following (the values query_id and plan_id will likely be different)
+
+ 
+
+ The query_id uniquely identifies the SELECT TOP.. statement. The plan_id shows that two different query plans were used for the same query. One with database compatibility of 130 and one with 150.
+
+ Notice the avg_duration_ms is much higher for the row using the plan for compatibility_level = 130. The reason? Look at the avg_logical_io which is the number of database pages read needed for the query. The number is much higher for the longer running query, typically due to an inefficient query plan.
+
+ So these results show the *same* query but two different query plans, one faster on average and more efficient than the other. The plan for compatibility_level = 150 was using table variable deferred compilation. The last columns shows the T-SQL query text within the stored procedure.
+
+ One of the columns on the right is the estimated query plan in XML. Reading the details of this XML is beyond the scope of this workshop but contains the details of how the query plan was built using table variable deferred compilation. In addition, in these XML details is the use of an Adaptive Join, which is another capability of Intelligent Query Processing.
+
+ If you saved the XML output as a .sqlplan file and opened up the file in SQL Server Management Studio (SSMS), you would see the following graphical plan for the query using compatibility level = 130
+
+ 
+
+ In this plan, the query processor chooses to use the table variable as the outer table of a Nested Loops Join which means it iterates through each row in the outer table to join to the inner table. Since the query processor estimates one row for the table variable, this is reasonable. But in reality, there are over 200,000 rows in the table variable so this ends up with many unnecessary logical reads (hence the high number of logical IO from query store)
+
+ The following output is an example for the stored procedure execution with compatibiilty level of 150
+
+ 
+
+ In this example SQL has recognized the table variable has more than 1 row and has chosen a different join method called a hash join. Furthermore, it has injected into the plan the concept of an **Adaptive Join** so that if there is small enough rowset in the table variable it could dynamically and automatically choose a Nested Loops Join. What is not seen from this diagram (which you can see from the properties detail in the XML plan) that is the query processor is using a third concept called **batch mode processing on rowstore** (rowstore is normal table as opposed to a columnstore).
+
+As a **post workshop exercise** you can go through this activity in a SQL notebook. Uuse the File menu of Azure Data Studio (Open File option) to open the **03_Query_Store.ipynb** notebook in the **sqlworkshops/SQLOnOpenShift/sqlonopenshift/iqp/03_performance** folder. Follow the steps provided in the notebook to complete the activity.
+
+In this activity you have seen how to use the Query Store for performance insights including the ability to see differences for the same query text of different query plans, including those that benefit from Intelligent Query Processing.
+
+You can now proceed to Next Steps to learn about High Availability for SQL Server on OpenShift.
+
+
+
+
For Further Study
+
+- [Intelligent Query Processing in SQL Server](https://docs.microsoft.com/en-us/sql/relational-databases/performance/intelligent-query-processing)
+- [Q&A about Intelligent Query Processing](https://techcommunity.microsoft.com/t5/Azure-SQL-Database/Intelligent-Query-Processing-Q-amp-A/ba-p/446657)
+- [Monitoring performance of SQL Server using the Query Store](https://docs.microsoft.com/en-us/sql/relational-databases/performance/monitoring-performance-by-using-the-query-store)
+- [What is Azure Data Studio?](https://docs.microsoft.com/en-us/sql/azure-data-studio/what-is)
+- [How to use Notebooks in Azure Data Studio](https://docs.microsoft.com/en-us/sql/azure-data-studio/sql-notebooks)
+
+
+
+
Next Steps
+
+Next, Continue to SQL Server High Availabilty on OpenShift.
diff --git a/SQLonOpenShift/sqlonopenshift/03_performance/README.md b/SQLonOpenShift/sqlonopenshift/03_performance/README.md
new file mode 100644
index 00000000..e69de29b
diff --git a/SQLonOpenShift/sqlonopenshift/03_performance/iqp/03_IQP_Table_Variable.ipynb b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/03_IQP_Table_Variable.ipynb
new file mode 100644
index 00000000..6ebb61d9
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/03_IQP_Table_Variable.ipynb
@@ -0,0 +1,1571 @@
+{
+ "metadata": {
+ "kernelspec": {
+ "name": "SQL",
+ "display_name": "SQL",
+ "language": "sql"
+ },
+ "language_info": {
+ "name": "sql",
+ "version": ""
+ }
+ },
+ "nbformat_minor": 2,
+ "nbformat": 4,
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": "# Intelligent Query Processing in SQL Server 2019 - Table Variable Deferred Compilation",
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Step 1 - Create the stored procedure\r\nThis procedure uses a table variable populated from a user table and then joins it with a user table to provide output. T-SQL functions like COUNT and SUM are often seen in analytic queries that benefit from Intelligent Query Processing. Note: In this example the TOP 1 T-SQL syntax is used so that the procedure only produces 1 row. This is only done to make the output easier to read using this workshop and demo since this procedure will be executed multiple times. Normal execution of this procedure may not include TOP.\r\n\r\n",
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": "USE WideWorldImporters\r\nGO\r\nCREATE or ALTER PROCEDURE [Sales].[CustomerProfits]\r\nAS\r\nBEGIN\r\n-- Declare the table variable\r\nDECLARE @ilines TABLE\r\n(\t[InvoiceLineID] [int] NOT NULL primary key,\r\n\t[InvoiceID] [int] NOT NULL,\r\n\t[StockItemID] [int] NOT NULL,\r\n\t[Description] [nvarchar](100) NOT NULL,\r\n\t[PackageTypeID] [int] NOT NULL,\r\n\t[Quantity] [int] NOT NULL,\r\n\t[UnitPrice] [decimal](18, 2) NULL,\r\n\t[TaxRate] [decimal](18, 3) NOT NULL,\r\n\t[TaxAmount] [decimal](18, 2) NOT NULL,\r\n\t[LineProfit] [decimal](18, 2) NOT NULL,\r\n\t[ExtendedPrice] [decimal](18, 2) NOT NULL,\r\n\t[LastEditedBy] [int] NOT NULL,\r\n\t[LastEditedWhen] [datetime2](7) NOT NULL\r\n)\r\n\r\n-- Insert all the rows from InvoiceLines into the table variable\r\nINSERT INTO @ilines SELECT * FROM Sales.InvoiceLines\r\n\r\n-- Find my total profile by customer\r\nSELECT TOP 1 COUNT(i.CustomerID) as customer_count, SUM(il.LineProfit) as total_profit\r\nFROM Sales.Invoices i\r\nINNER JOIN @ilines il\r\nON i.InvoiceID = il.InvoiceID\r\nGROUP By i.CustomerID\r\nEND\r\nGO",
+ "metadata": {},
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Commands completed successfully."
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Total execution time: 00:00:00.0416289"
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Commands completed successfully."
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Total execution time: 00:00:00.1910054"
+ },
+ "metadata": {}
+ }
+ ],
+ "execution_count": 1
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Step 2 - Run the stored procedure with database compatibility of 130\r\nou have been told this procedure executes fairly quickly with a single execution in a few seconds but over several iterations the total duration, over 20 seconds, is not acceptable to the application.\r\n\r\nThe script will ensure the database is in a compatibility mode that is less than 150 so Intelligent Query Processing will NOT be enabled. The script also turns off rowcount messages to be returned to the client to reduce network traffic for this test. Then the script executes the stored procedure. Notice the syntax of **GO 25**. This is a client tool tip that says to run the batch 25 times (avoids having to construct a loop).\r\n\r\nWhen you click Play to run the script look for these messages on the total elapsted time (your time may vary)\r\n\r\n
"
+ }
+ }
+ ],
+ "execution_count": 2
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Step 3 - Run the stored procedure with database compatibility of 150\r\nNow let's run the same exact test but with database compatibility of 150. You will not make any changes to the stored procedure.\r\n\r\nNotice this is the same script except database compatibility of 150 is used. This time, the query processor in SQL Server will enable table variable deferred compilation so a better query plan can be chosen\r\n\r\n The script should execute far faster than before. Your speeds can vary but should be 15 seconds or less.\r\n\r\n When you click Play to run the script look for these messages on the total elapsted time (your time may vary)\r\n\r\n
SELECT TOP 1 COUNT(i.CustomerID) as customer_count, SUM(il.LineProfit) as total_profit\r\nFROM Sales.Invoices i\r\nINNER JOIN @ilines il\r\nON i.InvoiceID = il.InvoiceID\r\nGROUP By i.CustomerID
SELECT TOP 1 COUNT(i.CustomerID) as customer_count, SUM(il.LineProfit) as total_profit\r\nFROM Sales.Invoices i\r\nINNER JOIN @ilines il\r\nON i.InvoiceID = il.InvoiceID\r\nGROUP By i.CustomerID
"
+ }
+ }
+ ],
+ "execution_count": 1
+ }
+ ]
+}
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/03_performance/iqp/proc.sql b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/proc.sql
new file mode 100644
index 00000000..0b0d61f5
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/proc.sql
@@ -0,0 +1,33 @@
+USE WideWorldImporters
+GO
+CREATE or ALTER PROCEDURE [Sales].[CustomerProfits]
+AS
+BEGIN
+-- Declare the table variable
+DECLARE @ilines TABLE
+( [InvoiceLineID] [int] NOT NULL primary key,
+ [InvoiceID] [int] NOT NULL,
+ [StockItemID] [int] NOT NULL,
+ [Description] [nvarchar](100) NOT NULL,
+ [PackageTypeID] [int] NOT NULL,
+ [Quantity] [int] NOT NULL,
+ [UnitPrice] [decimal](18, 2) NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [TaxAmount] [decimal](18, 2) NOT NULL,
+ [LineProfit] [decimal](18, 2) NOT NULL,
+ [ExtendedPrice] [decimal](18, 2) NOT NULL,
+ [LastEditedBy] [int] NOT NULL,
+ [LastEditedWhen] [datetime2](7) NOT NULL
+)
+
+-- Insert all the rows from InvoiceLines into the table variable
+INSERT INTO @ilines SELECT * FROM Sales.InvoiceLines
+
+-- Find my total profile by customer
+SELECT TOP 1 COUNT(i.CustomerID) as customer_count, SUM(il.LineProfit) as total_profit
+FROM Sales.Invoices i
+INNER JOIN @ilines il
+ON i.InvoiceID = il.InvoiceID
+GROUP By i.CustomerID
+END
+GO
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/03_performance/iqp/querystore.sql b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/querystore.sql
new file mode 100644
index 00000000..af2fc234
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/querystore.sql
@@ -0,0 +1,17 @@
+USE WideWorldImporters
+GO
+SELECT qsp.query_id, qsp.plan_id, qsp.compatibility_level, AVG(qsrs.avg_duration)/1000 as avg_duration_ms, AVG(qsrs.avg_logical_io_reads) as avg_logical_io,CAST (qsp.query_plan as XML),qsqt.query_sql_text
+FROM sys.query_store_plan qsp
+INNER JOIN sys.query_store_runtime_stats qsrs
+ON qsp.plan_id = qsrs.plan_id
+INNER JOIN sys.query_store_runtime_stats_interval qsrsi
+ON qsrs.runtime_stats_interval_id = qsrsi.runtime_stats_interval_id
+AND qsrsi.start_time between DATEADD(HOUR, -1, GETDATE()) and GETDATE()
+INNER JOIN sys.query_store_query qsq
+ON qsp.query_id = qsq. query_id
+INNER JOIN sys.query_store_query_text qsqt
+ON qsq.query_text_id = qsqt.query_text_id
+AND query_sql_text LIKE '%Invoices%'
+GROUP BY qsp.query_id, qsp.plan_id, qsrs.avg_duration, qsp.compatibility_level, qsp.query_plan, query_sql_text, qsrs.last_execution_time
+ORDER BY qsrs.last_execution_time DESC, query_id DESC, qsp.plan_id DESC
+GO
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/03_performance/iqp/repro130.sql b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/repro130.sql
new file mode 100644
index 00000000..738059bf
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/repro130.sql
@@ -0,0 +1,12 @@
+USE master
+GO
+ALTER DATABASE wideworldimporters SET compatibility_level = 130
+GO
+USE WideWorldImporters
+GO
+SET NOCOUNT ON
+GO
+EXEC [Sales].[CustomerProfits]
+GO 25
+SET NOCOUNT OFF
+GO
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/03_performance/iqp/repro150.sql b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/repro150.sql
new file mode 100644
index 00000000..d7124bb9
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/03_performance/iqp/repro150.sql
@@ -0,0 +1,12 @@
+USE master
+GO
+ALTER DATABASE wideworldimporters SET compatibility_level = 150
+GO
+USE WideWorldImporters
+GO
+SET NOCOUNT ON
+GO
+EXEC [Sales].[CustomerProfits]
+GO 25
+SET NOCOUNT OFF
+GO
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/04_HA.md b/SQLonOpenShift/sqlonopenshift/04_HA.md
new file mode 100644
index 00000000..67eaab8f
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_HA.md
@@ -0,0 +1,280 @@
+
+
+# Workshop: SQL Server on OpenShift
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
SQL Server High Availability on OpenShift
+
+You'll cover the following topics in this Module:
+
+
+
+In this module you will learn how to use the built-in capabilities of OpenShift for high availability for SQL Server.
+
+
Note: This Module assumes you have at minimum, completed Module 01, Deploy SQL Server on OpenShift, and Module 02, Connect and Query.
+
+OpenShift, which is based on Kubernetes, allows you to deploy a pod and specify a declaration for a number of replicas to be deployed. In previous modules, you deployed a SQL Server container in a single pod on OpenShift. You used a declarative yaml file to instruct OpenShift how to deploy SQL Server with a replica. The fragment of the yaml file looks like
+
+
+
+By stating **replicas of 1**, OpenShift will attempt to ensure at least one pod for SQL Server is always running. Since you used PersistentVolumeClaim for SQL Server databases, your data will always be available even if OpenShift has to start new SQL Server containers. In section 4.1, you will learn the signifance of the **type: Recreate** syntax.
+
+In addition, you created a load balancer service with the following yaml fragment
+
+
+
+The LoadBalancer service in OpenShift will have a consistent IP address and port (31433) that will always map to the private IP address for the SQL Server container and port 1433, which is the default port for the SQL Server database engine. So even if OpenShift has to deploy a new SQL Server container with a new private IP address, the LoadBalancer will always be directed to the correct SQL Server.
+
+The following diagram is a simple visual on how OpenShift provides built-in High Availability
+
+
+
+Proceed to the Activity to see examples of how OpenShift provides high availability for SQL Server.
+
+
+
+
+
+Follow these steps to see how high availability works with SQL Server on OpenShift using replicas, PersistentVolumeClaims, and Load Balancers.
+
+1. Open a shell prompt and change to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/04_ha** directory.
+
+2. One of the simple methods for High Availability for OpenShift is at the container level. Since SQL Server is the main program in the container, if SQL Server is shutdown or crashes, OpenShift will automatically restart the container in the same pod and node. Run the following commands to see the status of the current pod deployed for the project you created in Module 01.
+
+ `oc get pods -o wide`
+
+ The output of this command should look something like the following. Take note of the xxxxxx
+
+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
+ mssql-deployment-6d8844988b-j9r84 1/1 Running 0 3m 10.129.0.15 ocpcluster-node01 none
+
+ Take note of the pod name (under the NAME column), the IP address, and the NODE name (If your cluster has only one node, the node name will not change throughout this activity).
+
+3. Run the following commands to shutdown SQL Server. You can also use the script **step1_stop_container.sh**:
+
+ `SERVERIP=$(kubectl get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SHUTDOWN WITH NOWAIT"`
+
+ You should see the following output from the shell
+
+
Server shut down by NOWAIT request from login sa
+
+ Now run the command again to see the pod status
+
+ `oc get pods -o wide`
+
+ Note the pod name, IP address, and node have not changed. The STATUS should be Running.
+
+4. Execute the following commands to test SQL Server is running. You can also use the script **step2_test_sql.sh**
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT @@version"`
+
+ Your output should look like the following
+
+
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Microsoft SQL Server 2019 (CTP2.2) - 15.0.1200.24 (X64)
+ Dec 5 2018 16:51:26
+ Copyright (C) 2018 Microsoft Corporation
+ Developer Edition (64-bit) on Linux (Red Hat Enterprise Linux Server 7.6 (Maipo)) X64
+
+ (1 rows affected)
+
+ Depending on how long it takes OpenShift to restart the container, your reconnection make take longer than a few seconds.
+
+5. Since you deployed the SQL Server container in a pod using 1 replica, if there is a failure of the pod in a node, OpenShift will attempt to to create a new pod and schedule it on the appropriate available node. For SQL Server, this is like a *failover* since OpenShift will start a new SQL Server container but pointing to the same PersistentVolumeClaim, hence the same system and user databases. And the LoadBalancer acts like a *listener* so if the new SQL Server container has a new IP address, the application still connects to the same LoadBalancer IP address.
+
+ **Note**: Ths is not the same as a SQL Server Availability Group which will be discussed in Module 05, Using an Operator with SQL Server on OpenShift.
+
+ Run the following commands to take note again of the pod details including name, status, IP, and node
+
+ `oc get pods -o wide`
+
+ Run the following commands to delete the node for the pod for SQL Server container. You can also use the script **step3_pod_failure.sh**. In this example, you can take advantage of the label used to create the SQL Server pod. This was defined in the following yaml fragment from Module 01
+
+
template:
+ metadata:
+ labels:
+ app: mssql
+
+ `oc delete pod -l app=mssql`
+
+ The output of this command should look like the following:
+
+
pod "mssql-deployment-6d8844988b-j9r84" deleted
+
+ and then you are returned to the command prompt
+
+ Now run the command again to see the pod status
+
+ `oc get pods -o wide`
+
+ Starting a new pod on the same node should take a matter of seconds but if OpenShift has to start a new pod on a different node, it could take several minutes especially if the SQL Server container image does not exist on that new node.
+
+ The output of this command should have a new pod name but start with the name **mssql-deployment**. The IP address should also be different and depending on your OpenShift cluster configuration the pod could be scheduled on a new node. When the pod status is Running, you should be able to access SQL Server
+
+6. If you have completed the steps in Module 02, then you restored the database WideWorldImporters to the container. And since the container uses a PersistentVolumeClaim, the WideWorldImporters should have the same data. Execute the following commands or the script **step4_find_people.sh** to query the database:
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"USE WideWorldImporters;SELECT TOP 10 FullName, PhoneNumber, EmailAddress FROM [Application].[People] ORDER BY FullName;" -Y30`
+
+ You should get the same results of data as you did in Module 02. Notice in this example, you connect to SQL Server with the LoadBalancer service IP address and port since you cannot determine what the IP address will be of the SQL Server container as it gets recreated on the same or other nodes.
+
+Proceed to the next section of this module to learn how to update SQL Server while keeping it highly available.
+
+
+
+
+
+To update the version of the SQL Server software outside a container, you would normally use a package manger command such as **yum update**.
+
+However, package management software is not normally installed in the SQL Server container. Furthermore, the principle of containers is to use a consistent image. So the challenge to update the SQL Server software is that you want your running SQL Server container to keep the same set of databases and data but you want a new SQL server container with a new updated image to connect to the databases.
+
+Fortunately, the SQL Server engine is aware of the version it is running under when it starts up and can detect the compatible versions of metadata within system and user databases. Therefore, if you start a new container using the same PersistentVolumeClaim of your databases from the container with the newer version image, SQL Server can be updated in a highly available fashion.
+
+When you deploy your pod with SQL Server on OpenShift you can declare the deployment to stop the container and recreate a new one if you indicate to OpenShift that the image should be updated for the deployment.
+
+The following yaml fragment from Module 01 shows this declaration of strategy to **Recreate**.
+
+
+
+**Note**: SQL Server can be updated from a lower major release to a new one such as SQL Server 2017 to SQL Server 2019. However, you cannot rollback a major version update. SQL Server has a concept of minor updates called a *cumulative update*. Cumulative updates with a major SQL Server version are compatible to update and rollback. SQL Server releases in preview called Community Technology Preview (CTP) builds are treated like major releases. So you can update for example from SQL Server 2019 CTP 2.2 to 2.3 but you cannot rollback.
+
+This workshop as of April 2019 uses SQL 2019 CTP builds and therefore there is no exercise to rollback but future versions of this workshop will include a rollback exercise between SQL 2019 cumulative updates.
+
+Proceed to the Activity to learn how to update SQL Server but keep your databases intact making it highly available and consistent while updating the SQL Server software version.
+
+
+
+
+
+In this activity, you will use OpenShift commands to deploy an update to the release of SQL Server with no change to user data and minimize application downtime.
+
+1. Open a shell prompt and change to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/04_ha** directory.
+
+2. Execute the following commands to take note of the version of SQL Server. You can also execute the script **step2_test_sql.sh**
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT @@version"`
+
+ Depending on what image was used for your workshop, the output should be reflective of the version of SQL Server deployed. In this example, the image used was for SQL Server 2019 CTP 2.2 for Red Hat Enterprise Linux.
+
+
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Microsoft SQL Server 2019 (CTP2.2) - 15.0.1200.24 (X64)
+ Dec 5 2018 16:51:26
+ Copyright (C) 2018 Microsoft Corporation
+ Developer Edition (64-bit) on Linux (Red Hat Enterprise Linux Server 7.6 (Maipo)) X64
+
+3. Execute the following commands to update the image of the SQL Server deployment from Module 01 to SQL Server 2019 CTP 2.3. You can also use the script **step5_update_sql.sh**
+
+ `oc --record deployment set image mssql-deployment mssql=mcr.microsoft.com/mssql/rhel/server:2019-CTP2.3`
+
+ The output of this command should look like
+
+
+
+4. Execute the commands to see the status of the image update to the deployment
+
+ `oc get all`
+
+ The output of this command while the new pod with the new image is getting created should look something like this
+
+
NAME READY STATUS RESTARTS AGE
+ pod/mssql-deployment-6748c8d9ff-wqstc 0/1 ContainerCreating 0 1m
+
+ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+ service/mssql-service LoadBalancer 172.30.241.155 13.82.210.117 31433:31372/TCP 38m
+
+ NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
+ deployment.apps/mssql-deployment 1 1 1 0 38m
+
+ NAME DESIRED CURRENT READY AGE
+ replicaset.apps/mssql-deployment-6748c8d9ff 1 1 0 1m
+ replicaset.apps/mssql-deployment-6d8844988b 0 0 0 38m
+
+ The time it takes to run the new pod and container could be several minutes as the OpenShift cluster may not have the new container image for SQL 2019 CTP 2.3 in its local docker registry. Once the new container is Running, you can connect to SQL Server. Notice also in this output the previous pod is stopped but available to restart, which is part of the process to rollback should you be using a cumulative update strategy.
+
+5. Run the following commands to see the SQL Server version updates. You can also use the script **step2_test_sql.sh**
+
+ `SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})`
+ `PORT=31433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT @@version"`
+
+ The SQL Server container will move to a state of running, but internally the SQL Server engine must update the metadata for system and user databases. This process should only take a few minutes, but any attempt to connect to SQL Server before this completes can result in the following error
+
+
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time..
+
+ Within a few minutes, the above command or script if run again should product results like the following:
+
+
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Microsoft SQL Server 2019 (CTP2.3) - 15.0.1300.359 (X64)
+ Feb 15 2019 23:50:43
+ Copyright (C) 2019 Microsoft Corporation
+ Developer Edition (64-bit) on Linux (Red Hat Enterprise Linux Server 7.6 (Maipo)) X64
+
+ (1 rows affected)
+
+6. If you are going to proceed and complete Module 05, you need to cleanup resources. This can be done by running the following commands. You can also execute the script **step6_cleanup.sh**
+
+ `oc delete project mssql`
+ `oc project default`
+
+ When this completes, you should see the following output and be placed back at the shell prompt
+
+
project.project.openshift.io "mssql" deleted Now using project "default" on server "https://[masterconsoleurl]:443".
+
+You can now proceed to Next Steps to learn more about using operators and SQL Server Always On Availability Groups.
+
+
+
+
For Further Study
+
+- [High Availability for SQL Server on Kubernetes](https://docs.microsoft.com/en-us/sql/linux/tutorial-sql-server-containers-kubernetes?view=sql-server-2017#ha-solution-on-kubernetes-running-in-azure-kubernetes-service)
+- [High Availability for SQL Server Containers](https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-container-ha-overview)
+
+
+
+
Next Steps
+
+Next, Continue to Using an Operator with SQL Server on OpenShift.
diff --git a/SQLonOpenShift/sqlonopenshift/04_ha/step1_stop_container.sh b/SQLonOpenShift/sqlonopenshift/04_ha/step1_stop_container.sh
new file mode 100644
index 00000000..8a4f5093
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_ha/step1_stop_container.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(kubectl get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SHUTDOWN WITH NOWAIT"
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/04_ha/step2_test_sql.sh b/SQLonOpenShift/sqlonopenshift/04_ha/step2_test_sql.sh
new file mode 100644
index 00000000..26de9969
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_ha/step2_test_sql.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT @@version"
diff --git a/SQLonOpenShift/sqlonopenshift/04_ha/step3_pod_failure.sh b/SQLonOpenShift/sqlonopenshift/04_ha/step3_pod_failure.sh
new file mode 100644
index 00000000..b9e885cd
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_ha/step3_pod_failure.sh
@@ -0,0 +1 @@
+oc delete pod -l app=mssql
diff --git a/SQLonOpenShift/sqlonopenshift/04_ha/step4_find_people.sh b/SQLonOpenShift/sqlonopenshift/04_ha/step4_find_people.sh
new file mode 100644
index 00000000..aa91c162
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_ha/step4_find_people.sh
@@ -0,0 +1,3 @@
+SERVERIP=$(oc get service | grep mssql-service | awk {'print $4'})
+PORT=31433
+sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"USE WideWorldImporters;SELECT TOP 10 FullName, PhoneNumber, EmailAddress FROM [Application].[People] ORDER BY FullName;" -Y30
\ No newline at end of file
diff --git a/SQLonOpenShift/sqlonopenshift/04_ha/step5_update_sql.sh b/SQLonOpenShift/sqlonopenshift/04_ha/step5_update_sql.sh
new file mode 100644
index 00000000..3a12d833
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_ha/step5_update_sql.sh
@@ -0,0 +1 @@
+oc --record deployment set image mssql-deployment mssql=mcr.microsoft.com/mssql/rhel/server:2019-CTP2.3
diff --git a/SQLonOpenShift/sqlonopenshift/04_ha/step6_cleanup.sh b/SQLonOpenShift/sqlonopenshift/04_ha/step6_cleanup.sh
new file mode 100644
index 00000000..ff499978
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/04_ha/step6_cleanup.sh
@@ -0,0 +1,2 @@
+oc delete project mssql
+oc project default
diff --git a/SQLonOpenShift/sqlonopenshift/05_Operator.md b/SQLonOpenShift/sqlonopenshift/05_Operator.md
new file mode 100644
index 00000000..6d57a3ff
--- /dev/null
+++ b/SQLonOpenShift/sqlonopenshift/05_Operator.md
@@ -0,0 +1,385 @@
+
+
+# Workshop: SQL Server on OpenShift
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Using an Operator with SQL Server on OpenShift
+
+You'll cover the following topics in this Module:
+
+
+
+In this module you will learn more about what, how, and why you would deploy an Operator that assists in deployment and management of an Always On Availability Group.
+
+You learned in Module 04 the built-in capabilities of OpenShift to detect the health of the SQL Server container, pod, or node to provide High Availability.
+
+There are a few limitations with these capabilities:
+
+- OpenShift doesn't understand the health of the SQL Server engine
+- There is only one SQL Server instance so any restart of SQL Server within the container, in a new pod, or a new node could result in several minutes of downtime.
+
+Always On Availability Groups is the flagship HADR feature for SQL Server. It provides the fastest uptime capabilities using multiple SQL Server instances called *replicas*. You can read more about Always On Availability Groups at https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/overview-of-always-on-availability-groups-sql-server
+
+Installing and configuring an Always On Availability Group can take some time and involve many steps, especially when you need to configure a failover software package (such as Pacemaker) to help support automated failover capabilities. You can read more about Always On Availability Groups on Linux at https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-availability-group-overview
+
+In Module 04 your learned OpenShift has built-in high availability capabilities. SQL Server 2019 provides the necessary software to cooperate with OpenShift to provide high availability with Always On Availability Groups. In addition, SQL Server 2019 provides an *Operator* to assist in deployment of the Always On Availability Group and coordinate failover activities. This is non-shared storage solujtion as each SQL Server replic has its own storage. Combined with LoadBalancer services, this provides a complete High Availability solution with less downtime and read capabilities on secondary replicas. The following diagram shows a visual of this design
+
+
+
+Proceed to the Activity to learn how to use an Operator to deploy and configure an Availability Group on OpenShift.
+
+
+
+
+
+Follow these steps to deploy an Always On Availability Group on OpenShift using an operator.
+
+1. Open a shell prompt and change directories to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/05_operator** folder.
+
+2. First, create a project for this activity. Use the following command or execute the **step1_create_project.sh** script:
+
+ `oc new-project ag1`
+
+ When this completes, you should see output like the following
+
+
Now using project "ag1" on server "https://masterdnsx5rquio6c54pu.eastus.cloudapp.azure.com:443".
+
+ You can add applications to this project with the 'new-app' command. For example, try:
+
+ oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git
+
+ to build a new example application in Ruby.
+
+3. Create the SQL Server operator by executing the following command or execute the **step2_apply_operator.sh** script:
+
+ `oc apply -f operator.yaml --namespace ag1`
+
+ When this completes, you should see output like the following
+
+
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
+ namespace/ag1 configured
+ serviceaccount/mssql-operator created
+ clusterrole.rbac.authorization.k8s.io/mssql-operator-ag1 configured
+ clusterrolebinding.rbac.authorization.k8s.io/mssql-operator-ag1 configured
+ deployment.apps/mssql-operator created
+
+ You can ignore the Warning. To ensure the operator was deployed successfully, run the following command
+
+ `oc get pods`
+
+ The STATUS of the mssql-operator should be **Running**. Don't proceed to the next step until the pod is running.
+
+4. Next, create a secret for the system administrator (sa) password and a password for the master key using the following command or the script **step3_generate_secret.sh**
+
+ `oc create secret generic sql-secrets --from-literal=sapassword="Sql2019isfast" --from-literal=masterkeypassword="Sql2019isfast" --namespace ag1`
+
+ When this completes, you should see output like the following
+
+
secret/sql-secrets created
+
+5. Using the **sqlserver.yaml** file you will deploy the SQL Server instances for a primary and two secondary replicas. This will coordinate with the deployed operator to install a SQL Server Always On Availability Group configuration. No database will be deployed. That will be done in the next section of this Module. Run the following command or use the script **step4_apply_sqlserver.sh**
+
+ `oc apply -f sqlserver.yaml --namespace ag1`
+
+ When this completes, you should see the following
+
+
sqlserver.mssql.microsoft.com/mssql1 created
+ service/mssql1 created
+ sqlserver.mssql.microsoft.com/mssql2 created
+ service/mssql2 created
+ sqlserver.mssql.microsoft.com/mssql3 created
+ service/mssql3 created
+
+ To proceed to the rest of the activity, you need to ensure the deployment of all pods and LoadBalancer services are successful. The deployment could take several minutes. Execute the following command
+
+ `oc get all`
+
+ When the deployment is successful, these pods should be in a STATUS of Running
+
+
+
+ Run the `oc get all` command until the pods and LoadBalancer services are in this state.
+
+ Note: You will see some pods taht start with a name of mssql-initialize. You can ignore these. They are used to deploy the SQL Server Availability Group but may not be needed in the final design of the operator.
+
+ In addition, notice that there are three objects from the oc get all output that look like this
+
+
NAME DESIRED CURRENT AGE
+ statefulset.apps/mssql1 1 1 43s
+ statefulset.apps/mssql2 1 1 40s
+ statefulset.apps/mssql3 1 1 37s
+
+ As part of the design for SQL Server Availability groups, a concept called a *statefulset* is used. A stateful set provides the capabilities to manage a set of pods that belong in a group like Availability Groups. You can read more about StatefulSets at https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/.
+
+6. Even though each SQL Server pod has a LoadBalancer service, you need to create a LoadBalancer service that will always point to the primary replica and a LoadBalancer service that will point to any secondary replicas. Execute the following command or the script **step5_apply_agservices.sh**
+
+ `oc apply -f ag_services.yaml --namespace ag1`
+
+ When this completes, you should see the following
+
+
service/ag1-primary created
+ service/ag1-secondary created
+
+ Use the following command to check the status of these services. Wait until each of these services has a valid IP address for EXTERNAL-IP before proceeding to the next step. This could take several minutes
+
+ `oc get service`
+
+ At this point, a SQL Server installation is complete with a Always On Availability enabled and created for use. In the next module, you will learn how to add a database to the availability group, add data, and then query both the primary and secondary replicas.
+
+
+
+
+
+In this section, you will learn how to connect, add databases, add data, and query data to replicas in an Always On Availability Group deployed in OpenShift.
+
+Once you have an Always On Availability Group deployed on OpenShift, you will want to add a database to the group, add some data, and see the changes reflected in secondary replicas.
+
+SQL Server provides T-SQL statements, such as `ALTER AVAILABILITY GROUP`, to interact and manage the Availability Group including adding one more databases. An Availability Group can have one more or databases and is the unit of failover.
+
+Once you have added your database to the availability group, you will create a database backup, which will initiate a concept called *direct seeding*. This process will send data to the secondary replicas making them in *sync*. You can now connect to the primary replica SQL Server and interact with it like any SQL Server database. You can also connect to the secondary replica(s) with a special connection option to read data. This allows you to offload read users (e.g. reporting users) from your primary SQL Server.
+
+Proceed to the activity to see how this works.
+
+
+
+
+
+Follow the steps in his activity to connect, add databases, add data, and query data to replicas in an availability group deployed in OpenShift.
+
+1. Open a shell prompt and change directories to the **sqlworkshops/SQLonOpenShift/sqlonopenshift/05_operator** folder.
+
+2. T-SQL provides capabilities so you can see which SQL Servers instances are currently the primary vs secondary replica.
+
+ An example of a T-SQL command to see replica status looks like the following
+
+
+ ```sql
+ SELECT ar.replica_server_name, hars.role_desc, hars.operational_state_desc
+ FROM sys.dm_hadr_availability_replica_states hars
+ JOIN sys.availability_replicas ar
+ ON hars.replica_id = ar.replica_id
+ GO
+ ```
+ Run the following command or execute the script **step6_check_replicas.sh** to see the replica status of the Availability Group deployed
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -icheckreplicas.sql -Y30`
+
+ In this example, you are using the LoadBalancer directed to the primary replica. Your results should look like the following
+
+
+
+ It is possible that the replica_server_name for your deployment is any of these replicas. In most cases, it will be mssql1-0. You will use this same command later to see that the status of the Availability Group is after a failover.
+
+3. Now it is time to create a new database, backup the database, and then add the database to the Availability Group. Examine the contents of the script **setupag.sql** to see the T-SQL commands. Run the following command or script **step7_setupag.sh**
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -isetupag.sql`
+
+ You should see results like the following
+
+
Changed database context to 'master'.
+ Changed database context to 'master'.
+ Processed 328 pages for database 'testag', file 'testag' on file 1.
+ Processed 2 pages for database 'testag', file 'testag_log' on file 1.
+ BACKUP DATABASE successfully processed 330 pages in 1.239 seconds (2.077 MB/sec).
+
+ Direct seeding should happen almost instantly because there is no user database in the database.
+
+4. It's time to create a table in the database and add in data connected to the primary replica SQL Server. SQL Server will automatically send changes to secondary replicas.
+
+ Examine the T-SQL script **testag.sql** to see the table and data that will be added. Run the following commands or execute the script **step8_testag.sh**
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -itestag.sql`
+
+ Your results should look the following
+
+
Changed database context to 'testag'.
+ (1 rows affected)
+
+5. Now you want to query the primary replica to see your data but also query secondary replicas to see if the changes were synchronized and also learn how to run read-only queries against secondary replicas, one of the benefits of using Availability Groups.
+
+ In order to do this, you need to know the LoadBalancer of the primary replica and secondary replica. So far, you have been using scripts to connect to the primary replica. You created in the previous section a LoadBalancer for the secondary replicas. But, since there are two secondary replicas, which one will be chosen?. That is the benefit of the LoadBalancer service. You can have multiple users connect to the secondary replica LoadBalancer and these connections will be balanced across available replicas.
+
+ Consider the following T-SQL script you will use to query the data and to determine which SQL Server you are connected to
+
+ `SELECT 'Connected to Primary = '+@@SERVERNAME;USE testag;SELECT * FROM ilovesql`
+
+ In this example, you will use the -Q option of sqlcmd to run a query directly without a script file. The use of the ";" serves as a batch delimiter much like the keyword GO. In example, the built-in T-SQL function @@SERVERNAME indicates what SQL Server host the query is running on and the remaining queries select the data you have inserted. This way even though you are connected to the LoadBalancer you can see which exact replica you are being redirected to.
+
+ Execute the following commands or the script **step9_queryag.sh** to see the results
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT 'Connected to Primary = '+@@SERVERNAME;USE testag;SELECT * FROM ilovesql" -Y60`
+ `SERVERIP=$(oc get service | grep ag1-secondary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT 'Connected to Secondary = '+@@SERVERNAME;USE testag;SELECT * FROM ilovesql" -K ReadOnly -Y60`
+
+ The first set of commands connect to the primary replica LoadBalancer. The second set of commands to the secondary replica LoadBalancer. In the case of the secondary replica, the -K option is used to state the intention of only reading data. This is required to query a secondary replica. You can read more about how applications specify ReadOnly intent at https://docs.microsoft.com/en-us/sql/database-engine/availability-groups/windows/listeners-client-connectivity-application-failover?view=sql-server-2017#ReadOnlyAppIntent.
+
+ Your results should look like the following (your server names may be different depending on how the primary was elected when the Availability Group was deployed)
+
+
------------------------------------------------------------
+ Connected to Primary = mssql1-0
+ (1 rows affected)
+ Changed database context to 'testag'.
+ col1 col2
+ ----------- ------------------------------------------------------------
+ 1 SQL Server 2019 is fast, secure, and highly available
+ (1 rows affected)
+
+
------------------------------------------------------------
+ Connected to Secondary = mssql2-0
+ (1 rows affected)
+ Changed database context to 'testag'.
+ col1 col2
+ ----------- ------------------------------------------------------------
+ 1 SQL Server 2019 is fast, secure, and highly available
+ (1 rows affected)
+
+Now that you have successfully created a database, added it to the Availability Group, and synchronized data, you an proceed to the next section to test how a failover works.
+
+
+
+
+
+
+
+In this section, you will learn how to simulate a failover of SQL Server in an Availability Group deployed on OpenShift.
+
+Since Availability Groups provide high availability and lower downtime due to its non-shared storage replica design, it is important to see a failover in action to understand the application experience.
+
+In previous sections of this module, you deployed an Availability Group using an operator. The operator also coordinates with *agents* in containers running in the same pod as the SQL Server replicas to detect failover conditions and if necessary trigger a failover. A failover will cause one of the synchronized secondary replicas to shift roles as a new primary replica. The previous primary replica, once it is healthy, can be synchronized and become a secondary replica.
+
+Proceed to the Activity to simulate a failover and test a new primary replica is active.
+
+
+
+
+
+Follow the steps in his activity to simulate a failover and see how a new secondary replica has switched roles to become a primary, resulting in almost no downtime to the application.
+
+1. From the last step in the previous section activity, the primary and secondary replicas were synchronized. Verify the replica state by running the following commands or the script **step6_check_replicas.sh**
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -icheckreplicas.sql -Y30`
+
+ Your results should look like the following
+
+
+
+ 2. If in your results, if the replica_server_name is mssql2-0 for the PRIMARY, you will need to edit the **failover.yaml** file. If not, you can skip to step 3.
+
+ The **failover.yaml** file declares how to manually trigger a failover to another replica but is setup so you must specify which replica to target as the new primary. The workshop was built so that mssql2-0 would be the new primary. In some rare cases, mssql2-0 might have been chosen as the original primary. If this is the case, you need to edit this part of the failover.yaml file and change the name mssql2-0 to mssql3-0
+
+
+
+ Once you have edited and saved the changes, proceed to step 3.
+
+3. To manually trigger the failover, apply the failover.yaml file which will submit a *job* to trigger the change for a failover. Run the following command or the script **step10_failover.sh**
+
+ `oc apply -f failover.yaml`
+
+ You should see results like the following
+
+
serviceaccount/manual-failover created
+ role.rbac.authorization.k8s.io/manual-failover created
+ rolebinding.rbac.authorization.k8s.io/manual-failover created
+ job.batch/manual-failover created
+
+4. The failover should happen fairly quickly. You can run the following commands or script **step6_check_replicas.sh** to verify the new PRIMARY switch has occurred
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -icheckreplicas.sql -Y30`
+
+ Your results should show a new PRIMARY like the following
+
+
+
+5. Now run the following commands or use the script **step9_queryag.sh** to see your data is still synchronized and by using the LoadBalancers the application has not changed even though you are connected to a new primary and secondary
+
+ `SERVERIP=$(oc get service | grep ag1-primary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT 'Connected to Primary = '+@@SERVERNAME;USE testag;SELECT * FROM ilovesql" -Y60`
+ `SERVERIP=$(oc get service | grep ag1-secondary | awk {'print $4'})`
+ `PORT=1433`
+ `sqlcmd -Usa -PSql2019isfast -S$SERVERIP,$PORT -Q"SELECT 'Connected to Secondary = '+@@SERVERNAME;USE testag;SELECT * FROM ilovesql" -K ReadOnly -Y60`
+
+ Your results should look similar to the time you ran this query from the previous section activity except the server names should be different and reflect the new primary and secondary replicas.
+
+
------------------------------------------------------------
+ Connected to Primary = mssql2-0
+ (1 rows affected)
+ Changed database context to 'testag'.
+ col1 col2
+ ----------- ------------------------------------------------------------
+ 1 SQL Server 2019 is fast, secure, and highly available
+ (1 rows affected)
+
+
------------------------------------------------------------
+ Connected to Secondary = mssql1-0
+ (1 rows affected)
+ Changed database context to 'testag'.
+ col1 col2
+ ----------- ------------------------------------------------------------
+ 1 SQL Server 2019 is fast, secure, and highly available
+ (1 rows affected)
+
+In this activity, you have simulated a failover for your deployed Availability Group and seen your data is still consistent, intact, and your application has connected with no changes and almost no downtime.
+
+
+
+
+
+
The Knox Gateway provides a single access point for all REST interactions, used for security
-
Apache Livy
Tob submission system for Apache Spark
+
Apache Livy
Job submission system for Apache Spark
Apache Spark
In-memory large-scale, scale-out data processing architecture used by SQL Server
Python, R, Java, SparkML
ML/AI programming languages used for Machine Learning and AI Model creation
Azure Data Studio
Tooling for SQL Server, HDFS, Kubernetes cluster management, T-SQL, R, Python, and SparkML languages
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/00 - Prerequisites.md b/sqlserver2019bigdataclusters/SQL2019BDC/00 - Prerequisites.md
index 0090161f..e6239fb0 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/00 - Prerequisites.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/00 - Prerequisites.md
@@ -1,6 +1,6 @@
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
#### A Microsoft Course from the SQL Server team
@@ -89,7 +89,7 @@ Get-WindowsUpdate
Install-WindowsUpdate
-*Note: If you get an error during this update process, evaluate it to see if it is fatal. You may receive certain driver errors if you are using a Virtual Machine, this can be safely ignored.*
+*Note: If you get an error during this update process, evaluate it to see if it is fatal. You may recieve certain driver errors if you are using a Virtual Machine, this can be safely ignored.*
Install Chocolatey Windows package Manager
@@ -109,7 +109,7 @@ Your environment variables control how the cluster will be built.
The variables for **name**, **password** and **e-mail** for the big data cluster is provided to you when you request access to the Early Adopter program.
-*(Note that in production, you'll set these environment variables permanently using the Control Panel or by adding them with a Registry command, and may be handled by an improved installation experience)*
+*(Note that in production, you'll set these environment variables permanently using the Control Panel or by adding them with a Registry command, and will be handled by an improved and secure installation experience)*
Activity 4: Install Azure CLI
@@ -138,9 +138,7 @@ Note: Python can install in multiple locations based on various conditions. To s
`where python`
-(In Linux, `which Python`)
-
-In most cases, Choclaty will add Python and pip to your path. Otherwise, you may want to find and add it manually.
+Note that the Chocolatey Package Manager should have set the Path variable for Python and pip, but if they are not in your path you can learn how to set that here.
Activity 6: Install kubectl
@@ -160,6 +158,8 @@ the `mssqlctl` program then deploys the SQL Server big data cluster environment
You must delete the old version before the class. It is updated quite frequently during the preview phase.
You may need to run these steps in CMD rather than PowerShell.
+
+You may have `pip3` instead of `pip` as the command.
@@ -172,9 +172,14 @@ choco upgrade kubernetes-cli
python -m pip install --upgrade pip
+REM for 2.7 and lower:
pip uninstall mssqlctl
-pip install --extra-index-url https://private-repo.microsoft.com/python/ctp-2.2 mssqlctl
+REM for 2.3:
+pip uninstall -r https://private-repo.microsoft.com/python/ctp-2.3/mssqlctl/requirements.txt
+
+pip install -r https://private-repo.microsoft.com/python/ctp-2.4/mssqlctl/requirements.txt
+
Activity 8: Install Azure Data Studio and Extensions
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/01 - The Big Data Landscape.md b/sqlserver2019bigdataclusters/SQL2019BDC/01 - The Big Data Landscape.md
index 692ab9ea..3213ea8b 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/01 - The Big Data Landscape.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/01 - The Big Data Landscape.md
@@ -1,523 +1,525 @@
-
-
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
-
-#### A Microsoft Course from the SQL Server team
-
-
-
-
The Big Data Landscape
-
-In this workshop you'll cover using a Process and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic. There's a lot here - so focus on understanding the overall system first, then come back and explore each section.
-
-(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
-
-You'll cover the following topics in this Module:
-
-
-
-Businesses require near real-time insights from ever-larger sets of data. Large-scale data ingestion requires scale-out storage and processing in ways that allow fast response times. In addition to simply querying this data, organizations want full analysis and even predictive capabilities over their data. A few industry examples of applications of technology that create large data sets are:
-
-
-
-While solutions for large-scale data processing exist, they are often batch-based, which has a lag in the time from query to response. Also, batch systems such as Hadoop are complicated to set up and manage. Operational data is often stored in Relational Database systems on-premises, and joining that data to larger-scale cloud systems exposes security weaknesses and brittle architectures.
-
-
Activity: Define Your Use-Case
-
-In this activity, you will define the use-case that best fits the industry where you work.
-
-While you will review the design for a complete solution in this workshop, it extrapolates to many other scenarios. You need to find the top scenarios specific your industry.
-
-Steps
-
-
Open a web browser and any professional notes or resources you use at work.
-
Spend 5 minutes looking online for the terms "Big Data" and "Machine Learning" and "Top" for your industry. For example: Hospital Big Data Machine Learning Top
-
Record your findings in your personal workshop notes.
-
-This solution uses an example of a retail organization that has multiple data sources, but it has many applications to the other industries listed above. It serves as an end-to-end scenario where you will learn the technologies and processes you can use to create multiple solutions.
-
-
-
-Wide World Importers (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world. They have a great training program for new employees, that focuses on connecting with their customers and providing great face-to-face customer service. This strong focus on customer relationships has helped set WWI apart from their competitors.
-
-WWI has now added web and mobile commerce to their platform, which has generated a significant amount of additional data, and data formats. These new platforms were added without integrating into the OLTP system data or Business Intelligence infrastructures. As a result, "silos" of data stores have developed.
-
-
-
-Now they want to expand their reach to customers around the world through web and mobile e-commerce. But they don't want to just simply make their inventory available online. They want to build upon their track record of strong customer connections, and engage with their customers through personalized, high-quality application experiences that incorporate data and intelligence.
-
-The technology team at WWI has recognized that moving to an omni-channel strategy has quickly outgrown their ability to handle data. They anticipate the following solutions needed to reach more customers and grow the business:
-
- - Scale data systems to reach more consumers
- - Unlock business insights from multiple sources of structured and unstructured data
- - Apply deep analytics with high-performance responses
- - Enable AI into apps to actively engage with customers
-
-Prior to expanding to their current omni-channel strategy, WWI had a simple Point of Sale (POS) application that handled customer orders at each retail store. The back-end was a series of service layers used to process orders and store them in a SQL database. They had designed their systems and tuned them to handle this level of data.
-
-
-
-As they added new e-commerce channels to expand the customer base, consumer demand also increased. This increased demand from more customers ordering products through more channels generated more data. Now WWI has new challenges to address:
-
- - Increased consumer demand, leading to increased app data
- - They are unable to determine business trends because of siloed insights
- - They have a rising data management footprint, increasing cost and complexity
- - New development challenges resulting from more deployment targets and duplicated code
-
-
-
-
-
-In the 04 - Operationalization Module, you'll learn how the team solves each of these challenges.
-
-You can read more about Wide World Importers here.
-
-
-
-
-
-
-
-In this lab you will deploy a SQL Server 2019 big data cluster to the Azure Kubernetes Service. You will need a client machine and a subscription to Microsoft Azure where you can create assets. This will take some time, so you'll do this now as you work through the next few modules, and then return to the installation process in a later module.
-
-Using the following steps, you will create a Resource Group in Azure that will hold your BDC on AKS. When you complete your lab you can delete this Resource Group which will stop the Azure charges for this course.
-
-
-
-In this section you will learn more about the designs of two primary operating systems (Microsoft Windows and Linux) and how they are suited to working with Big Data.
-
-NOTE: This is not meant to be a comprehensive discussion of the merits of an operating system or ecostructure. The goal is to understand the salient features in each architecture as they pertain to processing large sets of data.
-
-When working with large-scale, distributed data processing systems, there are two primary concepts to keep in mind: You should move as much processing to the data location as possible, and the storage system should be abstracted so that the code does not have to determine where to get its data from each node. Both Windows and Linux have specific attributes to keep in mind as you select hardware, drivers, and configurations for your systems.
-
-
Storage
-
-The general rules for storage in a distributed data processing environment are that the disks should be as fast as possible, you should optimize for the best number of nodes that will store and process the data, and that the abstraction system (in the case of SQL Server BDC, this is HDFS in many cases) be at the latest version and optimized settings.
-
-For general concepts for Windows storage see the following resources:
-
-
-
-Both Windows and Linux (in the x86 architecture) are Symmetric Multiprocessing systems, which means that all processors are addressed as a single unit. In general, distributed processing systems should have larger, and more, processors at the "head" node. General Purpose (GP) processors are normally used in these systems, but for certain uses such as Deep Learning or Artificial Intelligence, Purpose-Built processors such as Graphics Processing Unit (GPU's) are used within the environment, and in some cases, Advanced RISC Machines (ARM) chips can be brought into play for specific tasks. For the purposes of this workshop, you will not need these latter two technologies, although both Windows and Linux support them.
-
-NOTE: Linux can be installed on chipsets other than x86 compatible systems. In this workshop, you will focus on x86 systems.
-
-
-
-Storage Nodes in a distributed processing system need a nominal amount of memory, and the processing nodes in the framework (Spark) included with SQL Server BDC need more.
-Both Linux and Windows support large amounts of memory (most often as much as the system can hold) natively, and no special configuration for memory is needed.
-
-Modern operating systems use a temporary area on storage to act as memory for light caching and also as being able to extend RAM memory. This should be avoided at all costs in both Windows and Linux.
-
-While technically a Processor system, NUMA (Non-Uniform Memory Access) systems allow for special memory configurations to be mixed in a single server by a processor set. Both Linux and Windows support NUMA access.
-
-
-
-The general guidance for large-scale distributed processing systems is that the network between them be as fast and collision-free (in the case of TCP/IP) as possible. The networking stacks in both Windows and Linux require no special settings within the operating system, but you should thoroughly understand the driver settings as they apply to each NIC installed in your system, that each setting is best optimized for your networking hardware and topology and that the drivers are at the latest tested versions.
-
-Although a full discussion of the TCP/IP protocol is beyond the scope of this workshop, it's important that you have a good understanding of how it works, since it permeates every part of the SQL Server BDC architecture. You can get a quick overview of TCP/IP here.
-
-The other rule you should keep in mind is that in general you should have only the presentation of the data handled by the workstation or device that accesses the solution. Do not move large amounts of data back and forth over the network to have the data processed locally. That being said, there are times (such as certain IoT scenarios) where subsets of data should be moved to the client, but you will not face those scenarios in this workshop.
-
-
5-Minute Linux Overview for the Windows Professional
-
-The best way to learn an operating system is to install it and perform real-world tasks. (A good place to learn a lot more about Linux is here). For this workshop, the essential concepts you need from the SQL Server perspective are:
-
-
Unlike Windows, which is written and controlled by Microsoft, Linux is comprised only of a small Kernel, and then all other parts of the operating system are created by commercial entities or the public, and packaged up into a Distribution. These Distributions have all of the complementary functions to the operating system, and in some cases a graphical interface and other files. The Distributions supported by SQL Server are RedHat, Ubuntu, and SuSE.
Software installation on Linux can be done manually by copying files or compiling source code. A Package Manager is a tool that simplifies this process, and is based on the Distribution. The two package managers you will see most often in SQL Server are yum and apt.
Like Windows, organized as a tree, but referenced by a forward-slash /. There are no drive letters in Linux - everything is "mounted" to what looks like a directory.
Users and Groups are stored in protected files, called /etc/passwd and /etc/group. These files and locations may be augmented or slightly different based in the distribution. By default, each user has very low privileges and must be granted access to files or directories. The sudo command allows you to run as a privileged user (known as root) or as another user.
-
-
-
-The essential commands you should know for this workshop are below. In Linux you can often send the results of one command to another using the "pipe" symbol, similar to PowerShell: |.
-
-
Find the mounted file systems, and then show the free space in them.
-
Show the current directory.
-
Show the files in the current directory.
-
Create a new directory, navigate to it, and create a file called test.txt with the words This is a test in it. (hint: us the nano editor or the echo command)
-
-Bare-metal installations of an operating system such as Windows are deployed on hardware using a Kernel, and additional software to bring all of the hardware into a set of calls.
-
-
Virtual machines
-
-One abstraction layer above installing software directly on hardware is using a Hypervisor. In essence, this layer uses the base operating system to emulate hardware. You install an operating system (called a *Guest* OS) on the Hypervisor (called the *Host*), and the Guest OS acts as if it is on bare-metal.
-
-
-
-
-
-In this abstraction level, you have full control (and responsibility) for the entire operating system, but not the hardware. This isolates all process space and provides an entire "Virtual Machine" to applications. For scale-out systems, a Virtual Machine allows for a distribution and control of complete computer environments using only software.
-
-You can read the details of Microsoft's Hypervisor here.
-
-
Containers (Docker)
-
-The next level of Abstraction is a Container. There are various types of Container technologies, in this workshop, you will focus on Docker.
-
-A Docker Container is provided by the Docker runtime engine, which sits above the operating system (Windows or Linux). In this abstraction, you do not control the hardware or the operating system. The Container has a very small Kernel in it, and can contain binaries such as Python, R, SQL Server, or other binaries. A Container with all its binaries is called an Image.
-
-(NOTE: The Container Image Kernel can run on Windows or Linux, but you will focus on the Linux Kernel Containers in this workshop.)
-
-
-
-
-
-This abstraction holds everything for an application to isolate it from other running processes. It is also completely portable - you can create an image on one system, and another system can run it so long as the Docker Runtime is installed. Containers also start very quickly, are easy to create (called Composing) using a simple text file with instructions of what to install on the image. The instructions pull the base Kernel, and then any binaries you want to install. Several pre-built Containers are already available, SQL Server is one of these. You can read more about installing SQL Server on Docker here.
-
-You can have several Containers running at any one time, based on the amount of hardware resources where you run it. For scale-out systems, a Container allows for distribution and control of complete applications using only declarative commands.
-
-You can read more about Docker here.
-
-
Container Orchestration (Kubernetes)
-
-For Big Data systems, having lots of Containers is very advantageous to segment purpose and performance profiles. However, dealing with many Container Images, allowing persisted storage, and interconnecting them for network and internetwork communications is a complex task. Kubernetes is an open source Container orchestrator, which can scale Container deployments according to need. The following table defines some important Kubernetes terminology:
-
-
-
-
Component
Used for
-
-
Cluster
A Kubernetes cluster is a set of machines, known as nodes. One node controls the cluster and is designated the master node; the remaining nodes are worker nodes. The Kubernetes master is responsible for distributing work between the workers, and for monitoring the health of the cluster.
-
Node
A node runs containerized applications. It can be either a physical machine or a virtual machine. A Kubernetes cluster can contain a mixture of physical machine and virtual machine nodes.
-
Pod
A pod is the atomic deployment unit of Kubernetes. A pod is a logical group of one or more containers-and associated resources-needed to run an application. Each pod runs on a node; a node can run one or more pods. The Kubernetes master automatically assigns pods to nodes in the cluster.
-
-
-
-
-
-
-
-You can learn much more about Kubernetes here.
-
-
-In SQL Server big data clusters, Kubernetes is responsible for the state of the SQL Server big data clusters; Kubernetes builds and configures the cluster nodes, assigns pods to nodes, and monitors the health of the cluster.
-
-(You'll cover the storage aspects of Kubernetes clusters in a moment.)
-
-
Activity: Familiarize Yourself with Kubernetes using minikube
-
-To practice with Kubernetes, you will use an online emulator to work with the minikube platform.
-
-
-
-Traditional storage uses a call from the operating system to an underlying I/O system, as you learned earlier. These file systems are either directly connected to the operating system or appear to be connected directly using a Storage Area Network. The blocks of data are stored and managed by the operating system.
-
-For large scale-out data systems, the mounting point for an I/O is another abstraction. For SQL Server BDC, the most commonly used scale-out file system is the Hadoop Data File System, or HDFS. HDFS is a set of Java code that gathers disparate disk subsystems into a Cluster which is comprised of various Nodes - a NameNode, which manages the cluster's metadata, and DataNodes that physically store the data. Files and directories are represented on the NameNode by a structure called inodes. Inodes record attributes such as permissions, modification and access times, and namespace and diskspace quotas.
-
-
-
-There are two primary storage concepts you will work with in SQL Server Big Data Clusters: the HDFS layer, and SQL Server. Fro HDFS, it's important to know the basics of how it works.
-
-
-
-There are three primary tools and utilities you will use to control the SQL Server big data cluster:
-
- - kubectl
- - mssqlctl
- - Azure Data Studio
-
-
Managing and Monitoring the SQL Server big data cluster (mssqlctl)
-
-The **mssqlctl** command-line utility is written in Python and can be installed on your workstation using the **pip** command in Python. You will see how to install this utility in the *Planning, Installation and Configuration* module.
-
-The **mssqlctl** utility enables cluster administrators to bootstrap and manage big data clusters via the REST APIs exposed by the Controller service. The controller is deployed and hosted in the same Kubernetes namespace where the customer wants to build out a big data cluster. The Controller is responsible for core logic for deploying and managing a big data cluster.
-
-The Controller service is installed by a Kubernetes administrator during cluster bootstrap, using the mssqlctl command-line utility.
-
-You'll explore further operations with these tools in the Management and Monitoring module.
-
-
SQL Server BDC Programming and GUI Surface (Azure Data Studio and Jupyter Notebooks)
-
-
Jupyter Notebooks
-
-A Jupyter *Notebook* is a web-page-based interface consisting of *Cells* that can contain text (using the Markdown specification) or code. The code depends on the Kernel that has been installed for the Notebook. Traditionally, Python and R Kernels are installed by default.
-
-*Notebook Servers* run **.ipynb** files (the Notebooks). You can install a Notebook Server locally, remotely, or you can use something like Azure Notebooks which provides a free, quick, easy way to work with and share Notebooks. (It's all a bit like a specific kind of web server).
-
-*Libraries* are a container on your Notebook server where you can have Notebooks, code, directories and other files.
-
-Notebooks are JSON files that contain areas called *Cells*, which have text or code in them. When you double-click a Notebook, the Notebook server renders and can display text or run code, such as R or Python, using a Kernel. Cells can hold text (such as *Markdown*, *HTML*, or *LaTeX*) which you can mix together, or Code. Double-click a Cell in a Notebook to edit it, and then click the "Run" button to render what you typed. Code runs and displays an output below the cell. You can toggle the result for code to show or hide it.
-
-*Markdown* is a simplified markup language for text. Use it for general text and simple graphics. You can read more about Markdown here, and there's a great cheat-sheet on Markdown here.
-
-You'll use Notebooks within Azure Data Studio to work with Spark, which you'll learn about in a moment. Learn more about Jupyter Notebooks here.
-
-
Azure Data Studio
-
-*Azure Data Studio* is a cross-platform database tool to manage and program on-premises and cloud data platforms on Windows, MacOS, and Linux. It is extensible, and one of these extensions is how you work with Azure Data Studio code and Jupyter Notebooks. It is built on the Visual Studio Code shell. The editor in Azure Data Studio has Intellisense, code snippets, source control integration, and an integrated terminal.
-
-If you have not completed the prerequisites for this workshop you can install Azure Data Studio from this location, and you will install the Extension to work with SQL Server big data clusters in a future module.
-
-You can learn more about Azure Data Studio here.
-
-
-
-
-
-You'll explore further operations with the Azure Data Studio in the Operationalization module.
-
-
-
-
-In any large data system, you will need a way to bring the data in. In some cases, you will edit the data either on the way in, or after it is staged using a process called "Extract, Transform and Load" (ETL) or leave the data "pure" and unaltered (common in Data Science projects), using a process called "Extract, Load and Transform" (ELT).
-
-In SQL Server big data clusters, you'll learn about three ways that the system interacts with large sets of data:
-
- - "Virtualize" the data by pushing down the query to the source system (no data is ingested in this scenario)
- - Ingesting data into SQL Server Tables, using the PolyBase feature or into standard SQL Server constructs
- - Loading data into HDFS
-
-In Data Virtualization, no data is brought into storage - it is queried from it's source. It's important to think about the network bandwidth per query in this scenario. The next two scenarios do bring data into the system, either into SQL Server tables within the big data cluster or into the HDFS mount points.
-
-In both cases, the first considerations for loading data are source-data locality and network bandwidth, utilization, and predictability of the path to the SQL Server BDC destination. Depending on where the data originates, network bandwidth will play a major part in your loading performance. For source data residing on premises, network throughput performance and predictability can be enhanced with a service such as a dedicated network path or "hot potato routing". You must consider the current average bandwidth, utilization, predictability, and maximum capabilities of your current public Internet-facing, source-to-destination route, regardless of the method you are using.
-
-In this section you'll learn more about working with the last two options for loading data into the SQL Server BDC architecture. In the *Operationalization* module you'll learn more about Data Virtualization, and also see a practical method for working with data ingestion and pipelines. You can use multiple methods to control the pipeline or workflow of your system, depending on the requirements and other parts of the Data Architecture at your location. In general, you will use the Team Data Science Process to define your objectives, identify and explore data sources, perform any transforms and do any Feature Engineering for Machine Learning, and then Operationalize your solution.
-
-Next you'll explore the two locations you can store data in the system, and then you'll learn about creating a Pipeline system to standardize and automate the process. In the Operationalization Module you'll see practical examples of each of these.
-
-
-
-As described earlier, you can use various methods to ingest data ad-hoc and as-needed for your two data targets (HDFS and SQL Server Tables. A more holistic archicture is to use a Pipeline system that can define sources, triggers and events, transforms, targets, and has logging and tracking capabilities. The Microsoft Azure Data Factory provides all of the capabilities, and often serves as the mechanism to transfer data to and from on-premises, in-cloud, and other sources and targets. ADF can serve as a full data pipeline system, as described here.
-
-
-
-
-
-A full description of Azure Data Factory is here.
-
-
Data Pipelines using Apache Spark
-
-Apache Spark is an analytics engine for large-scale data. It can be used with data stored in HDFS, and has connectors to work with data in SQL Server as well.
-
-
-
-
-
-While Spark is used for all phases of the data processing lifecycle and can compliment and extend the capabilities of SQL Server, it is also often used to ingest and transform data as a data pipeline. An important concept to keep in mind when you ingest data using Spark is that Spark has three main data storage representations it works with once they are ingested:
-
- - RDD (2011): Distribution of JVM objects, Functional Operators (map, filter, etc)
- - DataFrame (2013): Distribution of Row objects, Expression-based operations and UDF’s, Logical plans and an optimizer, Fast, efficient internal representations
- - Dataset (2015): Internally rows, externally JVM objects – type-safe + fast, Slower than DF’s, not as suited for interactive analysis
-
-You can learn more about these data representations here. You'll see an example of ingesting data using Apache Spark in the Operationalization module.
-
-
-
-
-
-
-
-Next, Continue to SQL Server big data cluster Components.
+
+
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
+
+#### A Microsoft Course from the SQL Server team
+
+
+
+
The Big Data Landscape
+
+In this workshop you'll cover using a Process and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic. There's a lot here - so focus on understanding the overall system first, then come back and explore each section.
+
+(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
+
+You'll cover the following topics in this Module:
+
+
+
+Businesses require near real-time insights from ever-larger sets of data. Large-scale data ingestion requires scale-out storage and processing in ways that allow fast response times. In addition to simply querying this data, organizations want full analysis and even predictive capabilities over their data. A few industry examples of applications of technology that create large data sets are:
+
+
+
+
Industry Sector
Primary Use-Cases
+
+
Retail
Demand prediction
+
In-store analytics
+
Supply chain optimization
+
Customer retention
+
Cost/Revenue analytics
+
HR analytics
+
Inventory control
+
Finance
Cyberattack Prevention
+
Fraud detection
+
Customer segmentation
+
Market analysis
+
Risk analysis
+
Blockchain
+
Customer retention
+
Healthcare
Fiscal control analytics
+
Disease Prevention prediction and classification
+
Clinical Trials optimization
+
Patient load analysis
+
Episode analytics
+
Public Sector
Revenue prediction
+
Education effectiveness analysis
+
Transportation analysis and prediction
+
Energy demand and supply prediction and control
+
Defense readiness predictions and threat analysis
+
Manufacturing
Predictive Maintenance (PdM)
+
Anomaly Detection
+
Pattern analysis
+
Agriculture
Food Safety analysis
+
Crop forecasting
+
Market forecasting
+
Pipeline Optimization
+
+
+
+While solutions for large-scale data processing exist, they are often batch-based, which has a lag in the time from query to response. Also, batch systems such as Hadoop are complicated to set up and manage. Operational data is often stored in Relational Database systems on-premises, and joining that data to larger-scale cloud systems exposes security weaknesses and brittle architectures.
+
+
Activity: Define Your Use-Case
+
+In this activity, you will define the use-case that best fits the industry where you work.
+
+While you will review the design for a complete solution in this workshop, it extrapolates to many other scenarios. You need to find the top scenarios specific your industry.
+
+Steps
+
+
Open a web browser and any professional notes or resources you use at work.
+
Spend 5 minutes looking online for the terms "Big Data" and "Machine Learning" and "Top" for your industry. For example: Hospital Big Data Machine Learning Top
+
Record your findings in your personal workshop notes.
+
+This solution uses an example of a retail organization that has multiple data sources, but it has many applications to the other industries listed above. It serves as an end-to-end scenario where you will learn the technologies and processes you can use to create multiple solutions.
+
+
+
+Wide World Importeres (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world. They have a great training program for new employees, that focuses on connecting with their customers and providing great face-to-face customer service. This strong focus on customer relationships has helped set WWI apart from their competitors.
+
+WWI has now added web and mobile commerce to their platform, which has generated a significant amount of additional data, and data formats. These new platforms were added without integrating into the OLTP system data or Business Intelligence infrastructures. As a result, "silos" of data stores have developed.
+
+
+
+Now they want to expand their reach to customers around the world through web and mobile e-commerce. But they don't want to just simply make their inventory available online. They want to build upon their track record of strong customer connections, and engage with their customers through personalized, high-quality application experiences that incorporate data and intelligence.
+
+The technology team at WWI has recognized that moving to an omni-channel strategy has quickly outgrown their ability to handle data. They anticipate the following solutions needed to reach more customers and grow the business:
+
+ - Scale data systems to reach more consumers
+ - Unlock business insights from multiple sources of structured and unstructured data
+ - Apply deep analytics with high-performance responses
+ - Enable AI into apps to actively engage with customers
+
+Prior to expanding to their current omni-channel strategy, WWI had a simple Point of Sale (POS) application that handled customer orders at each retail store. The back-end was a series of service layers used to process orders and store them in a SQL database. They had designed their systems and tuned them to handle this level of data.
+
+
+
+As they added new e-commerce channels to expand the customer base, consumer demand also increased. This increased demand from more customers ordering products through more channels generated more data. Now WWI has new challenges to address:
+
+ - Increased consumer demand, leading to increased app data
+ - They are unable to determine business trends because of siloed insights
+ - They have a rising data management footprint, increasing cost and complexity
+ - New development challenges resulting from more deployment targets and duplicated code
+
+
+
+
+
+In the 04 - Operationalization Module, you'll learn how the team solves each of these challenges.
+
+You can read more about Wide World Importers here.
+
+
+
+
+
+
+
+In this lab you will deploy a SQL Server 2019 big data cluster to the Azure Kubernetes Service. You will need a client machine and a subscription to Microsoft Azure where you can create assets. This will take some time, so you'll do this now as you work through the next few modules, and then return to the installation process in a later module.
+
+*NOTE: Your instructor may walk through these steps if the class environment does not have enough resources or time for each person to deploy. If so, they will provide you credentials to work with a pre-configured system in class.*
+
+Using the following steps, you will create a Resource Group in Azure that will hold your BDC on AKS. When you complete your lab you can delete this Resource Group which will stop the Azure charges for this course.
+
+
+
+In this section you will learn more about the designs of two primary operating systems (Microsoft Windows and Linux) and how they are suited to working with Big Data.
+
+NOTE: This is not meant to be a comprehensive discussion of the merits of an operating system or ecostructure. The goal is to understand the salient features in each architecture as they pertain to processing large sets of data.
+
+When working with large-scale, distributed data processing systems, there are two primary concepts to keep in mind: You should move as much processing to the data location as possible, and the storage system should be abstracted so that the code does not have to determine where to get its data from each node. Both Windows and Linux have specific attributes to keep in mind as you select hardware, drivers, and configurations for your systems.
+
+
Storage
+
+The general rules for storage in a distributed data processing environment are that the disks should be as fast as possible, you should optimize for the best number of nodes that will store and process the data, and that the abstraction system (in the case of SQL Server BDC, this is HDFS in many cases) be at the latest version and optimized settings.
+
+For general concepts for Windows storage see the following resources:
+
+
+
+Both Windows and Linux (in the x86 architecture) are Symmetric Multiprocessing systems, which means that all processors are addressed as a single unit. In general, distributed processing systems should have larger, and more, processors at the "head" node. General Purpose (GP) processors are normally used in these systems, but for certain uses such as Deep Learning or Artificial Intelligence, Purpose-Built processors such as Graphics Processing Unit (GPU's) are used within the environment, and in some cases, Advanced RISC Machines (ARM) chips can be brought into play for specific tasks. For the purposes of this workshop, you will not need these latter two technologies, although both Windows and Linux support them.
+
+NOTE: Linux can be installed on chipsets other than x86 compatible systems. In this workshop, you will focus on x86 systems.
+
+
+
+Storage Nodes in a distributed processing system need a nominal amount of memory, and the processing nodes in the framework (Spark) included with SQL Server BDC need more.
+Both Linux and Windows support large amounts of memory (most often as much as the system can hold) natively, and no special configuration for memory is needed.
+
+Modern operating systems use a temporary area on storage to act as memory for light caching and also as being able to extend RAM memory. This should be avoided at all costs in both Windows and Linux.
+
+While technically a Processor system, NUMA (Non-Uniform Memory Access) systems allow for special memory configurations to be mixed in a single server by a processor set. Both Linux and Windows support NUMA access.
+
+
+
+The general guidance for large-scale distributed processing systems is that the network between them be as fast and collision-free (in the case of TCP/IP) as possible. The networking stacks in both Windows and Linux require no special settings within the operating system, but you should thoroughly understand the driver settings as they apply to each NIC installed in your system, that each setting is best optimized for your networking hardware and topology and that the drivers are at the latest tested versions.
+
+Although a full discussion of the TCP/IP protocol is beyond the scope of this workshop, it's important that you have a good understanding of how it works, since it permeates every part of the SQL Server BDC architecture. You can get a quick overview of TCP/IP here.
+
+The other rule you should keep in mind is that in general you should have only the presentation of the data handled by the workstation or device that accesses the solution. Do not move large amounts of data back and forth over the network to have the data processed locally. That being said, there are times (such as certain IoT scenarios) where subsets of data should be moved to the client, but you will not face those scenarios in this workshop.
+
+
5-Minute Linux Overview for the Windows Professional
+
+The best way to learn an operating system is to install it and perform real-world tasks. (A good place to learn a lot more about Linux is here). For this workshop, the essential concepts you need from the SQL Server perspective are:
+
+
Unlike Windows, which is written and controlled by Microsoft, Linux is comprised only of a small Kernel, and then all other parts of the operating system are created by commercial entities or the public, and packaged up into a Distribution. These Distributions have all of the complementary functions to the operating system, and in some cases a graphical interface and other files. The Distributions supported by SQL Server are RedHat, Ubuntu, and SuSE.
Software installation on Linux can be done manually by copying files or compiling source code. A Package Manager is a tool that simplifies this process, and is based on the Distribution. The two package managers you will see most often in SQL Server are yum and apt.
Like Windows, organized as a tree, but referenced by a forward-slash /. There are no drive letters in Linux - everything is "mounted" to what looks like a directory.
Users and Groups are stored in protected files, called /etc/passwd and /etc/group. These files and locations may be augmented or slightly different based in the distribution. By default, each user has very low privileges and must be granted access to files or directories. The sudo command allows you to run as a privileged user (known as root) or as another user.
+
+
+
+The essential commands you should know for this workshop are below. In Linux you can often send the results of one command to another using the "pipe" symbol, similar to PowerShell: |.
+
+
Find the mounted file systems, and then show the free space in them.
+
Show the current directory.
+
Show the files in the current directory.
+
Create a new directory, navigate to it, and create a file called test.txt with the words This is a test in it. (hint: us the nano editor or the echo command)
+
+Bare-metal installations of an operating system such as Windows are deployed on hardware using a Kernel, and additional software to bring all of the hardware into a set of calls.
+
+
Virtual machines
+
+One abstraction layer above installing software directly on hardware is using a Hypervisor. In essence, this layer uses the base operating system to emulate hardware. You install an operating system (called a *Guest* OS) on the Hypervisor (called the *Host*), and the Guest OS acts as if it is on bare-metal.
+
+
+
+
+
+In this abstraction level, you have full control (and responsibility) for the entire operating system, but not the hardware. This isolates all process space and provides an entire "Virtual Machine" to applications. For scale-out systems, a Virtual Machine allows for a distribution and control of complete computer environments using only software.
+
+You can read the details of Microsoft's Hypervisor here.
+
+
Containers (Docker)
+
+The next level of Abstraction is a Container. There are various types of Container technologies, in this workshop, you will focus on Docker.
+
+A Docker Container is provided by the Docker runtime engine, which sits above the operating system (Windows or Linux). In this abstraction, you do not control the hardware or the operating system. The Container has a very small Kernel in it, and can contain binaries such as Python, R, SQL Server, or other binaries. A Container with all its binaries is called an Image.
+
+(NOTE: The Container Image Kernel can run on Windows or Linux, but you will focus on the Linux Kernel Containers in this workshop.)
+
+
+
+
+
+This abstraction holds everything for an application to isolate it from other running processes. It is also completely portable - you can create an image on one system, and another system can run it so long as the Docker Runtime is installed. Containers also start very quickly, are easy to create (called Composing) using a simple text file with instructions of what to install on the image. The instructions pull the base Kernel, and then any binaries you want to install. Several pre-built Containers are already available, SQL Server is one of these. You can read more about installing SQL Server on Docker here.
+
+You can have several Containers running at any one time, based on the amount of hardware resources where you run it. For scale-out systems, a Container allows for distribution and control of complete applications using only declarative commands.
+
+You can read more about Docker here.
+
+
Container Orchestration (Kubernetes)
+
+For Big Data systems, having lots of Containers is very advantageous to segment purpose and performance profiles. However, dealing with many Container Images, allowing persisted storage, and interconnecting them for network and internetwork communications is a complex task. Kubernetes is an open source Container orchestrator, which can scale Container deployments according to need. The following table defines some important Kubernetes terminology:
+
+
+
+
Component
Used for
+
+
Cluster
A Kubernetes cluster is a set of machines, known as nodes. One node controls the cluster and is designated the master node; the remaining nodes are worker nodes. The Kubernetes master is responsible for distributing work between the workers, and for monitoring the health of the cluster.
+
Node
A node runs containerized applications. It can be either a physical machine or a virtual machine. A Kubernetes cluster can contain a mixture of physical machine and virtual machine nodes.
+
Pod
A pod is the atomic deployment unit of Kubernetes. A pod is a logical group of one or more containers-and associated resources-needed to run an application. Each pod runs on a node; a node can run one or more pods. The Kubernetes master automatically assigns pods to nodes in the cluster.
+
+
+
+
+
+
+
+You can learn much more about Kubernetes here. We're using the Azure Kubernetes Service (AKS) in this workshop, and they have a great tutorial here.
+
+
+In SQL Server big data clusters, Kubernetes is responsible for the state of the SQL Server big data clusters; Kubernetes builds and configures the cluster nodes, assigns pods to nodes, and monitors the health of the cluster.
+
+(You'll cover the storage aspects of Kubernetes clusters in a moment.)
+
+
Activity: Familiarize Yourself with Kubernetes using minikube
+
+To practice with Kubernetes, you will use an online emulator to work with the minikube platform.
+
+
+
+Traditional storage uses a call from the operating system to an underlying I/O system, as you learned earlier. These file systems are either directly connected to the operating system or appear to be connected directly using a Storage Area Network. The blocks of data are stored and managed by the operating system.
+
+For large scale-out data systems, the mounting point for an I/O is another abstraction. For SQL Server BDC, the most commonly used scale-out file system is the Hadoop Data File System, or HDFS. HDFS is a set of Java code that gathers disparate disk subsystems into a Cluster which is comprised of various Nodes - a NameNode, which manages the cluster's metadata, and DataNodes that physically store the data. Files and directories are represented on the NameNode by a structure called inodes. Inodes record attributes such as permissions, modification and access times, and namespace and diskspace quotas.
+
+
+
+There are two primary storage concepts you will work with in SQL Server Big Data Clusters: the HDFS layer, and SQL Server. Fro HDFS, it's important to know the basics of how it works.
+
+
+
+There are three primary tools and utilities you will use to control the SQL Server big data cluster:
+
+ - kubectl
+ - mssqlctl
+ - Azure Data Studio
+
+
Managing and Monitoring the SQL Server big data cluster (mssqlctl)
+
+The **mssqlctl** command-line utility is written in Python and can be installed on your workstation using the **pip** command in Python. You will see how to install this utility in the *Planning, Installation and Configuration* module.
+
+The **mssqlctl** utility enables cluster administrators to bootstrap and manage big data clusters via the REST APIs exposed by the Controller service. The controller is deployed and hosted in the same Kubernetes namespace where the customer wants to build out a big data cluster. The Controller is responsible for core logic for deploying and managing a big data cluster.
+
+The Controller service is installed by a Kubernetes administrator during cluster bootstrap, using the mssqlctl command-line utility.
+
+You'll explore further operations with these tools in the Management and Monitoring module.
+
+
SQL Server BDC Programming and GUI Surface (Azure Data Studio and Jupyter Notebooks)
+
+
Jupyter Notebooks
+
+A Jupyter *Notebook* is a web-page-based interface consisting of *Cells* that can contain text (using the Markdown specification) or code. The code depends on the Kernel that has been installed for the Notebook. Traditionally, Python and R Kernels are installed by default.
+
+*Notebook Servers* run **.ipynb** files (the Notebooks). You can install a Notebook Server locally, remotely, or you can use something like Azure Notebooks which provides a free, quick, easy way to work with and share Notebooks. (It's all a bit like a specific kind of web server).
+
+*Libraries* are a container on your Notebook server where you can have Notebooks, code, directories and other files.
+
+Notebooks are JSON files that contain areas called *Cells*, which have text or code in them. When you double-click a Notebook, the Notebook server renders and can display text or run code, such as R or Python, using a Kernel. Cells can hold text (such as *Markdown*, *HTML*, or *LaTeX*) which you can mix together, or Code. Double-click a Cell in a Notebook to edit it, and then click the "Run" button to render what you typed. Code runs and displays an output below the cell. You can toggle the result for code to show or hide it.
+
+*Markdown* is a simplified markup language for text. Use it for general text and simple graphics. You can read more about Markdown here, and there's a great cheat-sheet on Markdown here.
+
+You'll use Notebooks within Azure Data Studio to work with Spark, which you'll learn about in a moment. Learn more about Jupyter Notebooks here.
+
+
Azure Data Studio
+
+*Azure Data Studio* is a cross-platform database tool to manage and program on-premises and cloud data platforms on Windows, MacOS, and Linux. It is extensible, and one of these extensions is how you work with Azure Data Studio code and Jupyter Notebooks. It is built on the Visual Studio Code shell. The editor in Azure Data Studio has Intellisense, code snippets, source control integration, and an integrated terminal.
+
+If you have not completed the prerequisites for this workshop you can install Azure Data Studio from this location, and you will install the Extension to work with SQL Server big data clusters in a future module.
+
+You can learn more about Azure Data Studio here.
+
+
+
+
+
+You'll explore further operations with the Azure Data Studio in the Operationalization module.
+
+
+
+
+In any large data system, you will need a way to bring the data in. In some cases, you will edit the data either on the way in, or after it is staged using a process called "Extract, Transform and Load" (ETL) or leave the data "pure" and unaltered (common in Data Science projects), using a process called "Extract, Load and Transform" (ELT).
+
+In SQL Server big data clusters, you'll learn about three ways that the system interacts with large sets of data:
+
+ - "Virtualize" the data by pushing down the query to the source system (no data is ingested in this scenario)
+ - Ingesting data into SQL Server Tables, using the PolyBase feature or into standard SQL Server constructs
+ - Loading data into HDFS
+
+In Data Virtualization, no data is brought into storage - it is queried from it's source. It's important to think about the network bandwidth per query in this scenario. The next two scenarios do bring data into the system, either into SQL Server tables within the big data cluster or into the HDFS mount points.
+
+In both cases, the first considerations for loading data are source-data locality and network bandwidth, utilization, and predictability of the path to the SQL Server BDC destination. Depending on where the data originates, network bandwidth will play a major part in your loading performance. For source data residing on premises, network throughput performance and predictability can be enhanced with a service such as a dedicated network path or "hot potato routing". You must consider the current average bandwidth, utilization, predictability, and maximum capabilities of your current public Internet-facing, source-to-destination route, regardless of the method you are using.
+
+In this section you'll learn more about working with the last two options for loading data into the SQL Server BDC architecture. In the *Operationalization* module you'll learn more about Data Virtualization, and also see a practical method for working with data ingestion and pipelines. You can use multiple methods to control the pipeline or workflow of your system, depending on the requirements and other parts of the Data Architecture at your location. In general, you will use the Team Data Science Process to define your objectives, identify and explore data sources, perform any transforms and do any Feature Engineering for Machine Learning, and then Operationalize your solution.
+
+Next you'll explore the two locations you can store data in the system, and then you'll learn about creating a Pipeline system to standardize and automate the process. In the Operationalization Module you'll see practical examples of each of these.
+
+
+
+As described earlier, you can use various methods to ingest data ad-hoc and as-needed for your two data targets (HDFS and SQL Server Tables. A more holistic archicture is to use a Pipeline system that can define sources, triggers and events, transforms, targets, and has logging and tracking capabilities. The Microsoft Azure Data Factory provides all of the capabilities, and often serves as the mechanism to transfer data to and from on-premises, in-cloud, and other sources and targets. ADF can serve as a full data pipeline system, as described here.
+
+
+
+
+
+A full description of Azure Data Factory is here.
+
+
Data Pipelines using Apache Spark
+
+Apache Spark is an analytics engine for large-scale data. It can be used with data stored in HDFS, and has connectors to work with data in SQL Server as well.
+
+
+
+
+
+While Spark is used for all phases of the data processing lifecycle and can compliment and extend the capabilities of SQL Server, it is also often used to ingest and transform data as a data pipeline. An important concept to keep in mind when you ingest data using Spark is that Spark has three main data storage representations it works with once they are ingested:
+
+ - RDD (2011): Distribution of JVM objects, Functional Operators (map, filter, etc)
+ - DataFrame (2013): Distribution of Row objects, Expression-based operations and UDF’s, Logical plans and an optimizer, Fast, efficient internal representations
+ - Dataset (2015): Internally rows, externally JVM objects – type-safe + fast, Slower than DF’s, not as suited for interactive analysis
+
+You can learn more about these data representations here. You'll see an example of ingesting data using Apache Spark in the Operationalization module.
+
+
+
+
+
+
+
+Next, Continue to SQL Server big data cluster Components.
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/02 - SQL Server BDC Components.md b/sqlserver2019bigdataclusters/SQL2019BDC/02 - SQL Server BDC Components.md
index 663607f8..4e26726a 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/02 - SQL Server BDC Components.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/02 - SQL Server BDC Components.md
@@ -1,225 +1,231 @@
-
-
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
-
-#### A Microsoft Course from the SQL Server team
-
-
-
-
SQL Server big data cluster Components
-
-In this workshop you'll cover using a Process and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
-
-(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
-
-You'll cover the following topics in this Module:
-
-
-
-SQL Server (starting with version 2019) provides three ways to work with large sets of data:
-
- - **Data Virtualization**: Query multiple sources of data technologies using the Polybase SQL Server feature (data left at source)
- - **Storage Pools**: Create sets of disparate data sources that can be queried as a single "Data Mart" (data ingested into sharded databases using PolyBase)
- - **SQL Server big data cluster**: Create, manage and control clusters of SQL Server Instances that co-exist in a Kubernetes cluster with Apache Spark and other technologies to access and process large sets of data (Data left in place, ingested through PolyBase, and into/through HDFS)
-
-Each of these functions are available separately based on the requirements of your solution. You'll cover each of these components in the sections that follow, and learn more about how each is used within the SQL Server big data cluster (BDC).
-
-
-
-In both a Stand-Alone Instance configuration and within the SQL Server big data cluster (BDC) configuration, you can use a series of *Connectors* to query data using the PolyBase feature. PolyBase enables your SQL Server instance to process Transact-SQL queries that read data from external data sources. Starting in SQL Server 2019, you can access external data in Hadoop, Azure Blob Storage and also external data in SQL Server, Oracle, Teradata, and MongoDB - as well as Generic ODBC sources. PolyBase pushes as much of the query as possible to the source system, which optimizes the query.
-
-
-
-
-
-To leverage PolyBase, you first define the external table using a specific set of statements, then configure the connection and security, and then use standard Transact-SQL statements work with the data as if it were an standard SQL Server table. The components used in this configuration are as Follows:
-
-Components used in Data Virtualization
-
-
Creates and runs HDFS Java jobs at the external tables
-
-
-
-
-
Activity: Review PolyBase Solution
-
-In this section you will review the solution tutorial you will perform in the 04 Operationalization Module. You'll see how to create a reference to an HDFS file store and query it within SQL Server as if it were a standard internal table.
-
-
-
-
-The Big Data capabilities for SQL Server can be used in a stand-alone Instance by leveraging the Data Virtualization feature described above. To perform scale-out Big Data, SQL Server implements a big data cluster by leveraging Kubernetes with several other components.
-
-
-
-
-
-SQL Server big data clusters can be installed in three ways:
-
- - Locally for Testing
- - In a Cloud Service
- - On premises
-
-These architectures are not mutually exclusive - you can install some components on-premises, and others as a service. Your connections can interconnect across these environments. You'll explore more about installing SQL Server BDC in the 03 Planning, Installation and Configuration module.
-
-
Control Plane: Kubernetes Components
-
-As you saw in the Big Data Landscape Module of this workshop, there are multiple components within Kubernetes that work with the SQL Server big data cluster. These components are used to manage and monitor the solution:
-
-
A dedicated Node within the Kubernetes Cluster that Manages and Controls the Kubernetes Cluster. In addition to the Master Node, controlling services installed on each Node include kublet, kube-proxy and the Docker runtime engine.
The Apache Knox Gateway is an Application Gateway for interacting with the REST APIs and UIs of the cluster. The Knox Gateway provides a single access point for all REST interactions in BDC's.
Kibana is an open source data visualization plugin used by the SQL Cluster Administration Portal.
-
-
-
-
Control Plane: Controller Service
-
-The Controller in a SQL Server BDC is a service that is deployed with the mssqlctl utility. It bridges the interactions with SQL Server, Kubernetes, Spark and HDFS.
-
-The controller service provides the following core functionality:
-
- - Manage cluster lifecycle: cluster bootstrap & delete, update configurations
- - Manage master SQL Server instances
- - Manage compute, data, and storage pools
- - Expose monitoring tools to observe the state of the cluster
- - Expose troubleshooting tools to detect and repair unexpected issues
- - Manage cluster security: ensure secure cluster endpoints, manage users and roles, configure credentials for intra-cluster communication
- - Manage the workflow of upgrades so that they are implemented safely
- - Manage high availability and DR for statefull services in the cluster
-
-You have two ways of working with the Controller service: the mssqlctl utility, and the Cluster Adminnstration Portal. All communication to the controller service is conducted via a REST API over HTTPS. A self-signed certificate will be automatically generated for you at bootstrap time. Authentication to the controller service endpoint is based on username and password. These credentials are provisioned at cluster bootstrap time using the input for environment variables CONTROLLER_USERNAME and CONTROLLER_PASSWORD.
-
-These components are used in the Controller of the SQL Server big data cluster:
-
-
The controller hosts the core logic for deploying and managing a big data cluster.
-
-
-
-
Control Plane: SQL Server Master Instance
-
-The SQL Server Master Instance is an installation of SQL Server 2019 in a Pod on a Node in the Kubernetes cluster. You access it the same way as any SQL Server Instance, and use it for high-value, OLTP, OLAP or other types of workloads. It has Machine Learning Services already configured, so you have the full range of R, Python, and Java to work with on the data in the Cluster environment.
-
-In addition, the Master Instance stores meta-data which is outside the scope of the meta-data HIVE is storing. It also contains the PolyBase definition tables.
-
-These components are used in the Controller of the SQL Server Master Instance:
-
-
Used for executing Machine Learning and other functionality in Java, R and Python code in SQL Server.
-
-
-
-
-
Compute Plane: Compute Pool
-
-The Compute Pool holds one or more SQL Server Pods used for distributed processing under the direction of the SQL Server Master Instance. It makes the calls out to the PolyBase connectors for a distributed Compute layer of the BDC.
-
-These components are used in the Compute Pool of the SQL Server big data cluster:
-
-
Provides scale-out computational resources for a big data cluster.
-
-
-
-
Compute Plane: App Pool
-
-The App Pool is a set of Pods within a Node that hold multiple types of end-points into the system. SQL Server Integration Services lives in the App Pool, and other Job systems are possible. You could instatiate a long-running job (such as IoT streaming) or Machine Learning (ML) endpoints used for scoring a prediction or returning a classification.
-
-These components are used in the Compute Pool of the SQL Server big data cluster:
-
-
Provides computational resources for Operationalizing a big data cluster.
-
-
-
-
Data Plane: Data Pool
-
-The Data Pool in a SQL Server big data cluster consists of one or more SQL Server data pool instances. SQL data pool instances provide persistent SQL Server storage for the cluster. A data pool is used to ingest data from SQL queries or Spark jobs. To provide better performance across large data sets, data in a data pool is distributed into shards across the member SQL data pool instances.
-
-
-
-
-
-These components are used in the Data Pool of the SQL Server big data cluster:
-
-
-
-The storage pool consists of storage nodes comprised of SQL Server on Linux, Spark, and HDFS. All the storage nodes in a SQL big data cluster are members of an HDFS cluster. You can use these as a "Data Lake" construct to work with large sets of data stored on disparate data sources.
-
-Inside the Storage Pool, the Storage nodes are responsible for data ingestion through Spark, data storage in HDFS (Parquet format). HDFS also provides data persistency, as HDFS data is spread across all the storage nodes in the SQL big data cluster. The Storage Nodes also provide data access through HDFS and SQL Server endpoints.
-
-
-
-
-
-These components are used in the Storage Pool of the SQL Server big data cluster:
-
-
Apache Spark is an analytics engine for processing large-scale data. It can be used with data stored in HDFS, and has connectors to work with data in SQL Server as well.
-
-
-
-
-
Activity: Review Data Pool Solution
-
-In this section you will review the solution tutorial you will perform in the 04 Operationalization Module. You'll see how to load data into the Data Pool.
-
-
-
-
-Next, Continue to Planning, Installation and Configuration.
+
+
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
+
+#### A Microsoft Course from the SQL Server team
+
+
+
+
SQL Server big data cluster Components
+
+In this workshop you'll cover using a Process and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
+
+(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
+
+You'll cover the following topics in this Module:
+
+
+
+SQL Server (starting with version 2019) provides three ways to work with large sets of data:
+
+ - **Data Virtualization**: Query multiple sources of data technologies using the Polybase SQL Server feature (data left at source)
+ - **Storage Pools**: Create sets of disparate data sources that can be queried as a single "Data Mart" (data ingested into sharded databases using PolyBase)
+ - **SQL Server big data cluster**: Create, manage and control clusters of SQL Server Instances that co-exist in a Kubernetes cluster with Apache Spark and other technologies to access and process large sets of data (Data left in place, ingested through PolyBase, and into/through HDFS)
+
+Each of these functions are available separately based on the requirements of your solution. You'll cover each of these components in the sections that follow, and learn more about how each is used within the SQL Server big data cluster (BDC).
+
+
+
+In both a Stand-Alone Instance configuration and within the SQL Server big data cluster (BDC) configuration, you can use a series of *Connectors* to query data using the PolyBase feature. PolyBase enables your SQL Server instance to process Transact-SQL queries that read data from external data sources. Starting in SQL Server 2019, you can access external data in Hadoop, Azure Blob Storage and also external data in SQL Server, Oracle, Teradata, and MongoDB - as well as Generic ODBC sources. PolyBase pushes as much of the query as possible to the source system, which optimizes the query.
+
+
+
+
+
+To leverage PolyBase, you first define the external table using a specific set of statements, then configure the connection and security, and then use standard Transact-SQL statements work with the data as if it were an standard SQL Server table. The components used in this configuration are as Follows:
+
+Components used in Data Virtualization
+
+
Creates and runs HDFS Java jobs at the external tables
+
+
+
+
+
Activity: Review PolyBase Solution
+
+In this section you will review the solution tutorial you will perform in the 04 Operationalization Module. You'll see how to create a reference to an HDFS file store and query it within SQL Server as if it were a standard internal table.
+
+
+
+
+The Big Data capabilities for SQL Server can be used in a stand-alone Instance by leveraging the Data Virtualization feature described above. To perform scale-out Big Data, SQL Server implements a big data cluster by leveraging Kubernetes with several other components.
+
+
+
+
+
+SQL Server big data clusters can be installed in three ways:
+
+ - Locally for Testing
+ - In a Cloud Service
+ - On premises
+
+These architectures are not mutually exclusive - you can install some components on-premises, and others as a service. Your connections can interconnect across these environments. You'll explore more about installing SQL Server BDC in the 03 Planning, Installation and Configuration module.
+
+
Control Plane: Kubernetes Components
+
+As you saw in the Big Data Landscape Module of this workshop, there are multiple components within Kubernetes that work with the SQL Server big data cluster. These components are used to manage and monitor the solution:
+
+
A dedicated Node within the Kubernetes Cluster that Manages and Controls the Kubernetes Cluster. In addition to the Master Node, controlling services installed on each Node include kublet, kube-proxy and the Docker runtime engine.
The Apache Knox Gateway is an Application Gateway for interacting with the REST APIs and UIs of the cluster. The Knox Gateway provides a single access point for all REST interactions in BDC's.
Kibana is an open source log visualization and search plugin used by the SQL Cluster Administration Portal.
+
+
+
+
Control Plane: Controller Service
+
+The Controller in a SQL Server BDC is a service that is deployed with the mssqlctl utility. It bridges the interactions with SQL Server, Kubernetes, Spark and HDFS.
+
+The controller service provides the following core functionality:
+
+ - Manage cluster lifecycle: cluster bootstrap & delete, update configurations
+ - Manage master SQL Server instances
+ - Manage compute, data, and storage pools
+ - Expose monitoring tools to observe the state of the cluster
+ - Expose troubleshooting tools to detect and repair unexpected issues
+ - Manage cluster security: ensure secure cluster endpoints, manage users and roles, configure credentials for intra-cluster communication
+ - Manage the workflow of upgrades so that they are implemented safely
+ - Manage high availability and DR for statefull services in the cluster
+
+You have two ways of working with the Controller service: the mssqlctl utility, and the Cluster Adminnstration Portal. All communication to the controller service is conducted via a REST API over HTTPS. A self-signed certificate will be automatically generated for you at bootstrap time. Authentication to the controller service endpoint is based on username and password. These credentials are provisioned at cluster bootstrap time using the input for environment variables CONTROLLER_USERNAME and CONTROLLER_PASSWORD.
+
+These components are used in the Controller of the SQL Server big data cluster:
+
+
The controller hosts the core logic for deploying and managing a big data cluster.
+
+
+
+
Control Plane: SQL Server Master Instance
+
+The SQL Server Master Instance is an installation of SQL Server 2019 in a Pod on a Node in the Kubernetes cluster. You access it the same way as any SQL Server Instance, and use it for high-value, OLTP, OLAP or other types of workloads. It has Machine Learning Services already configured, so you have the full range of R, Python, and Java to work with on the data in the Cluster environment.
+
+The Master Instance stores meta-data which is outside the scope of the meta-data HIVE is storing. It also contains the PolyBase definition tables, and in addition to the standard SQL Server system databases, the SQL master instance also contains the following:
+
+- A metadata database that holds HDFS-table metadata
+- A data plane shard map
+- Details of external tables that provide access to the cluster data plane.
+- PolyBase external data sources and external tables defined in user databases.
+
+
+These components are used in the Controller of the SQL Server Master Instance:
+
+
Used for executing Machine Learning and other functionality in Java, R and Python code in SQL Server.
+
+
+
+
+
Compute Plane: Compute Pool
+
+The Compute Pool holds one or more SQL Server Pods used for distributed processing under the direction of the SQL Server Master Instance. It makes the calls out to the PolyBase connectors for a distributed Compute layer of the BDC.
+
+These components are used in the Compute Pool of the SQL Server big data cluster:
+
+
Provides scale-out computational resources for a big data cluster.
+
+
+
+
Compute Plane: App Pool
+
+The App Pool is a set of Pods within a Node that hold multiple types of end-points into the system. SQL Server Integration Services lives in the App Pool, and other Job systems are possible. You could instatiate a long-running job (such as IoT streaming) or Machine Learning (ML) endpoints used for scoring a prediction or returning a classification.
+
+These components are used in the Compute Pool of the SQL Server big data cluster:
+
+
Provides computational resources for Operationalizing a big data cluster.
+
+
+
+
Data Plane: Data Pool
+
+The Data Pool in a SQL Server big data cluster consists of one or more SQL Server data pool instances. SQL data pool instances provide persistent SQL Server storage for the cluster. A data pool is used to ingest data from SQL queries or Spark jobs. To provide better performance across large data sets, data in a data pool is distributed into shards across the member SQL data pool instances.
+
+
+
+
+
+These components are used in the Data Pool of the SQL Server big data cluster:
+
+
+
+The storage pool consists of storage nodes comprised of SQL Server on Linux, Spark, and HDFS. All the storage nodes in a SQL big data cluster are members of an HDFS cluster. You can use these as a "Data Lake" construct to work with large sets of data stored on disparate data sources.
+
+Inside the Storage Pool, the Storage nodes are responsible for data ingestion through Spark, data storage in HDFS (Parquet format). HDFS also provides data persistency, as HDFS data is spread across all the storage nodes in the SQL big data cluster. The Storage Nodes also provide data access through HDFS and SQL Server endpoints.
+
+
+
+
+
+These components are used in the Storage Pool of the SQL Server big data cluster:
+
+
Apache Spark is an analytics engine for processing large-scale data. It can be used with data stored in HDFS, and has connectors to work with data in SQL Server as well.
+
+
+
+
+
Activity: Review Data Pool Solution
+
+In this section you will review the solution tutorial you will perform in the 04 Operationalization Module. You'll see how to load data into the Data Pool.
+
+
+
+
+Next, Continue to Planning, Installation and Configuration.
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/03 - Planning, Installation and Configuration.md b/sqlserver2019bigdataclusters/SQL2019BDC/03 - Planning, Installation and Configuration.md
index b89206b9..670a998c 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/03 - Planning, Installation and Configuration.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/03 - Planning, Installation and Configuration.md
@@ -1,147 +1,148 @@
-
-
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Planning, Installation and Configuration
-
-In this workshop you'll cover using a Process and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
-
-(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
-
-You'll cover the following topics in this Module:
-
-
-
-NOTE: The following Module is based on the Private Preview of the Microsoft SQL Server 2019 big data cluster feature. These instructions will change as the product is updated for the final release. The latest installation instructions are located here.
-
-A SQL Server big data cluster (BDC) is deployed onto a Kubernetes Cluster using the mssqlctl utility which creates the appropriate Docker Containers and other constructs. The installation uses various switches on the mssqlctl utility, and reads from several environment variables which you will define before you run the command.
-
-For planning, it is essential that you understand the SQL Server BDC components, and have a firm understanding of Kubernetes and TCP/IP networking. You should also have an understanding of how SQL Server and Apache Spark use the "Big Four" (*CPU, I/O, Memory and Networking*).
-
-Since the Kubernetes Cluster is often made up of Virtual Machines that host the Docker Images, they must be as large as possible. For the best possible performance, large physical machines that are tuned for optimal performance is a recommended physical architecture.
-
-You can deploy Kubernetes in three general ways:
- - In a Cloud Platform such as Azure Kubernetes Service (AKS)
-
- - In your own Kubernetes deployment using KubeADM
-
- - In your own Kubernetes deployment using minikube (*to be used only for training and testing*)
-
-Regardless of the Kubernetes target, the general steps for setting up the system are:
-
- - Set up Kubernetes cluster
-
- - Install the cluster configuration tool mssqlctl on the administration machine
-
- - Deploy the SQL Server big data cluster on the Kubernetes cluster
-
-In the sections that follow, you'll cover the general process for each of these deployments. The official documentation referenced above have the specific steps for each deployment, and the Activity section of this Module has the steps for deploying the BDC on AKS for the classroom enviornment.
-
-
-
-
-
-The Azure Kubernetes Service provides the ability to create a Kubernetes cluster in the Azure portal, with the Azure CLI, or template driven deployment options such as Resource Manager templates and Terraform. When you deploy an AKS cluster, the Kubernetes master and all nodes are deployed and configured for you. Additional features such as advanced networking, Azure Active Directory integration, and monitoring can also be configured during the deployment process.
-
-An AKS cluster is divided into two components: The *Cluster master nodes* which provide the core Kubernetes services and orchestration of application workloads; and the *Nodes* which run your application workloads.
-
-
-
-
-
-The cluster master includes the following core Kubernetes components:
-
- - *kube-apiserver* - The API server is how the underlying Kubernetes APIs are exposed. This component provides the interaction for management tools, such as kubectl or the Kubernetes dashboard.
-
- - *etcd* - To maintain the state of your Kubernetes cluster and configuration, the highly available etcd is a key value store within Kubernetes.
-
- - *kube-scheduler* - When you create or scale applications, the Scheduler determines what nodes can run the workload and starts them.
-
- - *kube-controller-manager* - The Controller Manager oversees a number of smaller Controllers that perform actions such as replicating pods and handling node operations.
-
-The Nodes include the following components:
-
- - The *kubelet* is the Kubernetes agent that processes the orchestration requests from the cluster master and scheduling of running the requested containers.
-
- - Virtual networking is handled by the *kube-proxy* on each node. The proxy routes network traffic and manages IP addressing for services and pods.
-
- - The *container runtime* is the component that allows containerized applications to run and interact with additional resources such as the virtual network and storage. In AKS, Docker is used as the container runtime.
-
-
-
-
-
-For a SQL Server BDC in an AKS environment, for an optimal experience while validating basic scenarios, you should use at least three agent VMs with at least 4 vCPUs and 32 GB of memory each.
-
-With this background, you can find the latest specific steps to deploy a SQL Server big data cluster on AKS here.
-
-
-
-
-
-The kubeadm toolbox helps you bootstrap a Kubernetes cluster that conforms to best practices. Kubeadm also supports other cluster lifecycle functions, such as upgrades, downgrade, and managing bootstrap tokens.
-
-The kubeadm toolbox can deploy a Kubernetes cluster to physical or virtual machines. It works by specifying the TCP/IP addresses of the targets.
-
-With this background, you can find the latest specific steps to deploy a SQL Server big data cluster using kubeadm here.
-
-
-
-
-
-Minikube is a tool that simplifies creating and running Kubernetes locally. Minikube creates a single-node Kubernetes cluster inside a VM. It uses Hyper-V, VirtualBox, or other Hypervisor products to create and operate the cluster.
-
-While SQL Server BDC can be installed on minikube, it requires at least 32GB of RAM to run, and really needs 64GB of RAM or more to perform any real testing. In any case, it is reserved only for testing or learning.
-
-Minikube supports the following Kubernetes features:
-
- - DNS
- - NodePorts
- - ConfigMaps and Secrets
- - Dashboards
- - Container Runtime: Docker, rkt, CRI-O and containerd
- - Enabling CNI (Container Network Interface)
- - Data Ingress
-
-With this background, you can find the latest specific steps to deploy a SQL Server big data cluster using minikube here.
-
-
-
-
-
-In this lab you will check your deployment you performed in Module 01 of a SQL Server 2019 big data cluster on the Azure Kubernetes Service.
-
-Using the following steps, you will evaluate your Resource Group in Azure that holds your BDC on AKS that you deployed earlier. When you complete your course you can delete this Resource Group which will stop the Azure charges for this course.
-
-Steps
-
-
-
-Next, Continue to Operationalization.
+
+
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Planning, Installation and Configuration
+
+In this workshop you'll cover using a Process and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
+
+(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
+
+You'll cover the following topics in this Module:
+
+
+
+NOTE: The following Module is based on the Private Preview of the Microsoft SQL Server 2019 big data cluster feature. These instructions will change as the product is updated for the final release. The latest installation instructions are located here.
+
+A SQL Server big data cluster (BDC) is deployed onto a Kubernetes Cluster using the `mssqlctl` utility which creates the appropriate Docker Containers and other constructs for the system. The installation uses various switches on the `mssqlctl` utility, and reads from several environment variables which you will define before you run the command. Note that these environment variables will be replaced with a different system at General Availability (GA) release.
+
+For planning, it is essential that you understand the SQL Server BDC components, and have a firm understanding of Kubernetes and TCP/IP networking. You should also have an understanding of how SQL Server and Apache Spark use the "Big Four" (*CPU, I/O, Memory and Networking*).
+
+Since the Kubernetes Cluster is often made up of Virtual Machines that host the Docker Images, they must be as large as possible. For the best possible performance, large physical machines that are tuned for optimal performance is a recommended physical architecture. The least viable production system is a Minimum of 3 Linux physical machines or virtual machines. The recommended configuration per machine is 8 CPUs, 32 GB of memory and 100GB of storage. This configuration would support only one or two users with a standard workload, and you would want to increase the system for each additional user or heavier workload.
+
+
+You can deploy Kubernetes in three general ways:
+ - In a Cloud Platform such as Azure Kubernetes Service (AKS)
+
+ - In your own Kubernetes deployment using KubeADM
+
+ - In your own Kubernetes deployment using minikube (*to be used only for training and testing*)
+
+Regardless of the Kubernetes target, the general steps for setting up the system are:
+
+ - Set up Kubernetes cluster
+
+ - Install the cluster configuration tool `mssqlctl` on the administration machine
+
+ - Deploy the SQL Server big data cluster onto the Kubernetes cluster
+
+In the sections that follow, you'll cover the general process for each of these deployments. The official documentation referenced above have the specific steps for each deployment, and the Activity section of this Module has the steps for deploying the BDC on AKS for the classroom enviornment.
+
+
+
+
+
+The Azure Kubernetes Service provides the ability to create a Kubernetes cluster in the Azure portal, with the Azure CLI, or template driven deployment options such as Resource Manager templates and Terraform. When you deploy an AKS cluster, the Kubernetes master and all nodes are deployed and configured for you. Additional features such as advanced networking, Azure Active Directory integration, and monitoring can also be configured during the deployment process.
+
+An AKS cluster is divided into two components: The *Cluster master nodes* which provide the core Kubernetes services and orchestration of application workloads; and the *Nodes* which run your application workloads.
+
+
+
+
+
+The cluster master includes the following core Kubernetes components:
+
+ - *kube-apiserver* - The API server is how the underlying Kubernetes APIs are exposed. This component provides the interaction for management tools, such as kubectl or the Kubernetes dashboard.
+
+ - *etcd* - To maintain the state of your Kubernetes cluster and configuration, the highly available etcd is a key value store within Kubernetes.
+
+ - *kube-scheduler* - When you create or scale applications, the Scheduler determines what nodes can run the workload and starts them.
+
+ - *kube-controller-manager* - The Controller Manager oversees a number of smaller Controllers that perform actions such as replicating pods and handling node operations.
+
+The Nodes include the following components:
+
+ - The *kubelet* is the Kubernetes agent that processes the orchestration requests from the cluster master and scheduling of running the requested containers.
+
+ - Virtual networking is handled by the *kube-proxy* on each node. The proxy routes network traffic and manages IP addressing for services and pods.
+
+ - The *container runtime* is the component that allows containerized applications to run and interact with additional resources such as the virtual network and storage. In AKS, Docker is used as the container runtime.
+
+
+
+
+
+For a SQL Server BDC in an AKS environment, for an optimal experience while validating basic scenarios, you should use at least three agent VMs with at least 4 vCPUs and 32 GB of memory each.
+
+With this background, you can find the latest specific steps to deploy a SQL Server big data cluster on AKS here.
+
+
+
+
+
+The kubeadm toolbox helps you bootstrap a Kubernetes cluster that conforms to best practices. Kubeadm also supports other cluster lifecycle functions, such as upgrades, downgrade, and managing bootstrap tokens.
+
+The kubeadm toolbox can deploy a Kubernetes cluster to physical or virtual machines. It works by specifying the TCP/IP addresses of the targets.
+
+With this background, you can find the latest specific steps to deploy a SQL Server big data cluster using kubeadm here.
+
+
+
+
+
+Minikube is a tool that simplifies creating and running Kubernetes locally. Minikube creates a single-node Kubernetes cluster inside a VM. It uses Hyper-V, VirtualBox, or other Hypervisor products to create and operate the cluster.
+
+While SQL Server BDC can be installed on minikube, it requires at least 32GB of RAM to run, and really needs 64GB of RAM or more to perform any real testing. In any case, it is reserved only for testing or training.
+
+Minikube supports the following Kubernetes features:
+
+ - DNS
+ - NodePorts
+ - ConfigMaps and Secrets
+ - Dashboards
+ - Container Runtime: Docker, rkt, CRI-O and containerd
+ - Enabling CNI (Container Network Interface)
+ - Data Ingress
+
+With this background, you can find the latest specific steps to deploy a SQL Server big data cluster using minikube here.
+
+
+
+
+
+In this lab you will check your deployment you performed in Module 01 of a SQL Server 2019 big data cluster on the Azure Kubernetes Service.
+
+Using the following steps, you will evaluate your Resource Group in Azure that holds your BDC on AKS that you deployed earlier. When you complete your course you can delete this Resource Group which will stop the Azure charges for this course.
+
+Steps
+
+
+
+Next, Continue to Operationalization.
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/04 - Operationalization.md b/sqlserver2019bigdataclusters/SQL2019BDC/04 - Operationalization.md
index b40cf394..9df86ef6 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/04 - Operationalization.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/04 - Operationalization.md
@@ -1,156 +1,156 @@
-
-
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
-
-#### A Microsoft Course from the SQL Server team
-
-
-
-
Operationalization
-
-In this workshop you'll cover using a Process and and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
-
-(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
-
-You'll cover the following topics in this Module:
-
-
-
-Recall from The Big Data Landscape module that you learned about the Wide World Importers company. Wide World Importers (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world. They have a traditional N-tier application that uses a front-end (mobile, web and installed) that interacts with a scale-out middle-tier software product, which in turn stores data in a large SQL Server database that has been scaled-up to meet demand.
-
-
-
-
-
-WWI has now added web and mobile commerce to their platform, which has generated a significant amount of additional data, and data formats. These new platforms were added without integrating into the OLTP system data or Business Intelligence infrastructures. As a result, "silos" of data stores have developed, and ingesting all of this data exceeds the scale of their current RDBMS server:
-
-
-
-
-
-This presented the following four challenges - the IT team at WWI needs to:
-
- - Scale data systems to reach more consumers
-
- - Unlock business insights from multiple sources of structured and unstructured data
-
- - Apply deep analytics with high-performance responses
-
- - Enable AI into apps to actively engage with customers
-
-
-
-
-
-
Solution - Challenge 1: Scale Data System
-
-To meet these challenges, the following solution is proposed. Using the SQL Server 2019 big data cluster platform you learned about in the 02 - SQL Server BDC Components Module, the solution allows the company to keep it's current codebase, while enabling a flexible scale-out architecture. This answers the first challenge of working with a scale-out system for larger data environments.
-
-The following diagram illustrates the complete solution that you can use to brief your audience with:
-
-
-
-
-
-In the following sections you'll dive deeper into how this scale is used to solve the rest of the challenges.
-
-
-
-
-
-The next challenge the IT team must solve is to enable a single data query to work across multiple disparate systems, optionally joining to internal SQL Server Tables, and also at scale.
-
-Using the Data Virtualization capability you saw in the 02 - SQL Server BDC Components Module, the IT team creates External Tables using the PolyBase feature. These External Table definitions are stored in the database on the SQL Server Master Instance within the cluster. When queried by the user, the queries are engaged from the SQL Server Master Instance through the Compute Pool in the SQL Server BDC, which holds Kubernetes Nodes containing the Pods running SQL Server Instances. These Instances send the query to the PolyBase Connector at the target data system, which processes the query based on the type of target system. The results are processed and returned through the PolyBase Connector to the Compute Pool and then on to the Master Instance, and then on to the user.
-
-
-
-
-
-This process allows not only a query to disparate systems, but also those remote systems can hold extremely large sets of data. Normally you are querying a subset of that data, so the results are all that are sent back over the network. These results can be joined with internal tables for a single view, and all from within the same Transact-SQL statements.
-
-
Activity: Load and query data in an External Table
-
-In this activity, you will load the sample data into your big data cluster environment, and then create and use an External table to query the data in HDFS. This process is similar to connecting to any Polybase target.
-
-Steps
-
-
-
-Ad-hoc queries are very useful for many scenarios. There are times when you would like to bring the data into storage, so that you can create denormalized representations of datasets, aggregated data, and other purpose-specific data tasks. Storing data in this fashion is called a "Data Mart".
-
-
-
-
-
-Using the Data Virtualization capability you saw in the 02 - SQL Server BDC Components Module, the IT team creates External Tables using PolyBase statements. These External Table definitions are stored in the database on the SQL Server Master Instance within the cluster. When queried by the user, the queries are engaged from the SQL Server Master Instance through the Compute Pool in the SQL Server BDC, which holds Kubernetes Nodes containing the Pods running SQL Server Instances. These Instances send the query to the PolyBase Connector at the target data system, which processes the query based on the type of target system. The results are processed and returned through the PolyBase Connector to the Compute Pool and then on to the Master Instance, and the PolyBase statements can specify the target of the Data Pool. The SQL Server Instances in the Data Pool store the data in a distributed fashion across multiple databases, called Shards.
-
-
Activity: Load and query data into the Data Pool
-
-In this activity, you will load the sample data into your big data cluster environment, and then create and use an External table to load data into the Data Pool.
-
-Steps
-
-
-
-There are three primary uses for a large cluster of data processing systems for Machine Learning and AI applications. The first is that the users will involved in the creation of the Features used in various ML and AI algorithms, and are often tasked to Label the data. These users can access the Data Pool and Data Storage data stores directly to query and assist with this task.
-
-The SQL Server Master Instance in the BDC installs with Machine Learning Services, which allow creation, training, evaluation and presisting of Machine Learning Models. Data from all parts of the BDC are available, and Data Science oriented languages and libraries in R, Python and Java are enabled. In this scenario, the Data Scientist creates the R or Python code, and the Transact-SQL Developer wraps that code in a Stored Procedure. This code can be used to train, evaluate and create Machine Learning Models. The Models can be stored in the Master Instance for scoring, or sent on to the App Pool where the Machine Learning Server is running, waiting to accept REST-based calls from applications.
-
-
-
-
-
-The Data Scientist has another option to create and train ML and AI models. The Spark platform within the Storage Pool is accessible through the Knox gateway, using Livy to send Spark Jobs as you learned about in the 02 - SQL Server BDC Components Module. This gives access to the full Spark platform, using Jupyter Notebooks (included in Azure Data Studio) or any other standard tools that can access Spark through REST calls.
-
-
-
-
Activity: Load data with Spark, run a Spark Notebook
-
-
-In this activity, you will load the sample data into your big data cluster environment using Spark, and use a Notebook in Azure Data Studio to work with it.
-
-Steps
-
-
- Note: curl is available in Windows 10 build 17063 or higher. If your version does not have curl, you can find out more about it here: https://blogs.msdn.microsoft.com/commandline/2018/01/18/tar-and-curl-come-to-windows
-
-
+
+
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
+
+#### A Microsoft Course from the SQL Server team
+
+
+
+
Operationalization
+
+In this workshop you'll cover using a Process and and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
+
+(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
+
+You'll cover the following topics in this Module:
+
+
+
+Recall from The Big Data Landscape module that you learned about the Wide World Importers company. Wide World Importers (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world. They have a traditional N-tier application that uses a front-end (mobile, web and installed) that interacts with a scale-out middle-tier software product, which in turn stores data in a large SQL Server database that has been scaled-up to meet demand.
+
+
+
+
+
+WWI has now added web and mobile commerce to their platform, which has generated a significant amount of additional data, and data formats. These new platforms were added without integrating into the OLTP system data or Business Intelligence infrastructures. As a result, "silos" of data stores have developed, and ingesting all of this data exceeds the scale of their current RDBMS server:
+
+
+
+
+
+This presented the following four challenges - the IT team at WWI needs to:
+
+ - Scale data systems to reach more consumers
+
+ - Unlock business insights from multiple sources of structured and unstructured data
+
+ - Apply deep analytics with high-performance responses
+
+ - Enable AI into apps to actively engage with customers
+
+
+
+
+
+
Solution - Challenge 1: Scale Data System
+
+To meet these challenges, the following solution is proposed. Using the SQL Server 2019 big data cluster platform you learned about in the 02 - SQL Server BDC Components Module, the solution allows the company to keep it's current codebase, while enabling a flexible scale-out architecture. This answers the first challenge of working with a scale-out system for larger data environments.
+
+The following diagram illustrates the complete solution that you can use to brief your audience with:
+
+
+
+
+
+In the following sections you'll dive deeper into how this scale is used to solve the rest of the challenges.
+
+
+
+
+
+The next challenge the IT team must solve is to enable a single data query to work across multiple disparate systems, optionally joining to internal SQL Server Tables, and also at scale.
+
+Using the Data Virtualization capability you saw in the 02 - SQL Server BDC Components Module, the IT team creates External Tables using the PolyBase feature. These External Table definitions are stored in the database on the SQL Server Master Instance within the cluster. When queried by the user, the queries are engaged from the SQL Server Master Instance through the Compute Pool in the SQL Server BDC, which holds Kubernetes Nodes containing the Pods running SQL Server Instances. These Instances send the query to the PolyBase Connector at the target data system, which processes the query based on the type of target system. The results are processed and returned through the PolyBase Connector to the Compute Pool and then on to the Master Instance, and then on to the user.
+
+
+
+
+
+This process allows not only a query to disparate systems, but also those remote systems can hold extremely large sets of data. Normally you are querying a subset of that data, so the results are all that are sent back over the network. These results can be joined with internal tables for a single view, and all from within the same Transact-SQL statements.
+
+
Activity: Load and query data in an External Table
+
+In this activity, you will load the sample data into your big data cluster environment, and then create and use an External table to query the data in HDFS. This process is similar to connecting to any Polybase target.
+
+Steps
+
+
+
+Ad-hoc queries are very useful for many scenarios. There are times when you would like to bring the data into storage, so that you can create denormalized representations of datasets, aggregated data, and other purpose-specific data tasks. Storing data in this fashion is called a "Data Mart".
+
+
+
+
+
+Using the Data Virtualization capability you saw in the 02 - SQL Server BDC Components Module, the IT team creates External Tables using PolyBase statements. These External Table definitions are stored in the database on the SQL Server Master Instance within the cluster. When queried by the user, the queries are engaged from the SQL Server Master Instance through the Compute Pool in the SQL Server BDC, which holds Kubernetes Nodes containing the Pods running SQL Server Instances. These Instances send the query to the PolyBase Connector at the target data system, which processes the query based on the type of target system. The results are processed and returned through the PolyBase Connector to the Compute Pool and then on to the Master Instance, and the PolyBase statements can specify the target of the Data Pool. The SQL Server Instances in the Data Pool store the data in a distributed fashion across multiple databases, called Shards.
+
+
Activity: Load and query data into the Data Pool
+
+In this activity, you will load the sample data into your big data cluster environment, and then create and use an External table to load data into the Data Pool.
+
+Steps
+
+
+
+There are three primary uses for a large cluster of data processing systems for Machine Learning and AI applications. The first is that the users will involved in the creation of the Features used in various ML and AI algorithms, and are often tasked to Label the data. These users can access the Data Pool and Data Storage data stores directly to query and assist with this task.
+
+The SQL Server Master Instance in the BDC installs with Machine Learning Services, which allow creation, training, evaluation and presisting of Machine Learning Models. Data from all parts of the BDC are available, and Data Science oriented languages and libraries in R, Python and Java are enabled. In this scenario, the Data Scientist creates the R or Python code, and the Transact-SQL Developer wraps that code in a Stored Procedure. This code can be used to train, evaluate and create Machine Learning Models. The Models can be stored in the Master Instance for scoring, or sent on to the App Pool where the Machine Learning Server is running, waiting to accept REST-based calls from applications.
+
+
+
+
+
+The Data Scientist has another option to create and train ML and AI models. The Spark platform within the Storage Pool is accessible through the Knox gateway, using Livy to send Spark Jobs as you learned about in the 02 - SQL Server BDC Components Module. This gives access to the full Spark platform, using Jupyter Notebooks (included in Azure Data Studio) or any other standard tools that can access Spark through REST calls.
+
+
+
+
Activity: Load data with Spark, run a Spark Notebook
+
+
+In this activity, you will load the sample data into your big data cluster environment using Spark, and use a Notebook in Azure Data Studio to work with it.
+
+Steps
+
+
+
Next, Continue to Management and Monitoring.
\ No newline at end of file
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/05 - Management and Monitoring.md b/sqlserver2019bigdataclusters/SQL2019BDC/05 - Management and Monitoring.md
index dfdd0bc5..3ebbfcba 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/05 - Management and Monitoring.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/05 - Management and Monitoring.md
@@ -1,145 +1,145 @@
-
-
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Management and Monitoring
-In this workshop you'll cover using a Process and and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
-
-(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
-
-You'll cover the following topics in this Module:
-
-
-
-There are two primary areas for monitoring your SQL Server 2019 big data cluster. The first deals with SQL Server 2019, and the second deals with the set of elements in the Cluster.
-
-For SQL Server, management is much as you would normally perform for any SQL Server system. You have the same type of services, surface points, security areas and other control vectors as in a stand-alone installation of SQL Server. The tools you have avalaible for managing the Master Instance in the SQL Server BDC are the same as managing a stand-alone installation, including SQL Server Management Studio, command-line interfaces, Azure Data Studio, and third party tools.
-
-For the cluster components, you have three primary interfaces to use, which you will review next.
-
-
-
-
-
-Since the SQL Server big data cluster lives within a Kubernetes cluster, you'll work with the kubectl command to deal with those specific components. The following list is a short version of some of the commands you can use to manage and monitor the SQL Server BDC implementation of a Kubernetes cluster:
-
-
-
-
Command
Description
-
-
az aks get-credentials --name --resource-group
Download the Kubernetes cluster configuration file and set the cluster context
-
kubectl get pods --all-namespaces
Get the status of pods in the cluster for either all namespaces or the big data cluster namespace
-
kubectl describe pod -n
Get a detailed description of a specific pod in json format output. It includes details, such as the current Kubernetes node that the pod is placed on, the containers running within the pod, and the image used to bootstrap the containers. It also shows other details, such as labels, status, and persisted volumes claims that are associated with the pod
-
kubectl get svc -n
Get details for the big data cluster services. These details include their type and the IPs associated with respective services and ports. Note that SQL Server big data cluster services are created in a new namespace created at cluster bootstrap time based on the cluster name specified in the mssqlctl create cluster command
-
kubectl describe pod -n
Get a detailed description of a service in json format output. It will include details like labels, selector, IP, external-IP (if the service is of LoadBalancer type), port, etc.
-
kubectl exec -it -c -n -- /bin/bash
If existing tools or the infrastructure does not enable you to perform a certain task without actually being in the context of the container, you can log in to the container using kubectl exec command. For example, you might need to check if a specific file exists, or you might need to restart services in the container
Copy files from the container to your local machine. Reverse the source and destination to copy into the container
-
kubectl delete pods -n --grace-period=0 --force
For testing availability, resiliency, or data persistence, you can delete a pod to simulate a pod failure with the kubectl delete pods command. Not recommended for production, only to simulate failure
-
kubectl get pods -o yaml -n | grep hostIP
Get the IP of the node a pod is currently running on
-
-The **mssqlctl** utility enables cluster administrators to bootstrap and manage big data clusters via the REST APIs exposed by the Controller service. The controller is deployed and hosted in the same Kubernetes namespace where the customer wants to build out a big data cluster. The Controller is responsible for core logic for deploying and managing a big data cluster.
-
-The Controller service is installed by a Kubernetes administrator during cluster bootstrap, using the mssqlctl command-line utility.
-
-You can find a list of the switches and commands by typing:
-
-
-mssqlctl --h
-
-
-You used the mssqlctl commands to deploy your cluster.
-
-
-
-
-
-
-
-Using Grafana and Kibana systems as you learned about in Module 01, Microsoft created a single portal you can use to interact with both the SQL Server-specific and Kubernetes portions of the BDC.
-
-
-
-
-
-
Activity: Start dashboard when cluster is running in AKS
-
-
-To launch the Kubernetes dashboard run the following commands:
-
-
-az aks browse --resource-group --name
-
-
-Note:
-
-If you get the following error:
-
-
Unable to listen on port 8001: All listeners failed to create with the following errors: Unable to create listener: Error listen tcp4 127.0.0.1:8001: >bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. Unable to create listener: Error listen tcp6: address [[::1]]:8001: missing port in >address error: Unable to listen on any of the requested ports: [{8001 9090}]
-
-
-make sure you did not start the dashboard already from another window.
-
-When you launch the dashboard on your browser, you might get permission warnings due to RBAC being enabled by default in AKS clusters, and the service account used by the dashboard does not have enough permissions to access all resources (for example, pods is forbidden: User "system:serviceaccount:kube-system:kubernetes-dasboard" cannot list pods in the namespace "default"). Run the following command to give the necessary permissions to kubernetes-dashboard, and then restart the dashboard:
-
-
-
-Next, Continue to Security.
+
+
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Management and Monitoring
+In this workshop you'll cover using a Process and and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
+
+(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
+
+You'll cover the following topics in this Module:
+
+
+
+There are two primary areas for monitoring your SQL Server 2019 big data cluster. The first deals with SQL Server 2019, and the second deals with the set of elements in the Cluster.
+
+For SQL Server, management is much as you would normally perform for any SQL Server system. You have the same type of services, surface points, security areas and other control vectors as in a stand-alone installation of SQL Server. The tools you have avalaible for managing the Master Instance in the SQL Server BDC are the same as managing a stand-alone installation, including SQL Server Management Studio, command-line interfaces, Azure Data Studio, and third party tools.
+
+For the cluster components, you have three primary interfaces to use, which you will review next.
+
+
+
+
+
+Since the SQL Server big data cluster lives within a Kubernetes cluster, you'll work with the kubectl command to deal with those specific components. The following list is a short version of some of the commands you can use to manage and monitor the SQL Server BDC implementation of a Kubernetes cluster:
+
+
+
+
Command
Description
+
+
az aks get-credentials --name --resource-group
Download the Kubernetes cluster configuration file and set the cluster context
+
kubectl get pods --all-namespaces
Get the status of pods in the cluster for either all namespaces or the big data cluster namespace
+
kubectl describe pod -n
Get a detailed description of a specific pod in json format output. It includes details, such as the current Kubernetes node that the pod is placed on, the containers running within the pod, and the image used to bootstrap the containers. It also shows other details, such as labels, status, and persisted volumes claims that are associated with the pod
+
kubectl get svc -n
Get details for the big data cluster services. These details include their type and the IPs associated with respective services and ports. Note that SQL Server big data cluster services are created in a new namespace created at cluster bootstrap time based on the cluster name specified in the mssqlctl create cluster command
+
kubectl describe pod -n
Get a detailed description of a service in json format output. It will include details like labels, selector, IP, external-IP (if the service is of LoadBalancer type), port, etc.
+
kubectl exec -it -c -n -- /bin/bash
If existing tools or the infrastructure does not enable you to perform a certain task without actually being in the context of the container, you can log in to the container using kubectl exec command. For example, you might need to check if a specific file exists, or you might need to restart services in the container
Copy files from the container to your local machine. Reverse the source and destination to copy into the container
+
kubectl delete pods -n --grace-period=0 --force
For testing availability, resiliency, or data persistence, you can delete a pod to simulate a pod failure with the kubectl delete pods command. Not recommended for production, only to simulate failure
+
kubectl get pods -o yaml -n | grep hostIP
Get the IP of the node a pod is currently running on
+
+The **mssqlctl** utility enables cluster administrators to bootstrap and manage big data clusters via the REST APIs exposed by the Controller service. The controller is deployed and hosted in the same Kubernetes namespace where the customer wants to build out a big data cluster. The Controller is responsible for core logic for deploying and managing a big data cluster.
+
+The Controller service is installed by a Kubernetes administrator during cluster bootstrap, using the mssqlctl command-line utility.
+
+You can find a list of the switches and commands by typing:
+
+
+mssqlctl --h
+
+
+You used the mssqlctl commands to deploy your cluster.
+
+
+
+
+
+
+
+Using Grafana and Kibana systems as you learned about in Module 01, Microsoft created a single portal you can use to interact with both the SQL Server-specific and Kubernetes portions of the BDC.
+
+
+
+
+
+
Activity: Start dashboard when cluster is running in AKS
+
+
+To launch the Kubernetes dashboard run the following commands:
+
+
+az aks browse --resource-group --name
+
+
+Note:
+
+If you get the following error:
+
+
Unable to listen on port 8001: All listeners failed to create with the following errors: Unable to create listener: Error listen tcp4 127.0.0.1:8001: >bind: Only one usage of each socket address (protocol/network address/port) is normally permitted. Unable to create listener: Error listen tcp6: address [[::1]]:8001: missing port in >address error: Unable to listen on any of the requested ports: [{8001 9090}]
+
+
+make sure you did not start the dashboard already from another window.
+
+When you launch the dashboard on your browser, you might get permission warnings due to RBAC being enabled by default in AKS clusters, and the service account used by the dashboard does not have enough permissions to access all resources (for example, pods is forbidden: User "system:serviceaccount:kube-system:kubernetes-dasboard" cannot list pods in the namespace "default"). Run the following command to give the necessary permissions to kubernetes-dashboard, and then restart the dashboard:
+
+
+
+Next, Continue to Security.
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/06 - Security.md b/sqlserver2019bigdataclusters/SQL2019BDC/06 - Security.md
index b2c0304c..4c801e4d 100644
--- a/sqlserver2019bigdataclusters/SQL2019BDC/06 - Security.md
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/06 - Security.md
@@ -1,102 +1,103 @@
-
-
-# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.2)
-
-#### A Microsoft workshop from the SQL Server team
-
-
-
-
Security
-
-In this workshop you'll cover using a Process and and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
-
-(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
-
-You'll cover the following topics in this Module:
-
-
-
-Authentication is the process of verifying the identity of a user or service and ensuring they are who they are claiming to be. Authorization refers to granting or denying of access to specific resources based on the requesting user's identity. This step is performed after a user is identified through authentication.
-
-
-
-
-
-There are three endpoints for entry points to the big data cluster:
-
-
-
-
Endpoint
Description
-
-
HDFS/Spark (Knox) gateway
An HTTPS-based endpoint that proxies other endpoints. The HDFS/Spark gateway is used for accessing services like webHDFS and Livy. Wherever you see references to Knox, this is the endpoint
-
Controller endpoint
The endpoint for the big data cluster management service that exposes REST APIs for managing the cluster. Some tools, such as the Admin portal, are also accessed through this endpoint
-
Master Instance
Get a detailed description of a specific pod in json format output. It includes details, such as the current Kubernetes node that the pod is placed on, the containers running within the pod, and the image used to bootstrap the containers. It also shows other details, such as labels, status, and persisted volumes claims that are associated with the pod
-
-
-
-You can see these endpoints in this diagram:
-
-
-
-
-
-
-
-
-
-
-When you create the cluster, a number of logins are created. Some of these logins are for services to communicate with each other, and others are for end users to access the cluster.
-End-user passwords are currently set using environment variables. These are passwords that SQL administrators and cluster administrators use to access services:
-
-
-
-
Use
Variable
-
-
Controller username
CONTROLLER_USERNAME=controller_username
-
Controller password
CONTROLLER_PASSWORD=controller_password
-
SQL Master SA password
MSSQL_SA_PASSWORD=controller_sa_password
-
Password for accessing the HDFS/Spark endpoint
KNOX_PASSWORD=knox_password
-
-
-
-Note that in the final release, these will not be stored as environment variables but in a more secure fashion..
-
-Intra-cluster authentication
-Upon deployment of the cluster, a number of SQL logins are created:
-
-A special SQL login is created in the Controller SQL instance that is system managed, with sysadmin role. The password for this login is captured as a K8s secret. A sysadmin login is created in all SQL instances in the cluster, that Controller owns and manages. It is required for Controller to perform administrative tasks, such as HA setup or upgrade, on these instances. These logins are also used for intra-cluster communication between SQL instances, such as the SQL master instance communicating with a data pool.
-
-Note: In current release, only basic authentication is supported. Fine-grained access control to HDFS objects, and SQL big data cluster compute and data pools, is not yet available.
-
-For Intra-cluster communication with non-SQL services within the big data cluster, such as Livy to Spark or Spark to the storage pool, security uses certificates. All SQL Server to SQL Server communication is secured using SQL logins.
-
-
-
Activity: Review Security Endpoints
-
-
-In this activity, you will review the endpoints exposed on the cluster.
-
-Steps
-
-
-
-Congratulations! You have completed this workshop on SQL Server big data clusters Architecture. You now have the tools, assets, and processes you need to extrapolate this information into other applications.
+
+
+# Workshop: Microsoft SQL Server big data clusters Architecture (CTP 2.3)
+
+#### A Microsoft workshop from the SQL Server team
+
+
+
+
Security
+
+In this workshop you'll cover using a Process and and various Platform components to create a SQL Server big data cluster solution you can deploy on premises, in the cloud, or in a hybrid architecture. In each module you'll get more references, which you should follow up on to learn more. Also watch for links within the text - click on each one to explore that topic.
+
+(Make sure you check out the prerequisites page before you start. You'll need all of the items loaded there before you can proceed with the workshop.)
+
+You'll cover the following topics in this Module:
+
+
+
+Authentication is the process of verifying the identity of a user or service and ensuring they are who they are claiming to be. Authorization refers to granting or denying of access to specific resources based on the requesting user's identity. This step is performed after a user is identified through authentication.
+
+*NOTE: Security will change prior to the General Availability (GA) Release. Active Directory integration is planned for production implementations.*
+
+
+
+
+
+There are three endpoints for entry points to the big data cluster:
+
+
+
+
Endpoint
Description
+
+
HDFS/Spark (Knox) gateway
An HTTPS-based endpoint that proxies other endpoints. The HDFS/Spark gateway is used for accessing services like webHDFS and Livy. Wherever you see references to Knox, this is the endpoint
+
Controller endpoint
The endpoint for the big data cluster management service that exposes REST APIs for managing the cluster. Some tools, such as the Admin portal, are also accessed through this endpoint
+
Master Instance
Get a detailed description of a specific pod in json format output. It includes details, such as the current Kubernetes node that the pod is placed on, the containers running within the pod, and the image used to bootstrap the containers. It also shows other details, such as labels, status, and persisted volumes claims that are associated with the pod
+
+
+
+You can see these endpoints in this diagram:
+
+
+
+
+
+
+
+
+
+
+When you create the cluster, a number of logins are created. Some of these logins are for services to communicate with each other, and others are for end users to access the cluster.
+Non-SQL Server End-user passwords currently are set using environment variables. These are passwords that cluster administrators use to access services:
+
+
+
+
Use
Variable
+
+
Controller username
CONTROLLER_USERNAME=controller_username
+
Controller password
CONTROLLER_PASSWORD=controller_password
+
SQL Master SA password
MSSQL_SA_PASSWORD=controller_sa_password
+
Password for accessing the HDFS/Spark endpoint
KNOX_PASSWORD=knox_password
+
+
+
+
+Intra-cluster authentication
+Upon deployment of the cluster, a number of SQL logins are created:
+
+A special SQL login is created in the Controller SQL instance that is system managed, with sysadmin role. The password for this login is captured as a K8s secret. A sysadmin login is created in all SQL instances in the cluster, that Controller owns and manages. It is required for Controller to perform administrative tasks, such as HA setup or upgrade, on these instances. These logins are also used for intra-cluster communication between SQL instances, such as the SQL master instance communicating with a data pool.
+
+Note: In current release, only basic authentication is supported. Fine-grained access control to HDFS objects, and SQL big data cluster compute and data pools, is not yet available.
+
+For Intra-cluster communication with non-SQL services within the big data cluster, such as Livy to Spark or Spark to the storage pool, security uses certificates. All SQL Server to SQL Server communication is secured using SQL logins.
+
+
+
Activity: Review Security Endpoints
+
+
+In this activity, you will review the endpoints exposed on the cluster.
+
+Steps
+
+
+
+Congratulations! You have completed this workshop on SQL Server big data clusters Architecture. You now have the tools, assets, and processes you need to extrapolate this information into other applications.
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/data/adventureworksproducts.csv b/sqlserver2019bigdataclusters/SQL2019BDC/data/adventureworksproducts.csv
deleted file mode 100644
index 01d4e69b..00000000
--- a/sqlserver2019bigdataclusters/SQL2019BDC/data/adventureworksproducts.csv
+++ /dev/null
@@ -1,296 +0,0 @@
-ProductID,Name,ProductNumber,Color,StandardCost,ListPrice,Size,Weight
-680,"HL Road Frame - Black, 58",FR-R92B-58,Black,1059.3100,1431.5000,58,1016.04
-706,"HL Road Frame - Red, 58",FR-R92R-58,Red,1059.3100,1431.5000,58,1016.04
-707,"Sport-100 Helmet, Red",HL-U509-R,Red,13.0863,34.9900,NULL,NULL
-708,"Sport-100 Helmet, Black",HL-U509,Black,13.0863,34.9900,NULL,NULL
-709,"Mountain Bike Socks, M",SO-B909-M,White,3.3963,9.5000,M,NULL
-710,"Mountain Bike Socks, L",SO-B909-L,White,3.3963,9.5000,L,NULL
-711,"Sport-100 Helmet, Blue",HL-U509-B,Blue,13.0863,34.9900,NULL,NULL
-712,AWC Logo Cap,CA-1098,Multi,6.9223,8.9900,NULL,NULL
-713,"Long-Sleeve Logo Jersey, S",LJ-0192-S,Multi,38.4923,49.9900,S,NULL
-714,"Long-Sleeve Logo Jersey, M",LJ-0192-M,Multi,38.4923,49.9900,M,NULL
-715,"Long-Sleeve Logo Jersey, L",LJ-0192-L,Multi,38.4923,49.9900,L,NULL
-716,"Long-Sleeve Logo Jersey, XL",LJ-0192-X,Multi,38.4923,49.9900,XL,NULL
-717,"HL Road Frame - Red, 62",FR-R92R-62,Red,868.6342,1431.5000,62,1043.26
-718,"HL Road Frame - Red, 44",FR-R92R-44,Red,868.6342,1431.5000,44,961.61
-719,"HL Road Frame - Red, 48",FR-R92R-48,Red,868.6342,1431.5000,48,979.75
-720,"HL Road Frame - Red, 52",FR-R92R-52,Red,868.6342,1431.5000,52,997.90
-721,"HL Road Frame - Red, 56",FR-R92R-56,Red,868.6342,1431.5000,56,1016.04
-722,"LL Road Frame - Black, 58",FR-R38B-58,Black,204.6251,337.2200,58,1115.83
-723,"LL Road Frame - Black, 60",FR-R38B-60,Black,204.6251,337.2200,60,1124.90
-724,"LL Road Frame - Black, 62",FR-R38B-62,Black,204.6251,337.2200,62,1133.98
-725,"LL Road Frame - Red, 44",FR-R38R-44,Red,187.1571,337.2200,44,1052.33
-726,"LL Road Frame - Red, 48",FR-R38R-48,Red,187.1571,337.2200,48,1070.47
-727,"LL Road Frame - Red, 52",FR-R38R-52,Red,187.1571,337.2200,52,1088.62
-728,"LL Road Frame - Red, 58",FR-R38R-58,Red,187.1571,337.2200,58,1115.83
-729,"LL Road Frame - Red, 60",FR-R38R-60,Red,187.1571,337.2200,60,1124.90
-730,"LL Road Frame - Red, 62",FR-R38R-62,Red,187.1571,337.2200,62,1133.98
-731,"ML Road Frame - Red, 44",FR-R72R-44,Red,352.1394,594.8300,44,1006.97
-732,"ML Road Frame - Red, 48",FR-R72R-48,Red,352.1394,594.8300,48,1025.11
-733,"ML Road Frame - Red, 52",FR-R72R-52,Red,352.1394,594.8300,52,1043.26
-734,"ML Road Frame - Red, 58",FR-R72R-58,Red,352.1394,594.8300,58,1070.47
-735,"ML Road Frame - Red, 60",FR-R72R-60,Red,352.1394,594.8300,60,1079.54
-736,"LL Road Frame - Black, 44",FR-R38B-44,Black,204.6251,337.2200,44,1052.33
-737,"LL Road Frame - Black, 48",FR-R38B-48,Black,204.6251,337.2200,48,1070.47
-738,"LL Road Frame - Black, 52",FR-R38B-52,Black,204.6251,337.2200,52,1088.62
-739,"HL Mountain Frame - Silver, 42",FR-M94S-42,Silver,747.2002,1364.5000,42,1233.76
-740,"HL Mountain Frame - Silver, 44",FR-M94S-44,Silver,706.8110,1364.5000,44,1251.91
-741,"HL Mountain Frame - Silver, 48",FR-M94S-52,Silver,706.8110,1364.5000,48,1270.05
-742,"HL Mountain Frame - Silver, 46",FR-M94S-46,Silver,747.2002,1364.5000,46,1288.20
-743,"HL Mountain Frame - Black, 42",FR-M94B-42,Black,739.0410,1349.6000,42,1233.76
-744,"HL Mountain Frame - Black, 44",FR-M94B-44,Black,699.0928,1349.6000,44,1251.91
-745,"HL Mountain Frame - Black, 48",FR-M94B-48,Black,699.0928,1349.6000,48,1270.05
-746,"HL Mountain Frame - Black, 46",FR-M94B-46,Black,739.0410,1349.6000,46,1288.20
-747,"HL Mountain Frame - Black, 38",FR-M94B-38,Black,739.0410,1349.6000,38,1215.62
-748,"HL Mountain Frame - Silver, 38",FR-M94S-38,Silver,747.2002,1364.5000,38,1215.62
-749,"Road-150 Red, 62",BK-R93R-62,Red,2171.2942,3578.2700,62,6803.85
-750,"Road-150 Red, 44",BK-R93R-44,Red,2171.2942,3578.2700,44,6245.93
-751,"Road-150 Red, 48",BK-R93R-48,Red,2171.2942,3578.2700,48,6409.23
-752,"Road-150 Red, 52",BK-R93R-52,Red,2171.2942,3578.2700,52,6540.77
-753,"Road-150 Red, 56",BK-R93R-56,Red,2171.2942,3578.2700,56,6658.70
-754,"Road-450 Red, 58",BK-R68R-58,Red,884.7083,1457.9900,58,8069.37
-755,"Road-450 Red, 60",BK-R68R-60,Red,884.7083,1457.9900,60,8119.26
-756,"Road-450 Red, 44",BK-R68R-44,Red,884.7083,1457.9900,44,7606.70
-757,"Road-450 Red, 48",BK-R68R-48,Red,884.7083,1457.9900,48,7770.00
-758,"Road-450 Red, 52",BK-R68R-52,Red,884.7083,1457.9900,52,7901.54
-759,"Road-650 Red, 58",BK-R50R-58,Red,486.7066,782.9900,58,8976.55
-760,"Road-650 Red, 60",BK-R50R-60,Red,486.7066,782.9900,60,9026.44
-761,"Road-650 Red, 62",BK-R50R-62,Red,486.7066,782.9900,62,9071.80
-762,"Road-650 Red, 44",BK-R50R-44,Red,486.7066,782.9900,44,8513.88
-763,"Road-650 Red, 48",BK-R50R-48,Red,486.7066,782.9900,48,8677.18
-764,"Road-650 Red, 52",BK-R50R-52,Red,486.7066,782.9900,52,8808.72
-765,"Road-650 Black, 58",BK-R50B-58,Black,486.7066,782.9900,58,8976.55
-766,"Road-650 Black, 60",BK-R50B-60,Black,486.7066,782.9900,60,9026.44
-767,"Road-650 Black, 62",BK-R50B-62,Black,486.7066,782.9900,62,9071.80
-768,"Road-650 Black, 44",BK-R50B-44,Black,486.7066,782.9900,44,8513.88
-769,"Road-650 Black, 48",BK-R50B-48,Black,486.7066,782.9900,48,8677.18
-770,"Road-650 Black, 52",BK-R50B-52,Black,486.7066,782.9900,52,8808.72
-771,"Mountain-100 Silver, 38",BK-M82S-38,Silver,1912.1544,3399.9900,38,9230.56
-772,"Mountain-100 Silver, 42",BK-M82S-42,Silver,1912.1544,3399.9900,42,9421.06
-773,"Mountain-100 Silver, 44",BK-M82S-44,Silver,1912.1544,3399.9900,44,9584.36
-774,"Mountain-100 Silver, 48",BK-M82S-48,Silver,1912.1544,3399.9900,48,9715.90
-775,"Mountain-100 Black, 38",BK-M82B-38,Black,1898.0944,3374.9900,38,9230.56
-776,"Mountain-100 Black, 42",BK-M82B-42,Black,1898.0944,3374.9900,42,9421.06
-777,"Mountain-100 Black, 44",BK-M82B-44,Black,1898.0944,3374.9900,44,9584.36
-778,"Mountain-100 Black, 48",BK-M82B-48,Black,1898.0944,3374.9900,48,9715.90
-779,"Mountain-200 Silver, 38",BK-M68S-38,Silver,1265.6195,2319.9900,38,10591.33
-780,"Mountain-200 Silver, 42",BK-M68S-42,Silver,1265.6195,2319.9900,42,10781.83
-781,"Mountain-200 Silver, 46",BK-M68S-46,Silver,1265.6195,2319.9900,46,10945.13
-782,"Mountain-200 Black, 38",BK-M68B-38,Black,1251.9813,2294.9900,38,10591.33
-783,"Mountain-200 Black, 42",BK-M68B-42,Black,1251.9813,2294.9900,42,10781.83
-784,"Mountain-200 Black, 46",BK-M68B-46,Black,1251.9813,2294.9900,46,10945.13
-785,"Mountain-300 Black, 38",BK-M47B-38,Black,598.4354,1079.9900,38,11498.51
-786,"Mountain-300 Black, 40",BK-M47B-40,Black,598.4354,1079.9900,40,11689.01
-787,"Mountain-300 Black, 44",BK-M47B-44,Black,598.4354,1079.9900,44,11852.31
-788,"Mountain-300 Black, 48",BK-M47B-48,Black,598.4354,1079.9900,48,11983.85
-789,"Road-250 Red, 44",BK-R89R-44,Red,1518.7864,2443.3500,44,6699.52
-790,"Road-250 Red, 48",BK-R89R-48,Red,1518.7864,2443.3500,48,6862.82
-791,"Road-250 Red, 52",BK-R89R-52,Red,1518.7864,2443.3500,52,6994.36
-792,"Road-250 Red, 58",BK-R89R-58,Red,1554.9479,2443.3500,58,7162.19
-793,"Road-250 Black, 44",BK-R89B-44,Black,1554.9479,2443.3500,44,6699.52
-794,"Road-250 Black, 48",BK-R89B-48,Black,1554.9479,2443.3500,48,6862.82
-795,"Road-250 Black, 52",BK-R89B-52,Black,1554.9479,2443.3500,52,6994.36
-796,"Road-250 Black, 58",BK-R89B-58,Black,1554.9479,2443.3500,58,7112.29
-797,"Road-550-W Yellow, 38",BK-R64Y-38,Yellow,713.0798,1120.4900,38,7869.79
-798,"Road-550-W Yellow, 40",BK-R64Y-40,Yellow,713.0798,1120.4900,40,8060.29
-799,"Road-550-W Yellow, 42",BK-R64Y-42,Yellow,713.0798,1120.4900,42,8223.59
-800,"Road-550-W Yellow, 44",BK-R64Y-44,Yellow,713.0798,1120.4900,44,8355.13
-801,"Road-550-W Yellow, 48",BK-R64Y-48,Yellow,713.0798,1120.4900,48,8473.06
-802,LL Fork,FK-1639,NULL,65.8097,148.2200,NULL,NULL
-803,ML Fork,FK-5136,NULL,77.9176,175.4900,NULL,NULL
-804,HL Fork,FK-9939,NULL,101.8936,229.4900,NULL,NULL
-805,LL Headset,HS-0296,NULL,15.1848,34.2000,NULL,NULL
-806,ML Headset,HS-2451,NULL,45.4168,102.2900,NULL,NULL
-807,HL Headset,HS-3479,NULL,55.3801,124.7300,NULL,NULL
-808,LL Mountain Handlebars,HB-M243,NULL,19.7758,44.5400,NULL,NULL
-809,ML Mountain Handlebars,HB-M763,NULL,27.4925,61.9200,NULL,NULL
-810,HL Mountain Handlebars,HB-M918,NULL,53.3999,120.2700,NULL,NULL
-811,LL Road Handlebars,HB-R504,NULL,19.7758,44.5400,NULL,NULL
-812,ML Road Handlebars,HB-R720,NULL,27.4925,61.9200,NULL,NULL
-813,HL Road Handlebars,HB-R956,NULL,53.3999,120.2700,NULL,NULL
-814,"ML Mountain Frame - Black, 38",FR-M63B-38,Black,185.8193,348.7600,38,1238.30
-815,LL Mountain Front Wheel,FW-M423,Black,26.9708,60.7450,NULL,NULL
-816,ML Mountain Front Wheel,FW-M762,Black,92.8071,209.0250,NULL,NULL
-817,HL Mountain Front Wheel,FW-M928,Black,133.2955,300.2150,NULL,NULL
-818,LL Road Front Wheel,FW-R623,Black,37.9909,85.5650,NULL,900.00
-819,ML Road Front Wheel,FW-R762,Black,110.2829,248.3850,NULL,850.00
-820,HL Road Front Wheel,FW-R820,Black,146.5466,330.0600,NULL,650.00
-821,Touring Front Wheel,FW-T905,Black,96.7964,218.0100,NULL,NULL
-822,"ML Road Frame-W - Yellow, 38",FR-R72Y-38,Yellow,360.9428,594.8300,38,988.83
-823,LL Mountain Rear Wheel,RW-M423,Black,38.9588,87.7450,NULL,NULL
-824,ML Mountain Rear Wheel,RW-M762,Black,104.7951,236.0250,NULL,NULL
-825,HL Mountain Rear Wheel,RW-M928,Black,145.2835,327.2150,NULL,NULL
-826,LL Road Rear Wheel,RW-R623,Black,49.9789,112.5650,NULL,1050.00
-827,ML Road Rear Wheel,RW-R762,Black,122.2709,275.3850,NULL,1000.00
-828,HL Road Rear Wheel,RW-R820,Black,158.5346,357.0600,NULL,890.00
-829,Touring Rear Wheel,RW-T905,Black,108.7844,245.0100,NULL,NULL
-830,"ML Mountain Frame - Black, 40",FR-M63B-40,Black,185.8193,348.7600,40,1256.44
-831,"ML Mountain Frame - Black, 44",FR-M63B-44,Black,185.8193,348.7600,44,1274.59
-832,"ML Mountain Frame - Black, 48",FR-M63B-48,Black,185.8193,348.7600,48,1292.73
-833,"ML Road Frame-W - Yellow, 40",FR-R72Y-40,Yellow,360.9428,594.8300,40,1006.97
-834,"ML Road Frame-W - Yellow, 42",FR-R72Y-42,Yellow,360.9428,594.8300,42,1025.11
-835,"ML Road Frame-W - Yellow, 44",FR-R72Y-44,Yellow,360.9428,594.8300,44,1043.26
-836,"ML Road Frame-W - Yellow, 48",FR-R72Y-48,Yellow,360.9428,594.8300,48,1061.40
-837,"HL Road Frame - Black, 62",FR-R92B-62,Black,868.6342,1431.5000,62,1043.26
-838,"HL Road Frame - Black, 44",FR-R92B-44,Black,868.6342,1431.5000,44,961.61
-839,"HL Road Frame - Black, 48",FR-R92B-48,Black,868.6342,1431.5000,48,979.75
-840,"HL Road Frame - Black, 52",FR-R92B-52,Black,868.6342,1431.5000,52,997.90
-841,"Men's Sports Shorts, S",SH-M897-S,Black,24.7459,59.9900,S,NULL
-842,"Touring-Panniers, Large",PA-T100,Grey,51.5625,125.0000,NULL,NULL
-843,Cable Lock,LO-C100,NULL,10.3125,25.0000,NULL,NULL
-844,Minipump,PU-0452,NULL,8.2459,19.9900,NULL,NULL
-845,Mountain Pump,PU-M044,NULL,10.3084,24.9900,NULL,NULL
-846,Taillights - Battery-Powered,LT-T990,NULL,5.7709,13.9900,NULL,NULL
-847,Headlights - Dual-Beam,LT-H902,NULL,14.4334,34.9900,NULL,NULL
-848,Headlights - Weatherproof,LT-H903,NULL,18.5584,44.9900,NULL,NULL
-849,"Men's Sports Shorts, M",SH-M897-M,Black,24.7459,59.9900,M,NULL
-850,"Men's Sports Shorts, L",SH-M897-L,Black,24.7459,59.9900,L,NULL
-851,"Men's Sports Shorts, XL",SH-M897-X,Black,24.7459,59.9900,XL,NULL
-852,"Women's Tights, S",TG-W091-S,Black,30.9334,74.9900,S,NULL
-853,"Women's Tights, M",TG-W091-M,Black,30.9334,74.9900,M,NULL
-854,"Women's Tights, L",TG-W091-L,Black,30.9334,74.9900,L,NULL
-855,"Men's Bib-Shorts, S",SB-M891-S,Multi,37.1209,89.9900,S,NULL
-856,"Men's Bib-Shorts, M",SB-M891-M,Multi,37.1209,89.9900,M,NULL
-857,"Men's Bib-Shorts, L",SB-M891-L,Multi,37.1209,89.9900,L,NULL
-858,"Half-Finger Gloves, S",GL-H102-S,Black,9.1593,24.4900,S,NULL
-859,"Half-Finger Gloves, M",GL-H102-M,Black,9.1593,24.4900,M,NULL
-860,"Half-Finger Gloves, L",GL-H102-L,Black,9.1593,24.4900,L,NULL
-861,"Full-Finger Gloves, S",GL-F110-S,Black,15.6709,37.9900,S,NULL
-862,"Full-Finger Gloves, M",GL-F110-M,Black,15.6709,37.9900,M,NULL
-863,"Full-Finger Gloves, L",GL-F110-L,Black,15.6709,37.9900,L,NULL
-864,"Classic Vest, S",VE-C304-S,Blue,23.7490,63.5000,S,NULL
-865,"Classic Vest, M",VE-C304-M,Blue,23.7490,63.5000,M,NULL
-866,"Classic Vest, L",VE-C304-L,Blue,23.7490,63.5000,L,NULL
-867,"Women's Mountain Shorts, S",SH-W890-S,Black,26.1763,69.9900,S,NULL
-868,"Women's Mountain Shorts, M",SH-W890-M,Black,26.1763,69.9900,M,NULL
-869,"Women's Mountain Shorts, L",SH-W890-L,Black,26.1763,69.9900,L,NULL
-870,Water Bottle - 30 oz.,WB-H098,NULL,1.8663,4.9900,NULL,NULL
-871,Mountain Bottle Cage,BC-M005,NULL,3.7363,9.9900,NULL,NULL
-872,Road Bottle Cage,BC-R205,NULL,3.3623,8.9900,NULL,NULL
-873,Patch Kit/8 Patches,PK-7098,NULL,0.8565,2.2900,NULL,NULL
-874,"Racing Socks, M",SO-R809-M,White,3.3623,8.9900,M,NULL
-875,"Racing Socks, L",SO-R809-L,White,3.3623,8.9900,L,NULL
-876,Hitch Rack - 4-Bike,RA-H123,NULL,44.8800,120.0000,NULL,NULL
-877,Bike Wash - Dissolver,CL-9009,NULL,2.9733,7.9500,NULL,NULL
-878,Fender Set - Mountain,FE-6654,NULL,8.2205,21.9800,NULL,NULL
-879,All-Purpose Bike Stand,ST-1401,NULL,59.4660,159.0000,NULL,NULL
-880,Hydration Pack - 70 oz.,HY-1023-70,Silver,20.5663,54.9900,70,NULL
-881,"Short-Sleeve Classic Jersey, S",SJ-0194-S,Yellow,41.5723,53.9900,S,NULL
-882,"Short-Sleeve Classic Jersey, M",SJ-0194-M,Yellow,41.5723,53.9900,M,NULL
-883,"Short-Sleeve Classic Jersey, L",SJ-0194-L,Yellow,41.5723,53.9900,L,NULL
-884,"Short-Sleeve Classic Jersey, XL",SJ-0194-X,Yellow,41.5723,53.9900,XL,NULL
-885,"HL Touring Frame - Yellow, 60",FR-T98Y-60,Yellow,601.7437,1003.9100,60,1397.06
-886,"LL Touring Frame - Yellow, 62",FR-T67Y-62,Yellow,199.8519,333.4200,62,1451.49
-887,"HL Touring Frame - Yellow, 46",FR-T98Y-46,Yellow,601.7437,1003.9100,46,1342.63
-888,"HL Touring Frame - Yellow, 50",FR-T98Y-50,Yellow,601.7437,1003.9100,50,1360.77
-889,"HL Touring Frame - Yellow, 54",FR-T98Y-54,Yellow,601.7437,1003.9100,54,1378.91
-890,"HL Touring Frame - Blue, 46",FR-T98U-46,Blue,601.7437,1003.9100,46,1342.63
-891,"HL Touring Frame - Blue, 50",FR-T98U-50,Blue,601.7437,1003.9100,50,1360.77
-892,"HL Touring Frame - Blue, 54",FR-T98U-54,Blue,601.7437,1003.9100,54,1378.91
-893,"HL Touring Frame - Blue, 60",FR-T98U-60,Blue,601.7437,1003.9100,60,1397.06
-894,Rear Derailleur,RD-2308,Silver,53.9282,121.4600,NULL,215.00
-895,"LL Touring Frame - Blue, 50",FR-T67U-50,Blue,199.8519,333.4200,50,1406.13
-896,"LL Touring Frame - Blue, 54",FR-T67U-54,Blue,199.8519,333.4200,54,1424.27
-897,"LL Touring Frame - Blue, 58",FR-T67U-58,Blue,199.8519,333.4200,58,1433.34
-898,"LL Touring Frame - Blue, 62",FR-T67U-62,Blue,199.8519,333.4200,62,1451.49
-899,"LL Touring Frame - Yellow, 44",FR-T67Y-44,Yellow,199.8519,333.4200,44,1369.84
-900,"LL Touring Frame - Yellow, 50",FR-T67Y-50,Yellow,199.8519,333.4200,50,1406.13
-901,"LL Touring Frame - Yellow, 54",FR-T67Y-54,Yellow,199.8519,333.4200,54,1424.27
-902,"LL Touring Frame - Yellow, 58",FR-T67Y-58,Yellow,199.8519,333.4200,58,1433.34
-903,"LL Touring Frame - Blue, 44",FR-T67U-44,Blue,199.8519,333.4200,44,1369.84
-904,"ML Mountain Frame-W - Silver, 40",FR-M63S-40,Silver,199.3757,364.0900,40,1256.44
-905,"ML Mountain Frame-W - Silver, 42",FR-M63S-42,Silver,199.3757,364.0900,42,1274.59
-906,"ML Mountain Frame-W - Silver, 46",FR-M63S-46,Silver,199.3757,364.0900,46,1292.73
-907,Rear Brakes,RB-9231,Silver,47.2860,106.5000,NULL,317.00
-908,LL Mountain Seat/Saddle,SE-M236,NULL,12.0413,27.1200,NULL,NULL
-909,ML Mountain Seat/Saddle,SE-M798,NULL,17.3782,39.1400,NULL,NULL
-910,HL Mountain Seat/Saddle,SE-M940,NULL,23.3722,52.6400,NULL,NULL
-911,LL Road Seat/Saddle,SE-R581,NULL,12.0413,27.1200,NULL,NULL
-912,ML Road Seat/Saddle,SE-R908,NULL,17.3782,39.1400,NULL,NULL
-913,HL Road Seat/Saddle,SE-R995,NULL,23.3722,52.6400,NULL,NULL
-914,LL Touring Seat/Saddle,SE-T312,NULL,12.0413,27.1200,NULL,NULL
-915,ML Touring Seat/Saddle,SE-T762,NULL,17.3782,39.1400,NULL,NULL
-916,HL Touring Seat/Saddle,SE-T924,NULL,23.3722,52.6400,NULL,NULL
-917,"LL Mountain Frame - Silver, 42",FR-M21S-42,Silver,144.5938,264.0500,42,1324.48
-918,"LL Mountain Frame - Silver, 44",FR-M21S-44,Silver,144.5938,264.0500,44,1342.63
-919,"LL Mountain Frame - Silver, 48",FR-M21S-48,Silver,144.5938,264.0500,48,1360.77
-920,"LL Mountain Frame - Silver, 52",FR-M21S-52,Silver,144.5938,264.0500,52,1378.91
-921,Mountain Tire Tube,TT-M928,NULL,1.8663,4.9900,NULL,NULL
-922,Road Tire Tube,TT-R982,NULL,1.4923,3.9900,NULL,NULL
-923,Touring Tire Tube,TT-T092,NULL,1.8663,4.9900,NULL,NULL
-924,"LL Mountain Frame - Black, 42",FR-M21B-42,Black,136.7850,249.7900,42,1324.48
-925,"LL Mountain Frame - Black, 44",FR-M21B-44,Black,136.7850,249.7900,44,1342.63
-926,"LL Mountain Frame - Black, 48",FR-M21B-48,Black,136.7850,249.7900,48,1360.77
-927,"LL Mountain Frame - Black, 52",FR-M21B-52,Black,136.7850,249.7900,52,1378.91
-928,LL Mountain Tire,TI-M267,NULL,9.3463,24.9900,NULL,NULL
-929,ML Mountain Tire,TI-M602,NULL,11.2163,29.9900,NULL,NULL
-930,HL Mountain Tire,TI-M823,NULL,13.0900,35.0000,NULL,NULL
-931,LL Road Tire,TI-R092,NULL,8.0373,21.4900,NULL,NULL
-932,ML Road Tire,TI-R628,NULL,9.3463,24.9900,NULL,NULL
-933,HL Road Tire,TI-R982,NULL,12.1924,32.6000,NULL,NULL
-934,Touring Tire,TI-T723,NULL,10.8423,28.9900,NULL,NULL
-935,LL Mountain Pedal,PD-M282,Silver/Black,17.9776,40.4900,NULL,218.00
-936,ML Mountain Pedal,PD-M340,Silver/Black,27.5680,62.0900,NULL,215.00
-937,HL Mountain Pedal,PD-M562,Silver/Black,35.9596,80.9900,NULL,185.00
-938,LL Road Pedal,PD-R347,Silver/Black,17.9776,40.4900,NULL,189.00
-939,ML Road Pedal,PD-R563,Silver/Black,27.5680,62.0900,NULL,168.00
-940,HL Road Pedal,PD-R853,Silver/Black,35.9596,80.9900,NULL,149.00
-941,Touring Pedal,PD-T852,Silver/Black,35.9596,80.9900,NULL,NULL
-942,"ML Mountain Frame-W - Silver, 38",FR-M63S-38,Silver,199.3757,364.0900,38,1238.30
-943,"LL Mountain Frame - Black, 40",FR-M21B-40,Black,136.7850,249.7900,40,1306.34
-944,"LL Mountain Frame - Silver, 40",FR-M21S-40,Silver,144.5938,264.0500,40,1306.34
-945,Front Derailleur,FD-2342,Silver,40.6216,91.4900,NULL,88.00
-946,LL Touring Handlebars,HB-T721,NULL,20.4640,46.0900,NULL,NULL
-947,HL Touring Handlebars,HB-T928,NULL,40.6571,91.5700,NULL,NULL
-948,Front Brakes,FB-9873,Silver,47.2860,106.5000,NULL,317.00
-949,LL Crankset,CS-4759,Black,77.9176,175.4900,NULL,600.00
-950,ML Crankset,CS-6583,Black,113.8816,256.4900,NULL,635.00
-951,HL Crankset,CS-9183,Black,179.8156,404.9900,NULL,575.00
-952,Chain,CH-0234,Silver,8.9866,20.2400,NULL,NULL
-953,"Touring-2000 Blue, 60",BK-T44U-60,Blue,755.1508,1214.8500,60,12655.16
-954,"Touring-1000 Yellow, 46",BK-T79Y-46,Yellow,1481.9379,2384.0700,46,11398.72
-955,"Touring-1000 Yellow, 50",BK-T79Y-50,Yellow,1481.9379,2384.0700,50,11530.26
-956,"Touring-1000 Yellow, 54",BK-T79Y-54,Yellow,1481.9379,2384.0700,54,11648.19
-957,"Touring-1000 Yellow, 60",BK-T79Y-60,Yellow,1481.9379,2384.0700,60,11747.98
-958,"Touring-3000 Blue, 54",BK-T18U-54,Blue,461.4448,742.3500,54,13462.55
-959,"Touring-3000 Blue, 58",BK-T18U-58,Blue,461.4448,742.3500,58,13562.34
-960,"Touring-3000 Blue, 62",BK-T18U-62,Blue,461.4448,742.3500,62,13607.70
-961,"Touring-3000 Yellow, 44",BK-T18Y-44,Yellow,461.4448,742.3500,44,13049.78
-962,"Touring-3000 Yellow, 50",BK-T18Y-50,Yellow,461.4448,742.3500,50,13213.08
-963,"Touring-3000 Yellow, 54",BK-T18Y-54,Yellow,461.4448,742.3500,54,13344.62
-964,"Touring-3000 Yellow, 58",BK-T18Y-58,Yellow,461.4448,742.3500,58,13512.45
-965,"Touring-3000 Yellow, 62",BK-T18Y-62,Yellow,461.4448,742.3500,62,13607.70
-966,"Touring-1000 Blue, 46",BK-T79U-46,Blue,1481.9379,2384.0700,46,11398.72
-967,"Touring-1000 Blue, 50",BK-T79U-50,Blue,1481.9379,2384.0700,50,11530.26
-968,"Touring-1000 Blue, 54",BK-T79U-54,Blue,1481.9379,2384.0700,54,11648.19
-969,"Touring-1000 Blue, 60",BK-T79U-60,Blue,1481.9379,2384.0700,60,11747.98
-970,"Touring-2000 Blue, 46",BK-T44U-46,Blue,755.1508,1214.8500,46,12305.90
-971,"Touring-2000 Blue, 50",BK-T44U-50,Blue,755.1508,1214.8500,50,12437.44
-972,"Touring-2000 Blue, 54",BK-T44U-54,Blue,755.1508,1214.8500,54,12555.37
-973,"Road-350-W Yellow, 40",BK-R79Y-40,Yellow,1082.5100,1700.9900,40,6962.61
-974,"Road-350-W Yellow, 42",BK-R79Y-42,Yellow,1082.5100,1700.9900,42,7153.11
-975,"Road-350-W Yellow, 44",BK-R79Y-44,Yellow,1082.5100,1700.9900,44,7316.41
-976,"Road-350-W Yellow, 48",BK-R79Y-48,Yellow,1082.5100,1700.9900,48,7447.95
-977,"Road-750 Black, 58",BK-R19B-58,Black,343.6496,539.9900,58,9430.14
-978,"Touring-3000 Blue, 44",BK-T18U-44,Blue,461.4448,742.3500,44,13049.78
-979,"Touring-3000 Blue, 50",BK-T18U-50,Blue,461.4448,742.3500,50,13213.08
-980,"Mountain-400-W Silver, 38",BK-M38S-38,Silver,419.7784,769.4900,38,11952.10
-981,"Mountain-400-W Silver, 40",BK-M38S-40,Silver,419.7784,769.4900,40,12142.60
-982,"Mountain-400-W Silver, 42",BK-M38S-42,Silver,419.7784,769.4900,42,12305.90
-983,"Mountain-400-W Silver, 46",BK-M38S-46,Silver,419.7784,769.4900,46,12437.44
-984,"Mountain-500 Silver, 40",BK-M18S-40,Silver,308.2179,564.9900,40,12405.69
-985,"Mountain-500 Silver, 42",BK-M18S-42,Silver,308.2179,564.9900,42,12596.19
-986,"Mountain-500 Silver, 44",BK-M18S-44,Silver,308.2179,564.9900,44,12759.49
-987,"Mountain-500 Silver, 48",BK-M18S-48,Silver,308.2179,564.9900,48,12891.03
-988,"Mountain-500 Silver, 52",BK-M18S-52,Silver,308.2179,564.9900,52,13008.96
-989,"Mountain-500 Black, 40",BK-M18B-40,Black,294.5797,539.9900,40,12405.69
-990,"Mountain-500 Black, 42",BK-M18B-42,Black,294.5797,539.9900,42,12596.19
-991,"Mountain-500 Black, 44",BK-M18B-44,Black,294.5797,539.9900,44,12759.49
-992,"Mountain-500 Black, 48",BK-M18B-48,Black,294.5797,539.9900,48,12891.03
-993,"Mountain-500 Black, 52",BK-M18B-52,Black,294.5797,539.9900,52,13008.96
-994,LL Bottom Bracket,BB-7421,NULL,23.9716,53.9900,NULL,223.00
-995,ML Bottom Bracket,BB-8107,NULL,44.9506,101.2400,NULL,168.00
-996,HL Bottom Bracket,BB-9108,NULL,53.9416,121.4900,NULL,170.00
-997,"Road-750 Black, 44",BK-R19B-44,Black,343.6496,539.9900,44,8967.47
-998,"Road-750 Black, 48",BK-R19B-48,Black,343.6496,539.9900,48,9130.77
-999,"Road-750 Black, 52",BK-R19B-52,Black,343.6496,539.9900,52,9262.31
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/ML_Sample.sql b/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/ML_Sample.sql
new file mode 100644
index 00000000..67272ca7
--- /dev/null
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/ML_Sample.sql
@@ -0,0 +1,265 @@
+/* Enable Machinbe Learning Servicews */
+EXEC sp_configure 'external scripts enabled', 1;
+GO
+RECONFIGURE WITH OVERRIDE;
+GO
+
+/* Load Prediction Data */
+USE [master]
+RESTORE DATABASE [NYC]
+FROM DISK = N'/var/opt/mssql/data/NYC.bak'
+WITH FILE = 1
+, REPLACE
+, MOVE N'NYCTaxi_Sample' TO N'/var/opt/mssql/data/NYC.mdf'
+, MOVE N'NYCTaxi_Sample_log' TO N'/var/opt/mssql/data/NYC.ldf'
+, NOUNLOAD, STATS = 5;
+GO
+
+/* Run some Python. This just returns a a couple of library versions */
+EXECUTE sp_execute_external_script
+@language = N'Python',
+@script = N'
+import numpy as np
+import pandas as pd
+print(np.__version__)
+print(pd.__version__)
+',
+@input_data_1 = N'SELECT 1 as Col1';
+GO
+
+/* Creates Plot Binary data */
+DROP PROCEDURE IF EXISTS PyPlotMatplotlib;
+GO
+
+CREATE PROCEDURE [dbo].[PyPlotMatplotlib]
+AS
+BEGIN
+ SET NOCOUNT ON;
+ DECLARE @query nvarchar(max) =
+ N'SELECT cast(tipped as int) as tipped, tip_amount, fare_amount FROM [dbo].[nyctaxi_sample]'
+ EXECUTE sp_execute_external_script
+ @language = N'Python',
+ @script = N'
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import pandas as pd
+import pickle
+
+fig_handle = plt.figure()
+plt.hist(InputDataSet.tipped)
+plt.xlabel("Tipped")
+plt.ylabel("Counts")
+plt.title("Histogram, Tipped")
+plot0 = pd.DataFrame(data =[pickle.dumps(fig_handle)], columns =["plot"])
+plt.clf()
+
+plt.hist(InputDataSet.tip_amount)
+plt.xlabel("Tip amount ($)")
+plt.ylabel("Counts")
+plt.title("Histogram, Tip amount")
+plot1 = pd.DataFrame(data =[pickle.dumps(fig_handle)], columns =["plot"])
+plt.clf()
+
+plt.hist(InputDataSet.fare_amount)
+plt.xlabel("Fare amount ($)")
+plt.ylabel("Counts")
+plt.title("Histogram, Fare amount")
+plot2 = pd.DataFrame(data =[pickle.dumps(fig_handle)], columns =["plot"])
+plt.clf()
+
+plt.scatter( InputDataSet.fare_amount, InputDataSet.tip_amount)
+plt.xlabel("Fare Amount ($)")
+plt.ylabel("Tip Amount ($)")
+plt.title("Tip amount by Fare amount")
+plot3 = pd.DataFrame(data =[pickle.dumps(fig_handle)], columns =["plot"])
+plt.clf()
+
+OutputDataSet = plot0.append(plot1, ignore_index=True).append(plot2, ignore_index=True).append(plot3, ignore_index=True)
+',
+@input_data_1 = @query
+WITH RESULT SETS ((plot varbinary(max)))
+END
+GO
+
+/* Call that procedure */
+EXEC [dbo].[PyPlotMatplotlib];
+GO
+
+/* Split the Sample */
+DROP PROCEDURE IF EXISTS PyTrainTestSplit;
+GO
+
+CREATE PROCEDURE [dbo].[PyTrainTestSplit] (@pct int)
+AS
+
+DROP TABLE IF EXISTS dbo.nyctaxi_sample_training
+SELECT * into nyctaxi_sample_training FROM nyctaxi_sample WHERE (ABS(CAST(BINARY_CHECKSUM(medallion,hack_license) as int)) % 100) < @pct
+
+DROP TABLE IF EXISTS dbo.nyctaxi_sample_testing
+SELECT * into nyctaxi_sample_testing FROM nyctaxi_sample
+WHERE (ABS(CAST(BINARY_CHECKSUM(medallion,hack_license) as int)) % 100) > @pct
+GO
+
+/* Call proc using 60% Split */
+EXEC PyTrainTestSplit 60
+GO
+
+/* Create Linear Regression Model */
+DROP PROCEDURE IF EXISTS PyTrainScikit;
+GO
+
+CREATE PROCEDURE [dbo].[PyTrainScikit] (@trained_model varbinary(max) OUTPUT)
+AS
+BEGIN
+EXEC sp_execute_external_script
+ @language = N'Python',
+ @script = N'
+import numpy
+import pickle
+from sklearn.linear_model import LogisticRegression
+
+##Create SciKit-Learn logistic regression model
+X = InputDataSet[["passenger_count", "trip_distance", "trip_time_in_secs", "direct_distance"]]
+y = numpy.ravel(InputDataSet[["tipped"]])
+
+SKLalgo = LogisticRegression()
+logitObj = SKLalgo.fit(X, y)
+
+##Serialize model
+trained_model = pickle.dumps(logitObj)
+',
+@input_data_1 = N'
+select tipped, fare_amount, passenger_count, trip_time_in_secs, trip_distance,
+dbo.fnCalculateDistance(pickup_latitude, pickup_longitude, dropoff_latitude, dropoff_longitude) as direct_distance
+from nyctaxi_sample_training
+',
+@input_data_1_name = N'InputDataSet',
+@params = N'@trained_model varbinary(max) OUTPUT',
+@trained_model = @trained_model OUTPUT;
+;
+END;
+GO
+
+/* Store the models. You could version, tag, or stamp here if you want */
+DECLARE @model VARBINARY(MAX);
+EXEC PyTrainScikit @model OUTPUT;
+INSERT INTO nyc_taxi_models (name, model) VALUES('SciKit_model', @model);
+
+/* Train using RevoScalePy */
+DROP PROCEDURE IF EXISTS TrainTipPredictionModelRxPy;
+GO
+
+CREATE PROCEDURE [dbo].[TrainTipPredictionModelRxPy] (@trained_model varbinary(max) OUTPUT)
+AS
+BEGIN
+EXEC sp_execute_external_script
+ @language = N'Python',
+ @script = N'
+import numpy
+import pickle
+from revoscalepy.functions.RxLogit import rx_logit
+
+## Create a logistic regression model using rx_logit function from revoscalepy package
+logitObj = rx_logit("tipped ~ passenger_count + trip_distance + trip_time_in_secs + direct_distance", data = InputDataSet);
+
+## Serialize model
+trained_model = pickle.dumps(logitObj)
+',
+@input_data_1 = N'
+select tipped, fare_amount, passenger_count, trip_time_in_secs, trip_distance,
+dbo.fnCalculateDistance(pickup_latitude, pickup_longitude, dropoff_latitude, dropoff_longitude) as direct_distance
+from nyctaxi_sample_training
+',
+@input_data_1_name = N'InputDataSet',
+@params = N'@trained_model varbinary(max) OUTPUT',
+@trained_model = @trained_model OUTPUT;
+;
+END;
+GO
+
+/* Now save that model out */
+DECLARE @model VARBINARY(MAX);
+EXEC TrainTipPredictionModelRxPy @model OUTPUT;
+INSERT INTO nyc_taxi_models (name, model) VALUES('revoscalepy_model', @model);
+GO
+
+/* Create Scoring function from SciKit Learn */
+DROP PROCEDURE IF EXISTS PredictTipSciKitPy;
+GO
+
+CREATE PROCEDURE [dbo].[PredictTipSciKitPy] (@model varchar(50), @inquery nvarchar(max))
+AS
+BEGIN
+DECLARE @lmodel2 varbinary(max) = (select model from nyc_taxi_models where name = @model);
+EXEC sp_execute_external_script
+ @language = N'Python',
+ @script = N'
+import pickle;
+import numpy;
+from sklearn import metrics
+
+mod = pickle.loads(lmodel2)
+X = InputDataSet[["passenger_count", "trip_distance", "trip_time_in_secs", "direct_distance"]]
+y = numpy.ravel(InputDataSet[["tipped"]])
+
+probArray = mod.predict_proba(X)
+probList = []
+for i in range(len(probArray)):
+ probList.append((probArray[i])[1])
+
+probArray = numpy.asarray(probList)
+fpr, tpr, thresholds = metrics.roc_curve(y, probArray)
+aucResult = metrics.auc(fpr, tpr)
+print ("AUC on testing data is: " + str(aucResult))
+
+OutputDataSet = pandas.DataFrame(data = probList, columns = ["predictions"])
+',
+ @input_data_1 = @inquery,
+ @input_data_1_name = N'InputDataSet',
+ @params = N'@lmodel2 varbinary(max)',
+ @lmodel2 = @lmodel2
+WITH RESULT SETS ((Score float));
+END
+GO
+
+/* Score with RevoScalePy */
+DROP PROCEDURE IF EXISTS PredictTipRxPy;
+GO
+
+CREATE PROCEDURE [dbo].[PredictTipRxPy] (@model varchar(50), @inquery nvarchar(max))
+AS
+BEGIN
+DECLARE @lmodel2 varbinary(max) = (select model from nyc_taxi_models where name = @model);
+EXEC sp_execute_external_script
+ @language = N'Python',
+ @script = N'
+import pickle;
+import numpy;
+from sklearn import metrics
+from revoscalepy.functions.RxPredict import rx_predict;
+
+mod = pickle.loads(lmodel2)
+X = InputDataSet[["passenger_count", "trip_distance", "trip_time_in_secs", "direct_distance"]]
+y = numpy.ravel(InputDataSet[["tipped"]])
+
+probArray = rx_predict(mod, X)
+probList = probArray["tipped_Pred"].values
+
+probArray = numpy.asarray(probList)
+fpr, tpr, thresholds = metrics.roc_curve(y, probArray)
+aucResult = metrics.auc(fpr, tpr)
+print ("AUC on testing data is: " + str(aucResult))
+
+OutputDataSet = pandas.DataFrame(data = probList, columns = ["predictions"])
+',
+ @input_data_1 = @inquery,
+ @input_data_1_name = N'InputDataSet',
+ @params = N'@lmodel2 varbinary(max)',
+ @lmodel2 = @lmodel2
+WITH RESULT SETS ((Score float));
+END
+GO
+
+
+
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/bdc_tutorial_00.ipynb b/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/bdc_tutorial_00.ipynb
new file mode 100644
index 00000000..dc009ef6
--- /dev/null
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/bdc_tutorial_00.ipynb
@@ -0,0 +1,95 @@
+{
+ "metadata": {
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.6.6",
+ "mimetype": "text/x-python",
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "pygments_lexer": "ipython3",
+ "nbconvert_exporter": "python",
+ "file_extension": ".py"
+ }
+ },
+ "nbformat_minor": 2,
+ "nbformat": 4,
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": "\r\n \r\n\r\n# SQL Server 2019 big data cluster Tutorial\r\n## 00 - Scenario Overview and System Setup\r\n\r\nIn this set of tutorials you'll work with an end-to-end scenario that uses SQL Server 2019's big data clusters to solve real-world problems. \r\n",
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Wide World Importers\r\n\r\nWide World Importers (WWI) is a traditional brick and mortar business that makes specialty items for other companies to use in their products. They design, sell and ship these products worldwide.\r\n\r\nWWI corporate has now added a new partnership with a company called \"AdventureWorks\", which sells bicycles both online and in-store. The AdventureWorks company has asked WWI to produce super-hero themed baskets, seats and other bicycle equipment for a new line of bicycles. WWI corporate has asked the IT department to develop a pilot program with these goals: \r\n\r\n- Integrate the large amounts of data from the AdventureWorks company including customers, products and sales\r\n- Allow a cross-selling strategy so that current WWI customers and AdventureWorks customers see their information without having to re-enter it\r\n- Incorporate their online sales information for deeper analysis\r\n- Provide a historical data set so that the partnership can be evaluated\r\n- Ensure this is a \"framework\" approach, so that it can be re-used with other partners\r\n\r\nWWI has a typical N-Tier application that provides a series of terminals, a Business Logic layer, and a Database back-end. They use on-premises systems, and are interested in linking these to the cloud. \r\n\r\nIn this series of tutorials, you will build a solution using the scale-out features of SQL Server 2019, Data Virtualization, Data Marts, and the Data Lake features. ",
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Running these Tutorials\r\n\r\n- You can read through the output of these completed tutorials if you wish - or:\r\n\r\n- You can follow along with the steps you see in these tutorials by copying the code into a SQL Query window and Spark Notebook using the Azure Data Studio tool, or you can click here to download these Jupyter Notebooks and run them in Azure Data Studio for a hands-on experience.\r\n \r\n- If you would like to run the tutorials, you'll need a SQL Server 2019 big data cluster and the client tools installed. If you want to set up your own cluster, click this reference and follow the steps you see there for the server and tools you need.\r\n\r\n- You will need to have the following: \r\n - Your **Knox Password**\r\n - The **Knox IP Address**\r\n - The `sa` **Username** and **Password** to your Master Instance\r\n - The **IP address** to the SQL Server big data cluster Master Instance \r\n - The **name** of your big data cluster\r\n\r\nFor a complete workshop on SQL Server 2019's big data clusters, check out this resource.",
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Copy Database backups to the SQL Server 2019 big data cluster Master Instance\r\n\r\nThe first step for the solution is to copy the database backups from WWI from their location on the cloud and then up to your cluster. \r\n\r\nThese commands use the `curl`program to pull the files down. [You can read more about curl here](https://curl.haxx.se/). \r\n\r\nThe next set of commands use the `kubectl` command to copy the files from where you downloaded them to the data directory of the SQL Server 2019 bdc Master Instance. [You can read more about kubectl here](https://kubernetes.io/docs/reference/kubectl/overview/). \r\n\r\nNote that you will need to replace the section of the script marked with `*ReplaceWithClusterName*` with the name of your SQL Server 2019 bdc. (It does not need single or double quotes, just the letters of your cluster name.)\r\n\r\nNotice also that these commands assume a `d:` drive, if you want to use another drive or directory, edit accordingly.\r\n\r\nOnce you have edited these commands, open a Command Prompt on your system and copy and paste each block, one at a time.\r\n\r\nIn the next tutorial you will restore these databases on the Master Instance.",
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": "md d:\\temp\r\nd:\r\ncd \\temp\r\n\r\ncurl -G \"https://cs7a9736a9346a1x44c6xb00.blob.core.windows.net/backups/WWI.bak\" -o WWI.bak",
+ "metadata": {},
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stderr",
+ "text": "A subdirectory or file d:\\temp already exists.\n"
+ }
+ ],
+ "execution_count": 2
+ },
+ {
+ "cell_type": "code",
+ "source": "d:\r\ncd \\temp\r\n\r\nkubectl cp WWI.bak mssql-master-pool-0:/var/opt/mssql/data -c mssql-server -n \r\n",
+ "metadata": {},
+ "outputs": [],
+ "execution_count": 0
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Copy Exported Data to Storage Pool\r\n\r\nNext, you'll download a few text files that will form the \"external\" data to be ingested into the Storage Pool HDFS store. In production environments, you have multiple options for moving data into HDFS, such as Spark Streaming or the Azure Data Factory.\r\n\r\nThe first code block creates directories in the HDFS store. The second block downloads the source data from a web location. And in the final block, you'll copy the data from your local system to the SQL Server 2019 big data cluster Storage Pool.\r\n\r\nNote that you need to replace the `ReplaceWithKnoxPassword`, `ReplaceWithKnoxwEdnpoint`, and potentially the driver letter and directory values with teh appropriate information.",
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": "curl -i -L -k -u root: -X PUT \"https://:30443/gateway/default/webhdfs/v1/product_review_data?op=MKDIRS\"\r\ncurl -i -L -k -u root: -X PUT \"https://:30443/gateway/default/webhdfs/v1/adventureworks_customers?op=MKDIRS\"\r\ncurl -i -L -k -u root: -X PUT \"https://:30443/gateway/default/webhdfs/v1/adventureworks_products?op=MKDIRS\"",
+ "metadata": {},
+ "outputs": [],
+ "execution_count": 0
+ },
+ {
+ "cell_type": "code",
+ "source": "curl -G \"https://cs7a9736a9346a1x44c6xb00.blob.core.windows.net/backups/product_reviews_sample.csv\" -o product_reviews.csv\r\ncurl -G \"https://cs7a9736a9346a1x44c6xb00.blob.core.windows.net/backups/customer.csv\" -o customers.csv\r\ncurl -G \"https://cs7a9736a9346a1x44c6xb00.blob.core.windows.net/backups/stockitemholdings.csv\" -o stockitemholdings.csv",
+ "metadata": {},
+ "outputs": [],
+ "execution_count": 0
+ },
+ {
+ "cell_type": "code",
+ "source": "curl -i -L -k -u root: -X PUT \"https://:30443/gateway/default/webhdfs/v1/product_review_data/product_reviews.csv?op=create&overwrite=true\" -H \"Content-Type: application/octet-stream\" -T \"product_reviews.csv\"\r\ncurl -i -L -k -u root: -X PUT \"https://:30443/gateway/default/webhdfs/v1/adventureworks_customers/customers.csv?op=create&overwrite=true\" -H \"Content-Type: application/octet-stream\" -T \"customers.csv\"\r\ncurl -i -L -k -u root: -X PUT \"https://:30443/gateway/default/webhdfs/v1/adventureworks_products/stockitemholdings.csv?op=create&overwrite=true\" -H \"Content-Type: application/octet-stream\" -T \"stockitemholdings.csv\"",
+ "metadata": {},
+ "outputs": [],
+ "execution_count": 0
+ },
+ {
+ "cell_type": "markdown",
+ "source": "## Next Step: Working with the SQL Server 2019 big data cluster Master Instance\r\n\r\n**TODO:** Add Link",
+ "metadata": {}
+ }
+ ]
+}
\ No newline at end of file
diff --git a/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/bdc_tutorial_01.ipynb b/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/bdc_tutorial_01.ipynb
new file mode 100644
index 00000000..3b54ffce
--- /dev/null
+++ b/sqlserver2019bigdataclusters/SQL2019BDC/notebooks/bdc_tutorial_01.ipynb
@@ -0,0 +1,1396 @@
+{
+ "metadata": {
+ "kernelspec": {
+ "name": "SQL",
+ "display_name": "SQL",
+ "language": "sql"
+ },
+ "language_info": {
+ "name": "sql",
+ "version": ""
+ }
+ },
+ "nbformat_minor": 2,
+ "nbformat": 4,
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": "\r\n \r\n\r\n# SQL Server 2019 big data cluster Tutorial\r\n## 01 - SQL Server Master Instance Queries\r\n\r\nIn this tutorial you will learn how to run standard SQL Server Queries against the Master Instance (MI) in a SQL Server big data cluster. \r\n\r\nWe'll start with a simple set of queries to explore the Instance: \r\n\r\n",
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": "/* Instance Version */\r\nSELECT @@VERSION; \r\nGO\r\n\r\n/* General Configuration */\r\nUSE master; \r\nGO \r\nEXEC sp_configure;\r\nGO\r\n\r\n/* Databases on this Instance */\r\nSELECT db.name AS 'Database Name'\r\n, Physical_Name AS 'Location on Disk'\r\n, Cast(Cast(Round(cast(mf.size as decimal) * 8.0/1024000.0,2) as decimal(18,2)) as nvarchar) 'Size (GB)'\r\nFROM sys.master_files mf\r\nINNER JOIN \r\n sys.databases db ON db.database_id = mf.database_id\r\nWHERE mf.type_desc = 'ROWS';\r\nGO\r\n\r\nSELECT * from sys.master_files\r\n",
+ "metadata": {},
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "(1 row affected)"
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Total execution time: 00:00:00.1399588"
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Commands completed successfully."
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/html": "Total execution time: 00:00:00.0121076"
+ },
+ "metadata": {}
+ },
+ {
+ "output_type": "execute_result",
+ "execution_count": 1,
+ "data": {
+ "application/vnd.dataresource+json": {
+ "schema": {
+ "fields": [
+ {
+ "name": "(No column name)"
+ }
+ ]
+ },
+ "data": [
+ {
+ "0": "Microsoft SQL Server 2019 (CTP2.4) - 15.0.1400.75 (X64) \n\tMar 16 2019 11:53:26 \n\tCopyright (C) 2019 Microsoft Corporation\n\tDeveloper Edition (64-bit) on Linux (Ubuntu 16.04.6 LTS) "
+ }
+ ]
+ },
+ "text/html": "
(No column name)
Microsoft SQL Server 2019 (CTP2.4) - 15.0.1400.75 (X64) \n\tMar 16 2019 11:53:26 \n\tCopyright (C) 2019 Microsoft Corporation\n\tDeveloper Edition (64-bit) on Linux (Ubuntu 16.04.6 LTS) <X64>
Activity 1: TBD
Option 1 - TBD
For Further Study
Next Steps
2.0 Automatic Tuning
Next Steps
# SQL Server Workshops
+## (https://aka.ms/sqlworkshops)
This site is a map of learning content produced by and curated by the SQL Server team in Microsoft Engineering. These materials are meant to be instructor-led, but you can work through the materials on a test system on your own if desired. You can view all materials directly in this interface, or you can [view the raw github site for this content here](https://github.com/Microsoft/sqlworkshops).
-See the license information at the bottom of this README.md file.
+*See the license information at the bottom of this README.md file*
Find a problem? Spot a bug? [Post an issue here](https://github.com/Microsoft/sqlworkshops/issues) and we'll try and fix it.
@@ -29,12 +30,11 @@ Find a problem? Spot a bug? [Post an issue here](https://github.com/Microsoft/sq
- [In Development](https://github.com/Microsoft/sqlworkshops)
-## Resources, Source Slide Decks and Code Examples
-
-- [PowerPoint Decks from SQL Server Team Sessions](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#decks)
-- [Sample Code](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#code)
-- [References and Resources](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#links)
+## Presentation Materials, Code, and References
+- [Presentation Materials - *(PowerPoint Decks, etc.)*](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#decks)
+- [Example Code](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#code)
+- [References and Tools from the Microsoft SQL Server Team](https://github.com/Microsoft/sqlworkshops/tree/master/References/README.MD#links)
## Learning how to self-learn
@@ -66,13 +66,13 @@ You can follow the steps below to clone individual files from a git repo using a
Example:
-```
+
-# References: Microsoft SQL Server and Machine Learning in SQL Server (On-prem and Azure)
+# Resources from the Microsoft SQL Server and Azure Data Services Teams
#### Microsoft References from the Azure Data Services Team
@@ -13,7 +13,7 @@ The Microsoft SQL Server organization (along with its parent, the Microsoft Azur
*(Note: These scripts, tools and information site are a not guarenteed to work in every situation, or to be placed into production without thoroughly testing on your own. Microsoft assumes no risk associated with your use of these tools. These sites have adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.)*
-
+
+3. The first time you launch Azure Data Studio, you may see the following choices. For the purposes of this workshop, select No to not load the preview feature and use x to close out the 2nd choice to collect usage data
+
+ 
+
+4. You will now be presented with the following screen to enter in your connection details for SQL Server. For Server, put in the values for EXTERNAL IP, PORT from step 1 above. Change the Authentication type to SQL Login, Put in a User name of sa with the Password you used for the secret in Module 01 when you deployed SQL Server. I also recommend you click the checkbox for Remember Password so you will not have to do it again for future connections. Click the Connect button to connect. An example of a connection looks like the following
+
+ 
+
+5. A successful connection should look like the following
+
+ 
+
+6. Open the script **proc.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder
+
+ The stored procedure looks like the following
+
+ ```sql
+ USE WideWorldImporters
+ GO
+ CREATE or ALTER PROCEDURE [Sales].[CustomerProfits]
+ AS
+ BEGIN
+ -- Declare the table variable
+ DECLARE @ilines TABLE
+ ([InvoiceLineID] [int] NOT NULL primary key,
+ [InvoiceID] [int] NOT NULL,
+ [StockItemID] [int] NOT NULL,
+ [Description] [nvarchar](100) NOT NULL,
+ [PackageTypeID] [int] NOT NULL,
+ [Quantity] [int] NOT NULL,
+ [UnitPrice] [decimal](18, 2) NULL,
+ [TaxRate] [decimal](18, 3) NOT NULL,
+ [TaxAmount] [decimal](18, 2) NOT NULL,
+ [LineProfit] [decimal](18, 2) NOT NULL,
+ [ExtendedPrice] [decimal](18, 2) NOT NULL,
+ [LastEditedBy] [int] NOT NULL,
+ [LastEditedWhen] [datetime2](7) NOT NULL
+ )
+
+ -- Insert all the rows from InvoiceLines into the table variable
+ INSERT INTO @ilines SELECT * FROM Sales.InvoiceLines
+ -- Find my total profile by customer
+ SELECT TOP 1 COUNT(i.CustomerID) as customer_count, SUM(il.LineProfit) as total_profit
+ FROM Sales.Invoices i
+ INNER JOIN @ilines il
+ ON i.InvoiceID = il.InvoiceID
+ GROUP By i.CustomerID
+ END
+ GO
+ ```
+
+ This procedure uses a table variable populated from a user table and then joins it with a user table to provide output. T-SQL functions like COUNT and SUM are often seen in analytic queries that benefit from Intelligent Query Processing. **Note**: In this example the TOP 1 T-SQL syntax is used so that the procedure only produces 1 row. This is only done to make the output easier to read using this workshop and demo since this procedure will be executed multiple times. Normal execution of this procedure may not include TOP.
+
+ Click the Run button to execute the script. You will be prompted to pick the connection to execute the script. Select the connection you created in Step 4.
+
+ When you execute this script the results should look like the following
+
+ 
+
+7. You have been told this procedure executes fairly quickly with a single execution in a few seconds but over several iterations, the total duration is over 20 seconds, which is not acceptable to the application.
+
+ Open the script **repro130.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder.
+
+ The script looks like the following
+
+ ```sql
+ USE master
+ GO
+ ALTER DATABASE wideworldimporters SET compatibility_level = 130
+ GO
+ USE WideWorldImporters
+ GO
+ SET NOCOUNT ON
+ GO
+ EXEC [Sales].[CustomerProfits]
+ GO 25
+ SET NOCOUNT OFF
+ GO
+ ```
+ The script will ensure the database is in a compatibility mode that is less than 150 so Intelligent Query Processing will NOT be enabled. The script also turns off rowcount messages to be returned to the client to reduce network traffic for this test. Then the script executes the stored procedure. Notice the syntax of **GO 25**. This is a client tool tip that says to run the batch 25 times (avoids having to construct a loop)
+
+ Click the Run button to execute the script to observe the results. Choose the connection by clicking on the IP,PORT you created for the SQL Server container and click Connect.
+
+ You will see while the query is executing in the bottom status bar the current elapsed execution time, the server connection details, a status of **Executing Query**, and number of rows being returned to the client.
+
+ 
+
+ The query should complete in well over 30 seconds. Given result sets are being sent back to the client, it can take even longer depending on connectivity speeds of your client workstation.
+
+ The final results of Azure Data Studio should look like this
+
+ 
+
+ You can scroll in the RESULTS or MESSAGES pane. If you scroll down to the bottom of the MESSAGES pane, you can see the total execution time (this includes time to execute the stored procedure 25 times on the server but also return results to the client). Your results should look something like this (30 seconds or greater)
+
+ 
+
+8. Now let's run the same exact test but with database compatibility of 150. You will not make any changes to the stored procedure.
+
+ Open the script **repro150.sql** by using the File Menu/Open File option of Azure Data Studio. The file can be found in the **sqlworkshops/SQLonOpenShift/sqlonopenshift/03_performance/iqp** folder.
+
+ Use the SQL container connection as you have done in previous steps. The script should look like this
+
+
+ ```sql
+ USE master
+ GO
+ ALTER DATABASE wideworldimporters SET compatibility_level = 150
+ GO
+ USE WideWorldImporters
+ GO
+ SET NOCOUNT ON
+ GO
+ EXEC [Sales].[CustomerProfits]
+ GO 25
+ SET NOCOUNT OFF
+ GO
+ ```
+ Notice this is the same script except database compatibility of 150 is used. This time, the query processor in SQL Server will enable table variable deferred compilation so a better query plan can be chosen
+
+9. Run the script and choose the SQL Server container connection. Go through the same steps as in Step 8 to analyze the results. The script should execute far faster than before. Your speeds can vary but should be 15 seconds or less.
+
+As a **post workshop exercise** you can go through this activity in a SQL notebook. For SQL notebooks,use the File menu of Azure Data Studio (Open File option) to open the **03_IQP_Table_Variable.ipynb** notebook in the **sqlworkshops/SQLOnOpenShift/sqlonopenshift/iqp/03_performance** folder. Follow the steps provided in the notebook to complete the activity.
+
+The SQL notebook experience looks like the following:
+
+
+
+Read each step in the notebook and use the Play button to execute each T-SQL script. Use the scrollbar to scroll through the notebook.
+
+The rest of the steps in this section of the Module is for the experience without using a notebook. Proceed to section 3.1.
+
+
-
-Wide World Importers (WWI) is a traditional brick and mortar business with a long track record of success, generating profits through strong retail store sales of their unique offering of affordable products from around the world. They have a great training program for new employees, that focuses on connecting with their customers and providing great face-to-face customer service. This strong focus on customer relationships has helped set WWI apart from their competitors.
-
-WWI has now added web and mobile commerce to their platform, which has generated a significant amount of additional data, and data formats. These new platforms were added without integrating into the OLTP system data or Business Intelligence infrastructures. As a result, "silos" of data stores have developed.
-
-
-
-Now they want to expand their reach to customers around the world through web and mobile e-commerce. But they don't want to just simply make their inventory available online. They want to build upon their track record of strong customer connections, and engage with their customers through personalized, high-quality application experiences that incorporate data and intelligence.
-
-The technology team at WWI has recognized that moving to an omni-channel strategy has quickly outgrown their ability to handle data. They anticipate the following solutions needed to reach more customers and grow the business:
-
- - Scale data systems to reach more consumers
- - Unlock business insights from multiple sources of structured and unstructured data
- - Apply deep analytics with high-performance responses
- - Enable AI into apps to actively engage with customers
-
-Prior to expanding to their current omni-channel strategy, WWI had a simple Point of Sale (POS) application that handled customer orders at each retail store. The back-end was a series of service layers used to process orders and store them in a SQL database. They had designed their systems and tuned them to handle this level of data.
-
-
-
-As they added new e-commerce channels to expand the customer base, consumer demand also increased. This increased demand from more customers ordering products through more channels generated more data. Now WWI has new challenges to address:
-
- - Increased consumer demand, leading to increased app data
- - They are unable to determine business trends because of siloed insights
- - They have a rising data management footprint, increasing cost and complexity
- - New development challenges resulting from more deployment targets and duplicated code
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-![\"Microsoft\"](\"https://github.com/Microsoft/sqlworkshops/blob/master/graphics/solutions-microsoft-logo-small.png?raw=true\"){kind=logo}
\r\n