Skip to content

Update to pyspark package requirements #798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rajmera3 merged 10 commits into from
Aug 12, 2020
Merged

Update to pyspark package requirements #798

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
47c8d2c
Merge pull request #5 from microsoft/master
bweissman Mar 10, 2020
1f2e6c1
separate SP creation
bweissman Mar 10, 2020
8da98bf
Update deploy-sql-big-data-aks.py
bweissman Mar 10, 2020
0a07a79
adjusted for CU5
bweissman Jun 24, 2020
1f04075
change back
bweissman Jun 24, 2020
fc688c9
Merge pull request #6 from microsoft/master
bweissman Jun 24, 2020
8123bd2
update to CU5
bweissman Jun 24, 2020
92ce87c
Merge pull request #7 from microsoft/master
bweissman Jul 22, 2020
f676462
Merge pull request #8 from microsoft/master
bweissman Jul 31, 2020
cf627c8
Updated requirements for pyspark installation in CU5
bweissman Jul 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion samples/features/sql-big-data-cluster/spark/config-install/installpackage_Spark.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,20 @@
"The following code can be used to install packages on each executor node at runtime. \\\n",
"**Note**: This functionality is not available on a non-root BDC deployment (including OpenShift). This installation is temporary, and must be performed each time a new Spark session is invoked.\n",
"\n",
"If you want to use this from CU5 upwards, you must add two settings pre-deployment.\n",
"\n",
"In contron.json, add (under security):\n",
"\n",
"_\"allowRunAsRoot\": true_\n",
"\n",
"In BDC.json, add (under spec.services.spark.settings): \n",
"\n",
"_\"yarn-site.yarn.nodemanager.container-executor.class\": \"org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor\"_\n",
"\n",
"``` Python\n",
"import subprocess\n",
"\n",
"import os\n",
"os.environ[\"XDG_CACHE_HOME\"]=\"/tmp\"\n",
"# Install TensorFlow\n",
"stdout = subprocess.check_output(\n",
" \"pip3 install tensorflow\",\n",
Expand Down