From 0d528fa077377232224bcc1458a74a09d5fd693d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nuno=20Gon=C3=A7alves?= Date: Fri, 6 Dec 2024 17:21:12 +0000 Subject: [PATCH 1/4] Add a notebook to demonstrate how to run notebooks from another notebook --- .../meta.toml | 13 + .../notebook.ipynb | 306 ++++++++++++++++++ 2 files changed, 319 insertions(+) create mode 100644 notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml create mode 100644 notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb diff --git a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml b/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml new file mode 100644 index 0000000..f5d80e4 --- /dev/null +++ b/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml @@ -0,0 +1,13 @@ +[meta] +authors=["singlestore"] +title="Executing Notebooks from Another Notebook with Fusion SQL" +description="""\ + Learn how to execute Notebooks from another Notebook + in SingleStoreDB Cloud using Fusion SQL. + """ +icon="files" +difficulty="intermediate" +tags=["notebooks", "jobs", "python", "fusion", "starter"] +lesson_areas=["Python SDK"] +destinations=["spaces"] +minimum_tier="standard" diff --git a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb b/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb new file mode 100644 index 0000000..1100aeb --- /dev/null +++ b/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb @@ -0,0 +1,306 @@ +{ + "cells": [ + { + "id": "55bb0386", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "
\n", + " \n", + "
\n", + "
\n", + "
SingleStore Notebooks
\n", + "

Executing Notebooks from Another Notebook with Fusion SQL

\n", + "
\n", + "
" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "id": "18ed6021", + "metadata": {}, + "source": [ + "In this notebook, we demonstrate how to use **[Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/)** to execute a notebook from another notebook, either within the **same session** (useful to avoid code duplication) or in a **new session** (useful for parallel execution)." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating the Sample Notebook\n", + "In this section, we will create a sample notebook to be used in our examples." + ], + "id": "54d88908" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Create the sample notebook in the local filesystem:" + ], + "id": "a2c382f9" + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import nbformat as nbf\n", + "\n", + "nb = nbf.v4.new_notebook()\n", + "\n", + "cell = nbf.v4.new_code_cell(\"\"\"# This is a code cell\n", + "if 'sample_var' not in globals():\n", + " sample_var = 'sample value'\n", + "print('Sample Notebook has been executed!')\"\"\")\n", + "\n", + "cell.metadata = {\n", + " \"language\": \"python\"\n", + "}\n", + "\n", + "nb.cells.append(cell)\n", + "\n", + "# Save the notebook to a file\n", + "with open('sample_notebook.ipynb', 'w') as f:\n", + " nbf.write(nb, f)\n", + "\n", + "print(\"Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.\")" + ], + "id": "dd837f57" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Upload it to the **Data Studio** shared files for later download:" + ], + "id": "bb858912" + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "\n", + "sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000))\n", + "\n", + "%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';\n", + "\n", + "print(\"Notebook '{}' has been created in the Data Studio shared files.\".format(sample_notebook_name))" + ], + "id": "d39b0564" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Executing the Sample Notebook in the Current Session\n", + "\n", + "In this example, we will execute the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions." + ], + "id": "4f0cbad1" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, we download the notebook to the local filesystem:" + ], + "id": "baefb864" + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "%sql DOWNLOAD SHARED FILE '{{ sample_notebook_name }}' TO 'sample_notebook.ipynb' OVERWRITE\n", + "\n", + "print(\"Notebook 'sample_notebook.ipynb' has been created in the local filesystem.\")" + ], + "id": "b04db0d5" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, we execute the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" + ], + "id": "9c08ff46" + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "if 'sample_var' in globals():\n", + " del sample_var\n", + "\n", + "%run sample_notebook.ipynb\n", + "\n", + "print(\"The value of 'sample_var' is '{}'.\\n\".format(sample_var))" + ], + "id": "85ad1f06" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Executing the Sample Notebook in a New Session\n", + "\n", + "Instead of executing a notebook in the current session, we can run it in a new session \u2014 either synchronously or asynchronously \u2014 using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.\n", + "\n", + "
\n", + " \n", + "
\n", + "

Note

\n", + "

SingleStore enforces limits on the number of compute sessions that can be created, which can be increased upon request. Please contact Support for more information.

\n", + "
\n", + "
" + ], + "id": "1a059fcb" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Run **two jobs** in parallel and wait for their completion (each job will run on a separate session):" + ], + "id": "b5d35764" + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "job_ids = []\n", + "for x in range(2):\n", + " print(\"Running job for {}...\".format(x))\n", + " job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {\"sample_var\": \"{{x}}\"}\n", + " job_ids.append(job_res[0].JobID)\n", + "\n", + "print(f'Waiting for jobs to complete... {job_ids}')\n", + "success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES\n", + "\n", + "print(f'All jobs completed with success: {bool(success[0].Success)}')" + ], + "id": "2cdaea66" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "View the executions of the jobs we run:" + ], + "id": "eaa1de71" + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "for job_id in job_ids:\n", + " execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1\n", + " print(execs)" + ], + "id": "b81fa588" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The jobs can now be dropped:" + ], + "id": "f64a5020" + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "for id in job_ids:\n", + " print(f\"Dropping job '{id}'...\")\n", + " %sql DROP JOBS {{id}}" + ], + "id": "5f7b74a6" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Cleanup" + ], + "id": "48372f6b" + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Delete the sample notebook from the **Data Studio** shared files:" + ], + "id": "00b5f172" + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "%%sql\n", + "DROP SHARED FILE '{{ sample_notebook_name }}'" + ], + "id": "c368e8a6" + }, + { + "id": "7259e797", + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
\n", + "
" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 1d5ebb10b618ee0a28e0d874a5ff2aeb8dad80cf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nuno=20Gon=C3=A7alves?= Date: Fri, 6 Dec 2024 21:11:17 +0000 Subject: [PATCH 2/4] fix typo --- .../notebook.ipynb | 82 +++++++++---------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb b/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb index 1100aeb..46415a8 100644 --- a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb +++ b/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb @@ -28,25 +28,26 @@ { "attachments": {}, "cell_type": "markdown", + "id": "54d88908", "metadata": {}, "source": [ "## Creating the Sample Notebook\n", "In this section, we will create a sample notebook to be used in our examples." - ], - "id": "54d88908" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "a2c382f9", "metadata": {}, "source": [ "Create the sample notebook in the local filesystem:" - ], - "id": "a2c382f9" + ] }, { "cell_type": "code", "execution_count": 1, + "id": "dd837f57", "metadata": {}, "outputs": [], "source": [ @@ -70,21 +71,21 @@ " nbf.write(nb, f)\n", "\n", "print(\"Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.\")" - ], - "id": "dd837f57" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "bb858912", "metadata": {}, "source": [ "Upload it to the **Data Studio** shared files for later download:" - ], - "id": "bb858912" + ] }, { "cell_type": "code", "execution_count": 2, + "id": "d39b0564", "metadata": {}, "outputs": [], "source": [ @@ -95,53 +96,53 @@ "%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';\n", "\n", "print(\"Notebook '{}' has been created in the Data Studio shared files.\".format(sample_notebook_name))" - ], - "id": "d39b0564" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "4f0cbad1", "metadata": {}, "source": [ "## Executing the Sample Notebook in the Current Session\n", "\n", "In this example, we will execute the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions." - ], - "id": "4f0cbad1" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "baefb864", "metadata": {}, "source": [ "First, we download the notebook to the local filesystem:" - ], - "id": "baefb864" + ] }, { "cell_type": "code", "execution_count": 3, + "id": "b04db0d5", "metadata": {}, "outputs": [], "source": [ "%sql DOWNLOAD SHARED FILE '{{ sample_notebook_name }}' TO 'sample_notebook.ipynb' OVERWRITE\n", "\n", "print(\"Notebook 'sample_notebook.ipynb' has been created in the local filesystem.\")" - ], - "id": "b04db0d5" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "9c08ff46", "metadata": {}, "source": [ "Next, we execute the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" - ], - "id": "9c08ff46" + ] }, { "cell_type": "code", "execution_count": 4, + "id": "85ad1f06", "metadata": {}, "outputs": [], "source": [ @@ -151,12 +152,12 @@ "%run sample_notebook.ipynb\n", "\n", "print(\"The value of 'sample_var' is '{}'.\\n\".format(sample_var))" - ], - "id": "85ad1f06" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "1a059fcb", "metadata": {}, "source": [ "## Executing the Sample Notebook in a New Session\n", @@ -170,21 +171,21 @@ "

SingleStore enforces limits on the number of compute sessions that can be created, which can be increased upon request. Please contact Support for more information.

\n", " \n", "" - ], - "id": "1a059fcb" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "b5d35764", "metadata": {}, "source": [ "Run **two jobs** in parallel and wait for their completion (each job will run on a separate session):" - ], - "id": "b5d35764" + ] }, { "cell_type": "code", "execution_count": 5, + "id": "2cdaea66", "metadata": {}, "outputs": [], "source": [ @@ -198,79 +199,78 @@ "success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES\n", "\n", "print(f'All jobs completed with success: {bool(success[0].Success)}')" - ], - "id": "2cdaea66" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "eaa1de71", "metadata": {}, "source": [ - "View the executions of the jobs we run:" - ], - "id": "eaa1de71" + "View the executions of the jobs we ran:" + ] }, { "cell_type": "code", "execution_count": 6, + "id": "b81fa588", "metadata": {}, "outputs": [], "source": [ "for job_id in job_ids:\n", " execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 1\n", " print(execs)" - ], - "id": "b81fa588" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "f64a5020", "metadata": {}, "source": [ "The jobs can now be dropped:" - ], - "id": "f64a5020" + ] }, { "cell_type": "code", "execution_count": 7, + "id": "5f7b74a6", "metadata": {}, "outputs": [], "source": [ "for id in job_ids:\n", " print(f\"Dropping job '{id}'...\")\n", " %sql DROP JOBS {{id}}" - ], - "id": "5f7b74a6" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "48372f6b", "metadata": {}, "source": [ "## Cleanup" - ], - "id": "48372f6b" + ] }, { "attachments": {}, "cell_type": "markdown", + "id": "00b5f172", "metadata": {}, "source": [ "Delete the sample notebook from the **Data Studio** shared files:" - ], - "id": "00b5f172" + ] }, { "cell_type": "code", "execution_count": 8, + "id": "c368e8a6", "metadata": {}, "outputs": [], "source": [ "%%sql\n", "DROP SHARED FILE '{{ sample_notebook_name }}'" - ], - "id": "c368e8a6" + ] }, { "id": "7259e797", From 7ba9428128daa9d4da0332596066075ada1a0bfa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nuno=20Gon=C3=A7alves?= Date: Mon, 9 Dec 2024 09:41:47 +0000 Subject: [PATCH 3/4] rename occurrences of execute/executing to run/running --- .../meta.toml | 4 ++-- .../notebook.ipynb | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) rename notebooks/{executing-notebooks-from-another-notebook-with-fusion-sql => running-notebooks-from-another-notebook-with-fusion-sql}/meta.toml (69%) rename notebooks/{executing-notebooks-from-another-notebook-with-fusion-sql => running-notebooks-from-another-notebook-with-fusion-sql}/notebook.ipynb (89%) diff --git a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml similarity index 69% rename from notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml rename to notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml index f5d80e4..8feda32 100644 --- a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/meta.toml +++ b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/meta.toml @@ -1,8 +1,8 @@ [meta] authors=["singlestore"] -title="Executing Notebooks from Another Notebook with Fusion SQL" +title="Running Notebooks from Another Notebook with Fusion SQL" description="""\ - Learn how to execute Notebooks from another Notebook + Learn how to run Notebooks from another Notebook in SingleStoreDB Cloud using Fusion SQL. """ icon="files" diff --git a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb similarity index 89% rename from notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb rename to notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb index 46415a8..28867bf 100644 --- a/notebooks/executing-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb +++ b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb @@ -11,7 +11,7 @@ " \n", "
\n", "
SingleStore Notebooks
\n", - "

Executing Notebooks from Another Notebook with Fusion SQL

\n", + "

Running Notebooks from Another Notebook with Fusion SQL

\n", "
\n", "" ] @@ -22,7 +22,7 @@ "id": "18ed6021", "metadata": {}, "source": [ - "In this notebook, we demonstrate how to use **[Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/)** to execute a notebook from another notebook, either within the **same session** (useful to avoid code duplication) or in a **new session** (useful for parallel execution)." + "In this notebook, we demonstrate how to use **[Fusion SQL](https://www.singlestore.com/spaces/getting-started-with-fusion-sql/)** to run a notebook from another notebook, either within the **same session** (useful to avoid code duplication) or in a **new session** (useful for parallel execution)." ] }, { @@ -104,9 +104,9 @@ "id": "4f0cbad1", "metadata": {}, "source": [ - "## Executing the Sample Notebook in the Current Session\n", + "## Running the Sample Notebook in the Current Session\n", "\n", - "In this example, we will execute the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions." + "In this example, we will run the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions." ] }, { @@ -136,7 +136,7 @@ "id": "9c08ff46", "metadata": {}, "source": [ - "Next, we execute the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" + "Next, we run the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" ] }, { @@ -160,9 +160,9 @@ "id": "1a059fcb", "metadata": {}, "source": [ - "## Executing the Sample Notebook in a New Session\n", + "## Running the Sample Notebook in a New Session\n", "\n", - "Instead of executing a notebook in the current session, we can run it in a new session \u2014 either synchronously or asynchronously \u2014 using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.\n", + "Instead of running a notebook in the current session, we can run it in a new session \u2014 either synchronously or asynchronously \u2014 using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.\n", "\n", "
\n", " \n", From 57917c0bea42329d5adfd5a4f1b938246643008c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Nuno=20Gon=C3=A7alves?= Date: Tue, 17 Dec 2024 15:28:13 +0000 Subject: [PATCH 4/4] add %run_shared example --- .../notebook.ipynb | 35 ++++--------------- 1 file changed, 7 insertions(+), 28 deletions(-) diff --git a/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb index 28867bf..dc51591 100644 --- a/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb +++ b/notebooks/running-notebooks-from-another-notebook-with-fusion-sql/notebook.ipynb @@ -109,39 +109,18 @@ "In this example, we will run the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions." ] }, - { - "attachments": {}, - "cell_type": "markdown", - "id": "baefb864", - "metadata": {}, - "source": [ - "First, we download the notebook to the local filesystem:" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "b04db0d5", - "metadata": {}, - "outputs": [], - "source": [ - "%sql DOWNLOAD SHARED FILE '{{ sample_notebook_name }}' TO 'sample_notebook.ipynb' OVERWRITE\n", - "\n", - "print(\"Notebook 'sample_notebook.ipynb' has been created in the local filesystem.\")" - ] - }, { "attachments": {}, "cell_type": "markdown", "id": "9c08ff46", "metadata": {}, "source": [ - "Next, we run the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" + "To run the notebook, we can use the `%run_shared` magic command. Let's run the notebook and confirm that the `sample_var` variable set in the sample notebook is accessible in the current session:" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 3, "id": "85ad1f06", "metadata": {}, "outputs": [], @@ -149,7 +128,7 @@ "if 'sample_var' in globals():\n", " del sample_var\n", "\n", - "%run sample_notebook.ipynb\n", + "%run_shared {{ sample_notebook_name }}\n", "\n", "print(\"The value of 'sample_var' is '{}'.\\n\".format(sample_var))" ] @@ -184,7 +163,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 4, "id": "2cdaea66", "metadata": {}, "outputs": [], @@ -212,7 +191,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 5, "id": "b81fa588", "metadata": {}, "outputs": [], @@ -233,7 +212,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "id": "5f7b74a6", "metadata": {}, "outputs": [], @@ -263,7 +242,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "id": "c368e8a6", "metadata": {}, "outputs": [],