Skip to content

Commit

Permalink
Merge pull request #64 from LSSTDESC/issue/56/notebooks-ec
Browse files Browse the repository at this point in the history
Issue/56/notebooks ec
  • Loading branch information
OliviaLynn committed Aug 3, 2023
2 parents 231c360 + 978e7ca commit f220f17
Show file tree
Hide file tree
Showing 4 changed files with 129 additions and 110 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# RAIL Pipeline: Build, Save, Load, and Run\n",
"# Build, Save, Load, and Run a Pipeline\n",
"\n",
"author: Eric Charles\n",
"**Author:** Eric Charles\n",
"\n",
"last run successfully: August 2, 2023"
"**Last Run Successfully:** August 2, 2023"
]
},
{
Expand Down Expand Up @@ -71,6 +71,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Build the pipeline\n",
"\n",
"### Some configuration setup\n",
"\n",
"The example pipeline builds some of the RAIL creation functionality into a pipeline. \n",
Expand Down Expand Up @@ -184,7 +186,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Here are some examples of interactive introspection into the pipeline\n",
"### Interactive introspection\n",
"\n",
"Here are some examples of interactive introspection into the pipeline\n",
"\n",
"I.e., some functions that you can use to figure out what the pipeline is doing."
]
Expand Down Expand Up @@ -235,7 +239,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Okay, now let's connect up the pipeline stages\n",
"### Connect up the pipeline stages\n",
"\n",
"We can use the `RailStage.connect_input` function to connect one stage to another.\n",
"By default, this will connect the output data product called `output` for one stage."
Expand Down Expand Up @@ -283,7 +287,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Save the pipeline\n",
"## Save the pipeline\n",
"\n",
"This will actually write two files (as this is what `ceci` wants)\n",
"\n",
Expand All @@ -305,7 +309,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Read the saved pipeline"
"## Read the saved pipeline"
]
},
{
Expand All @@ -321,9 +325,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run the newly read pipeline\n",
"## Run the newly read pipeline\n",
"\n",
"This will actually launch Unix process to individually run each stage of the pipeline; you can see the commands that are being executed in each case."
"This will actually launch a Unix process to individually run each stage of the pipeline; you can see the commands that are being executed in each case."
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
"id": "8cab252c",
"metadata": {},
"source": [
"# Iterator Test\n",
"# Iterate Over Tabular Data\n",
"\n",
"author: Eric Charles\n",
"**Author:** Eric Charles\n",
"\n",
"last run successfully: April 26, 2022"
"**Last Run Successfully:** April 26, 2022"
]
},
{
Expand All @@ -18,7 +18,7 @@
"metadata": {},
"source": [
"\n",
"This notebook demonstrates three ways to iterate over tabular data\n",
"This notebook demonstrates three ways to iterate over tabular data:\n",
"\n",
"1. Using the `tables_io.iteratorNative` function\n",
"\n",
Expand Down Expand Up @@ -47,9 +47,9 @@
"id": "40332abb",
"metadata": {},
"source": [
"Get access to the rail DataStore, and set it to allow use to overwrite data.\n",
"Get access to the RAIL DataStore, and set it to allow us to overwrite data.\n",
"\n",
"Allowing overwrites will prevent errors when re-running cells in the notebook"
"Allowing overwrites will prevent errors when re-running cells in the notebook."
]
},
{
Expand Down Expand Up @@ -117,7 +117,7 @@
"id": "5728e08a",
"metadata": {},
"source": [
"## Iterate using the tables_io.iteratorNative function\n",
"## tables_io.iteratorNative function\n",
"\n",
"This will open the HDF5 file, and iterate over the file, returning chunks of data"
]
Expand All @@ -141,7 +141,7 @@
"id": "d3dfcb10",
"metadata": {},
"source": [
"## Iterate using the rail.core.data.TableHandle data handle object\n",
"## rail.core.data.TableHandle data handle object\n",
"\n",
"This will create a TableHandle object that points to the correct file, which can be use to iterate over that file."
]
Expand All @@ -165,7 +165,7 @@
"id": "b251b43e",
"metadata": {},
"source": [
"## Iterator using the rail.core.stage.RailStage functionality\n",
"## rail.core.stage.RailStage functionality\n",
"\n",
"This will create a RailStage pipeline stage, which takes as input an HDF5 file, \n",
"so the `input_iterator` function can be used to iterate over that file."
Expand Down
92 changes: 0 additions & 92 deletions examples/core_examples/Run_Pipe.ipynb

This file was deleted.

107 changes: 107 additions & 0 deletions examples/core_examples/Run_Saved_Pipeline.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ec47ab0a",
"metadata": {},
"source": [
"# Run a Saved Pipeline\n",
"\n",
"**Author:** Eric Charles\n",
"\n",
"**Last Run Successfully:** April 26, 2023"
]
},
{
"cell_type": "markdown",
"id": "ff1c1990",
"metadata": {},
"source": [
" \n",
"This notebook shows how to:\n",
"\n",
"1. Load a pipeline from a saved yaml file\n",
"\n",
"2. Run the loaded pipeline"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a29a1b21",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import ceci\n",
"from rail.core.utils import find_rail_file\n",
"\n",
"# To create a catalog, you need a model of what the distrubutions of the colors \n",
"# are--that's what this flow file is:\n",
"flow_file = find_rail_file('examples_data/goldenspike_data/data/pretrained_flow.pkl')\n",
"os.environ['FLOWDIR'] = os.path.dirname(flow_file)"
]
},
{
"cell_type": "markdown",
"id": "deaaff62",
"metadata": {},
"source": [
"Each pipeline file has an associated config file. Whenever ceci reads in a pipeline file called `[name].yml`, it will automatically look for a configuration file `[name]_config.yml` in the same directory.\n",
"\n",
"Here, we read in our `pipe_example.yml` pipeline, which is associated with `pipe_example.config.yml`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2657aca0",
"metadata": {},
"outputs": [],
"source": [
"\n",
"p = ceci.Pipeline.read('pipe_example.yml')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "38e445de",
"metadata": {},
"outputs": [],
"source": [
"p.run()"
]
},
{
"cell_type": "markdown",
"id": "cb8a7120",
"metadata": {},
"source": [
"Yep, that's it."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

0 comments on commit f220f17

Please sign in to comment.