Skip to content

Commit

Permalink
update hello-world example
Browse files Browse the repository at this point in the history
  • Loading branch information
@donaldcampbelljr
donaldcampbelljr committed Nov 8, 2023
1 parent 24c9079 commit 59e0d27
Showing 1 changed file with 130 additions and 38 deletions.
168 changes: 130 additions & 38 deletions docs_jupyter/hello-world.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,57 +21,79 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2020-05-21 08:23:43-- https://github.com/pepkit/hello_looper/archive/master.zip\n",
"Resolving github.com (github.com)... 140.82.112.4\n",
"Connecting to github.com (github.com)|140.82.112.4|:443... connected.\n",
"--2023-11-08 17:27:01-- https://github.com/pepkit/hello_looper/archive/refs/heads/master.zip\n",
"Resolving github.com (github.com)... 140.82.114.3\n",
"Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
"HTTP request sent, awaiting response... 302 Found\n",
"Location: https://codeload.github.com/pepkit/hello_looper/zip/master [following]\n",
"--2020-05-21 08:23:43-- https://codeload.github.com/pepkit/hello_looper/zip/master\n",
"Resolving codeload.github.com (codeload.github.com)... 140.82.114.10\n",
"Connecting to codeload.github.com (codeload.github.com)|140.82.114.10|:443... connected.\n",
"Location: https://codeload.github.com/pepkit/hello_looper/zip/refs/heads/master [following]\n",
"--2023-11-08 17:27:01-- https://codeload.github.com/pepkit/hello_looper/zip/refs/heads/master\n",
"Resolving codeload.github.com (codeload.github.com)... 140.82.113.10\n",
"Connecting to codeload.github.com (codeload.github.com)|140.82.113.10|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: unspecified [application/zip]\n",
"Saving to: ‘master.zip’\n",
"\n",
"master.zip [ <=> ] 5.20K --.-KB/s in 0.004s \n",
"master.zip [ <=> ] 13.37K --.-KB/s in 0.03s \n",
"\n",
"2020-05-21 08:23:44 (1.25 MB/s) - ‘master.zip’ saved [5328]\n",
"2023-11-08 17:27:01 (472 KB/s) - ‘master.zip’ saved [13693]\n",
"\n"
]
}
],
"source": [
"!wget https://github.com/pepkit/hello_looper/archive/master.zip"
"!wget https://github.com/pepkit/hello_looper/archive/refs/heads/master.zip"
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Archive: master.zip\r\n",
"c8c4088d6e14df05071fb99809dfc86b2a55d86a\r\n",
"73ef08e38d3e17fd3d4f940282c80e3ee4dbb91f\r\n",
" creating: hello_looper-master/\r\n",
" inflating: hello_looper-master/.gitignore \r\n",
" inflating: hello_looper-master/.looper.yaml \r\n",
" inflating: hello_looper-master/.looper_pephub.yaml \r\n",
" inflating: hello_looper-master/.looper_pipestat.yaml \r\n",
" inflating: hello_looper-master/.looper_pipestat_shell.yaml \r\n",
" inflating: hello_looper-master/README.md \r\n",
" creating: hello_looper-master/data/\r\n",
" inflating: hello_looper-master/data/frog1_data.txt \r\n",
" inflating: hello_looper-master/data/frog2_data.txt \r\n",
" inflating: hello_looper-master/looper_pipelines.md \r\n",
" inflating: hello_looper-master/output.txt \r\n",
" creating: hello_looper-master/old_specification/\r\n",
" inflating: hello_looper-master/old_specification/README.md \r\n",
" creating: hello_looper-master/old_specification/data/\r\n",
" inflating: hello_looper-master/old_specification/data/frog1_data.txt \r\n",
" inflating: hello_looper-master/old_specification/data/frog2_data.txt \r\n",
" creating: hello_looper-master/old_specification/pipeline/\r\n",
" inflating: hello_looper-master/old_specification/pipeline/count_lines.sh \r\n",
" inflating: hello_looper-master/old_specification/pipeline/pipeline_interface.yaml \r\n",
" creating: hello_looper-master/old_specification/project/\r\n",
" inflating: hello_looper-master/old_specification/project/project_config.yaml \r\n",
" inflating: hello_looper-master/old_specification/project/sample_annotation.csv \r\n",
" creating: hello_looper-master/pipeline/\r\n",
" inflating: hello_looper-master/pipeline/count_lines.sh \r\n",
" inflating: hello_looper-master/pipeline/pipeline_interface.yaml \r\n",
" inflating: hello_looper-master/pipeline/pipeline_interface_project.yaml \r\n",
" creating: hello_looper-master/pipeline_pipestat/\r\n",
" inflating: hello_looper-master/pipeline_pipestat/count_lines.py \r\n",
" inflating: hello_looper-master/pipeline_pipestat/count_lines_pipestat.sh \r\n",
" inflating: hello_looper-master/pipeline_pipestat/pipeline_interface.yaml \r\n",
" inflating: hello_looper-master/pipeline_pipestat/pipeline_interface_shell.yaml \r\n",
" inflating: hello_looper-master/pipeline_pipestat/pipestat_output_schema.yaml \r\n",
" creating: hello_looper-master/project/\r\n",
" inflating: hello_looper-master/project/project_config.yaml \r\n",
" inflating: hello_looper-master/project/sample_annotation.csv \r\n"
Expand All @@ -93,34 +115,45 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Looper version: 1.2.0-dev\r\n",
"Looper version: 1.5.2-dev\r\n",
"Command: run\r\n",
"Ignoring invalid pipeline interface source: ../pipeline/pipeline_interface.yaml. Caught exception: FileNotFoundError(2, 'No such file or directory')\r\n",
"> Not submitted: No pipeline interfaces defined\r\n",
"> Not submitted: No pipeline interfaces defined\r\n",
"Using default divvy config. You may specify in env var: ['DIVCFG']\r\n",
"Pipestat compatible: False\r\n",
"\u001b[36m## [1 of 2] sample: frog_1; pipeline: count_lines\u001b[0m\r\n",
"/home/drc/GITHUB/looper/master/looper/docs_jupyter/hello_looper-master/pipeline/count_lines.sh data/frog1_data.txt\r\n",
"Writing script to /home/drc/GITHUB/looper/master/looper/docs_jupyter/hello_looper-master/results/submission/count_lines_frog_1.sub\r\n",
"Job script (n=1; 0.00Gb): /home/drc/GITHUB/looper/master/looper/docs_jupyter/hello_looper-master/results/submission/count_lines_frog_1.sub\r\n",
"Compute node: databio\r\n",
"Start time: 2023-11-08 17:29:45\r\n",
"wc: data/frog1_data.txt: No such file or directory\r\n",
"Number of lines: \r\n",
"\u001b[36m## [2 of 2] sample: frog_2; pipeline: count_lines\u001b[0m\r\n",
"/home/drc/GITHUB/looper/master/looper/docs_jupyter/hello_looper-master/pipeline/count_lines.sh data/frog2_data.txt\r\n",
"Writing script to /home/drc/GITHUB/looper/master/looper/docs_jupyter/hello_looper-master/results/submission/count_lines_frog_2.sub\r\n",
"Job script (n=1; 0.00Gb): /home/drc/GITHUB/looper/master/looper/docs_jupyter/hello_looper-master/results/submission/count_lines_frog_2.sub\r\n",
"Compute node: databio\r\n",
"Start time: 2023-11-08 17:29:45\r\n",
"wc: data/frog2_data.txt: No such file or directory\r\n",
"Number of lines: \r\n",
"\r\n",
"Looper finished\r\n",
"Samples valid for job generation: 0 of 2\r\n",
"Commands submitted: 0 of 0\r\n",
"Jobs submitted: 0\r\n",
"\r\n",
"1 unique reasons for submission failure: No pipeline interfaces defined\r\n",
"\r\n",
"Summary of failures:\r\n",
"\u001B[91mNo pipeline interfaces defined\u001B[0m: frog_2, frog_1\r\n",
"\u001B[0m"
"Samples valid for job generation: 2 of 2\r\n",
"Commands submitted: 2 of 2\r\n",
"Jobs submitted: 2\r\n",
"{'Pipestat compatible': False, 'Commands submitted': '2 of 2', 'Jobs submitted': 2}\r\n",
"\u001b[0m"
]
}
],
"source": [
"!looper run hello_looper-master/project/project_config.yaml"
"!looper run --looper-config hello_looper-master/.looper.yaml"
]
},
{
Expand Down Expand Up @@ -196,33 +229,92 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Looper config\n",
"\n",
"The [looper config](looper-config.md) contains paths to the project config, the output_dir as well as any dfine pipeline interfaces. "
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"pep_config: project/project_config.yaml # local path to pep config\r\n",
"# pep_config: pepkit/hello_looper:default # you can also use a pephub registry path\r\n",
"output_dir: \"results\"\r\n",
"pipeline_interfaces:\r\n",
" sample: pipeline/pipeline_interface.yaml\r\n"
]
}
],
"source": [
"!cat hello_looper-master/.looper.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Pipeline outputs\n",
"\n",
"Outputs of pipeline runs will be under the directory specified in the `output_dir` variable under the `paths` section in the project config file (see [defining a project](defining-a-project.md)). Let's inspect that `project_config.yaml` file to see what it says under `output_dir`:\n"
"## Project Config\n",
"\n",
"The project config file contains the PEP version and sample annotation sheet. (see [defining a project](defining-a-project.md)).\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"metadata:\r\n",
" sample_annotation: sample_annotation.csv\r\n",
" output_dir: $HOME/hello_looper_results\r\n",
" pipeline_interfaces: ../pipeline/pipeline_interface.yaml\r\n"
"pep_version: 2.0.0\r\n",
"sample_table: sample_annotation.csv\r\n"
]
}
],
"source": [
"!cat hello_looper-master/project/project_config.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Pipeline Interface\n",
"\n",
"The [pipeline interface](pipeline-interface-specification.md) shows the pipeline_name, pipeline_type, as well as the var_templates and command_templates used for this pipeline.\n"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"pipeline_name: count_lines\r\n",
"pipeline_type: sample\r\n",
"var_templates:\r\n",
" pipeline: '{looper.piface_dir}/count_lines.sh'\r\n",
"command_template: >\r\n",
" {pipeline.var_templates.pipeline} {sample.file}\r\n"
]
}
],
"source": [
"!cat hello_looper-master/pipeline/pipeline_interface.yaml"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -305,7 +397,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -319,7 +411,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5"
"version": "3.10.12"
}
},
"nbformat": 4,
Expand Down

0 comments on commit 59e0d27

Please sign in to comment.