Revise dev-oriented onboarding instructions

[aimalz]

#16 is about user-facing instructions. I'm using this issue to centralize developer-facing onboarding documentation.

Suggestions from the retreat (list will be edited continuously):

• rail/docs/source/overview.rst should include an introduction to stages and pipelines (also remove outdated content there and to check that the descriptions of creation/estimation/evaluation are accurate) PR Updating overview, added section about Ceci pipelines and stages. #74
• mamba not conda so it's simpler
• rail --help, rail-nb --help [path] (suggest running on some notebooks to confirm it works)
• explain why the instructions are what they are (pip vs. conda)
• explain the command-line tool (purpose, what this gives you) rather than just jumping into the commands
• change pip install to pip install -e .
• instructions to get around NERSC ceci path issue
• install local kernel to notebook server instructions to introduce rail-nb command (insert this link)
• add a link to NERSC-specific instructions
• add a link to a git cheat sheet Addressing assorted items in issue 51 #106

[ztq1996]

Some notes I took for running rail pipeline on NERSC

Install rail:

module load python
git clone https://github.com/LSSTDESC/rail.git
cd rail
mamba env create -f environment.yml -n [env] # or mamba env create, which is much faster
conda activate [env]
pip install .
rail clone-source --package-file rail_packages.yml
rail install --package-file rail_packages.yml --from-source

add the rail kernel to jupyter:

conda activate [env] # . ./cenv/bin/activate in case of virtualenv
([env])$ conda install ipykernel
([env])$ ipython kernel install --user --name=<any_name_for_kernel>
([env])$ conda deactivate

If you run into issue with ceci or pipeline.run() in jupyterlab, it is a path issue with the jupyter.

you need to ssh into ssh username@perlmutter.nersc.gov , and run

module load python
conda activate [env]
Open your your_rail_pipeline.yml, add “name: local” under the site section. Then run
ceci your_rail_pipeline.yml

[OliviaLynn]

Additional changes we may want to make:

• separate mac-specific instructions
• Add an additional “check that the installation worked” section (as this is applicable after following any of the installation instruction sections) about running the two lines (wrote this in the RR - what are the two lines?)
  • the notebooks section includes an example, which, for now, feels like enough. would be nice to have an equivalent for other sections as well but not a priority
• Add line: struggling to figure out what repo some code is from? these ides will let you command + click to jump around repos: (list IDEs)
• in Installation page "RAIL packages > Installing with pip" and "RAIL packages > Installing from source" seem redundant (probably leftovers?)

More abstract:

• Something explaining the philosophy of rail, eg, “the usual way you pass things around in rail is handles, not the data itself” with a little explanation that yeah, you might be using toy data in notebooks, but this is built to handle zuntz level pipelines as well.

[drewoldag]

I've created a PR that addresses all but two of the requests in the first comment. If I could get some help from someone about what to include for the two NERSC-related requests, I would really appreciate it. I'm not familiar with what they are referring to.