From d02d49bc99b435951db054e41bf2e6efdad9b66f Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 20 Jun 2017 18:39:51 -0400 Subject: [PATCH 01/14] Reorganize homepage Install/Project info sections - Move links downwards to not distract from installation docs. - Separate "Installation" from "Project info" sections so that release notes and such don't distract from how-to pages for installing the stack. --- index.rst | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/index.rst b/index.rst index 27b99169..f12dca3e 100644 --- a/index.rst +++ b/index.rst @@ -5,30 +5,46 @@ The LSST Science Pipelines The LSST Science Pipelines enable optical and near-infrared astronomy in the big data era. We are building the Science Pipelines for the `Large Synoptic Survey Telescope (LSST) `_, but our command line task and Python API can be extended for any optical or near-infrared dataset. -- Join us on the `LSST Community forum, community.lsst.org `_. -- Fork our code on GitHub at https://github.com/lsst. -- Report issues in `JIRA `_. -- API documentation is currently published with `Doxygen `_. -- DM Developer guidance is at https://developer.lsst.io. -- Learn more about LSST Data Management by visiting http://dm.lsst.org. -- Help us improve our documentation! This guide is on GitHub at `lsst/pipelines_lsst_io `_. +The latest release is |current-release|: :doc:`learn what's new `. -Latest release -============== +.. _part-installation: -The latest release is |current-release|: :doc:`learn what's new `. +Installation +============ + +- :doc:`Overview of installation methods ` +- :doc:`Prerequisites ` +- :doc:`Installing with newinstall.sh ` (recommended) +- :doc:`Installing from source with lsstsw ` (for developers) .. toctree:: :hidden: + :caption: Installation - releases/index + install/index + +.. _part-release-details: .. toctree:: :maxdepth: 1 - :caption: Getting Started - :name: part-getting-started + :caption: Release details - install/index - known-issues releases/notes + known-issues metrics/index + +.. toctree:: + :hidden: + + releases/index + +More info +========= + +- Join us on the `LSST Community forum, community.lsst.org `_. +- Fork our code on GitHub at https://github.com/lsst. +- Report issues in `JIRA `_. +- API documentation is currently published with `Doxygen `_. +- DM Developer guidance is at https://developer.lsst.io. +- Learn more about LSST Data Management by visiting http://dm.lsst.org. +- Contribute to our documentation. This guide is on GitHub at `lsst/pipelines_lsst_io `_. From 7e50c68991f6eff5e8e17156b6bcc6fa78f16d7e Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 20 Jun 2017 18:39:51 -0400 Subject: [PATCH 02/14] Refactor prerequisite docs from newinstall docs This makes it easier to document pre-requisites from either the lsstsw or newinstall installation methods. Overall it makes the newinstall docs easier to follow, as well, making it possible to use the newinstall docs from a getting started context. --- install/index.rst | 3 +- install/lsstsw.rst | 3 +- install/newinstall.rst | 170 ++++--------------------------------- install/prereqs/centos.rst | 50 +++++++++++ install/prereqs/debian.rst | 24 ++++++ install/prereqs/index.rst | 38 +++++++++ install/prereqs/macos.rst | 49 +++++++++++ 7 files changed, 181 insertions(+), 156 deletions(-) create mode 100644 install/prereqs/centos.rst create mode 100644 install/prereqs/debian.rst create mode 100644 install/prereqs/index.rst create mode 100644 install/prereqs/macos.rst diff --git a/install/index.rst b/install/index.rst index 9f96caf0..97cbde14 100644 --- a/install/index.rst +++ b/install/index.rst @@ -4,7 +4,8 @@ Installing the LSST Science Pipelines .. toctree:: :hidden: - + + prereqs/index conda newinstall lsstsw diff --git a/install/lsstsw.rst b/install/lsstsw.rst index 256fbb75..c4328b8a 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -21,8 +21,9 @@ If you have difficulty installing LSST software: You need to install some prerequisites to build the LSST Stack from source: -- :ref:`Install build dependencies `. - `Install and configure Git LFS `_ for LSST DM's data servers. +- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. +- If you opt not to use ``lsstsw`` \’s default Python environment you need to :ref:`install these Python dependencies `. .. _lsstsw-deploy: diff --git a/install/newinstall.rst b/install/newinstall.rst index 1796ec6c..ed10278b 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -15,154 +15,16 @@ If you have difficulty installing LSST software: .. _source-install-prereqs: -Prerequisites -============= +1. Install prerequisites +======================== -This section lists system prerequisites for :ref:`macOS `, :ref:`Debian/Ubuntu `, and :ref:`RedHat/CentOS ` platforms. -All platforms also need :ref:`Python package dependencies ` listed here. - -.. note:: - - **New since 11.0**: The minimum gcc version required to compile the Stack is **gcc 4.8.** - If you using our previous factory platform, RedHat/CentOS 6, and you are unable to upgrade to version 7 (which comes with gcc 4.8 as default) consult :ref:`the section below on upgrading compilers in legacy Linux `. - -.. _source-install-mac-prereqs: - -macOS ------ - -To build LSST software, macOS systems need: - -1. :ref:`Xcode `, or command line tools. -2. :ref:`cmake `. - -Versions prior to OS X 10.9 and earlier have not been tested recently and may not work. - -.. _source-install-mac-prereqs-xcode: - -Xcode -^^^^^ - -You will need to install developer tools, which we recommend you obtain with Apple's Xcode command line tools package. -To do this, run from the command line (e.g. ``Terminal.app`` or similar): - -.. code-block:: bash - - xcode-select --install - -and follow the on-screen instructions. -You can verify where the tools are installed by running: - -.. code-block:: bash - - xcode-select -p - -.. _source-install-mac-prereqs-cmake: - -cmake -^^^^^ - -``cmake`` can be `installed directly `__, or though a package manager like `Homebrew `__. - -.. _source-install-debian-prereqs: - -Debian / Ubuntu ---------------- - -.. code-block:: bash - - apt-get install bison ca-certificates \ - cmake flex g++ gettext git libbz2-dev \ - libfontconfig1 libglib2.0-dev libncurses5-dev \ - libreadline6-dev libssl-dev libx11-dev libxrender1 \ - libxt-dev m4 openjdk-8-jre \ - perl-modules zlib1g-dev \ - - -.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp - -Prefix the :command:`apt-get` command with :command:`sudo` if necessary. - -.. _source-install-redhat-prereqs: - -RedHat / CentOS ---------------- - -.. code-block:: bash - - yum install bison curl blas bzip2-devel bzip2 flex fontconfig \ - freetype-devel gcc-c++ gcc-gfortran git libuuid-devel \ - libXext libXrender libXt-devel make openssl-devel patch perl \ - readline-devel tar zlib-devel ncurses-devel cmake glib2-devel \ - java-1.8.0-openjdk gettext perl-ExtUtils-MakeMaker - -.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp - -Prefix the :command:`yum` command with :command:`sudo` if necessary. - -.. _source-install-redhat-legacy: - -Upgrading compilers for legacy RedHat / CentOS 6 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The minimum gcc version required to compile the Stack is gcc 4.8. -This comes as standard in the LSST "factory" platform, Red Hat / CentOS 7. - -On our previous factory platform, Red Hat / CentOS 6, you will need to use a more current version of gcc that what is available with your system. -If you can go to Red Hat 7, we recommend that you do; if you cannot, we recommend that you use a newer gcc version for the stack by using a Software Collection (SCL) with a different version of devtoolset. -This will enable you to safely use a different version of gcc (4.9) for the stack than that used by your operating system (4.4). - -First, install ``devtoolset-3`` (after the :ref:`installing the standard pre-requisites (above) `): - -.. code-block:: bash - - sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm - sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/rh-java-common/epel-6-x86_64/download/rhscl-rh-java-common-epel-6-x86_64.noarch.rpm - sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/devtoolset-3/epel-6-x86_64/download/rhscl-devtoolset-3-epel-6-x86_64.noarch.rpm - sudo yum install -y scl-utils - sudo yum install -y devtoolset-3 - -Then enable ``devtoolset-3`` by including this line in your :file:`~/.bash_profile`: - -.. code-block:: bash - - scl enable devtoolset-3 bash - -.. _source-install-py-deps: - -Python dependencies -------------------- - -You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. - -.. _source-install-optional-deps: - -Optional dependencies ---------------------- - -Although not required, we recommend you install the `matplotlib `_ and `scipy `_ Python packages: - -.. code-block:: bash - - pip install -U matplotlib scipy - -.. FIXME - -Note these are included by default in `Anaconda `__, which :command:`newinstall.sh` *can* obtain for you. - -We also use `SAOImage DS9 `_ to display images for debugging. - -.. _install-from-source: - -Installing from Source with newinstall.sh -========================================= - -This section will guide you through installing the *current* release of the LSST Science Pipelines from source given that prerequisites have been installed. +- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. +- If you opt not to use :command:`newinstall` \’s default Python environment you need to :ref:`install these Python dependencies `. .. _install-from-source-dir: -1. Choose an installation directory ------------------------------------ +2. Choose an installation directory +=================================== First, choose where you want to install the LSST Science Pipelines. We'll use :file:`$HOME/lsst_stack` in this example. @@ -182,8 +44,8 @@ Individual users who install a personal Stack on their own machine need not worr .. _install-from-source-envvar: -2. Unset environment variables ------------------------------- +3. Unset environment variables +============================== If you've run the LSST Science Pipelines previously, you may have conflicting environment variables setup. To be safe, run: @@ -194,8 +56,8 @@ To be safe, run: .. _install-from-source-setup: -3. Installation set-up ----------------------- +4. Installation set-up +====================== Download and run the `installation setup script from GitHub `__, which installs the basic packages required to install other packages: @@ -228,8 +90,8 @@ where :file:`$LSST_INSTALL_DIR` is expanded to your installation directory. .. _install-from-source-packages: -4. Install packages -------------------- +5. Install lsst_distrib +======================= Finally, install components of the LSST Science Pipelines that are relevant for your work. A simple way to ensure that you have a fairly complete set of packages for this need is to install ``lsst_distrib``: @@ -244,8 +106,8 @@ See :ref:`source-install-testing-your-installation`. .. _install-from-source-loadlsst: -5. Source the LSST environment in each shell session ----------------------------------------------------- +6. Source the LSST environment in each shell session +==================================================== Whenever you want to run the installed LSST Science Pipelines in a new terminal session, be sure to :command:`source` the appropriate :file:`loadLSST.bash`, :file:`loadLSST.csh`, :file:`loadLSST.ksh` or :file:`loadLSST.zsh}` script. @@ -257,8 +119,8 @@ Then setup the EUPS packages you need, typically: .. _source-install-testing-your-installation: -Testing Your Installation -========================= +7. Testing Your Installation +============================ Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. This demo processes a small amount of SDSS data. diff --git a/install/prereqs/centos.rst b/install/prereqs/centos.rst new file mode 100644 index 00000000..950bcc7c --- /dev/null +++ b/install/prereqs/centos.rst @@ -0,0 +1,50 @@ +.. _source-install-redhat-prereqs: + +#################################### +RedHat / CentOS system prerequisites +#################################### + +.. code-block:: bash + + yum install bison curl blas bzip2-devel bzip2 flex fontconfig \ + freetype-devel gcc-c++ gcc-gfortran git libuuid-devel \ + libXext libXrender libXt-devel make openssl-devel patch perl \ + readline-devel tar zlib-devel ncurses-devel cmake glib2-devel \ + java-1.8.0-openjdk gettext perl-ExtUtils-MakeMaker + +.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp + +Prefix the :command:`yum` command with :command:`sudo` if necessary. + +.. note:: + + **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.** + If you using our previous factory platform, RedHat/CentOS 6, and you are unable to upgrade to version 7 (which comes with gcc 4.8 as default) consult :ref:`the section below on upgrading compilers in legacy Linux `. + +.. _source-install-redhat-legacy: + +Upgrading compilers for legacy RedHat / CentOS 6 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The minimum gcc version required to compile the Stack is gcc 4.8. +This comes as standard in the LSST "factory" platform, Red Hat / CentOS 7. + +On our previous factory platform, Red Hat / CentOS 6, you will need to use a more current version of gcc that what is available with your system. +If you can go to Red Hat 7, we recommend that you do; if you cannot, we recommend that you use a newer gcc version for the stack by using a Software Collection (SCL) with a different version of devtoolset. +This will enable you to safely use a different version of gcc (4.9) for the stack than that used by your operating system (4.4). + +First, install ``devtoolset-3`` (after the :ref:`installing the standard pre-requisites (above) `): + +.. code-block:: bash + + sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm + sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/rh-java-common/epel-6-x86_64/download/rhscl-rh-java-common-epel-6-x86_64.noarch.rpm + sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/devtoolset-3/epel-6-x86_64/download/rhscl-devtoolset-3-epel-6-x86_64.noarch.rpm + sudo yum install -y scl-utils + sudo yum install -y devtoolset-3 + +Then enable ``devtoolset-3`` by including this line in your :file:`~/.bash_profile`: + +.. code-block:: bash + + scl enable devtoolset-3 bash diff --git a/install/prereqs/debian.rst b/install/prereqs/debian.rst new file mode 100644 index 00000000..dbc39b67 --- /dev/null +++ b/install/prereqs/debian.rst @@ -0,0 +1,24 @@ +.. _source-install-debian-prereqs: + +#################################### +Debian / Ubuntu system prerequisites +#################################### + +Debian or Ubuntu systems require the following packages: + +.. code-block:: bash + + apt-get install bison ca-certificates \ + cmake flex g++ gettext git libbz2-dev \ + libfontconfig1 libglib2.0-dev libncurses5-dev \ + libreadline6-dev libssl-dev libx11-dev libxrender1 \ + libxt-dev m4 openjdk-8-jre \ + perl-modules zlib1g-dev \ + +.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp + +Prefix the :command:`apt-get` command with :command:`sudo` if necessary. + +.. note:: + + **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.** diff --git a/install/prereqs/index.rst b/install/prereqs/index.rst new file mode 100644 index 00000000..b0f2fcba --- /dev/null +++ b/install/prereqs/index.rst @@ -0,0 +1,38 @@ +############# +Prerequisites +############# + +This page lists software needed to install and use the LSST Science Pipelines. + +.. _system-prereqs: + +System prerequisites +==================== + +.. toctree:: + :maxdepth: 1 + + macos + debian + centos + +.. _python-deps: + +Python dependencies +=================== + +You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. + +If you opt to use your own Python, you can re-create the default Python environment made by :command:`newinstall.sh` and ``lsstsw`` with these Conda environments: + +- `macOS and Python 2.7 `_. +- `macOS and Python 3.5+ `_. +- `Linux and Python 2.7 `_. +- `Linux and Python 3.5+ `_. + +.. _optional-deps: + +Optional dependencies +===================== + +We use `SAOImage DS9 `_ to display images for debugging. diff --git a/install/prereqs/macos.rst b/install/prereqs/macos.rst new file mode 100644 index 00000000..89471b7c --- /dev/null +++ b/install/prereqs/macos.rst @@ -0,0 +1,49 @@ +.. _source-install-mac-prereqs: + +########################## +macOS system prerequisites +########################## + +To build LSST software, macOS systems need: + +1. :ref:`Xcode `, or command line tools. +2. :ref:`cmake `. + +.. note:: + + macOS versions 10.9 and earlier have not been tested recently and may not work. + +.. _source-install-mac-prereqs-xcode: + +Xcode +===== + +You will need to install developer tools, which we recommend you obtain with Apple's Xcode command line tools package. +To do this, run from the command line (e.g. ``Terminal.app`` or similar): + +.. code-block:: bash + + xcode-select --install + +You can verify where the tools are installed by running: + +.. code-block:: bash + + xcode-select -p + +.. _source-install-mac-prereqs-cmake: + +cmake +===== + +``cmake`` can be installed through a package manager like `Homebrew `_: + +.. code-block:: bash + + brew install cmake + +**Alternatively,** `Anaconda `_ users can install ``cmake`` with :command:`conda`: + +.. code-block:: bash + + conda install cmake From 4ec81671ee8f1eaa7a3d166ac0364c04b03a2662 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 12 Sep 2017 15:45:33 -0700 Subject: [PATCH 03/14] Remove conda installation docs A conda distribution is no longer provided with v14 of the stack. --- install/conda.rst | 113 --------------------------------------------- install/demo.rst | 3 +- install/index.rst | 5 -- releases/v12_0.rst | 2 +- 4 files changed, 2 insertions(+), 121 deletions(-) delete mode 100644 install/conda.rst diff --git a/install/conda.rst b/install/conda.rst deleted file mode 100644 index 4c651a1d..00000000 --- a/install/conda.rst +++ /dev/null @@ -1,113 +0,0 @@ -############################################################## -Installing and Using LSST Science Pipelines as a Conda Package -############################################################## - -This page will guide you through installing the LSST Science Pipelines as a Conda package for use in `Anaconda or Miniconda `__. -Anaconda is a popular Python distribution and package ecosystem. -With this installation method you don't compile source code, or even need to have an existing Python installation. - -.. note:: - - This documentation does not cover the LSST Simulations software. - See the `LSST Simulations installation documentation `_ instead. - -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. - -1. Install Anaconda or Miniconda -================================ - -You might already have Anaconda or Miniconda as your main Python distribution. -If not, you can quickly get started by following Continuum's official installation instructions: - -- `Install Anaconda `__ if you want a complete, science-ready Python installation with minimal setup. -- Otherwise, `install the leaner Miniconda `__ version and install packages as you need them. - -.. warning:: - - You need the Python 2 (e.g. Miniconda2) distribution. If you want to use the experimental Python 3 support, you should try our source installation. - -If you're new to Anaconda, Continuum's `30-minute test drive `_ tutorial will get you up-to-speed. - -.. warning:: - - Don't reuse the Miniconda you might have previously obtained from an :doc:`eups distrib `\ -based installation. - Open a new shell and install Anaconda/Miniconda from scratch. - -Upgrading conda ---------------- - -If you have an existing Anaconda or Miniconda installation, you'll want to make sure the :command:`conda` command itself is up to date: - -.. code-block:: bash - - conda update conda - -See the `Conda documentation for more information about installing and managing conda and Anaconda/Miniconda `__. - -2. Install Science Pipelines in a Conda Environment -=================================================== - -These commands will download and activate the LSST Science Pipelines in a new Conda environment: - -.. code-block:: bash - :linenos: - - conda config --add channels http://conda.lsst.codes/stack/0.13.0 - conda create --name lsst python=2 - source activate lsst - conda install lsst-distrib - source eups-setups.sh - setup lsst_distrib - - You can ignore warnings about pre-link and post-link scripts being dangerous and deprecated. Sorry about that. - -.. warning:: - - Installing the LSST simulation tools (including MAF) requires pointing to a different Conda channel. - See the `LSST Simulations installation documentation `_. - -Here's what these commands are doing, line-by-line: - -1. Tell :command:`conda` about the LSST channel for Conda packages. -2. Create a Conda environment called ``lsst`` with Python 2.7. - You can change the environment's name to anything you like. - Conda environments are recommended since they help keep the Python dependencies of your projects separate. - See the `Conda documentation on environments for more information `__. -3. Activate the ``lsst`` environment (use your environment's name if you chose a different one). - The :command:`activate` command is provided by Anaconda/Miniconda (e.g. at :file:`~/miniconda2/bin/activate`). -4. Install the full suite of LSST science software, including Science Pipelines (``lsst-distrib``). -5. Setup EUPS, LSST's package manager. -6. Setup LSST packages in your environment with EUPS (setting up ``lsst_distrib`` makes most packages available to you). - -.. warning:: - - If the install fails with an error, check that your shell does not have another EUPS Stack configured (try ``echo $EUPS_STACK``). - Conda packaged EUPS will use existing values of ``EUPS_PATH`` and ``EUPS_DIR``. - If they exist, unset them before installing or using Conda packages. - -.. _conda-install-activate: - -3. Activating Science Pipelines in a new Shell -============================================== - -Whenever you open a new shell or terminal session, use these commands to re-activate your previously-installed Science Pipelines: - -.. code-block:: bash - :linenos: - - source activate lsst - source eups-setups.sh - setup lsst_distrib - -These commands can also be used to switch from one Conda environment and LSST Science Pipelines installation to another. - -.. _conda-install-test: - -4. Testing Your Installation -============================ - -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. diff --git a/install/demo.rst b/install/demo.rst index c3f10c85..713c9604 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -10,8 +10,7 @@ This demo will allow you to quickly test your LSST Science Pipelines installatio Remember to first load the LSST Science Pipelines into your shell's environment. The method depends on how the Science Pipelines were installed: -- :ref:`Conda ` -- :ref:`newinstall.sh ` +- :doc:`newinstall.sh ` - :ref:`lsstsw ` 2. Download the Demo Project diff --git a/install/index.rst b/install/index.rst index 97cbde14..cfcd4263 100644 --- a/install/index.rst +++ b/install/index.rst @@ -6,7 +6,6 @@ Installing the LSST Science Pipelines :hidden: prereqs/index - conda newinstall lsstsw demo @@ -18,10 +17,6 @@ Choose an option below to get started. To install the LSST Simulation software, such as MAF, please follow the `LSST Simulations documentation `_ instead of these guides. -:doc:`Conda installation ` - Install the Pipelines as an `Anaconda/Miniconda `__ binary package. - This is the easiest installation option since no source compilation is required. - :doc:`Source installation (newinstall.sh) ` ``newinstall.sh`` allows you to build and install the LSST Science Pipelines from source. You can use your existing Python 2.7, or opt to use a built-in `Miniconda `__. diff --git a/releases/v12_0.rst b/releases/v12_0.rst index 4449ed88..154c665e 100644 --- a/releases/v12_0.rst +++ b/releases/v12_0.rst @@ -26,7 +26,7 @@ New: Install with Anaconda In addition to the customary source installation, this release features two binary installation methods: -- :doc:`Anaconda ` +- Anaconda - `CernVM FS `__ (contributed by Fabio Hernandez) If you are unsure which one to use, we recommend you try the conda binaries. From dcb0d8d36e063347a419110e2cb62b0fa10805aa Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 20 Jun 2017 18:39:51 -0400 Subject: [PATCH 04/14] Overhaul newinstall docs: simplify + v14 binaries - Simplify the newinstall.sh installation instructions to minimize commentary in the how-to steps. The idea is that this commentary and alternative installation strategies will be linked to an "advanced topics" section. - Update the newinstall.sh command so it uses binaries (-t). Also use (-c) to continue a failed build because Josh Hoblitt recommends it. - Point to the 14.0 release. - Adds sections on binaries, ABI compatibility, tag format, the Miniconda Python, and background/reference on newinstall.sh itself. --- install/newinstall.rst | 447 ++++++++++++++++++++++++++++++++++------- 1 file changed, 377 insertions(+), 70 deletions(-) diff --git a/install/newinstall.rst b/install/newinstall.rst index ed10278b..87083f7c 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -1,126 +1,433 @@ -###################################### -Source Installation with newinstall.sh -###################################### +########################################### +Install with newinstall.sh and eups distrib +########################################### -This page will guide you through installing the LSST Science Pipelines from source with :command:`newinstall.sh` (internally based on :command:`eups distrib`). +This page guides you through installing the LSST Science Pipelines software. +This installation method is recommended for anyone who uses or develops the Pipelines software. -The reference platform for the LSST Science Pipelines is CentOS 7 however individual developers compile on `a variety of Linux and macOS operating systems `_ so if you are on a similar platform you should be able to build from source successfully. +If you have installation issues, here are some ways to get help: -:doc:`We also offer Conda binaries and Docker images ` if you do not wish to install the Science Pipelines from source. +- Review the :ref:`known installation issues `. +- Ask a question in the `LSST Community support forum `_. -If you have difficulty installing LSST software: +.. _newinstall-prereqs: -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. +1. Prerequisites +================ -.. _source-install-prereqs: +You can install the LSST Science Pipelines on CentOS 7 (LSST's reference platform) as well as other Linux distributions and macOS (see the `LSST Stack Testing Status `_ for information on the platforms we have tested with). -1. Install prerequisites -======================== +Before you begin, install prerequisite software for your platform: -- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. -- If you opt not to use :command:`newinstall` \’s default Python environment you need to :ref:`install these Python dependencies `. +- :doc:`macOS ` +- :doc:`Centos / RedHat ` +- :doc:`Debian / Ubuntu ` -.. _install-from-source-dir: +.. + TK recommended memory, disk space, and build time. -2. Choose an installation directory -=================================== +.. _newinstall-source-dir: + +2. Make an installation directory +================================= First, choose where you want to install the LSST Science Pipelines. -We'll use :file:`$HOME/lsst_stack` in this example. Create and change into that directory: .. code-block:: bash - mkdir -p $HOME/lsst_stack - cd $HOME/lsst_stack + mkdir -p lsst_stack + cd lsst_stack + +If you are installing the software for multiple users (a shared stack), see :ref:`newinstall-shared-permissions`. + +.. _newinstall-run: + +3. Run newinstall.sh +==================== + +Open a new shell session for the installation (or ensure you haven't used LSST software in that shell before). +If you need to reuse a shell, see :ref:`newinstall-unset-variables`. + +Next, run :command:`newinstall.sh` to set up the environment you'll install the LSST Science Pipelines into. +For most use cases we recommend downloading and running :command:`newinstall.sh` like this: + +.. code-block:: bash + + curl -OL https://raw.githubusercontent.com/lsst/lsst/14.0/scripts/newinstall.sh + bash newinstall.sh -ct + +Always execute `newinstall.sh` with bash, as shown, regardless of what shell you are in. + +We recommend that you opt into the provided Miniconda Python environment (see the links below for more information). + +Then load this environment into your shell: + +.. TODO Use sphinx-tabs here? + +.. code-block:: bash + + source loadLSST.bash # for bash + source loadLSST.csh # for csh + source loadLSST.ksh # for ksh + source loadLSST.zsh # for zsh + +.. note:: + + Here are ways to customize the :command:`newinstall.sh` installation for specific needs: + + - The default Python environment is Python 3.5. + **Python 2.7 users,** see :ref:`newinstall-py2`. + - :ref:`newinstall-user-python`. + - The recommended installation uses precompiled binary tarballs if available for your platform (and falls back to a source build). + See :ref:`newinstall-binary-packages`. + If you will be compiling and linking C++ code against this installation you'll need to ensure your compilers match the distribution's. + **Developers should review** :ref:`newinstall-binary-compatibility`. + + For background information about :command:`newinstall.sh`, see: + + - :ref:`newinstall-background`. + - :ref:`newinstall-miniconda`. + - :ref:`newinstall-reference`. + +.. _newinstall-install: + +4. Install Science Pipelines packages +===================================== + +Install the LSST Science Pipelines packages by running :command:`eups distrib install` for a top-level package and a tagged version. + +This example installs the ``v14_0_rc1`` tagged version (current release) of the ``lsst_distrib`` top-level package: + +.. code-block:: bash + + eups distrib install lsst_distrib -t v14_0_rc1 lsst_distrib + curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python + setup lsst_distrib + +If pre-built binaries are available for your platform (and you ran :command:`newinstall.sh` with the :option:`-t ` argument) the installation should take roughly 10 minutes. +Otherwise, installation falls back to a source build which can take two hours, depending on the top-level package and your machine's performance. + +The last command, :command:`setup`, activates the installed packages in your shell environment. +You'll need to run :command:`setup` in each shell session you'll use the LSST Science Pipelines in. +See :doc:`setup` for more information. + +.. note:: -Installation for groups -^^^^^^^^^^^^^^^^^^^^^^^ - -Those in a system administration role, who are installing a writable stack for multiple users, will likely want to establish a separate group (perhaps lsst) with a umask of 002 (all access permissions for the group; allow other users to read+execute). -The installation directory must be owned by the group, have the SGID (2000) bit set, and allow group read/write/execute: that is, mode 2775. -Individual users who install a personal Stack on their own machine need not worry about this. + - ``lsst_distrib`` is a top-level package that brings-in most LSST Data Management pipelines software, but other top-level packages may be more applicable for your work, such as ``lsst_apps`` or ``lsst_sims``. + See :doc:`top-level-packages` for more information. -.. _install-from-source-envvar: + - ``v14_0_rc1`` is the current release. + You can install other tagged versions of the LSST Science Pipelines, though. + See :ref:`newinstall-other-tags`. -3. Unset environment variables -============================== + - If you're curious about the shebangtron, see its repository at `github.com/lsst/shebrangtron `_. -If you've run the LSST Science Pipelines previously, you may have conflicting environment variables setup. +.. _newinstall-test: + +5. Test your installation (optional) +==================================== + +Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo pipeline `. +This demo processes a small amount of SDSS data and verifies that measurements match expected values. + +See :doc:`demo` for instructions. + +.. _newinstall-next-steps: + +Next steps +========== + +Add links to topics the reader should go to next. +Examples: + +- A topic reminding a user to setup the stack in a new shell. +- Topic explaining top-level packages. +- Topic on compiling a package alongside an existing stack. + +Advanced installation topics +============================ + +The above steps guided you through LSST's recommended installation. +These topics provide additional information about the installation and ways to customize it: + +- :ref:`newinstall-shared-permissions`. +- :ref:`newinstall-unset-variables`. +- :ref:`newinstall-background`. +- :ref:`newinstall-miniconda`. +- :ref:`newinstall-py2`. +- :ref:`newinstall-user-python`. +- :ref:`newinstall-binary-packages`. +- :ref:`newinstall-binary-compatibility`. +- :ref:`newinstall-other-tags`. +- :ref:`newinstall-reference`. + +.. _newinstall-shared-permissions: + +Setting unix permissions for shared installations +------------------------------------------------- + +You can make the LSST Science Pipelines installation accessible to multiple users on the same machine. + +First, create a separate unix group (called ``lsst``, for example) with a ``umask`` of ``002`` (all access permissions for the group and allow other users to read/execute). + +Then set the ownership of the installation directory to the ``lsst`` group, have the ``SGID`` (2000) bit set, and allow group read/write/execute (mode 2775). + +.. _newinstall-unset-variables: + +Running newinstall.sh in an already set-up shell +------------------------------------------------ + +If you've run the LSST Science Pipelines previously, you may have conflicting environment variables in your shell. To be safe, run: .. code-block:: bash unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH -.. _install-from-source-setup: +Then return to the instructions step :ref:`newinstall-run`. -4. Installation set-up -====================== +.. _newinstall-background: -Download and run the `installation setup script from GitHub `__, which installs the basic packages required to install other packages: +What newinstall.sh does +----------------------- -.. code-block:: bash +:command:`newinstall.sh` creates a self-contained environment on your machine where you can install, run, and develop the LSST Science Pipelines. +You activate this environment in a shell by sourcing the :command:`loadLSST` script in the installation directory (see :ref:`setup-howto`). + +Here is how :command:`newinstall.sh` prepares the environment: + +- Identifies your operating system and compilers to determine what EUPS binary packages should be installed (the EUPS package root, see :ref:`newinstall-binary-packages`). +- Installs a specific version of Python, through Miniconda, that is compatible with EUPS binary packages (see :ref:`newinstall-miniconda`). +- Installs Conda packages that the LSST Science Pipelines depends on (see :ref:`python-deps`). +- Checks for :command:`git` on your systems and offers to install it if necessary. +- Installs EUPS_, the package manager used by the LSST software stack. + +For information about :command:`newinstall.sh`\ ’s arguments, see :ref:`newinstall-reference`. + +.. _newinstall-miniconda: - curl -OL https://raw.githubusercontent.com/lsst/lsst/13.0/scripts/newinstall.sh - bash newinstall.sh +About the Miniconda Python installed by newinstall.sh +----------------------------------------------------- -This installs the :command:`loadLSST.*` scripts, which you should source to ensure that LSST tools (e.g., the :command:`eups` command) are included in your path. +:command:`newinstall.sh` can install a dedicated Python environment for your LSST Science Pipelines installation. +This Python installation isn't required, but we recommend it. +See :ref:`newinstall-user-python` if required. -The install script will check your system to ensure that appropriate versions of critical packages are installed on your system, to enable bootstrapping the Science Pipelines, including :command:`git`, and :command:`python`. -If these packages are not available, the script will offer to install them for you (using the Anaconda Python distribution for the latter packages). +The Python environment installed by :command:`newinstall.sh` is Miniconda_, a minimal version of Anaconda_. +By default, :command:`newinstall.sh` installs Python 3.5.2. +If you need to work with your own Python 2.7-only packages, see :ref:`newinstall-py2`. -Allowing the installation of these core packages will not replace or modify any other version of these packages that may be installed on your system. -If you do not choose the Anaconda Python install, and subsequent package build steps fail, you can do one of two things: +In this Miniconda environment, :command:`newinstall.sh` installs the Science Pipeline's Python prerequisites. +See :ref:`python-deps` for more information. -* Report the problem to `community.lsst.org `_. Include your OS, a description of the problem, plus any error messages. Community members will provide assistance. -* Consider removing all contents of the install directory and start from scratch, and accepting the Anaconda Python installation option. +This Miniconda installation won't affect your other Python installations (like the system's Python, your own Anaconda or Miniconda, or virtual environments). +The LSST Miniconda environment is only active when you source the ``loadLSST`` script installed by :command:`newinstall.sh` (see :doc:`setup`). -Once :command:`newinstall.sh` has finished, source the LSST environment to continue the installation by running the appropriate command for your shell: +If you install other Python packages in a shell where the LSST Miniconda is activated (with :command:`pip install` or :command:`conda install`) those packages are installed into the LSST Miniconda's :file:`site-packages`, not your system's. +The Python installed by :command:`newinstall.sh` works like an isolated Python environment dedicated to LSST Science Pipelines code and your own related modules---effectively like a `Conda environment `_ or Python `venv `_. +This pattern is useful because it reduces the risk of having Python package version incompatibilities. + +.. _newinstall-py2: + +How to install a Python 2.7 environment with newinstall.sh +---------------------------------------------------------- + +LSST Science Pipelines is backwards compatible with Python 2.7. +If you need to run your own Python 2.7-only Python packages in conjunction with the Pipelines, you can have :command:`newinstall.sh` install a Python 2.7 environment for you instead of the default Python 3.5 environment. + +To select Python 2.7, run :command:`newinstall.sh` with the :option:`-2 ` flag (in addition to other flags, like :option:`-t `): .. code-block:: bash - source $LSST_INSTALL_DIR/loadLSST.bash # for bash users - source $LSST_INSTALL_DIR/loadLSST.csh # for csh users - source $LSST_INSTALL_DIR/loadLSST.ksh # for ksh users - source $LSST_INSTALL_DIR/loadLSST.zsh # for zsh users + bash newinstall.sh -2 + +Then follow the remaining instructions at :ref:`newinstall-run`. + +See also: :ref:`newinstall-miniconda`. + +.. _newinstall-user-python: -where :file:`$LSST_INSTALL_DIR` is expanded to your installation directory. +How to use your own Python with newinstall.sh +--------------------------------------------- -.. _install-from-source-packages: +:command:`newinstall.sh` creates a new Python environment by default (pre-configured with Python dependencies). +If necessary, you can use your own pre-existing Python environment. -5. Install lsst_distrib -======================= +To do so, run :command:`newinstall.sh` (see :command:`newinstall-run` for details and command arguments). +When :command:`newinstall-run` prompts you to install Miniconda, type ``no``. -Finally, install components of the LSST Science Pipelines that are relevant for your work. -A simple way to ensure that you have a fairly complete set of packages for this need is to install ``lsst_distrib``: +Be aware of these caveats when using your own Python installation: + +- You are responsible for installing Python package dependencies. + See :ref:`python-deps`. + +- Prebuilt binaries will not be available. + :command:`eups distrib install` will always install from source. + +.. _newinstall-binary-packages: + +About EUPS tarball packages +--------------------------- + +EUPS distrib binary (tarball) packages significantly speed up your installation. +Rather than compiling the LSST Science Pipelines from source, EUPS tarballs are prebuilt packages made specifically for supported platforms. + +Platforms are defined by four factors: + +1. Operating system. +2. Compiler. +3. Miniconda_ (Python) version. +4. lsstsw_ version (Git ref). + +EUPS distrib binary packages are currently being built for these platform combinations: + +.. csv-table:: EUPS distrib binary flavors + :header: "OS","Compiler","Python" + + "macOS ``osx/10.9``", "``clang-800.0.42.1``", "``miniconda3-4.2.12`` (Python 3)" + "macOS ``osx/10.9``", "``clang-800.0.42.1``", "``miniconda2-4.2.12`` (Python 2)" + "Redhat ``redhat/el7``", "``gcc-system``", "``miniconda3-4.2.12`` (Python 3)" + "Redhat ``redhat/el7``", "``gcc-system``", "``miniconda2-4.2.12`` (Python 2)" + "Redhat ``redhat/el6``", "``devtoolset-3``", "``miniconda2-4.2.12`` (Python 2)" + +When you run :command:`newinstall.sh`, it looks at your system to identify your operating system and compiler. +The version of :command:`newinstall.sh` you run also determines the Miniconda_ version and the lsstsw_ build system versions. + +Together, these four factors define the URL prefix (called an *EUPS package root*) that :command:`eups distrib install` looks for binary packages from. +If binary tarballs are unavailable for that EUPS package root, :command:`eups distrib install` automatically falls back to compiling LSST Science Pipelines packages from source. + +You can see the active EUPS package roots on your system by running: .. code-block:: bash - eups distrib install -t v13_0 lsst_distrib - setup lsst_distrib + eups distrib path + +Here is an example of the output: + +.. code-block:: text -After this initial setup, it is a good idea to test the installation. -See :ref:`source-install-testing-your-installation`. + https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67 + https://eups.lsst.codes/stack/src -.. _install-from-source-loadlsst: +Based on this example, :command:`eups distrib install` will preferentially install EUPS distrib binary packages for the macOS 10.9 system, ``clang-800.0.42.1`` compiler, and ``miniconda3-4.2.12-7c8e67`` Python and lsstsw combination. +If :command:`eups distrib install` cannot find packages at that EUPS package root it will look in the second EUPS package root (https://eups.lsst.codes/stack/src), which provides source packages. -6. Source the LSST environment in each shell session -==================================================== +.. _newinstall-binary-compatibility: -Whenever you want to run the installed LSST Science Pipelines in a new terminal session, be sure to :command:`source` the appropriate :file:`loadLSST.bash`, :file:`loadLSST.csh`, :file:`loadLSST.ksh` or :file:`loadLSST.zsh}` script. +EUPS tarball packages and compiler compatibility +------------------------------------------------ -Then setup the EUPS packages you need, typically: +EUPS binary tarball packages are prebuilt on LSST's continuous integration servers for a specific combination of operating system, compilers, Python, and Python dependencies. +If you are developing packages alongside this installation, you might encounter application binary interface (ABI) incompatibilities if are using a different compiler version or a different Python environment. + +In this case, the more reliable solution is to revert to a source installation. +To do this, repeat the installation but run :command:`newinstall.sh` *without* the :option:`-t ` argument: .. code-block:: bash - setup lsst_distrib + bash newinstall.sh -c -.. _source-install-testing-your-installation: +Without the :option:`-t ` argument to :command:`newinstall.sh`, :command:`eups distrib install` will always build and install packages from source, ensuring ABI compatibility. + +.. _newinstall-other-tags: + +Installing other releases (including daily and weekly tags) +----------------------------------------------------------- + +The instructions on this page guide you through installing the current release of the LSST Science Pipelines corresponding to this documentation. +You can, however, install other releases by running :command:`eups distrib install` with a different tag. + +The common types of tags are: + +- **Major releases,** tagged as ``v_`` (for example, ``v14_0``). +- **Weekly builds,** tagged as ``w__`` (for example, ``w_2017_33`` is the 33rd weekly build in 2017). +- **Daily builds,** tagged as ``d___`` (for example, ``d_2017_09_01`` is the daily build for September 1, 2017). + +There are also tags pointing to the most recent releases: + +- **Current major release,** tagged as ``current``. +- **Current weekly build,** tagged as ``w_latest``. +- **Current daily build,** tagged as ``d_latest``. + +You can see all available tags at https://eups.lsst.codes/stack/src/tags (each tag has a ``.list`` file). + +.. note:: + + Binary installations may not be available for all tags. + From https://eups.lsst.codes/stack, browse subdirectories corresponding to your platform and look for ``.list`` files of available tags. + :command:`eups distrib install` automatically falls back to a source build if binaries are not available. + +.. warning:: + + You need to ensure that the Python environment created by :command:`newinstall.sh` (see step :ref:`newinstall-run`) is compatible with the tagged software. + + For example, if you are installing a recent weekly you may need to download and run :command:`newinstall.sh` from master: + + .. code-block:: bash + + curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh + bash newinstall.sh -ct + + See https://github.com/lsst/lsst/tags for available tagged versions of :command:`newinstall.sh`. + +.. _newinstall-reference: + +newinstall.sh argument reference +-------------------------------- + +.. program:: newinstall.sh + +.. code-block:: text + + usage: newinstall.sh [-b] [-f] [-h] [-n] [-3|-2] [-t|-T] [-s|-S] [-P ] + +.. option:: -b + + Run in batch mode. Don't ask any questions and install all extra packages. + +.. option:: -c + + Attempt to continue a previously failed install. + +.. option:: -n + + No-op. Go through the motions but echo commands instead of running them. + +.. option:: -P + + Use a specific python interpreter for EUPS. + +.. option:: -2 + + Use Python 2 if the script is installing its own Python. + +.. option:: -3 + + Use Python 3 if the script is installing its own Python. (**default**) + +.. option:: -t + + Use pre-compiled EUPS "tarball" packages, if available. + +.. option:: -T + + **Do not** use pre-compiled EUPS "tarball" packages. (**default**) + +.. option:: -s + + Use EUPS source "eupspkg" packages, if available (compile from source). + +.. option:: -S + + **Do not** use EUPS source "eupspkg" packages (do not compile from source). + +.. option:: -h + + Display this help message. -7. Testing Your Installation -============================ -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. +.. _Miniconda: https://conda.io/miniconda.html +.. _Anaconda: https://docs.anaconda.com +.. _lsstsw: https://github.com/lsst/lsstsw +.. _EUPS: https://github.com/RobertLuptonTheGood/eups From d4617c98b87636065b214a7c494e303df4a48259 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Wed, 13 Sep 2017 10:58:17 -0700 Subject: [PATCH 05/14] Add page on top-level packages This explains and mentions that lsst_apps and lsst_distrib both exist. The page will get better when we have packages in the pipelines.lsst.io docs and they automatically report their package dependencies. --- index.rst | 1 + install/index.rst | 1 + install/top-level-packages.rst | 48 ++++++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) create mode 100644 install/top-level-packages.rst diff --git a/index.rst b/index.rst index f12dca3e..142946f6 100644 --- a/index.rst +++ b/index.rst @@ -15,6 +15,7 @@ Installation - :doc:`Overview of installation methods ` - :doc:`Prerequisites ` - :doc:`Installing with newinstall.sh ` (recommended) +- :doc:`install/top-level-packages` - :doc:`Installing from source with lsstsw ` (for developers) .. toctree:: diff --git a/install/index.rst b/install/index.rst index cfcd4263..8d11f3d1 100644 --- a/install/index.rst +++ b/install/index.rst @@ -7,6 +7,7 @@ Installing the LSST Science Pipelines prereqs/index newinstall + top-level-packages lsstsw demo diff --git a/install/top-level-packages.rst b/install/top-level-packages.rst new file mode 100644 index 00000000..ae33acc7 --- /dev/null +++ b/install/top-level-packages.rst @@ -0,0 +1,48 @@ +############################# +Top-level packages to install +############################# + +The LSST Science Pipelines are part of LSST's EUPS_ package stack. +This means that the Science Pipelines software is actually a collection of packages that you install and setup together. +By specifying different **top-level packages** to the :ref:`eups distrib install ` and :doc:`setup ` commands, you can control the size of the software installation or add new capabilities. + +This page describes the common top-level packages that make up the LSST Science Pipelines and related EUPS stacks. + +lsst\_apps +========== + +``lsst_apps`` provides the core packages that LSST will use in operations. + +Example installation (:ref:`more info `): + +.. code-block:: bash + + eups distrib install lsst_apps -t + setup lsst_apps + +lsst\_distrib +============= + +``lsst_distrib`` provides all the packages from ``lsst_apps`` along with these additional ones: + +.. https://github.com/lsst/lsst_distrib/blob/master/ups/lsst_distrib.table + +- `ctrl\_execute `_ +- `ctrl\_platform_lsstvc `_ +- `datarel `_ +- `meas\_extensions_convolved `_ +- `meas\_extensions_shapeHSM `_ +- `meas\_extensions_photometryKron `_ +- `pipe\_drivers `_ +- `lsst\_obs `_ +- `jointcal `_ +- `verify `_ + +Example installation (:ref:`more info `): + +.. code-block:: bash + + eups distrib install lsst_distrib -t + setup lsst_distrib + +.. _EUPS: https://github.com/RobertLuptonTheGood/eups From 0708d98b7294bf3757edece5e9ca460cb2180394 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Wed, 13 Sep 2017 13:37:01 -0700 Subject: [PATCH 06/14] Add topic on setting up the stack This is a standalone from the installation instructions to make it easier to understanding what steps need to be done in in each new shell, compared to a new installation. Also lets us add additional background on `setup` itself. --- index.rst | 1 + install/index.rst | 1 + install/setup.rst | 56 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) create mode 100644 install/setup.rst diff --git a/index.rst b/index.rst index 142946f6..a510f880 100644 --- a/index.rst +++ b/index.rst @@ -15,6 +15,7 @@ Installation - :doc:`Overview of installation methods ` - :doc:`Prerequisites ` - :doc:`Installing with newinstall.sh ` (recommended) +- :doc:`install/setup` - :doc:`install/top-level-packages` - :doc:`Installing from source with lsstsw ` (for developers) diff --git a/install/index.rst b/install/index.rst index 8d11f3d1..58d4246a 100644 --- a/install/index.rst +++ b/install/index.rst @@ -7,6 +7,7 @@ Installing the LSST Science Pipelines prereqs/index newinstall + setup top-level-packages lsstsw demo diff --git a/install/setup.rst b/install/setup.rst new file mode 100644 index 00000000..df54ca1f --- /dev/null +++ b/install/setup.rst @@ -0,0 +1,56 @@ +.. _setup: + +########################################### +Setting up installed LSST Science Pipelines +########################################### + +Whenever you start a new command-line shell you need to set up the LSST Science Pipelines software before you can use it. + +.. _setup-howto: + +Setting up +========== + +Setting the LSST Science Pipelines in a shell is a two-step process: + +1. Load the LSST environment by sourcing the ``loadLSST`` script in your installation directory: + + .. TODO Use sphinx-tabs here? + + .. code-block:: bash + + source loadLSST.bash # for bash + source loadLSST.csh # for csh + source loadLSST.ksh # for ksh + source loadLSST.zsh # for zsh + + .. note:: + + These installation are for :doc:`newinstall.sh `-based installations. + For ``lsstsw``, follow :ref:`these instructions ` instead. + +2. Set up a top-level package: + + .. code-block:: bash + + setup + + For example, ``setup lsst_apps`` or ``setup lsst_distrib``. + See :doc:`top-level-packages` for more about LSST's top-level packages. + +.. _setup-list: + +Listing what packages are set up +================================ + +To see what packages (and their versions) are currently set up: + +.. code-block:: bash + + setup list -s + +To see all packages that are installed, even if not currently set up, run: + +.. code-block:: bash + + setup list From d3ef922c2294d75d7fee6101757115d149901ffe Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:13:06 -0700 Subject: [PATCH 07/14] Add new topic on running from Docker This covers a basic quick start and how to find tags on Docker Hub. Future commits can add details on attaching volumes, developing against a containerized stack, and creating new images with Dockerfiles. --- index.rst | 1 + install/docker.rst | 77 ++++++++++++++++++++++++++++++++++++++++++++++ install/index.rst | 1 + 3 files changed, 79 insertions(+) create mode 100644 install/docker.rst diff --git a/index.rst b/index.rst index a510f880..dc5193ac 100644 --- a/index.rst +++ b/index.rst @@ -17,6 +17,7 @@ Installation - :doc:`Installing with newinstall.sh ` (recommended) - :doc:`install/setup` - :doc:`install/top-level-packages` +- :doc:`install/docker` - :doc:`Installing from source with lsstsw ` (for developers) .. toctree:: diff --git a/install/docker.rst b/install/docker.rst new file mode 100644 index 00000000..c68436c5 --- /dev/null +++ b/install/docker.rst @@ -0,0 +1,77 @@ +.. _docker: + +################### +Running with Docker +################### + +LSST provides versioned Docker images containing the Science Pipelines software. +With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. + +If you have issues, reach out on the `LSST Community support forum `_. + +.. _docker-prereqs: + +Prerequisites +============= + +To download Docker images and run containers, you need Docker's software. +The `Docker Community Edition `_ is freely available for most platforms, including macOS, Linux, and Windows. + +If you haven't used Docker before, you might want to learn more about Docker, images, and containers. +Docker's `Getting Started `_ documentation is a good resource. + +.. _docker-quick-start: + +Quick start +=========== + +This command downloads a weekly build of the LSST Science Pipelines, starts a container, and opens a prompt: + +.. code-block:: bash + + docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2017_35 + +Then in the container's shell, load the LSST environment and set up a :doc:`top-level package ` (``lsst_distrib`` in this case): + +.. code-block:: bash + + source /opt/lsst/software/stack/loadLSST.bash + setup lsst_distrib + +This step is equivalent to the :doc:`setup instructions ` for a :doc:`newinstall.sh `\ -based installation. +In fact, the images are internally based on :command:`newinstall.sh`. + +Next, learn more with these topics: + +- :ref:`docker-tags` + +.. _docker-tags: + +Finding images for different LSST Science Pipelines releases +============================================================ + +LSST Science Pipelines Docker images are published as `lsstsqre/centos`_ on Docker Hub. +These images are based on the CentOS base image. + +Docker images are versioned with tags, allowing you to run any release of the LSST Science Pipelines software. +The schema of these tags is: + +.. code-block:: text + + -stack-- + +For example: + +.. code-block:: text + + 7-stack-lsst_distrib-w_2017_35 + +This tag corresponds to: + +- CentOS 7 operating system. +- ``lsst_distrib`` :doc:`top-level package `. +- ``w_2017_35`` EUPS tag. See :ref:`newinstall-other-tags` for an overview of LSST's EUPS tag schema. + +You can see what tags are available by browsing `lsstsqre/centos on Docker Hub `_. + +.. _`lsstsqre/centos`: https://hub.docker.com/r/lsstsqre/centos/ diff --git a/install/index.rst b/install/index.rst index 58d4246a..abd2df8b 100644 --- a/install/index.rst +++ b/install/index.rst @@ -9,6 +9,7 @@ Installing the LSST Science Pipelines newinstall setup top-level-packages + docker lsstsw demo From 5305126ebbda08b846f805c243f5c020f060425d Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:24:41 -0700 Subject: [PATCH 08/14] Update demo for v14.0 release --- install/demo.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/install/demo.rst b/install/demo.rst index 713c9604..fdfb610c 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -28,8 +28,8 @@ Then download the demo's data (if you aren't running the current stable release, .. code-block:: bash - curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/13.0.tar.gz | tar xvzf - - cd lsst_dm_stack_demo-13.0 + curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/14.0.tar.gz | tar xvzf - + cd lsst_dm_stack_demo-14.0 .. note:: From 2a28db08968842b7126782b4a3cb787afd9dcde8 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:25:08 -0700 Subject: [PATCH 09/14] Demo page edits - Switch to sentence case headers. - Improve some of the technical writing style. --- install/demo.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/install/demo.rst b/install/demo.rst index fdfb610c..b5df28ac 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -1,5 +1,5 @@ ###################################################### -Testing the Science Pipelines Installation with a Demo +Testing the Science Pipelines installation with a demo ###################################################### This demo will allow you to quickly test your LSST Science Pipelines installation, :doc:`regardless of your installation method `. @@ -13,11 +13,11 @@ The method depends on how the Science Pipelines were installed: - :doc:`newinstall.sh ` - :ref:`lsstsw ` -2. Download the Demo Project +2. Download the demo project ============================ Choose a directory to run the demo in. -We'll call this directory :file:`demo_data`. +For example: .. code-block:: bash @@ -34,7 +34,7 @@ Then download the demo's data (if you aren't running the current stable release, .. note:: The demo's version should match your installed software. - If you installed from source (with :doc:`lsstsw `), clone the demo repository instead of downloading a release: + If you installed from source (with :doc:`lsstsw `) or with a :ref`newer tag `, clone the demo repository instead of downloading a release: .. code-block:: bash @@ -53,7 +53,7 @@ run camcol field filters *Filters in parentheses are not processed if run with the* ``--small`` *option, see below* -3. Run the Demo +3. Run the demo =============== Now setup the processing package and run the demo: @@ -65,11 +65,12 @@ Now setup the processing package and run the demo: For each input image the script performs the following operations: -* generate a subset of basic image characterization (e.g., determine photometric zero-point, detect sources, and measures positions, shapes, brightness with a variety of techniques), -* creates a :command:`./output` subdirectory containing subdirectories of configuration files, processing metadata, calibrated images, FITS tables of detected sources. These "raw" outputs are readable by other parts of the LSST pipeline, and -* generates a master comparison catalog in the working directory from the band-specific source catalogs in the ``output/sci-results/`` subdirectories. +- Generates a subset of basic image characterizations (such photometric zero-point estimation, source detection, and measurement). +- Creates a :file:`./output` directory containing subdirectories of configuration files, processing metadata, calibrated images, and FITS tables of detected sources. + These ``raw`` outputs are readable by other parts of the LSST pipeline. +- Generates a master comparison catalog in the working directory from the band-specific source catalogs in the ``output/sci-results/`` subdirectories. -4. Check the Demo Results +4. Check the demo results ========================= The demo will take a minute or two to execute (depending upon your machine), and will generate a large number of status messages. From 48e2d7a2ffe9b2a3a622252fd851ddc83af23f5d Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:44:20 -0700 Subject: [PATCH 10/14] Bump CURRENT_RELEASE to 14.0 --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 88df4bcc..eb8ad508 100644 --- a/conf.py +++ b/conf.py @@ -23,7 +23,7 @@ sys.setrecursionlimit(2000) # master variable for current Stack release -CURRENT_RELEASE = '13.0' +CURRENT_RELEASE = '14.0' # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the From 8020236d1c31088711e9f3ecdbb2a3eae83877ec Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 14:34:40 -0700 Subject: [PATCH 11/14] Update lsstsw-based install docs - Update for 14.0 - Move advanced how-to and reference material from the basic installation steps. - Add overviews of deploy and rebuild, along with command references. --- install/lsstsw.rst | 240 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 193 insertions(+), 47 deletions(-) diff --git a/install/lsstsw.rst b/install/lsstsw.rst index c4328b8a..1741b085 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -19,7 +19,7 @@ If you have difficulty installing LSST software: 1. Prerequisites ================ -You need to install some prerequisites to build the LSST Stack from source: +Before you begin: - `Install and configure Git LFS `_ for LSST DM's data servers. - Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. @@ -27,45 +27,30 @@ You need to install some prerequisites to build the LSST Stack from source: .. _lsstsw-deploy: -2. Obtaining a Development Stack with lsstsw -============================================ +2. Deploy lsstsw +================ -Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). -`lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. +.. Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). +.. `lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. Begin by choosing a working directory, then deploy ``lsstsw`` into it: .. code-block:: bash - :linenos: git clone https://github.com/lsst/lsstsw.git cd lsstsw ./bin/deploy - . bin/setup.sh - -The ``deploy`` script automates several things for you: + source bin/setup.sh -1. installs a miniconda_ Python environment specific to this lsstsw workspace, -2. installs EUPS_ in :file:`eups/current/`, -3. clones `lsst-build`_, which will run the build process for us, -4. clones versiondb_, a robot-made Git repository of package dependency information, and -5. creates an empty Stack *installation* directory, :file:`stack/`. - -``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. -Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. +For more information about the :command:`deploy` command, such as using Python 2.7 instead of 3, see :ref:`lsstsw-about-deploy`. .. The ``setup.sh`` step enables EUPS_, the package manager used by LSST. .. **Whenever you open a new terminal session, you need to run '. bin/setup.sh' to activate your lsstsw environment.** -.. _lsst-build: https://github.com/lsst/lsst_build -.. _versiondb: https://github.com/lsst/versiondb -.. _EUPS: https://github.com/RobertLuptonTheGood/eups -.. _miniconda: http://conda.pydata.org/miniconda.html - .. _lsstsw-rebuild: -3. Build Science Pipelines -========================== +3. Build the Science Pipelines packages +======================================= From the :file:`lsstsw` directory, run: @@ -75,54 +60,215 @@ From the :file:`lsstsw` directory, run: Once the ``rebuild`` step finishes, note the build number printed on screen. It is formatted as "``bNNNN``." -Tell EUPS this is the current build by making a clone of the build's EUPS tag and calling it "``current``:" +Tag this build as ``current`` so that EUPS can set it up by default: .. code-block:: bash eups tags --clone bNNNN current -The ``rebuild`` command accomplishes the following: - -1. Clones all Science Pipelines packages from `github.com/lsst `__. - A high-bandwidth connection is helpful since the stack contains a non-trivial amount of code and test data. -2. Runs the Scons-based build process to compile C++, make Swig bindings, and ultimately create the :lmod:`lsst` Python package. - The Stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. - Finally, set up the packages with EUPS: .. code-block:: bash setup lsst_distrib +See :doc:`setup` for more information. + +.. _lsstsw-testing-your-installation: + +4. Testing Your installation (optional) +======================================= + +Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. +This demo processes a small amount of SDSS data. + .. _lsstsw-setup: -4. Sourcing the Pipelines in a New Shell -======================================== +Sourcing the Pipelines in a new shell +===================================== + +In every new shell session you will need to set up the Science Pipelines environment and EUPS package stack. + +Run these two steps: + +1. Activate the lsstsw software environment by sourcing the :file:`setup.sh` script in lsstsw's :file:`bin` directory: + + .. code-block:: bash + + source bin/setup.sh + + If you are running in a :command:`csh` or :command:`tcsh`, run this set up script instead: + + .. code-block:: bash + + source bin/setup.csh + +2. Set up a :doc:`top-level package `: + + .. code-block:: bash + + setup lsst_distrib + + Instead of ``lsst_distrib``, you can set up a different top-level package like ``lsst_apps`` or any individual EUPS package. + See :doc:`top-level-packages`. + +.. _lsstsw-next: + +Next steps and advanced topics +============================== + +- :ref:`lsstsw-about-deploy`. +- :ref:`lsstsw-py2`. +- :ref:`lsstsw-about-rebuild`. +- :ref:`lsstsw-branches`. +- :ref:`lsstsw-deploy-ref`. +- :ref:`lsstsw-rebuild-ref`. -In every new shell session you will need to 'setup' the Science Pipelines. -Do this by running the ``setup.sh`` from the ``lsstsw/`` directory: +.. _lsstsw-about-deploy: + +About the lsstsw deploy script +------------------------------ + +The ``deploy`` script automates several things to prepare an LSST development environment: + +1. Installs Git. +2. Installs Git LFS (*you* are still responsible for `configuring it `_). +3. Installs a Miniconda_ Python environment specific to this lsstsw workspace. + The default Python is Python 3, but you can switch to Python 2.7 if necessary. + See :ref:`lsstsw-py2`. +4. Installs EUPS_ into :file:`eups/current/`. +5. Clones `lsst-build`_, which runs the build process. +6. Clones versiondb_, a robot-managed Git repository of package dependency information. +7. Creates an empty stack *installation* directory, :file:`stack/`. + +This environment, including the EUPS, Miniconda, Git, and Git LFS software, is only activated when you source the :file:`bin/setup.sh` or :file:`bin/setup.csh` scripts in a shell. +Otherwise, lsstsw does not affect the software installed on your computer. + +See also: :ref:`lsstsw-deploy-ref`. + +.. _lsstsw-py2: + +How to use Python 2.7 +--------------------- + +The lsstsw :command:`deploy` script installs Miniconda_ as self-contained Python environment. +By default, :command:`deploy` installs a Python 3 version of Miniconda_. +For testing Python 2.7 compatibility, you can create an lsstsw deployment with a Python 2.7 version of Miniconda using the :option:`deploy -2` argument: .. code-block:: bash - . bin/setup.sh - setup lsst_distrib # or an alternative top-level package + ./bin/deploy -2 -.. note:: +.. _lsstsw-about-rebuild: - If you are using a tcsh shell, run ``. bin/setup.csh`` instead (note ``csh`` extension). +About the lsstsw rebuild command +-------------------------------- -.. _lsstsw-testing-your-installation: +The :command:`rebuild` command accomplishes the following: -5. Testing Your Installation -============================ +1. Clones all Science Pipelines packages from `github.com/lsst `__. + :command:`rebuild` -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. +2. Runs the Scons-based build process to compile C++, make Pybind11 bindings, and ultimately create the :lmod:`lsst` Python package. + The stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. + +``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. +Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. + +See also: :ref:`lsstsw-rebuild-ref`. + +.. _lsstsw-branches: + +Building from branches +---------------------- + +lsstsw's :command:`rebuild` enables you to clone and build development branches. + +To build ``lsst_distrb``, but use the Git branch ``my-feature`` when it's available in a package's repository: + +.. code-block:: bash + + rebuild -r my-feature lsst_distrib + +Multiple ticket branches across multiple products can be built in order of priority: + +.. code-block:: bash + + rebuild -r feature-1 -r feature-2 lsst_distrib + +In this example, a ``feature-1`` branch will be used in any package's Git repository. +A ``feature-2`` branch will be used secondarily in repositories where ``feature-1`` doesn't exist. +Finally, ``lsstsw`` falls back to using the ``master`` branch for repositories that lack both ``feature-1`` and ``feature-2``. + +.. _lsstsw-deploy-ref: + +lsstsw deploy command reference +------------------------------- + +.. program:: deploy + +.. code-block:: text + + usage: deploy.sh [-2|-3] [-b] [-h] + +.. option:: -2 + + Install a Python 2-based Miniconda_. + +.. option:: -3 + + Use a Python 3-based Miniconda_ (default). + +.. option:: -b + + Use bleeding-edge conda packages. + +.. option:: -h + + Print the help message. + +.. _lsstsw-rebuild-ref: + +lsstsw rebuild command reference +-------------------------------- + +.. program:: rebuild + +.. code-block:: text + + rebuild [-p] [-n] [-u] [-r [-r [...]]] [-t ] [product1 [product2 [...]]] + +.. option:: -p + + Prep only. + +.. option:: -n + + Do not run :command:`git fetch` in already-downloaded repositories. + +.. option:: -u + + Update the :file:`repos.yaml` package index to the ``master`` branch on GitHub of https://github.com/lsst/repos. + +.. option:: -r + + Rebuild using the Git ref. + A Git ref can be a branch name, tag, or commit SHA. + Multiple ``-r`` arguments can be given, in order or priority. + +.. option:: -t + + EUPS tag. + +.. _lsst-build: https://github.com/lsst/lsst_build +.. _versiondb: https://github.com/lsst/versiondb +.. _EUPS: https://github.com/RobertLuptonTheGood/eups +.. _Miniconda: http://conda.pydata.org/miniconda.html .. _lsstsw-development: -6. Bonus: Developing a Package -============================== +Developing a package +-------------------- An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. From 9c2998caf8f4470339665ee5e9e523df3c323c23 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 15:11:23 -0700 Subject: [PATCH 12/14] Move package development docs from lsstsw The same docs should apply to all base installations, where it's newinstall, lsstsw, or Docker. This commit moves the original content to an orphan page to give us time to make that change. --- install/lsstsw.rst | 65 -------------------------------- install/package-development.rst | 67 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 65 deletions(-) create mode 100644 install/package-development.rst diff --git a/install/lsstsw.rst b/install/lsstsw.rst index 1741b085..d4219db9 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -264,68 +264,3 @@ lsstsw rebuild command reference .. _versiondb: https://github.com/lsst/versiondb .. _EUPS: https://github.com/RobertLuptonTheGood/eups .. _Miniconda: http://conda.pydata.org/miniconda.html - -.. _lsstsw-development: - -Developing a package --------------------- - -An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. -The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. - -1. Stack packages are found in the :file:`lsstsw/build/` directory. - -2. Create a new branch in a package's Git repository, - - .. code-block:: bash - - git checkout -b {{ticket-name}} - - Then declare this package for EUPS and set it up: - - .. code-block:: bash - - eups declare -r . -t $USER {{package_name}} git - setup -r . -t $USER - - Unpacking the ``eups declare`` arguments: - - - ``-r .`` is the path to the package's repository, which is the current working directory. - You don't *need* to be in the repository's directory if you provide the path appropriately. - - ``-t $USER`` sets the EUPS *tag*. - We use this because your username (``$USER``) is an allowed EUPS tag. - - ``git`` is used as an EUPS *version*. - Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. - - In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." - In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. - If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. - This is why we initially declared the entire lsstsw repository to have the version ``current``. - -3. Build the package with Scons: - - .. code-block:: bash - - scons -Q -j 6 opt=3 - - These flags tell SCons to build with flags: - - - ``-Q``: reduce logging to the terminal, - - ``-j 6``: build in parallel (e.g., with '6' CPUs), - - ``opt=3``: build with level 3 optimization. - - This ``scons`` command will run several targets by default, in sequence: - - 1. ``lib``: build the C++ code and SWIG interface layer - 2. ``python``: install the Python code - 3. ``tests``: run the test suite - 4. ``example``: compile the examples, - 5. ``doc``: compile Doxygen-based documentation, and - 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). - - You can build a subset of these targets by specifying one explicitly. - To simply compile C++, SWIG, build the Python package and run tests: - - .. code-block:: bash - - scons -q -j 6 opt=3 tests diff --git a/install/package-development.rst b/install/package-development.rst new file mode 100644 index 00000000..873379f3 --- /dev/null +++ b/install/package-development.rst @@ -0,0 +1,67 @@ +:orphan: 1 + +.. NOTE: this page is not currently part of the built site, though it is intended for re-integration. + +############################################################### +Developing a package with the installed Science Pipelines stack +############################################################### + +An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. +The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. + +1. Stack packages are found in the :file:`lsstsw/build/` directory. + +2. Create a new branch in a package's Git repository, + + .. code-block:: bash + + git checkout -b {{ticket-name}} + + Then declare this package for EUPS and set it up: + + .. code-block:: bash + + eups declare -r . -t $USER {{package_name}} git + setup -r . -t $USER + + Unpacking the ``eups declare`` arguments: + + - ``-r .`` is the path to the package's repository, which is the current working directory. + You don't *need* to be in the repository's directory if you provide the path appropriately. + - ``-t $USER`` sets the EUPS *tag*. + We use this because your username (``$USER``) is an allowed EUPS tag. + - ``git`` is used as an EUPS *version*. + Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. + + In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." + In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. + If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. + This is why we initially declared the entire lsstsw repository to have the version ``current``. + +3. Build the package with Scons: + + .. code-block:: bash + + scons -Q -j 6 opt=3 + + These flags tell SCons to build with flags: + + - ``-Q``: reduce logging to the terminal, + - ``-j 6``: build in parallel (e.g., with '6' CPUs), + - ``opt=3``: build with level 3 optimization. + + This ``scons`` command will run several targets by default, in sequence: + + 1. ``lib``: build the C++ code and SWIG interface layer + 2. ``python``: install the Python code + 3. ``tests``: run the test suite + 4. ``example``: compile the examples, + 5. ``doc``: compile Doxygen-based documentation, and + 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). + + You can build a subset of these targets by specifying one explicitly. + To simply compile C++, SWIG, build the Python package and run tests: + + .. code-block:: bash + + scons -q -j 6 opt=3 tests From 14e0cc098b8eac70b0f2c430b97104a55dabf4d3 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 15:41:44 -0700 Subject: [PATCH 13/14] Make homepage the curated install entrypoint Before, both the homepage and install/index.rst served as entrypoints for the installation docs. Now I'm making the homepage the curated documentation entrypoint, but install/index.rst has the toctree so that a user landing at https://pipelines.lsst.io/install can still find all the topics. --- index.rst | 16 ++++++++++++---- install/index.rst | 33 +++------------------------------ 2 files changed, 15 insertions(+), 34 deletions(-) diff --git a/index.rst b/index.rst index dc5193ac..c48f27b4 100644 --- a/index.rst +++ b/index.rst @@ -12,13 +12,21 @@ The latest release is |current-release|: :doc:`learn what's new Installation ============ -- :doc:`Overview of installation methods ` -- :doc:`Prerequisites ` -- :doc:`Installing with newinstall.sh ` (recommended) +Recommended installation path: + +- :doc:`Installing with newinstall.sh ` - :doc:`install/setup` - :doc:`install/top-level-packages` + +Alternative distributions and installation methods: + - :doc:`install/docker` -- :doc:`Installing from source with lsstsw ` (for developers) +- :doc:`Installing from source with lsstsw ` +- `CernVM FS `_ (contributed by IN2P3) + +To install the LSST Simulation software, such as MAF, follow the `LSST Simulations documentation `_. + +.. This toctree is hidden to let us curate the section above, but still add the install/ pages to the Sphinx toctree .. toctree:: :hidden: diff --git a/install/index.rst b/install/index.rst index abd2df8b..ccb2894f 100644 --- a/install/index.rst +++ b/install/index.rst @@ -2,8 +2,10 @@ Installing the LSST Science Pipelines ##################################### +.. This page acts as an index.html for the install/ path, but the currated contents are on the homepage. + .. toctree:: - :hidden: + :maxdepth: 1 prereqs/index newinstall @@ -12,32 +14,3 @@ Installing the LSST Science Pipelines docker lsstsw demo - -We offer a few ways of installing the LSST Science Pipelines. -Choose an option below to get started. - -.. note:: - - To install the LSST Simulation software, such as MAF, please follow the `LSST Simulations documentation `_ instead of these guides. - -:doc:`Source installation (newinstall.sh) ` - ``newinstall.sh`` allows you to build and install the LSST Science Pipelines from source. - You can use your existing Python 2.7, or opt to use a built-in `Miniconda `__. - -`CernVM FS (external link) `__ - CernVM FS is a virtual machine that makes it easy to run the LSST Science Pipelines without compiling code. - This distribution is supported by Fabio Hernandez of IN2P3. - -`Docker Containers (external link) `__ - These images are convenient for using the LSST Science Pipelines in distributed or cloud-based computing workflows, or for one-off installations without modifying your environment. You can get them from our `DockerHub repo `__ - -:doc:`lsstsw installation ` - ``lsstsw`` (and ``lsst-build``) are the tools we use internally to build and test the LSST Science Pipelines. - Use this installation option if you're interested in developing the Science Pipelines since ``lsstsw`` presents the Pipelines as a directory of cloned repositories from `github.com/lsst `__. - -Next, :doc:`try out your Science Pipelines installation by running a demo `. - -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. From 38218835bcf09dcb5f5447626be8fb57c22c5cb4 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 15:50:33 -0700 Subject: [PATCH 14/14] Change prereq messaging about Miniconda This wording encourages users to use the provided Python instead of using their own. --- install/prereqs/index.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/install/prereqs/index.rst b/install/prereqs/index.rst index b0f2fcba..c0717d7a 100644 --- a/install/prereqs/index.rst +++ b/install/prereqs/index.rst @@ -21,8 +21,7 @@ System prerequisites Python dependencies =================== -You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. - +Both the :doc:`newinstall.sh <../newinstall>` and :doc:`lsstsw <../lsstsw>`\ -based installation methods provide dedicated Miniconda environments pre-loaded with Python dependencies. If you opt to use your own Python, you can re-create the default Python environment made by :command:`newinstall.sh` and ``lsstsw`` with these Conda environments: - `macOS and Python 2.7 `_.