diff --git a/conf.py b/conf.py index 88df4bcc..eb8ad508 100644 --- a/conf.py +++ b/conf.py @@ -23,7 +23,7 @@ sys.setrecursionlimit(2000) # master variable for current Stack release -CURRENT_RELEASE = '13.0' +CURRENT_RELEASE = '14.0' # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the diff --git a/index.rst b/index.rst index 27b99169..c48f27b4 100644 --- a/index.rst +++ b/index.rst @@ -5,30 +5,57 @@ The LSST Science Pipelines The LSST Science Pipelines enable optical and near-infrared astronomy in the big data era. We are building the Science Pipelines for the `Large Synoptic Survey Telescope (LSST) `_, but our command line task and Python API can be extended for any optical or near-infrared dataset. -- Join us on the `LSST Community forum, community.lsst.org `_. -- Fork our code on GitHub at https://github.com/lsst. -- Report issues in `JIRA `_. -- API documentation is currently published with `Doxygen `_. -- DM Developer guidance is at https://developer.lsst.io. -- Learn more about LSST Data Management by visiting http://dm.lsst.org. -- Help us improve our documentation! This guide is on GitHub at `lsst/pipelines_lsst_io `_. +The latest release is |current-release|: :doc:`learn what's new `. -Latest release -============== +.. _part-installation: -The latest release is |current-release|: :doc:`learn what's new `. +Installation +============ + +Recommended installation path: + +- :doc:`Installing with newinstall.sh ` +- :doc:`install/setup` +- :doc:`install/top-level-packages` + +Alternative distributions and installation methods: + +- :doc:`install/docker` +- :doc:`Installing from source with lsstsw ` +- `CernVM FS `_ (contributed by IN2P3) + +To install the LSST Simulation software, such as MAF, follow the `LSST Simulations documentation `_. + +.. This toctree is hidden to let us curate the section above, but still add the install/ pages to the Sphinx toctree .. toctree:: :hidden: + :caption: Installation - releases/index + install/index + +.. _part-release-details: .. toctree:: :maxdepth: 1 - :caption: Getting Started - :name: part-getting-started + :caption: Release details - install/index - known-issues releases/notes + known-issues metrics/index + +.. toctree:: + :hidden: + + releases/index + +More info +========= + +- Join us on the `LSST Community forum, community.lsst.org `_. +- Fork our code on GitHub at https://github.com/lsst. +- Report issues in `JIRA `_. +- API documentation is currently published with `Doxygen `_. +- DM Developer guidance is at https://developer.lsst.io. +- Learn more about LSST Data Management by visiting http://dm.lsst.org. +- Contribute to our documentation. This guide is on GitHub at `lsst/pipelines_lsst_io `_. diff --git a/install/conda.rst b/install/conda.rst deleted file mode 100644 index 4c651a1d..00000000 --- a/install/conda.rst +++ /dev/null @@ -1,113 +0,0 @@ -############################################################## -Installing and Using LSST Science Pipelines as a Conda Package -############################################################## - -This page will guide you through installing the LSST Science Pipelines as a Conda package for use in `Anaconda or Miniconda `__. -Anaconda is a popular Python distribution and package ecosystem. -With this installation method you don't compile source code, or even need to have an existing Python installation. - -.. note:: - - This documentation does not cover the LSST Simulations software. - See the `LSST Simulations installation documentation `_ instead. - -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. - -1. Install Anaconda or Miniconda -================================ - -You might already have Anaconda or Miniconda as your main Python distribution. -If not, you can quickly get started by following Continuum's official installation instructions: - -- `Install Anaconda `__ if you want a complete, science-ready Python installation with minimal setup. -- Otherwise, `install the leaner Miniconda `__ version and install packages as you need them. - -.. warning:: - - You need the Python 2 (e.g. Miniconda2) distribution. If you want to use the experimental Python 3 support, you should try our source installation. - -If you're new to Anaconda, Continuum's `30-minute test drive `_ tutorial will get you up-to-speed. - -.. warning:: - - Don't reuse the Miniconda you might have previously obtained from an :doc:`eups distrib `\ -based installation. - Open a new shell and install Anaconda/Miniconda from scratch. - -Upgrading conda ---------------- - -If you have an existing Anaconda or Miniconda installation, you'll want to make sure the :command:`conda` command itself is up to date: - -.. code-block:: bash - - conda update conda - -See the `Conda documentation for more information about installing and managing conda and Anaconda/Miniconda `__. - -2. Install Science Pipelines in a Conda Environment -=================================================== - -These commands will download and activate the LSST Science Pipelines in a new Conda environment: - -.. code-block:: bash - :linenos: - - conda config --add channels http://conda.lsst.codes/stack/0.13.0 - conda create --name lsst python=2 - source activate lsst - conda install lsst-distrib - source eups-setups.sh - setup lsst_distrib - - You can ignore warnings about pre-link and post-link scripts being dangerous and deprecated. Sorry about that. - -.. warning:: - - Installing the LSST simulation tools (including MAF) requires pointing to a different Conda channel. - See the `LSST Simulations installation documentation `_. - -Here's what these commands are doing, line-by-line: - -1. Tell :command:`conda` about the LSST channel for Conda packages. -2. Create a Conda environment called ``lsst`` with Python 2.7. - You can change the environment's name to anything you like. - Conda environments are recommended since they help keep the Python dependencies of your projects separate. - See the `Conda documentation on environments for more information `__. -3. Activate the ``lsst`` environment (use your environment's name if you chose a different one). - The :command:`activate` command is provided by Anaconda/Miniconda (e.g. at :file:`~/miniconda2/bin/activate`). -4. Install the full suite of LSST science software, including Science Pipelines (``lsst-distrib``). -5. Setup EUPS, LSST's package manager. -6. Setup LSST packages in your environment with EUPS (setting up ``lsst_distrib`` makes most packages available to you). - -.. warning:: - - If the install fails with an error, check that your shell does not have another EUPS Stack configured (try ``echo $EUPS_STACK``). - Conda packaged EUPS will use existing values of ``EUPS_PATH`` and ``EUPS_DIR``. - If they exist, unset them before installing or using Conda packages. - -.. _conda-install-activate: - -3. Activating Science Pipelines in a new Shell -============================================== - -Whenever you open a new shell or terminal session, use these commands to re-activate your previously-installed Science Pipelines: - -.. code-block:: bash - :linenos: - - source activate lsst - source eups-setups.sh - setup lsst_distrib - -These commands can also be used to switch from one Conda environment and LSST Science Pipelines installation to another. - -.. _conda-install-test: - -4. Testing Your Installation -============================ - -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. diff --git a/install/demo.rst b/install/demo.rst index c3f10c85..b5df28ac 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -1,5 +1,5 @@ ###################################################### -Testing the Science Pipelines Installation with a Demo +Testing the Science Pipelines installation with a demo ###################################################### This demo will allow you to quickly test your LSST Science Pipelines installation, :doc:`regardless of your installation method `. @@ -10,15 +10,14 @@ This demo will allow you to quickly test your LSST Science Pipelines installatio Remember to first load the LSST Science Pipelines into your shell's environment. The method depends on how the Science Pipelines were installed: -- :ref:`Conda ` -- :ref:`newinstall.sh ` +- :doc:`newinstall.sh ` - :ref:`lsstsw ` -2. Download the Demo Project +2. Download the demo project ============================ Choose a directory to run the demo in. -We'll call this directory :file:`demo_data`. +For example: .. code-block:: bash @@ -29,13 +28,13 @@ Then download the demo's data (if you aren't running the current stable release, .. code-block:: bash - curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/13.0.tar.gz | tar xvzf - - cd lsst_dm_stack_demo-13.0 + curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/14.0.tar.gz | tar xvzf - + cd lsst_dm_stack_demo-14.0 .. note:: The demo's version should match your installed software. - If you installed from source (with :doc:`lsstsw `), clone the demo repository instead of downloading a release: + If you installed from source (with :doc:`lsstsw `) or with a :ref`newer tag `, clone the demo repository instead of downloading a release: .. code-block:: bash @@ -54,7 +53,7 @@ run camcol field filters *Filters in parentheses are not processed if run with the* ``--small`` *option, see below* -3. Run the Demo +3. Run the demo =============== Now setup the processing package and run the demo: @@ -66,11 +65,12 @@ Now setup the processing package and run the demo: For each input image the script performs the following operations: -* generate a subset of basic image characterization (e.g., determine photometric zero-point, detect sources, and measures positions, shapes, brightness with a variety of techniques), -* creates a :command:`./output` subdirectory containing subdirectories of configuration files, processing metadata, calibrated images, FITS tables of detected sources. These "raw" outputs are readable by other parts of the LSST pipeline, and -* generates a master comparison catalog in the working directory from the band-specific source catalogs in the ``output/sci-results/`` subdirectories. +- Generates a subset of basic image characterizations (such photometric zero-point estimation, source detection, and measurement). +- Creates a :file:`./output` directory containing subdirectories of configuration files, processing metadata, calibrated images, and FITS tables of detected sources. + These ``raw`` outputs are readable by other parts of the LSST pipeline. +- Generates a master comparison catalog in the working directory from the band-specific source catalogs in the ``output/sci-results/`` subdirectories. -4. Check the Demo Results +4. Check the demo results ========================= The demo will take a minute or two to execute (depending upon your machine), and will generate a large number of status messages. diff --git a/install/docker.rst b/install/docker.rst new file mode 100644 index 00000000..c68436c5 --- /dev/null +++ b/install/docker.rst @@ -0,0 +1,77 @@ +.. _docker: + +################### +Running with Docker +################### + +LSST provides versioned Docker images containing the Science Pipelines software. +With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. + +If you have issues, reach out on the `LSST Community support forum `_. + +.. _docker-prereqs: + +Prerequisites +============= + +To download Docker images and run containers, you need Docker's software. +The `Docker Community Edition `_ is freely available for most platforms, including macOS, Linux, and Windows. + +If you haven't used Docker before, you might want to learn more about Docker, images, and containers. +Docker's `Getting Started `_ documentation is a good resource. + +.. _docker-quick-start: + +Quick start +=========== + +This command downloads a weekly build of the LSST Science Pipelines, starts a container, and opens a prompt: + +.. code-block:: bash + + docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2017_35 + +Then in the container's shell, load the LSST environment and set up a :doc:`top-level package ` (``lsst_distrib`` in this case): + +.. code-block:: bash + + source /opt/lsst/software/stack/loadLSST.bash + setup lsst_distrib + +This step is equivalent to the :doc:`setup instructions ` for a :doc:`newinstall.sh `\ -based installation. +In fact, the images are internally based on :command:`newinstall.sh`. + +Next, learn more with these topics: + +- :ref:`docker-tags` + +.. _docker-tags: + +Finding images for different LSST Science Pipelines releases +============================================================ + +LSST Science Pipelines Docker images are published as `lsstsqre/centos`_ on Docker Hub. +These images are based on the CentOS base image. + +Docker images are versioned with tags, allowing you to run any release of the LSST Science Pipelines software. +The schema of these tags is: + +.. code-block:: text + + -stack-- + +For example: + +.. code-block:: text + + 7-stack-lsst_distrib-w_2017_35 + +This tag corresponds to: + +- CentOS 7 operating system. +- ``lsst_distrib`` :doc:`top-level package `. +- ``w_2017_35`` EUPS tag. See :ref:`newinstall-other-tags` for an overview of LSST's EUPS tag schema. + +You can see what tags are available by browsing `lsstsqre/centos on Docker Hub `_. + +.. _`lsstsqre/centos`: https://hub.docker.com/r/lsstsqre/centos/ diff --git a/install/index.rst b/install/index.rst index 9f96caf0..ccb2894f 100644 --- a/install/index.rst +++ b/install/index.rst @@ -2,43 +2,15 @@ Installing the LSST Science Pipelines ##################################### +.. This page acts as an index.html for the install/ path, but the currated contents are on the homepage. + .. toctree:: - :hidden: - - conda + :maxdepth: 1 + + prereqs/index newinstall + setup + top-level-packages + docker lsstsw demo - -We offer a few ways of installing the LSST Science Pipelines. -Choose an option below to get started. - -.. note:: - - To install the LSST Simulation software, such as MAF, please follow the `LSST Simulations documentation `_ instead of these guides. - -:doc:`Conda installation ` - Install the Pipelines as an `Anaconda/Miniconda `__ binary package. - This is the easiest installation option since no source compilation is required. - -:doc:`Source installation (newinstall.sh) ` - ``newinstall.sh`` allows you to build and install the LSST Science Pipelines from source. - You can use your existing Python 2.7, or opt to use a built-in `Miniconda `__. - -`CernVM FS (external link) `__ - CernVM FS is a virtual machine that makes it easy to run the LSST Science Pipelines without compiling code. - This distribution is supported by Fabio Hernandez of IN2P3. - -`Docker Containers (external link) `__ - These images are convenient for using the LSST Science Pipelines in distributed or cloud-based computing workflows, or for one-off installations without modifying your environment. You can get them from our `DockerHub repo `__ - -:doc:`lsstsw installation ` - ``lsstsw`` (and ``lsst-build``) are the tools we use internally to build and test the LSST Science Pipelines. - Use this installation option if you're interested in developing the Science Pipelines since ``lsstsw`` presents the Pipelines as a directory of cloned repositories from `github.com/lsst `__. - -Next, :doc:`try out your Science Pipelines installation by running a demo `. - -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. diff --git a/install/lsstsw.rst b/install/lsstsw.rst index 256fbb75..d4219db9 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -19,52 +19,38 @@ If you have difficulty installing LSST software: 1. Prerequisites ================ -You need to install some prerequisites to build the LSST Stack from source: +Before you begin: -- :ref:`Install build dependencies `. - `Install and configure Git LFS `_ for LSST DM's data servers. +- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. +- If you opt not to use ``lsstsw`` \’s default Python environment you need to :ref:`install these Python dependencies `. .. _lsstsw-deploy: -2. Obtaining a Development Stack with lsstsw -============================================ +2. Deploy lsstsw +================ -Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). -`lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. +.. Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). +.. `lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. Begin by choosing a working directory, then deploy ``lsstsw`` into it: .. code-block:: bash - :linenos: git clone https://github.com/lsst/lsstsw.git cd lsstsw ./bin/deploy - . bin/setup.sh - -The ``deploy`` script automates several things for you: + source bin/setup.sh -1. installs a miniconda_ Python environment specific to this lsstsw workspace, -2. installs EUPS_ in :file:`eups/current/`, -3. clones `lsst-build`_, which will run the build process for us, -4. clones versiondb_, a robot-made Git repository of package dependency information, and -5. creates an empty Stack *installation* directory, :file:`stack/`. - -``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. -Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. +For more information about the :command:`deploy` command, such as using Python 2.7 instead of 3, see :ref:`lsstsw-about-deploy`. .. The ``setup.sh`` step enables EUPS_, the package manager used by LSST. .. **Whenever you open a new terminal session, you need to run '. bin/setup.sh' to activate your lsstsw environment.** -.. _lsst-build: https://github.com/lsst/lsst_build -.. _versiondb: https://github.com/lsst/versiondb -.. _EUPS: https://github.com/RobertLuptonTheGood/eups -.. _miniconda: http://conda.pydata.org/miniconda.html - .. _lsstsw-rebuild: -3. Build Science Pipelines -========================== +3. Build the Science Pipelines packages +======================================= From the :file:`lsstsw` directory, run: @@ -74,111 +60,207 @@ From the :file:`lsstsw` directory, run: Once the ``rebuild`` step finishes, note the build number printed on screen. It is formatted as "``bNNNN``." -Tell EUPS this is the current build by making a clone of the build's EUPS tag and calling it "``current``:" +Tag this build as ``current`` so that EUPS can set it up by default: .. code-block:: bash eups tags --clone bNNNN current -The ``rebuild`` command accomplishes the following: - -1. Clones all Science Pipelines packages from `github.com/lsst `__. - A high-bandwidth connection is helpful since the stack contains a non-trivial amount of code and test data. -2. Runs the Scons-based build process to compile C++, make Swig bindings, and ultimately create the :lmod:`lsst` Python package. - The Stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. - Finally, set up the packages with EUPS: .. code-block:: bash setup lsst_distrib +See :doc:`setup` for more information. + +.. _lsstsw-testing-your-installation: + +4. Testing Your installation (optional) +======================================= + +Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. +This demo processes a small amount of SDSS data. + .. _lsstsw-setup: -4. Sourcing the Pipelines in a New Shell -======================================== +Sourcing the Pipelines in a new shell +===================================== -In every new shell session you will need to 'setup' the Science Pipelines. -Do this by running the ``setup.sh`` from the ``lsstsw/`` directory: +In every new shell session you will need to set up the Science Pipelines environment and EUPS package stack. -.. code-block:: bash +Run these two steps: - . bin/setup.sh - setup lsst_distrib # or an alternative top-level package +1. Activate the lsstsw software environment by sourcing the :file:`setup.sh` script in lsstsw's :file:`bin` directory: -.. note:: + .. code-block:: bash + + source bin/setup.sh - If you are using a tcsh shell, run ``. bin/setup.csh`` instead (note ``csh`` extension). + If you are running in a :command:`csh` or :command:`tcsh`, run this set up script instead: -.. _lsstsw-testing-your-installation: + .. code-block:: bash + + source bin/setup.csh -5. Testing Your Installation -============================ +2. Set up a :doc:`top-level package `: -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. + .. code-block:: bash + + setup lsst_distrib -.. _lsstsw-development: + Instead of ``lsst_distrib``, you can set up a different top-level package like ``lsst_apps`` or any individual EUPS package. + See :doc:`top-level-packages`. -6. Bonus: Developing a Package +.. _lsstsw-next: + +Next steps and advanced topics ============================== -An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. -The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. +- :ref:`lsstsw-about-deploy`. +- :ref:`lsstsw-py2`. +- :ref:`lsstsw-about-rebuild`. +- :ref:`lsstsw-branches`. +- :ref:`lsstsw-deploy-ref`. +- :ref:`lsstsw-rebuild-ref`. -1. Stack packages are found in the :file:`lsstsw/build/` directory. +.. _lsstsw-about-deploy: -2. Create a new branch in a package's Git repository, +About the lsstsw deploy script +------------------------------ - .. code-block:: bash +The ``deploy`` script automates several things to prepare an LSST development environment: - git checkout -b {{ticket-name}} +1. Installs Git. +2. Installs Git LFS (*you* are still responsible for `configuring it `_). +3. Installs a Miniconda_ Python environment specific to this lsstsw workspace. + The default Python is Python 3, but you can switch to Python 2.7 if necessary. + See :ref:`lsstsw-py2`. +4. Installs EUPS_ into :file:`eups/current/`. +5. Clones `lsst-build`_, which runs the build process. +6. Clones versiondb_, a robot-managed Git repository of package dependency information. +7. Creates an empty stack *installation* directory, :file:`stack/`. - Then declare this package for EUPS and set it up: +This environment, including the EUPS, Miniconda, Git, and Git LFS software, is only activated when you source the :file:`bin/setup.sh` or :file:`bin/setup.csh` scripts in a shell. +Otherwise, lsstsw does not affect the software installed on your computer. - .. code-block:: bash +See also: :ref:`lsstsw-deploy-ref`. - eups declare -r . -t $USER {{package_name}} git - setup -r . -t $USER - - Unpacking the ``eups declare`` arguments: - - - ``-r .`` is the path to the package's repository, which is the current working directory. - You don't *need* to be in the repository's directory if you provide the path appropriately. - - ``-t $USER`` sets the EUPS *tag*. - We use this because your username (``$USER``) is an allowed EUPS tag. - - ``git`` is used as an EUPS *version*. - Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. - - In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." - In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. - If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. - This is why we initially declared the entire lsstsw repository to have the version ``current``. +.. _lsstsw-py2: -3. Build the package with Scons: +How to use Python 2.7 +--------------------- - .. code-block:: bash - - scons -Q -j 6 opt=3 - - These flags tell SCons to build with flags: - - - ``-Q``: reduce logging to the terminal, - - ``-j 6``: build in parallel (e.g., with '6' CPUs), - - ``opt=3``: build with level 3 optimization. - - This ``scons`` command will run several targets by default, in sequence: - - 1. ``lib``: build the C++ code and SWIG interface layer - 2. ``python``: install the Python code - 3. ``tests``: run the test suite - 4. ``example``: compile the examples, - 5. ``doc``: compile Doxygen-based documentation, and - 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). - - You can build a subset of these targets by specifying one explicitly. - To simply compile C++, SWIG, build the Python package and run tests: - - .. code-block:: bash - - scons -q -j 6 opt=3 tests +The lsstsw :command:`deploy` script installs Miniconda_ as self-contained Python environment. +By default, :command:`deploy` installs a Python 3 version of Miniconda_. +For testing Python 2.7 compatibility, you can create an lsstsw deployment with a Python 2.7 version of Miniconda using the :option:`deploy -2` argument: + +.. code-block:: bash + + ./bin/deploy -2 + +.. _lsstsw-about-rebuild: + +About the lsstsw rebuild command +-------------------------------- + +The :command:`rebuild` command accomplishes the following: + +1. Clones all Science Pipelines packages from `github.com/lsst `__. + :command:`rebuild` + +2. Runs the Scons-based build process to compile C++, make Pybind11 bindings, and ultimately create the :lmod:`lsst` Python package. + The stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. + +``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. +Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. + +See also: :ref:`lsstsw-rebuild-ref`. + +.. _lsstsw-branches: + +Building from branches +---------------------- + +lsstsw's :command:`rebuild` enables you to clone and build development branches. + +To build ``lsst_distrb``, but use the Git branch ``my-feature`` when it's available in a package's repository: + +.. code-block:: bash + + rebuild -r my-feature lsst_distrib + +Multiple ticket branches across multiple products can be built in order of priority: + +.. code-block:: bash + + rebuild -r feature-1 -r feature-2 lsst_distrib + +In this example, a ``feature-1`` branch will be used in any package's Git repository. +A ``feature-2`` branch will be used secondarily in repositories where ``feature-1`` doesn't exist. +Finally, ``lsstsw`` falls back to using the ``master`` branch for repositories that lack both ``feature-1`` and ``feature-2``. + +.. _lsstsw-deploy-ref: + +lsstsw deploy command reference +------------------------------- + +.. program:: deploy + +.. code-block:: text + + usage: deploy.sh [-2|-3] [-b] [-h] + +.. option:: -2 + + Install a Python 2-based Miniconda_. + +.. option:: -3 + + Use a Python 3-based Miniconda_ (default). + +.. option:: -b + + Use bleeding-edge conda packages. + +.. option:: -h + + Print the help message. + +.. _lsstsw-rebuild-ref: + +lsstsw rebuild command reference +-------------------------------- + +.. program:: rebuild + +.. code-block:: text + + rebuild [-p] [-n] [-u] [-r [-r [...]]] [-t ] [product1 [product2 [...]]] + +.. option:: -p + + Prep only. + +.. option:: -n + + Do not run :command:`git fetch` in already-downloaded repositories. + +.. option:: -u + + Update the :file:`repos.yaml` package index to the ``master`` branch on GitHub of https://github.com/lsst/repos. + +.. option:: -r + + Rebuild using the Git ref. + A Git ref can be a branch name, tag, or commit SHA. + Multiple ``-r`` arguments can be given, in order or priority. + +.. option:: -t + + EUPS tag. + +.. _lsst-build: https://github.com/lsst/lsst_build +.. _versiondb: https://github.com/lsst/versiondb +.. _EUPS: https://github.com/RobertLuptonTheGood/eups +.. _Miniconda: http://conda.pydata.org/miniconda.html diff --git a/install/newinstall.rst b/install/newinstall.rst index 1796ec6c..87083f7c 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -1,264 +1,433 @@ -###################################### -Source Installation with newinstall.sh -###################################### +########################################### +Install with newinstall.sh and eups distrib +########################################### -This page will guide you through installing the LSST Science Pipelines from source with :command:`newinstall.sh` (internally based on :command:`eups distrib`). +This page guides you through installing the LSST Science Pipelines software. +This installation method is recommended for anyone who uses or develops the Pipelines software. -The reference platform for the LSST Science Pipelines is CentOS 7 however individual developers compile on `a variety of Linux and macOS operating systems `_ so if you are on a similar platform you should be able to build from source successfully. +If you have installation issues, here are some ways to get help: -:doc:`We also offer Conda binaries and Docker images ` if you do not wish to install the Science Pipelines from source. +- Review the :ref:`known installation issues `. +- Ask a question in the `LSST Community support forum `_. -If you have difficulty installing LSST software: +.. _newinstall-prereqs: -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. +1. Prerequisites +================ -.. _source-install-prereqs: +You can install the LSST Science Pipelines on CentOS 7 (LSST's reference platform) as well as other Linux distributions and macOS (see the `LSST Stack Testing Status `_ for information on the platforms we have tested with). -Prerequisites -============= +Before you begin, install prerequisite software for your platform: -This section lists system prerequisites for :ref:`macOS `, :ref:`Debian/Ubuntu `, and :ref:`RedHat/CentOS ` platforms. -All platforms also need :ref:`Python package dependencies ` listed here. +- :doc:`macOS ` +- :doc:`Centos / RedHat ` +- :doc:`Debian / Ubuntu ` -.. note:: +.. + TK recommended memory, disk space, and build time. + +.. _newinstall-source-dir: - **New since 11.0**: The minimum gcc version required to compile the Stack is **gcc 4.8.** - If you using our previous factory platform, RedHat/CentOS 6, and you are unable to upgrade to version 7 (which comes with gcc 4.8 as default) consult :ref:`the section below on upgrading compilers in legacy Linux `. +2. Make an installation directory +================================= -.. _source-install-mac-prereqs: +First, choose where you want to install the LSST Science Pipelines. +Create and change into that directory: -macOS ------ +.. code-block:: bash -To build LSST software, macOS systems need: + mkdir -p lsst_stack + cd lsst_stack -1. :ref:`Xcode `, or command line tools. -2. :ref:`cmake `. +If you are installing the software for multiple users (a shared stack), see :ref:`newinstall-shared-permissions`. -Versions prior to OS X 10.9 and earlier have not been tested recently and may not work. +.. _newinstall-run: -.. _source-install-mac-prereqs-xcode: +3. Run newinstall.sh +==================== -Xcode -^^^^^ +Open a new shell session for the installation (or ensure you haven't used LSST software in that shell before). +If you need to reuse a shell, see :ref:`newinstall-unset-variables`. -You will need to install developer tools, which we recommend you obtain with Apple's Xcode command line tools package. -To do this, run from the command line (e.g. ``Terminal.app`` or similar): +Next, run :command:`newinstall.sh` to set up the environment you'll install the LSST Science Pipelines into. +For most use cases we recommend downloading and running :command:`newinstall.sh` like this: .. code-block:: bash - xcode-select --install + curl -OL https://raw.githubusercontent.com/lsst/lsst/14.0/scripts/newinstall.sh + bash newinstall.sh -ct + +Always execute `newinstall.sh` with bash, as shown, regardless of what shell you are in. -and follow the on-screen instructions. -You can verify where the tools are installed by running: +We recommend that you opt into the provided Miniconda Python environment (see the links below for more information). + +Then load this environment into your shell: + +.. TODO Use sphinx-tabs here? .. code-block:: bash - xcode-select -p + source loadLSST.bash # for bash + source loadLSST.csh # for csh + source loadLSST.ksh # for ksh + source loadLSST.zsh # for zsh + +.. note:: + + Here are ways to customize the :command:`newinstall.sh` installation for specific needs: -.. _source-install-mac-prereqs-cmake: + - The default Python environment is Python 3.5. + **Python 2.7 users,** see :ref:`newinstall-py2`. + - :ref:`newinstall-user-python`. + - The recommended installation uses precompiled binary tarballs if available for your platform (and falls back to a source build). + See :ref:`newinstall-binary-packages`. + If you will be compiling and linking C++ code against this installation you'll need to ensure your compilers match the distribution's. + **Developers should review** :ref:`newinstall-binary-compatibility`. -cmake -^^^^^ + For background information about :command:`newinstall.sh`, see: -``cmake`` can be `installed directly `__, or though a package manager like `Homebrew `__. + - :ref:`newinstall-background`. + - :ref:`newinstall-miniconda`. + - :ref:`newinstall-reference`. -.. _source-install-debian-prereqs: +.. _newinstall-install: -Debian / Ubuntu ---------------- +4. Install Science Pipelines packages +===================================== + +Install the LSST Science Pipelines packages by running :command:`eups distrib install` for a top-level package and a tagged version. + +This example installs the ``v14_0_rc1`` tagged version (current release) of the ``lsst_distrib`` top-level package: .. code-block:: bash - apt-get install bison ca-certificates \ - cmake flex g++ gettext git libbz2-dev \ - libfontconfig1 libglib2.0-dev libncurses5-dev \ - libreadline6-dev libssl-dev libx11-dev libxrender1 \ - libxt-dev m4 openjdk-8-jre \ - perl-modules zlib1g-dev \ + eups distrib install lsst_distrib -t v14_0_rc1 lsst_distrib + curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python + setup lsst_distrib +If pre-built binaries are available for your platform (and you ran :command:`newinstall.sh` with the :option:`-t ` argument) the installation should take roughly 10 minutes. +Otherwise, installation falls back to a source build which can take two hours, depending on the top-level package and your machine's performance. -.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp +The last command, :command:`setup`, activates the installed packages in your shell environment. +You'll need to run :command:`setup` in each shell session you'll use the LSST Science Pipelines in. +See :doc:`setup` for more information. -Prefix the :command:`apt-get` command with :command:`sudo` if necessary. +.. note:: -.. _source-install-redhat-prereqs: + - ``lsst_distrib`` is a top-level package that brings-in most LSST Data Management pipelines software, but other top-level packages may be more applicable for your work, such as ``lsst_apps`` or ``lsst_sims``. + See :doc:`top-level-packages` for more information. -RedHat / CentOS ---------------- + - ``v14_0_rc1`` is the current release. + You can install other tagged versions of the LSST Science Pipelines, though. + See :ref:`newinstall-other-tags`. -.. code-block:: bash + - If you're curious about the shebangtron, see its repository at `github.com/lsst/shebrangtron `_. - yum install bison curl blas bzip2-devel bzip2 flex fontconfig \ - freetype-devel gcc-c++ gcc-gfortran git libuuid-devel \ - libXext libXrender libXt-devel make openssl-devel patch perl \ - readline-devel tar zlib-devel ncurses-devel cmake glib2-devel \ - java-1.8.0-openjdk gettext perl-ExtUtils-MakeMaker +.. _newinstall-test: -.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp +5. Test your installation (optional) +==================================== -Prefix the :command:`yum` command with :command:`sudo` if necessary. +Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo pipeline `. +This demo processes a small amount of SDSS data and verifies that measurements match expected values. -.. _source-install-redhat-legacy: +See :doc:`demo` for instructions. -Upgrading compilers for legacy RedHat / CentOS 6 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. _newinstall-next-steps: -The minimum gcc version required to compile the Stack is gcc 4.8. -This comes as standard in the LSST "factory" platform, Red Hat / CentOS 7. +Next steps +========== -On our previous factory platform, Red Hat / CentOS 6, you will need to use a more current version of gcc that what is available with your system. -If you can go to Red Hat 7, we recommend that you do; if you cannot, we recommend that you use a newer gcc version for the stack by using a Software Collection (SCL) with a different version of devtoolset. -This will enable you to safely use a different version of gcc (4.9) for the stack than that used by your operating system (4.4). +Add links to topics the reader should go to next. +Examples: -First, install ``devtoolset-3`` (after the :ref:`installing the standard pre-requisites (above) `): +- A topic reminding a user to setup the stack in a new shell. +- Topic explaining top-level packages. +- Topic on compiling a package alongside an existing stack. -.. code-block:: bash +Advanced installation topics +============================ - sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm - sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/rh-java-common/epel-6-x86_64/download/rhscl-rh-java-common-epel-6-x86_64.noarch.rpm - sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/devtoolset-3/epel-6-x86_64/download/rhscl-devtoolset-3-epel-6-x86_64.noarch.rpm - sudo yum install -y scl-utils - sudo yum install -y devtoolset-3 +The above steps guided you through LSST's recommended installation. +These topics provide additional information about the installation and ways to customize it: -Then enable ``devtoolset-3`` by including this line in your :file:`~/.bash_profile`: +- :ref:`newinstall-shared-permissions`. +- :ref:`newinstall-unset-variables`. +- :ref:`newinstall-background`. +- :ref:`newinstall-miniconda`. +- :ref:`newinstall-py2`. +- :ref:`newinstall-user-python`. +- :ref:`newinstall-binary-packages`. +- :ref:`newinstall-binary-compatibility`. +- :ref:`newinstall-other-tags`. +- :ref:`newinstall-reference`. -.. code-block:: bash +.. _newinstall-shared-permissions: - scl enable devtoolset-3 bash +Setting unix permissions for shared installations +------------------------------------------------- -.. _source-install-py-deps: +You can make the LSST Science Pipelines installation accessible to multiple users on the same machine. -Python dependencies -------------------- +First, create a separate unix group (called ``lsst``, for example) with a ``umask`` of ``002`` (all access permissions for the group and allow other users to read/execute). -You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. +Then set the ownership of the installation directory to the ``lsst`` group, have the ``SGID`` (2000) bit set, and allow group read/write/execute (mode 2775). -.. _source-install-optional-deps: +.. _newinstall-unset-variables: -Optional dependencies ---------------------- +Running newinstall.sh in an already set-up shell +------------------------------------------------ -Although not required, we recommend you install the `matplotlib `_ and `scipy `_ Python packages: +If you've run the LSST Science Pipelines previously, you may have conflicting environment variables in your shell. +To be safe, run: .. code-block:: bash - pip install -U matplotlib scipy + unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH -.. FIXME +Then return to the instructions step :ref:`newinstall-run`. -Note these are included by default in `Anaconda `__, which :command:`newinstall.sh` *can* obtain for you. +.. _newinstall-background: -We also use `SAOImage DS9 `_ to display images for debugging. +What newinstall.sh does +----------------------- -.. _install-from-source: +:command:`newinstall.sh` creates a self-contained environment on your machine where you can install, run, and develop the LSST Science Pipelines. +You activate this environment in a shell by sourcing the :command:`loadLSST` script in the installation directory (see :ref:`setup-howto`). -Installing from Source with newinstall.sh -========================================= +Here is how :command:`newinstall.sh` prepares the environment: -This section will guide you through installing the *current* release of the LSST Science Pipelines from source given that prerequisites have been installed. +- Identifies your operating system and compilers to determine what EUPS binary packages should be installed (the EUPS package root, see :ref:`newinstall-binary-packages`). +- Installs a specific version of Python, through Miniconda, that is compatible with EUPS binary packages (see :ref:`newinstall-miniconda`). +- Installs Conda packages that the LSST Science Pipelines depends on (see :ref:`python-deps`). +- Checks for :command:`git` on your systems and offers to install it if necessary. +- Installs EUPS_, the package manager used by the LSST software stack. -.. _install-from-source-dir: +For information about :command:`newinstall.sh`\ ’s arguments, see :ref:`newinstall-reference`. -1. Choose an installation directory ------------------------------------ +.. _newinstall-miniconda: -First, choose where you want to install the LSST Science Pipelines. -We'll use :file:`$HOME/lsst_stack` in this example. -Create and change into that directory: +About the Miniconda Python installed by newinstall.sh +----------------------------------------------------- -.. code-block:: bash +:command:`newinstall.sh` can install a dedicated Python environment for your LSST Science Pipelines installation. +This Python installation isn't required, but we recommend it. +See :ref:`newinstall-user-python` if required. - mkdir -p $HOME/lsst_stack - cd $HOME/lsst_stack +The Python environment installed by :command:`newinstall.sh` is Miniconda_, a minimal version of Anaconda_. +By default, :command:`newinstall.sh` installs Python 3.5.2. +If you need to work with your own Python 2.7-only packages, see :ref:`newinstall-py2`. -Installation for groups -^^^^^^^^^^^^^^^^^^^^^^^ - -Those in a system administration role, who are installing a writable stack for multiple users, will likely want to establish a separate group (perhaps lsst) with a umask of 002 (all access permissions for the group; allow other users to read+execute). -The installation directory must be owned by the group, have the SGID (2000) bit set, and allow group read/write/execute: that is, mode 2775. -Individual users who install a personal Stack on their own machine need not worry about this. +In this Miniconda environment, :command:`newinstall.sh` installs the Science Pipeline's Python prerequisites. +See :ref:`python-deps` for more information. -.. _install-from-source-envvar: +This Miniconda installation won't affect your other Python installations (like the system's Python, your own Anaconda or Miniconda, or virtual environments). +The LSST Miniconda environment is only active when you source the ``loadLSST`` script installed by :command:`newinstall.sh` (see :doc:`setup`). -2. Unset environment variables ------------------------------- +If you install other Python packages in a shell where the LSST Miniconda is activated (with :command:`pip install` or :command:`conda install`) those packages are installed into the LSST Miniconda's :file:`site-packages`, not your system's. +The Python installed by :command:`newinstall.sh` works like an isolated Python environment dedicated to LSST Science Pipelines code and your own related modules---effectively like a `Conda environment `_ or Python `venv `_. +This pattern is useful because it reduces the risk of having Python package version incompatibilities. -If you've run the LSST Science Pipelines previously, you may have conflicting environment variables setup. -To be safe, run: +.. _newinstall-py2: + +How to install a Python 2.7 environment with newinstall.sh +---------------------------------------------------------- + +LSST Science Pipelines is backwards compatible with Python 2.7. +If you need to run your own Python 2.7-only Python packages in conjunction with the Pipelines, you can have :command:`newinstall.sh` install a Python 2.7 environment for you instead of the default Python 3.5 environment. + +To select Python 2.7, run :command:`newinstall.sh` with the :option:`-2 ` flag (in addition to other flags, like :option:`-t `): .. code-block:: bash - unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH + bash newinstall.sh -2 -.. _install-from-source-setup: +Then follow the remaining instructions at :ref:`newinstall-run`. -3. Installation set-up ----------------------- +See also: :ref:`newinstall-miniconda`. -Download and run the `installation setup script from GitHub `__, which installs the basic packages required to install other packages: +.. _newinstall-user-python: -.. code-block:: bash +How to use your own Python with newinstall.sh +--------------------------------------------- - curl -OL https://raw.githubusercontent.com/lsst/lsst/13.0/scripts/newinstall.sh - bash newinstall.sh +:command:`newinstall.sh` creates a new Python environment by default (pre-configured with Python dependencies). +If necessary, you can use your own pre-existing Python environment. -This installs the :command:`loadLSST.*` scripts, which you should source to ensure that LSST tools (e.g., the :command:`eups` command) are included in your path. +To do so, run :command:`newinstall.sh` (see :command:`newinstall-run` for details and command arguments). +When :command:`newinstall-run` prompts you to install Miniconda, type ``no``. -The install script will check your system to ensure that appropriate versions of critical packages are installed on your system, to enable bootstrapping the Science Pipelines, including :command:`git`, and :command:`python`. -If these packages are not available, the script will offer to install them for you (using the Anaconda Python distribution for the latter packages). +Be aware of these caveats when using your own Python installation: -Allowing the installation of these core packages will not replace or modify any other version of these packages that may be installed on your system. -If you do not choose the Anaconda Python install, and subsequent package build steps fail, you can do one of two things: +- You are responsible for installing Python package dependencies. + See :ref:`python-deps`. -* Report the problem to `community.lsst.org `_. Include your OS, a description of the problem, plus any error messages. Community members will provide assistance. -* Consider removing all contents of the install directory and start from scratch, and accepting the Anaconda Python installation option. +- Prebuilt binaries will not be available. + :command:`eups distrib install` will always install from source. -Once :command:`newinstall.sh` has finished, source the LSST environment to continue the installation by running the appropriate command for your shell: +.. _newinstall-binary-packages: -.. code-block:: bash +About EUPS tarball packages +--------------------------- + +EUPS distrib binary (tarball) packages significantly speed up your installation. +Rather than compiling the LSST Science Pipelines from source, EUPS tarballs are prebuilt packages made specifically for supported platforms. + +Platforms are defined by four factors: - source $LSST_INSTALL_DIR/loadLSST.bash # for bash users - source $LSST_INSTALL_DIR/loadLSST.csh # for csh users - source $LSST_INSTALL_DIR/loadLSST.ksh # for ksh users - source $LSST_INSTALL_DIR/loadLSST.zsh # for zsh users +1. Operating system. +2. Compiler. +3. Miniconda_ (Python) version. +4. lsstsw_ version (Git ref). -where :file:`$LSST_INSTALL_DIR` is expanded to your installation directory. +EUPS distrib binary packages are currently being built for these platform combinations: -.. _install-from-source-packages: +.. csv-table:: EUPS distrib binary flavors + :header: "OS","Compiler","Python" -4. Install packages -------------------- + "macOS ``osx/10.9``", "``clang-800.0.42.1``", "``miniconda3-4.2.12`` (Python 3)" + "macOS ``osx/10.9``", "``clang-800.0.42.1``", "``miniconda2-4.2.12`` (Python 2)" + "Redhat ``redhat/el7``", "``gcc-system``", "``miniconda3-4.2.12`` (Python 3)" + "Redhat ``redhat/el7``", "``gcc-system``", "``miniconda2-4.2.12`` (Python 2)" + "Redhat ``redhat/el6``", "``devtoolset-3``", "``miniconda2-4.2.12`` (Python 2)" -Finally, install components of the LSST Science Pipelines that are relevant for your work. -A simple way to ensure that you have a fairly complete set of packages for this need is to install ``lsst_distrib``: +When you run :command:`newinstall.sh`, it looks at your system to identify your operating system and compiler. +The version of :command:`newinstall.sh` you run also determines the Miniconda_ version and the lsstsw_ build system versions. + +Together, these four factors define the URL prefix (called an *EUPS package root*) that :command:`eups distrib install` looks for binary packages from. +If binary tarballs are unavailable for that EUPS package root, :command:`eups distrib install` automatically falls back to compiling LSST Science Pipelines packages from source. + +You can see the active EUPS package roots on your system by running: .. code-block:: bash - eups distrib install -t v13_0 lsst_distrib - setup lsst_distrib + eups distrib path + +Here is an example of the output: + +.. code-block:: text -After this initial setup, it is a good idea to test the installation. -See :ref:`source-install-testing-your-installation`. + https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67 + https://eups.lsst.codes/stack/src -.. _install-from-source-loadlsst: +Based on this example, :command:`eups distrib install` will preferentially install EUPS distrib binary packages for the macOS 10.9 system, ``clang-800.0.42.1`` compiler, and ``miniconda3-4.2.12-7c8e67`` Python and lsstsw combination. +If :command:`eups distrib install` cannot find packages at that EUPS package root it will look in the second EUPS package root (https://eups.lsst.codes/stack/src), which provides source packages. -5. Source the LSST environment in each shell session ----------------------------------------------------- +.. _newinstall-binary-compatibility: -Whenever you want to run the installed LSST Science Pipelines in a new terminal session, be sure to :command:`source` the appropriate :file:`loadLSST.bash`, :file:`loadLSST.csh`, :file:`loadLSST.ksh` or :file:`loadLSST.zsh}` script. +EUPS tarball packages and compiler compatibility +------------------------------------------------ -Then setup the EUPS packages you need, typically: +EUPS binary tarball packages are prebuilt on LSST's continuous integration servers for a specific combination of operating system, compilers, Python, and Python dependencies. +If you are developing packages alongside this installation, you might encounter application binary interface (ABI) incompatibilities if are using a different compiler version or a different Python environment. + +In this case, the more reliable solution is to revert to a source installation. +To do this, repeat the installation but run :command:`newinstall.sh` *without* the :option:`-t ` argument: .. code-block:: bash - setup lsst_distrib + bash newinstall.sh -c + +Without the :option:`-t ` argument to :command:`newinstall.sh`, :command:`eups distrib install` will always build and install packages from source, ensuring ABI compatibility. + +.. _newinstall-other-tags: + +Installing other releases (including daily and weekly tags) +----------------------------------------------------------- + +The instructions on this page guide you through installing the current release of the LSST Science Pipelines corresponding to this documentation. +You can, however, install other releases by running :command:`eups distrib install` with a different tag. + +The common types of tags are: + +- **Major releases,** tagged as ``v_`` (for example, ``v14_0``). +- **Weekly builds,** tagged as ``w__`` (for example, ``w_2017_33`` is the 33rd weekly build in 2017). +- **Daily builds,** tagged as ``d___`` (for example, ``d_2017_09_01`` is the daily build for September 1, 2017). + +There are also tags pointing to the most recent releases: + +- **Current major release,** tagged as ``current``. +- **Current weekly build,** tagged as ``w_latest``. +- **Current daily build,** tagged as ``d_latest``. + +You can see all available tags at https://eups.lsst.codes/stack/src/tags (each tag has a ``.list`` file). + +.. note:: + + Binary installations may not be available for all tags. + From https://eups.lsst.codes/stack, browse subdirectories corresponding to your platform and look for ``.list`` files of available tags. + :command:`eups distrib install` automatically falls back to a source build if binaries are not available. + +.. warning:: + + You need to ensure that the Python environment created by :command:`newinstall.sh` (see step :ref:`newinstall-run`) is compatible with the tagged software. + + For example, if you are installing a recent weekly you may need to download and run :command:`newinstall.sh` from master: + + .. code-block:: bash + + curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh + bash newinstall.sh -ct + + See https://github.com/lsst/lsst/tags for available tagged versions of :command:`newinstall.sh`. + +.. _newinstall-reference: + +newinstall.sh argument reference +-------------------------------- + +.. program:: newinstall.sh + +.. code-block:: text + + usage: newinstall.sh [-b] [-f] [-h] [-n] [-3|-2] [-t|-T] [-s|-S] [-P ] + +.. option:: -b + + Run in batch mode. Don't ask any questions and install all extra packages. + +.. option:: -c + + Attempt to continue a previously failed install. + +.. option:: -n + + No-op. Go through the motions but echo commands instead of running them. + +.. option:: -P + + Use a specific python interpreter for EUPS. + +.. option:: -2 + + Use Python 2 if the script is installing its own Python. + +.. option:: -3 + + Use Python 3 if the script is installing its own Python. (**default**) + +.. option:: -t + + Use pre-compiled EUPS "tarball" packages, if available. + +.. option:: -T + + **Do not** use pre-compiled EUPS "tarball" packages. (**default**) + +.. option:: -s + + Use EUPS source "eupspkg" packages, if available (compile from source). + +.. option:: -S + + **Do not** use EUPS source "eupspkg" packages (do not compile from source). + +.. option:: -h -.. _source-install-testing-your-installation: + Display this help message. -Testing Your Installation -========================= -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. +.. _Miniconda: https://conda.io/miniconda.html +.. _Anaconda: https://docs.anaconda.com +.. _lsstsw: https://github.com/lsst/lsstsw +.. _EUPS: https://github.com/RobertLuptonTheGood/eups diff --git a/install/package-development.rst b/install/package-development.rst new file mode 100644 index 00000000..873379f3 --- /dev/null +++ b/install/package-development.rst @@ -0,0 +1,67 @@ +:orphan: 1 + +.. NOTE: this page is not currently part of the built site, though it is intended for re-integration. + +############################################################### +Developing a package with the installed Science Pipelines stack +############################################################### + +An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. +The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. + +1. Stack packages are found in the :file:`lsstsw/build/` directory. + +2. Create a new branch in a package's Git repository, + + .. code-block:: bash + + git checkout -b {{ticket-name}} + + Then declare this package for EUPS and set it up: + + .. code-block:: bash + + eups declare -r . -t $USER {{package_name}} git + setup -r . -t $USER + + Unpacking the ``eups declare`` arguments: + + - ``-r .`` is the path to the package's repository, which is the current working directory. + You don't *need* to be in the repository's directory if you provide the path appropriately. + - ``-t $USER`` sets the EUPS *tag*. + We use this because your username (``$USER``) is an allowed EUPS tag. + - ``git`` is used as an EUPS *version*. + Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. + + In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." + In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. + If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. + This is why we initially declared the entire lsstsw repository to have the version ``current``. + +3. Build the package with Scons: + + .. code-block:: bash + + scons -Q -j 6 opt=3 + + These flags tell SCons to build with flags: + + - ``-Q``: reduce logging to the terminal, + - ``-j 6``: build in parallel (e.g., with '6' CPUs), + - ``opt=3``: build with level 3 optimization. + + This ``scons`` command will run several targets by default, in sequence: + + 1. ``lib``: build the C++ code and SWIG interface layer + 2. ``python``: install the Python code + 3. ``tests``: run the test suite + 4. ``example``: compile the examples, + 5. ``doc``: compile Doxygen-based documentation, and + 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). + + You can build a subset of these targets by specifying one explicitly. + To simply compile C++, SWIG, build the Python package and run tests: + + .. code-block:: bash + + scons -q -j 6 opt=3 tests diff --git a/install/prereqs/centos.rst b/install/prereqs/centos.rst new file mode 100644 index 00000000..950bcc7c --- /dev/null +++ b/install/prereqs/centos.rst @@ -0,0 +1,50 @@ +.. _source-install-redhat-prereqs: + +#################################### +RedHat / CentOS system prerequisites +#################################### + +.. code-block:: bash + + yum install bison curl blas bzip2-devel bzip2 flex fontconfig \ + freetype-devel gcc-c++ gcc-gfortran git libuuid-devel \ + libXext libXrender libXt-devel make openssl-devel patch perl \ + readline-devel tar zlib-devel ncurses-devel cmake glib2-devel \ + java-1.8.0-openjdk gettext perl-ExtUtils-MakeMaker + +.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp + +Prefix the :command:`yum` command with :command:`sudo` if necessary. + +.. note:: + + **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.** + If you using our previous factory platform, RedHat/CentOS 6, and you are unable to upgrade to version 7 (which comes with gcc 4.8 as default) consult :ref:`the section below on upgrading compilers in legacy Linux `. + +.. _source-install-redhat-legacy: + +Upgrading compilers for legacy RedHat / CentOS 6 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The minimum gcc version required to compile the Stack is gcc 4.8. +This comes as standard in the LSST "factory" platform, Red Hat / CentOS 7. + +On our previous factory platform, Red Hat / CentOS 6, you will need to use a more current version of gcc that what is available with your system. +If you can go to Red Hat 7, we recommend that you do; if you cannot, we recommend that you use a newer gcc version for the stack by using a Software Collection (SCL) with a different version of devtoolset. +This will enable you to safely use a different version of gcc (4.9) for the stack than that used by your operating system (4.4). + +First, install ``devtoolset-3`` (after the :ref:`installing the standard pre-requisites (above) `): + +.. code-block:: bash + + sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm + sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/rh-java-common/epel-6-x86_64/download/rhscl-rh-java-common-epel-6-x86_64.noarch.rpm + sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/devtoolset-3/epel-6-x86_64/download/rhscl-devtoolset-3-epel-6-x86_64.noarch.rpm + sudo yum install -y scl-utils + sudo yum install -y devtoolset-3 + +Then enable ``devtoolset-3`` by including this line in your :file:`~/.bash_profile`: + +.. code-block:: bash + + scl enable devtoolset-3 bash diff --git a/install/prereqs/debian.rst b/install/prereqs/debian.rst new file mode 100644 index 00000000..dbc39b67 --- /dev/null +++ b/install/prereqs/debian.rst @@ -0,0 +1,24 @@ +.. _source-install-debian-prereqs: + +#################################### +Debian / Ubuntu system prerequisites +#################################### + +Debian or Ubuntu systems require the following packages: + +.. code-block:: bash + + apt-get install bison ca-certificates \ + cmake flex g++ gettext git libbz2-dev \ + libfontconfig1 libglib2.0-dev libncurses5-dev \ + libreadline6-dev libssl-dev libx11-dev libxrender1 \ + libxt-dev m4 openjdk-8-jre \ + perl-modules zlib1g-dev \ + +.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp + +Prefix the :command:`apt-get` command with :command:`sudo` if necessary. + +.. note:: + + **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.** diff --git a/install/prereqs/index.rst b/install/prereqs/index.rst new file mode 100644 index 00000000..c0717d7a --- /dev/null +++ b/install/prereqs/index.rst @@ -0,0 +1,37 @@ +############# +Prerequisites +############# + +This page lists software needed to install and use the LSST Science Pipelines. + +.. _system-prereqs: + +System prerequisites +==================== + +.. toctree:: + :maxdepth: 1 + + macos + debian + centos + +.. _python-deps: + +Python dependencies +=================== + +Both the :doc:`newinstall.sh <../newinstall>` and :doc:`lsstsw <../lsstsw>`\ -based installation methods provide dedicated Miniconda environments pre-loaded with Python dependencies. +If you opt to use your own Python, you can re-create the default Python environment made by :command:`newinstall.sh` and ``lsstsw`` with these Conda environments: + +- `macOS and Python 2.7 `_. +- `macOS and Python 3.5+ `_. +- `Linux and Python 2.7 `_. +- `Linux and Python 3.5+ `_. + +.. _optional-deps: + +Optional dependencies +===================== + +We use `SAOImage DS9 `_ to display images for debugging. diff --git a/install/prereqs/macos.rst b/install/prereqs/macos.rst new file mode 100644 index 00000000..89471b7c --- /dev/null +++ b/install/prereqs/macos.rst @@ -0,0 +1,49 @@ +.. _source-install-mac-prereqs: + +########################## +macOS system prerequisites +########################## + +To build LSST software, macOS systems need: + +1. :ref:`Xcode `, or command line tools. +2. :ref:`cmake `. + +.. note:: + + macOS versions 10.9 and earlier have not been tested recently and may not work. + +.. _source-install-mac-prereqs-xcode: + +Xcode +===== + +You will need to install developer tools, which we recommend you obtain with Apple's Xcode command line tools package. +To do this, run from the command line (e.g. ``Terminal.app`` or similar): + +.. code-block:: bash + + xcode-select --install + +You can verify where the tools are installed by running: + +.. code-block:: bash + + xcode-select -p + +.. _source-install-mac-prereqs-cmake: + +cmake +===== + +``cmake`` can be installed through a package manager like `Homebrew `_: + +.. code-block:: bash + + brew install cmake + +**Alternatively,** `Anaconda `_ users can install ``cmake`` with :command:`conda`: + +.. code-block:: bash + + conda install cmake diff --git a/install/setup.rst b/install/setup.rst new file mode 100644 index 00000000..df54ca1f --- /dev/null +++ b/install/setup.rst @@ -0,0 +1,56 @@ +.. _setup: + +########################################### +Setting up installed LSST Science Pipelines +########################################### + +Whenever you start a new command-line shell you need to set up the LSST Science Pipelines software before you can use it. + +.. _setup-howto: + +Setting up +========== + +Setting the LSST Science Pipelines in a shell is a two-step process: + +1. Load the LSST environment by sourcing the ``loadLSST`` script in your installation directory: + + .. TODO Use sphinx-tabs here? + + .. code-block:: bash + + source loadLSST.bash # for bash + source loadLSST.csh # for csh + source loadLSST.ksh # for ksh + source loadLSST.zsh # for zsh + + .. note:: + + These installation are for :doc:`newinstall.sh `-based installations. + For ``lsstsw``, follow :ref:`these instructions ` instead. + +2. Set up a top-level package: + + .. code-block:: bash + + setup + + For example, ``setup lsst_apps`` or ``setup lsst_distrib``. + See :doc:`top-level-packages` for more about LSST's top-level packages. + +.. _setup-list: + +Listing what packages are set up +================================ + +To see what packages (and their versions) are currently set up: + +.. code-block:: bash + + setup list -s + +To see all packages that are installed, even if not currently set up, run: + +.. code-block:: bash + + setup list diff --git a/install/top-level-packages.rst b/install/top-level-packages.rst new file mode 100644 index 00000000..ae33acc7 --- /dev/null +++ b/install/top-level-packages.rst @@ -0,0 +1,48 @@ +############################# +Top-level packages to install +############################# + +The LSST Science Pipelines are part of LSST's EUPS_ package stack. +This means that the Science Pipelines software is actually a collection of packages that you install and setup together. +By specifying different **top-level packages** to the :ref:`eups distrib install ` and :doc:`setup ` commands, you can control the size of the software installation or add new capabilities. + +This page describes the common top-level packages that make up the LSST Science Pipelines and related EUPS stacks. + +lsst\_apps +========== + +``lsst_apps`` provides the core packages that LSST will use in operations. + +Example installation (:ref:`more info `): + +.. code-block:: bash + + eups distrib install lsst_apps -t + setup lsst_apps + +lsst\_distrib +============= + +``lsst_distrib`` provides all the packages from ``lsst_apps`` along with these additional ones: + +.. https://github.com/lsst/lsst_distrib/blob/master/ups/lsst_distrib.table + +- `ctrl\_execute `_ +- `ctrl\_platform_lsstvc `_ +- `datarel `_ +- `meas\_extensions_convolved `_ +- `meas\_extensions_shapeHSM `_ +- `meas\_extensions_photometryKron `_ +- `pipe\_drivers `_ +- `lsst\_obs `_ +- `jointcal `_ +- `verify `_ + +Example installation (:ref:`more info `): + +.. code-block:: bash + + eups distrib install lsst_distrib -t + setup lsst_distrib + +.. _EUPS: https://github.com/RobertLuptonTheGood/eups diff --git a/releases/v12_0.rst b/releases/v12_0.rst index 4449ed88..154c665e 100644 --- a/releases/v12_0.rst +++ b/releases/v12_0.rst @@ -26,7 +26,7 @@ New: Install with Anaconda In addition to the customary source installation, this release features two binary installation methods: -- :doc:`Anaconda ` +- Anaconda - `CernVM FS `__ (contributed by Fabio Hernandez) If you are unsure which one to use, we recommend you try the conda binaries.