From d2b63c1df88fcdf7f2d064aaa13521b4609ee7db Mon Sep 17 00:00:00 2001 From: Simon Krughoff Date: Thu, 14 Sep 2017 17:34:46 -0700 Subject: [PATCH 01/35] Add 14.0 characterization report This adds measurements of various measurements of KPMs for the RC1 for V14.0. We could add a few more measurements (v13 included y-band). Also, some of the SRD numbers changed in this new report. Those should be checked. We should also define the canonical place to find the KPM performance ramp. I assume LDM-240 is no longer the place. --- metrics/v14_0.rst | 128 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 metrics/v14_0.rst diff --git a/metrics/v14_0.rst b/metrics/v14_0.rst new file mode 100644 index 00000000..4f711f66 --- /dev/null +++ b/metrics/v14_0.rst @@ -0,0 +1,128 @@ +.. _metrics-v14-0: + +############################################################## +Characterization Metric Report: Science Pipelines Version 13.0 +############################################################## + +Measured using `validation_data_hsc`_, which consists of 8 HSC engineering images: 4 *r*-band, 2 *i*'-band, and 2 *y*-band. +Measurements were made on individual, separately-processed, single-frame images: Jointcal and/or meas_mosaic were not run. +For comparison, we provide the SRD required “minimum” value of each metric as defined in the Science Requirements Document +(`LPM-17`_), and, where available, the target for this release as defined in the Data Management Development Milestone Roadmap (`LDM-240`_). +All values were computed using the :file:`examples/runHscTest.sh` script in the `validate_drp`_ package. + +Photometric Performance +======================= + +*Submitted by Michael Wood-Vasey* + +procCalRep corresponds to requirement OSS-REQ-0275 (defined in `LSE-30`_). +All other photometric performance metrics follow LSS-REQ-0093 (`LSE-29`_) and `LPM-17`_ table 14. + ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| Metric | Unit | SRD Requirement | Release 14 Target | Value | Comments | ++================+======+===================+===================+==============+====================+ +| procCalRep | mmag | :math:`\leq 13` | --- | --- | Need simulations | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA1 (*u* band) | mmag | :math:`\leq 12` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA1 (*g* band) | mmag | :math:`\leq 8` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA1 (*r* band) | mmag | :math:`\leq 5` | 8.0 | 13.97 | | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA1 (*i* band) | mmag | :math:`\leq 5` | 8.0 | 11.8 | | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA1 (*z* band) | mmag | :math:`\leq 12` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA1 (*y* band) | mmag | :math:`\leq 12` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PF1 (*u* band) | % | :math:`\leq 20` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PF1 (*g* band) | % | :math:`\leq 20` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PF1 (*r* band) | % | :math:`\leq 10` | 10.0 | 30.39 | | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PF1 (*i* band) | % | :math:`\leq 10` | 10.0 | 25.83 | | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PF1 (*z* band) | % | :math:`\leq 20` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PF1 (*y* band) | % | :math:`\leq 20` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA2 (*u* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA2 (*g* band) | mmag | :math:`\leq 15` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA2 (*r* band) | mmag | :math:`\leq 15` | 22.5 | 26.65 | | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA2 (*i* band) | mmag | :math:`\leq 15` | 22.5 | 25.12 | | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA2 (*z* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ +| PA2 (*y* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | ++----------------+------+-------------------+-------------------+--------------+--------------------+ + +Astrometric Performance +======================= + +*Submitted by Michael Wood-Vasey* + +The following metrics are defined following LSR-REQ-0094 (`LSE-29`_) and `LPM-17`_ table 18. + ++----------------+------+-----------------+-------------------+--------------+ +| Metric | Unit | SRD Requirement | Release 14 Target | Value | ++================+======+=================+===================+==============+ +| AM1 (*r*-band) | mas | :math:`\leq 20` | 30.0 | 6.2 | ++----------------+------+-----------------+-------------------+--------------+ +| AM1 (*i*-band) | mas | :math:`\leq 20` | 30.0 | 10.01 | ++----------------+------+-----------------+-------------------+--------------+ +| AF1 (*r*-band) | % | :math:`\leq 20` | 10.0 | 1.86 | ++----------------+------+-----------------+-------------------+--------------+ +| AF1 (*i*-band) | % | :math:`\leq 20` | 10.0 | 3.12 | ++----------------+------+-----------------+-------------------+--------------+ +| AD1 (*r*-band) | mas | :math:`\leq 20` | 20.0 | 10.42 | ++----------------+------+-----------------+-------------------+--------------+ +| AD1 (*i*-band) | mas | :math:`\leq 20` | 20.0 | 11.75 | ++----------------+------+-----------------+-------------------+--------------+ +| AM2 (*r*-band) | mas | :math:`\leq 10` | 40.0 | 6.13 | ++----------------+------+-----------------+-------------------+--------------+ +| AM2 (*i*-band) | mas | :math:`\leq 10` | 40.0 | 10.23 | ++----------------+------+-----------------+-------------------+--------------+ +| AF2 (*r*-band) | % | :math:`\leq 10` | 10.0 | 1.19 | ++----------------+------+-----------------+-------------------+--------------+ +| AF2 (*i*-band) | % | :math:`\leq 10` | 10.0 | 2.53 | ++----------------+------+-----------------+-------------------+--------------+ +| AD2 (*r*-band) | mas | :math:`\leq 20` | 20.0 | 10.26 | ++----------------+------+-----------------+-------------------+--------------+ +| AD2 (*i*-band) | mas | :math:`\leq 20` | 20.0 | 11.34 | ++----------------+------+-----------------+-------------------+--------------+ + +Ellipticity Correlations +======================== + +*Submitted by Michael Wood-Vasey* + +The following metrics are defined following LSR-REQ-0097 (`LSE-29`_) and `LPM-17`_ table 27. + ++----------------+------+------------------------------+--------------------------+----------------------------+ +| Metric | Unit | SRD Requirement | Release 14 Target | Value | ++================+======+==============================+==========================+============================+ +| TE1 (*r*-band) | --- | :math:`\leq 3\times 10^{-5}` | --- | --- | ++----------------+------+------------------------------+--------------------------+----------------------------+ +| TE1 (*i*-band) | --- | :math:`\leq 3\times 10^{-5}` | --- | --- | ++----------------+------+------------------------------+--------------------------+----------------------------+ +| TE2 (*r*-band) | --- | :math:`\leq 3\times 10^{-7}` | --- | --- | ++----------------+------+------------------------------+--------------------------+----------------------------+ +| TE2 (*i*-band) | --- | :math:`\leq 3\times 10^{-7}` | --- | --- | ++----------------+------+------------------------------+--------------------------+----------------------------+ + +Computational Performance +========================= + +Computational performance metrics were not re-measured for this release. +We expect no significant changes relative to the :ref:`report on version 12 `. + +.. _validation_data_hsc: https://github.com/lsst/validation_data_hsc +.. _validate_drp: https://github.com/lsst/validate_drp +.. _LDM-240: http://ls.st/ldm-240 +.. _LPM-17: http://ls.st/lpm-17 +.. _LSE-29: http://ls.st/lse-29 +.. _LSE-30: http://ls.st/lse-30 From c9fbf039febb368052b5016421aad37102a93637 Mon Sep 17 00:00:00 2001 From: Simon Krughoff Date: Thu, 14 Sep 2017 17:35:34 -0700 Subject: [PATCH 02/35] Add V14 to the index --- metrics/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/metrics/index.rst b/metrics/index.rst index 05dfedbe..83952536 100644 --- a/metrics/index.rst +++ b/metrics/index.rst @@ -7,5 +7,6 @@ Metrics included in these reports are expected to increase in number and sophist .. toctree:: + v14_0 v13_0 v12_0 From b50264977745ce3d9bdc27fbfb7f4a5b5d039199 Mon Sep 17 00:00:00 2001 From: Simon Krughoff Date: Fri, 15 Sep 2017 15:22:18 -0700 Subject: [PATCH 03/35] Fix turndown values --- metrics/v14_0.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/metrics/v14_0.rst b/metrics/v14_0.rst index 4f711f66..9267f6b3 100644 --- a/metrics/v14_0.rst +++ b/metrics/v14_0.rst @@ -1,7 +1,7 @@ .. _metrics-v14-0: ############################################################## -Characterization Metric Report: Science Pipelines Version 13.0 +Characterization Metric Report: Science Pipelines Version 14.0 ############################################################## Measured using `validation_data_hsc`_, which consists of 8 HSC engineering images: 4 *r*-band, 2 *i*'-band, and 2 *y*-band. @@ -82,9 +82,9 @@ The following metrics are defined following LSR-REQ-0094 (`LSE-29`_) and `LPM-17 +----------------+------+-----------------+-------------------+--------------+ | AD1 (*i*-band) | mas | :math:`\leq 20` | 20.0 | 11.75 | +----------------+------+-----------------+-------------------+--------------+ -| AM2 (*r*-band) | mas | :math:`\leq 10` | 40.0 | 6.13 | +| AM2 (*r*-band) | mas | :math:`\leq 10` | 50.0 | 6.13 | +----------------+------+-----------------+-------------------+--------------+ -| AM2 (*i*-band) | mas | :math:`\leq 10` | 40.0 | 10.23 | +| AM2 (*i*-band) | mas | :math:`\leq 10` | 50.0 | 10.23 | +----------------+------+-----------------+-------------------+--------------+ | AF2 (*r*-band) | % | :math:`\leq 10` | 10.0 | 1.19 | +----------------+------+-----------------+-------------------+--------------+ From d02d49bc99b435951db054e41bf2e6efdad9b66f Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 20 Jun 2017 18:39:51 -0400 Subject: [PATCH 04/35] Reorganize homepage Install/Project info sections - Move links downwards to not distract from installation docs. - Separate "Installation" from "Project info" sections so that release notes and such don't distract from how-to pages for installing the stack. --- index.rst | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/index.rst b/index.rst index 27b99169..f12dca3e 100644 --- a/index.rst +++ b/index.rst @@ -5,30 +5,46 @@ The LSST Science Pipelines The LSST Science Pipelines enable optical and near-infrared astronomy in the big data era. We are building the Science Pipelines for the `Large Synoptic Survey Telescope (LSST) `_, but our command line task and Python API can be extended for any optical or near-infrared dataset. -- Join us on the `LSST Community forum, community.lsst.org `_. -- Fork our code on GitHub at https://github.com/lsst. -- Report issues in `JIRA `_. -- API documentation is currently published with `Doxygen `_. -- DM Developer guidance is at https://developer.lsst.io. -- Learn more about LSST Data Management by visiting http://dm.lsst.org. -- Help us improve our documentation! This guide is on GitHub at `lsst/pipelines_lsst_io `_. +The latest release is |current-release|: :doc:`learn what's new `. -Latest release -============== +.. _part-installation: -The latest release is |current-release|: :doc:`learn what's new `. +Installation +============ + +- :doc:`Overview of installation methods ` +- :doc:`Prerequisites ` +- :doc:`Installing with newinstall.sh ` (recommended) +- :doc:`Installing from source with lsstsw ` (for developers) .. toctree:: :hidden: + :caption: Installation - releases/index + install/index + +.. _part-release-details: .. toctree:: :maxdepth: 1 - :caption: Getting Started - :name: part-getting-started + :caption: Release details - install/index - known-issues releases/notes + known-issues metrics/index + +.. toctree:: + :hidden: + + releases/index + +More info +========= + +- Join us on the `LSST Community forum, community.lsst.org `_. +- Fork our code on GitHub at https://github.com/lsst. +- Report issues in `JIRA `_. +- API documentation is currently published with `Doxygen `_. +- DM Developer guidance is at https://developer.lsst.io. +- Learn more about LSST Data Management by visiting http://dm.lsst.org. +- Contribute to our documentation. This guide is on GitHub at `lsst/pipelines_lsst_io `_. From 7e50c68991f6eff5e8e17156b6bcc6fa78f16d7e Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 20 Jun 2017 18:39:51 -0400 Subject: [PATCH 05/35] Refactor prerequisite docs from newinstall docs This makes it easier to document pre-requisites from either the lsstsw or newinstall installation methods. Overall it makes the newinstall docs easier to follow, as well, making it possible to use the newinstall docs from a getting started context. --- install/index.rst | 3 +- install/lsstsw.rst | 3 +- install/newinstall.rst | 170 ++++--------------------------------- install/prereqs/centos.rst | 50 +++++++++++ install/prereqs/debian.rst | 24 ++++++ install/prereqs/index.rst | 38 +++++++++ install/prereqs/macos.rst | 49 +++++++++++ 7 files changed, 181 insertions(+), 156 deletions(-) create mode 100644 install/prereqs/centos.rst create mode 100644 install/prereqs/debian.rst create mode 100644 install/prereqs/index.rst create mode 100644 install/prereqs/macos.rst diff --git a/install/index.rst b/install/index.rst index 9f96caf0..97cbde14 100644 --- a/install/index.rst +++ b/install/index.rst @@ -4,7 +4,8 @@ Installing the LSST Science Pipelines .. toctree:: :hidden: - + + prereqs/index conda newinstall lsstsw diff --git a/install/lsstsw.rst b/install/lsstsw.rst index 256fbb75..c4328b8a 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -21,8 +21,9 @@ If you have difficulty installing LSST software: You need to install some prerequisites to build the LSST Stack from source: -- :ref:`Install build dependencies `. - `Install and configure Git LFS `_ for LSST DM's data servers. +- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. +- If you opt not to use ``lsstsw`` \’s default Python environment you need to :ref:`install these Python dependencies `. .. _lsstsw-deploy: diff --git a/install/newinstall.rst b/install/newinstall.rst index 1796ec6c..ed10278b 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -15,154 +15,16 @@ If you have difficulty installing LSST software: .. _source-install-prereqs: -Prerequisites -============= +1. Install prerequisites +======================== -This section lists system prerequisites for :ref:`macOS `, :ref:`Debian/Ubuntu `, and :ref:`RedHat/CentOS ` platforms. -All platforms also need :ref:`Python package dependencies ` listed here. - -.. note:: - - **New since 11.0**: The minimum gcc version required to compile the Stack is **gcc 4.8.** - If you using our previous factory platform, RedHat/CentOS 6, and you are unable to upgrade to version 7 (which comes with gcc 4.8 as default) consult :ref:`the section below on upgrading compilers in legacy Linux `. - -.. _source-install-mac-prereqs: - -macOS ------ - -To build LSST software, macOS systems need: - -1. :ref:`Xcode `, or command line tools. -2. :ref:`cmake `. - -Versions prior to OS X 10.9 and earlier have not been tested recently and may not work. - -.. _source-install-mac-prereqs-xcode: - -Xcode -^^^^^ - -You will need to install developer tools, which we recommend you obtain with Apple's Xcode command line tools package. -To do this, run from the command line (e.g. ``Terminal.app`` or similar): - -.. code-block:: bash - - xcode-select --install - -and follow the on-screen instructions. -You can verify where the tools are installed by running: - -.. code-block:: bash - - xcode-select -p - -.. _source-install-mac-prereqs-cmake: - -cmake -^^^^^ - -``cmake`` can be `installed directly `__, or though a package manager like `Homebrew `__. - -.. _source-install-debian-prereqs: - -Debian / Ubuntu ---------------- - -.. code-block:: bash - - apt-get install bison ca-certificates \ - cmake flex g++ gettext git libbz2-dev \ - libfontconfig1 libglib2.0-dev libncurses5-dev \ - libreadline6-dev libssl-dev libx11-dev libxrender1 \ - libxt-dev m4 openjdk-8-jre \ - perl-modules zlib1g-dev \ - - -.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp - -Prefix the :command:`apt-get` command with :command:`sudo` if necessary. - -.. _source-install-redhat-prereqs: - -RedHat / CentOS ---------------- - -.. code-block:: bash - - yum install bison curl blas bzip2-devel bzip2 flex fontconfig \ - freetype-devel gcc-c++ gcc-gfortran git libuuid-devel \ - libXext libXrender libXt-devel make openssl-devel patch perl \ - readline-devel tar zlib-devel ncurses-devel cmake glib2-devel \ - java-1.8.0-openjdk gettext perl-ExtUtils-MakeMaker - -.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp - -Prefix the :command:`yum` command with :command:`sudo` if necessary. - -.. _source-install-redhat-legacy: - -Upgrading compilers for legacy RedHat / CentOS 6 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The minimum gcc version required to compile the Stack is gcc 4.8. -This comes as standard in the LSST "factory" platform, Red Hat / CentOS 7. - -On our previous factory platform, Red Hat / CentOS 6, you will need to use a more current version of gcc that what is available with your system. -If you can go to Red Hat 7, we recommend that you do; if you cannot, we recommend that you use a newer gcc version for the stack by using a Software Collection (SCL) with a different version of devtoolset. -This will enable you to safely use a different version of gcc (4.9) for the stack than that used by your operating system (4.4). - -First, install ``devtoolset-3`` (after the :ref:`installing the standard pre-requisites (above) `): - -.. code-block:: bash - - sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm - sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/rh-java-common/epel-6-x86_64/download/rhscl-rh-java-common-epel-6-x86_64.noarch.rpm - sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/devtoolset-3/epel-6-x86_64/download/rhscl-devtoolset-3-epel-6-x86_64.noarch.rpm - sudo yum install -y scl-utils - sudo yum install -y devtoolset-3 - -Then enable ``devtoolset-3`` by including this line in your :file:`~/.bash_profile`: - -.. code-block:: bash - - scl enable devtoolset-3 bash - -.. _source-install-py-deps: - -Python dependencies -------------------- - -You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. - -.. _source-install-optional-deps: - -Optional dependencies ---------------------- - -Although not required, we recommend you install the `matplotlib `_ and `scipy `_ Python packages: - -.. code-block:: bash - - pip install -U matplotlib scipy - -.. FIXME - -Note these are included by default in `Anaconda `__, which :command:`newinstall.sh` *can* obtain for you. - -We also use `SAOImage DS9 `_ to display images for debugging. - -.. _install-from-source: - -Installing from Source with newinstall.sh -========================================= - -This section will guide you through installing the *current* release of the LSST Science Pipelines from source given that prerequisites have been installed. +- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. +- If you opt not to use :command:`newinstall` \’s default Python environment you need to :ref:`install these Python dependencies `. .. _install-from-source-dir: -1. Choose an installation directory ------------------------------------ +2. Choose an installation directory +=================================== First, choose where you want to install the LSST Science Pipelines. We'll use :file:`$HOME/lsst_stack` in this example. @@ -182,8 +44,8 @@ Individual users who install a personal Stack on their own machine need not worr .. _install-from-source-envvar: -2. Unset environment variables ------------------------------- +3. Unset environment variables +============================== If you've run the LSST Science Pipelines previously, you may have conflicting environment variables setup. To be safe, run: @@ -194,8 +56,8 @@ To be safe, run: .. _install-from-source-setup: -3. Installation set-up ----------------------- +4. Installation set-up +====================== Download and run the `installation setup script from GitHub `__, which installs the basic packages required to install other packages: @@ -228,8 +90,8 @@ where :file:`$LSST_INSTALL_DIR` is expanded to your installation directory. .. _install-from-source-packages: -4. Install packages -------------------- +5. Install lsst_distrib +======================= Finally, install components of the LSST Science Pipelines that are relevant for your work. A simple way to ensure that you have a fairly complete set of packages for this need is to install ``lsst_distrib``: @@ -244,8 +106,8 @@ See :ref:`source-install-testing-your-installation`. .. _install-from-source-loadlsst: -5. Source the LSST environment in each shell session ----------------------------------------------------- +6. Source the LSST environment in each shell session +==================================================== Whenever you want to run the installed LSST Science Pipelines in a new terminal session, be sure to :command:`source` the appropriate :file:`loadLSST.bash`, :file:`loadLSST.csh`, :file:`loadLSST.ksh` or :file:`loadLSST.zsh}` script. @@ -257,8 +119,8 @@ Then setup the EUPS packages you need, typically: .. _source-install-testing-your-installation: -Testing Your Installation -========================= +7. Testing Your Installation +============================ Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. This demo processes a small amount of SDSS data. diff --git a/install/prereqs/centos.rst b/install/prereqs/centos.rst new file mode 100644 index 00000000..950bcc7c --- /dev/null +++ b/install/prereqs/centos.rst @@ -0,0 +1,50 @@ +.. _source-install-redhat-prereqs: + +#################################### +RedHat / CentOS system prerequisites +#################################### + +.. code-block:: bash + + yum install bison curl blas bzip2-devel bzip2 flex fontconfig \ + freetype-devel gcc-c++ gcc-gfortran git libuuid-devel \ + libXext libXrender libXt-devel make openssl-devel patch perl \ + readline-devel tar zlib-devel ncurses-devel cmake glib2-devel \ + java-1.8.0-openjdk gettext perl-ExtUtils-MakeMaker + +.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp + +Prefix the :command:`yum` command with :command:`sudo` if necessary. + +.. note:: + + **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.** + If you using our previous factory platform, RedHat/CentOS 6, and you are unable to upgrade to version 7 (which comes with gcc 4.8 as default) consult :ref:`the section below on upgrading compilers in legacy Linux `. + +.. _source-install-redhat-legacy: + +Upgrading compilers for legacy RedHat / CentOS 6 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The minimum gcc version required to compile the Stack is gcc 4.8. +This comes as standard in the LSST "factory" platform, Red Hat / CentOS 7. + +On our previous factory platform, Red Hat / CentOS 6, you will need to use a more current version of gcc that what is available with your system. +If you can go to Red Hat 7, we recommend that you do; if you cannot, we recommend that you use a newer gcc version for the stack by using a Software Collection (SCL) with a different version of devtoolset. +This will enable you to safely use a different version of gcc (4.9) for the stack than that used by your operating system (4.4). + +First, install ``devtoolset-3`` (after the :ref:`installing the standard pre-requisites (above) `): + +.. code-block:: bash + + sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm + sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/rh-java-common/epel-6-x86_64/download/rhscl-rh-java-common-epel-6-x86_64.noarch.rpm + sudo yum install -y https://www.softwarecollections.org/en/scls/rhscl/devtoolset-3/epel-6-x86_64/download/rhscl-devtoolset-3-epel-6-x86_64.noarch.rpm + sudo yum install -y scl-utils + sudo yum install -y devtoolset-3 + +Then enable ``devtoolset-3`` by including this line in your :file:`~/.bash_profile`: + +.. code-block:: bash + + scl enable devtoolset-3 bash diff --git a/install/prereqs/debian.rst b/install/prereqs/debian.rst new file mode 100644 index 00000000..dbc39b67 --- /dev/null +++ b/install/prereqs/debian.rst @@ -0,0 +1,24 @@ +.. _source-install-debian-prereqs: + +#################################### +Debian / Ubuntu system prerequisites +#################################### + +Debian or Ubuntu systems require the following packages: + +.. code-block:: bash + + apt-get install bison ca-certificates \ + cmake flex g++ gettext git libbz2-dev \ + libfontconfig1 libglib2.0-dev libncurses5-dev \ + libreadline6-dev libssl-dev libx11-dev libxrender1 \ + libxt-dev m4 openjdk-8-jre \ + perl-modules zlib1g-dev \ + +.. from https://github.com/lsst-sqre/puppet-lsststack/blob/master/manifests/params.pp + +Prefix the :command:`apt-get` command with :command:`sudo` if necessary. + +.. note:: + + **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.** diff --git a/install/prereqs/index.rst b/install/prereqs/index.rst new file mode 100644 index 00000000..b0f2fcba --- /dev/null +++ b/install/prereqs/index.rst @@ -0,0 +1,38 @@ +############# +Prerequisites +############# + +This page lists software needed to install and use the LSST Science Pipelines. + +.. _system-prereqs: + +System prerequisites +==================== + +.. toctree:: + :maxdepth: 1 + + macos + debian + centos + +.. _python-deps: + +Python dependencies +=================== + +You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. + +If you opt to use your own Python, you can re-create the default Python environment made by :command:`newinstall.sh` and ``lsstsw`` with these Conda environments: + +- `macOS and Python 2.7 `_. +- `macOS and Python 3.5+ `_. +- `Linux and Python 2.7 `_. +- `Linux and Python 3.5+ `_. + +.. _optional-deps: + +Optional dependencies +===================== + +We use `SAOImage DS9 `_ to display images for debugging. diff --git a/install/prereqs/macos.rst b/install/prereqs/macos.rst new file mode 100644 index 00000000..89471b7c --- /dev/null +++ b/install/prereqs/macos.rst @@ -0,0 +1,49 @@ +.. _source-install-mac-prereqs: + +########################## +macOS system prerequisites +########################## + +To build LSST software, macOS systems need: + +1. :ref:`Xcode `, or command line tools. +2. :ref:`cmake `. + +.. note:: + + macOS versions 10.9 and earlier have not been tested recently and may not work. + +.. _source-install-mac-prereqs-xcode: + +Xcode +===== + +You will need to install developer tools, which we recommend you obtain with Apple's Xcode command line tools package. +To do this, run from the command line (e.g. ``Terminal.app`` or similar): + +.. code-block:: bash + + xcode-select --install + +You can verify where the tools are installed by running: + +.. code-block:: bash + + xcode-select -p + +.. _source-install-mac-prereqs-cmake: + +cmake +===== + +``cmake`` can be installed through a package manager like `Homebrew `_: + +.. code-block:: bash + + brew install cmake + +**Alternatively,** `Anaconda `_ users can install ``cmake`` with :command:`conda`: + +.. code-block:: bash + + conda install cmake From 4ec81671ee8f1eaa7a3d166ac0364c04b03a2662 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 12 Sep 2017 15:45:33 -0700 Subject: [PATCH 06/35] Remove conda installation docs A conda distribution is no longer provided with v14 of the stack. --- install/conda.rst | 113 --------------------------------------------- install/demo.rst | 3 +- install/index.rst | 5 -- releases/v12_0.rst | 2 +- 4 files changed, 2 insertions(+), 121 deletions(-) delete mode 100644 install/conda.rst diff --git a/install/conda.rst b/install/conda.rst deleted file mode 100644 index 4c651a1d..00000000 --- a/install/conda.rst +++ /dev/null @@ -1,113 +0,0 @@ -############################################################## -Installing and Using LSST Science Pipelines as a Conda Package -############################################################## - -This page will guide you through installing the LSST Science Pipelines as a Conda package for use in `Anaconda or Miniconda `__. -Anaconda is a popular Python distribution and package ecosystem. -With this installation method you don't compile source code, or even need to have an existing Python installation. - -.. note:: - - This documentation does not cover the LSST Simulations software. - See the `LSST Simulations installation documentation `_ instead. - -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. - -1. Install Anaconda or Miniconda -================================ - -You might already have Anaconda or Miniconda as your main Python distribution. -If not, you can quickly get started by following Continuum's official installation instructions: - -- `Install Anaconda `__ if you want a complete, science-ready Python installation with minimal setup. -- Otherwise, `install the leaner Miniconda `__ version and install packages as you need them. - -.. warning:: - - You need the Python 2 (e.g. Miniconda2) distribution. If you want to use the experimental Python 3 support, you should try our source installation. - -If you're new to Anaconda, Continuum's `30-minute test drive `_ tutorial will get you up-to-speed. - -.. warning:: - - Don't reuse the Miniconda you might have previously obtained from an :doc:`eups distrib `\ -based installation. - Open a new shell and install Anaconda/Miniconda from scratch. - -Upgrading conda ---------------- - -If you have an existing Anaconda or Miniconda installation, you'll want to make sure the :command:`conda` command itself is up to date: - -.. code-block:: bash - - conda update conda - -See the `Conda documentation for more information about installing and managing conda and Anaconda/Miniconda `__. - -2. Install Science Pipelines in a Conda Environment -=================================================== - -These commands will download and activate the LSST Science Pipelines in a new Conda environment: - -.. code-block:: bash - :linenos: - - conda config --add channels http://conda.lsst.codes/stack/0.13.0 - conda create --name lsst python=2 - source activate lsst - conda install lsst-distrib - source eups-setups.sh - setup lsst_distrib - - You can ignore warnings about pre-link and post-link scripts being dangerous and deprecated. Sorry about that. - -.. warning:: - - Installing the LSST simulation tools (including MAF) requires pointing to a different Conda channel. - See the `LSST Simulations installation documentation `_. - -Here's what these commands are doing, line-by-line: - -1. Tell :command:`conda` about the LSST channel for Conda packages. -2. Create a Conda environment called ``lsst`` with Python 2.7. - You can change the environment's name to anything you like. - Conda environments are recommended since they help keep the Python dependencies of your projects separate. - See the `Conda documentation on environments for more information `__. -3. Activate the ``lsst`` environment (use your environment's name if you chose a different one). - The :command:`activate` command is provided by Anaconda/Miniconda (e.g. at :file:`~/miniconda2/bin/activate`). -4. Install the full suite of LSST science software, including Science Pipelines (``lsst-distrib``). -5. Setup EUPS, LSST's package manager. -6. Setup LSST packages in your environment with EUPS (setting up ``lsst_distrib`` makes most packages available to you). - -.. warning:: - - If the install fails with an error, check that your shell does not have another EUPS Stack configured (try ``echo $EUPS_STACK``). - Conda packaged EUPS will use existing values of ``EUPS_PATH`` and ``EUPS_DIR``. - If they exist, unset them before installing or using Conda packages. - -.. _conda-install-activate: - -3. Activating Science Pipelines in a new Shell -============================================== - -Whenever you open a new shell or terminal session, use these commands to re-activate your previously-installed Science Pipelines: - -.. code-block:: bash - :linenos: - - source activate lsst - source eups-setups.sh - setup lsst_distrib - -These commands can also be used to switch from one Conda environment and LSST Science Pipelines installation to another. - -.. _conda-install-test: - -4. Testing Your Installation -============================ - -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. diff --git a/install/demo.rst b/install/demo.rst index c3f10c85..713c9604 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -10,8 +10,7 @@ This demo will allow you to quickly test your LSST Science Pipelines installatio Remember to first load the LSST Science Pipelines into your shell's environment. The method depends on how the Science Pipelines were installed: -- :ref:`Conda ` -- :ref:`newinstall.sh ` +- :doc:`newinstall.sh ` - :ref:`lsstsw ` 2. Download the Demo Project diff --git a/install/index.rst b/install/index.rst index 97cbde14..cfcd4263 100644 --- a/install/index.rst +++ b/install/index.rst @@ -6,7 +6,6 @@ Installing the LSST Science Pipelines :hidden: prereqs/index - conda newinstall lsstsw demo @@ -18,10 +17,6 @@ Choose an option below to get started. To install the LSST Simulation software, such as MAF, please follow the `LSST Simulations documentation `_ instead of these guides. -:doc:`Conda installation ` - Install the Pipelines as an `Anaconda/Miniconda `__ binary package. - This is the easiest installation option since no source compilation is required. - :doc:`Source installation (newinstall.sh) ` ``newinstall.sh`` allows you to build and install the LSST Science Pipelines from source. You can use your existing Python 2.7, or opt to use a built-in `Miniconda `__. diff --git a/releases/v12_0.rst b/releases/v12_0.rst index 4449ed88..154c665e 100644 --- a/releases/v12_0.rst +++ b/releases/v12_0.rst @@ -26,7 +26,7 @@ New: Install with Anaconda In addition to the customary source installation, this release features two binary installation methods: -- :doc:`Anaconda ` +- Anaconda - `CernVM FS `__ (contributed by Fabio Hernandez) If you are unsure which one to use, we recommend you try the conda binaries. From dcb0d8d36e063347a419110e2cb62b0fa10805aa Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 20 Jun 2017 18:39:51 -0400 Subject: [PATCH 07/35] Overhaul newinstall docs: simplify + v14 binaries - Simplify the newinstall.sh installation instructions to minimize commentary in the how-to steps. The idea is that this commentary and alternative installation strategies will be linked to an "advanced topics" section. - Update the newinstall.sh command so it uses binaries (-t). Also use (-c) to continue a failed build because Josh Hoblitt recommends it. - Point to the 14.0 release. - Adds sections on binaries, ABI compatibility, tag format, the Miniconda Python, and background/reference on newinstall.sh itself. --- install/newinstall.rst | 447 ++++++++++++++++++++++++++++++++++------- 1 file changed, 377 insertions(+), 70 deletions(-) diff --git a/install/newinstall.rst b/install/newinstall.rst index ed10278b..87083f7c 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -1,126 +1,433 @@ -###################################### -Source Installation with newinstall.sh -###################################### +########################################### +Install with newinstall.sh and eups distrib +########################################### -This page will guide you through installing the LSST Science Pipelines from source with :command:`newinstall.sh` (internally based on :command:`eups distrib`). +This page guides you through installing the LSST Science Pipelines software. +This installation method is recommended for anyone who uses or develops the Pipelines software. -The reference platform for the LSST Science Pipelines is CentOS 7 however individual developers compile on `a variety of Linux and macOS operating systems `_ so if you are on a similar platform you should be able to build from source successfully. +If you have installation issues, here are some ways to get help: -:doc:`We also offer Conda binaries and Docker images ` if you do not wish to install the Science Pipelines from source. +- Review the :ref:`known installation issues `. +- Ask a question in the `LSST Community support forum `_. -If you have difficulty installing LSST software: +.. _newinstall-prereqs: -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. +1. Prerequisites +================ -.. _source-install-prereqs: +You can install the LSST Science Pipelines on CentOS 7 (LSST's reference platform) as well as other Linux distributions and macOS (see the `LSST Stack Testing Status `_ for information on the platforms we have tested with). -1. Install prerequisites -======================== +Before you begin, install prerequisite software for your platform: -- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. -- If you opt not to use :command:`newinstall` \’s default Python environment you need to :ref:`install these Python dependencies `. +- :doc:`macOS ` +- :doc:`Centos / RedHat ` +- :doc:`Debian / Ubuntu ` -.. _install-from-source-dir: +.. + TK recommended memory, disk space, and build time. -2. Choose an installation directory -=================================== +.. _newinstall-source-dir: + +2. Make an installation directory +================================= First, choose where you want to install the LSST Science Pipelines. -We'll use :file:`$HOME/lsst_stack` in this example. Create and change into that directory: .. code-block:: bash - mkdir -p $HOME/lsst_stack - cd $HOME/lsst_stack + mkdir -p lsst_stack + cd lsst_stack + +If you are installing the software for multiple users (a shared stack), see :ref:`newinstall-shared-permissions`. + +.. _newinstall-run: + +3. Run newinstall.sh +==================== + +Open a new shell session for the installation (or ensure you haven't used LSST software in that shell before). +If you need to reuse a shell, see :ref:`newinstall-unset-variables`. + +Next, run :command:`newinstall.sh` to set up the environment you'll install the LSST Science Pipelines into. +For most use cases we recommend downloading and running :command:`newinstall.sh` like this: + +.. code-block:: bash + + curl -OL https://raw.githubusercontent.com/lsst/lsst/14.0/scripts/newinstall.sh + bash newinstall.sh -ct + +Always execute `newinstall.sh` with bash, as shown, regardless of what shell you are in. + +We recommend that you opt into the provided Miniconda Python environment (see the links below for more information). + +Then load this environment into your shell: + +.. TODO Use sphinx-tabs here? + +.. code-block:: bash + + source loadLSST.bash # for bash + source loadLSST.csh # for csh + source loadLSST.ksh # for ksh + source loadLSST.zsh # for zsh + +.. note:: + + Here are ways to customize the :command:`newinstall.sh` installation for specific needs: + + - The default Python environment is Python 3.5. + **Python 2.7 users,** see :ref:`newinstall-py2`. + - :ref:`newinstall-user-python`. + - The recommended installation uses precompiled binary tarballs if available for your platform (and falls back to a source build). + See :ref:`newinstall-binary-packages`. + If you will be compiling and linking C++ code against this installation you'll need to ensure your compilers match the distribution's. + **Developers should review** :ref:`newinstall-binary-compatibility`. + + For background information about :command:`newinstall.sh`, see: + + - :ref:`newinstall-background`. + - :ref:`newinstall-miniconda`. + - :ref:`newinstall-reference`. + +.. _newinstall-install: + +4. Install Science Pipelines packages +===================================== + +Install the LSST Science Pipelines packages by running :command:`eups distrib install` for a top-level package and a tagged version. + +This example installs the ``v14_0_rc1`` tagged version (current release) of the ``lsst_distrib`` top-level package: + +.. code-block:: bash + + eups distrib install lsst_distrib -t v14_0_rc1 lsst_distrib + curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python + setup lsst_distrib + +If pre-built binaries are available for your platform (and you ran :command:`newinstall.sh` with the :option:`-t ` argument) the installation should take roughly 10 minutes. +Otherwise, installation falls back to a source build which can take two hours, depending on the top-level package and your machine's performance. + +The last command, :command:`setup`, activates the installed packages in your shell environment. +You'll need to run :command:`setup` in each shell session you'll use the LSST Science Pipelines in. +See :doc:`setup` for more information. + +.. note:: -Installation for groups -^^^^^^^^^^^^^^^^^^^^^^^ - -Those in a system administration role, who are installing a writable stack for multiple users, will likely want to establish a separate group (perhaps lsst) with a umask of 002 (all access permissions for the group; allow other users to read+execute). -The installation directory must be owned by the group, have the SGID (2000) bit set, and allow group read/write/execute: that is, mode 2775. -Individual users who install a personal Stack on their own machine need not worry about this. + - ``lsst_distrib`` is a top-level package that brings-in most LSST Data Management pipelines software, but other top-level packages may be more applicable for your work, such as ``lsst_apps`` or ``lsst_sims``. + See :doc:`top-level-packages` for more information. -.. _install-from-source-envvar: + - ``v14_0_rc1`` is the current release. + You can install other tagged versions of the LSST Science Pipelines, though. + See :ref:`newinstall-other-tags`. -3. Unset environment variables -============================== + - If you're curious about the shebangtron, see its repository at `github.com/lsst/shebrangtron `_. -If you've run the LSST Science Pipelines previously, you may have conflicting environment variables setup. +.. _newinstall-test: + +5. Test your installation (optional) +==================================== + +Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo pipeline `. +This demo processes a small amount of SDSS data and verifies that measurements match expected values. + +See :doc:`demo` for instructions. + +.. _newinstall-next-steps: + +Next steps +========== + +Add links to topics the reader should go to next. +Examples: + +- A topic reminding a user to setup the stack in a new shell. +- Topic explaining top-level packages. +- Topic on compiling a package alongside an existing stack. + +Advanced installation topics +============================ + +The above steps guided you through LSST's recommended installation. +These topics provide additional information about the installation and ways to customize it: + +- :ref:`newinstall-shared-permissions`. +- :ref:`newinstall-unset-variables`. +- :ref:`newinstall-background`. +- :ref:`newinstall-miniconda`. +- :ref:`newinstall-py2`. +- :ref:`newinstall-user-python`. +- :ref:`newinstall-binary-packages`. +- :ref:`newinstall-binary-compatibility`. +- :ref:`newinstall-other-tags`. +- :ref:`newinstall-reference`. + +.. _newinstall-shared-permissions: + +Setting unix permissions for shared installations +------------------------------------------------- + +You can make the LSST Science Pipelines installation accessible to multiple users on the same machine. + +First, create a separate unix group (called ``lsst``, for example) with a ``umask`` of ``002`` (all access permissions for the group and allow other users to read/execute). + +Then set the ownership of the installation directory to the ``lsst`` group, have the ``SGID`` (2000) bit set, and allow group read/write/execute (mode 2775). + +.. _newinstall-unset-variables: + +Running newinstall.sh in an already set-up shell +------------------------------------------------ + +If you've run the LSST Science Pipelines previously, you may have conflicting environment variables in your shell. To be safe, run: .. code-block:: bash unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH -.. _install-from-source-setup: +Then return to the instructions step :ref:`newinstall-run`. -4. Installation set-up -====================== +.. _newinstall-background: -Download and run the `installation setup script from GitHub `__, which installs the basic packages required to install other packages: +What newinstall.sh does +----------------------- -.. code-block:: bash +:command:`newinstall.sh` creates a self-contained environment on your machine where you can install, run, and develop the LSST Science Pipelines. +You activate this environment in a shell by sourcing the :command:`loadLSST` script in the installation directory (see :ref:`setup-howto`). + +Here is how :command:`newinstall.sh` prepares the environment: + +- Identifies your operating system and compilers to determine what EUPS binary packages should be installed (the EUPS package root, see :ref:`newinstall-binary-packages`). +- Installs a specific version of Python, through Miniconda, that is compatible with EUPS binary packages (see :ref:`newinstall-miniconda`). +- Installs Conda packages that the LSST Science Pipelines depends on (see :ref:`python-deps`). +- Checks for :command:`git` on your systems and offers to install it if necessary. +- Installs EUPS_, the package manager used by the LSST software stack. + +For information about :command:`newinstall.sh`\ ’s arguments, see :ref:`newinstall-reference`. + +.. _newinstall-miniconda: - curl -OL https://raw.githubusercontent.com/lsst/lsst/13.0/scripts/newinstall.sh - bash newinstall.sh +About the Miniconda Python installed by newinstall.sh +----------------------------------------------------- -This installs the :command:`loadLSST.*` scripts, which you should source to ensure that LSST tools (e.g., the :command:`eups` command) are included in your path. +:command:`newinstall.sh` can install a dedicated Python environment for your LSST Science Pipelines installation. +This Python installation isn't required, but we recommend it. +See :ref:`newinstall-user-python` if required. -The install script will check your system to ensure that appropriate versions of critical packages are installed on your system, to enable bootstrapping the Science Pipelines, including :command:`git`, and :command:`python`. -If these packages are not available, the script will offer to install them for you (using the Anaconda Python distribution for the latter packages). +The Python environment installed by :command:`newinstall.sh` is Miniconda_, a minimal version of Anaconda_. +By default, :command:`newinstall.sh` installs Python 3.5.2. +If you need to work with your own Python 2.7-only packages, see :ref:`newinstall-py2`. -Allowing the installation of these core packages will not replace or modify any other version of these packages that may be installed on your system. -If you do not choose the Anaconda Python install, and subsequent package build steps fail, you can do one of two things: +In this Miniconda environment, :command:`newinstall.sh` installs the Science Pipeline's Python prerequisites. +See :ref:`python-deps` for more information. -* Report the problem to `community.lsst.org `_. Include your OS, a description of the problem, plus any error messages. Community members will provide assistance. -* Consider removing all contents of the install directory and start from scratch, and accepting the Anaconda Python installation option. +This Miniconda installation won't affect your other Python installations (like the system's Python, your own Anaconda or Miniconda, or virtual environments). +The LSST Miniconda environment is only active when you source the ``loadLSST`` script installed by :command:`newinstall.sh` (see :doc:`setup`). -Once :command:`newinstall.sh` has finished, source the LSST environment to continue the installation by running the appropriate command for your shell: +If you install other Python packages in a shell where the LSST Miniconda is activated (with :command:`pip install` or :command:`conda install`) those packages are installed into the LSST Miniconda's :file:`site-packages`, not your system's. +The Python installed by :command:`newinstall.sh` works like an isolated Python environment dedicated to LSST Science Pipelines code and your own related modules---effectively like a `Conda environment `_ or Python `venv `_. +This pattern is useful because it reduces the risk of having Python package version incompatibilities. + +.. _newinstall-py2: + +How to install a Python 2.7 environment with newinstall.sh +---------------------------------------------------------- + +LSST Science Pipelines is backwards compatible with Python 2.7. +If you need to run your own Python 2.7-only Python packages in conjunction with the Pipelines, you can have :command:`newinstall.sh` install a Python 2.7 environment for you instead of the default Python 3.5 environment. + +To select Python 2.7, run :command:`newinstall.sh` with the :option:`-2 ` flag (in addition to other flags, like :option:`-t `): .. code-block:: bash - source $LSST_INSTALL_DIR/loadLSST.bash # for bash users - source $LSST_INSTALL_DIR/loadLSST.csh # for csh users - source $LSST_INSTALL_DIR/loadLSST.ksh # for ksh users - source $LSST_INSTALL_DIR/loadLSST.zsh # for zsh users + bash newinstall.sh -2 + +Then follow the remaining instructions at :ref:`newinstall-run`. + +See also: :ref:`newinstall-miniconda`. + +.. _newinstall-user-python: -where :file:`$LSST_INSTALL_DIR` is expanded to your installation directory. +How to use your own Python with newinstall.sh +--------------------------------------------- -.. _install-from-source-packages: +:command:`newinstall.sh` creates a new Python environment by default (pre-configured with Python dependencies). +If necessary, you can use your own pre-existing Python environment. -5. Install lsst_distrib -======================= +To do so, run :command:`newinstall.sh` (see :command:`newinstall-run` for details and command arguments). +When :command:`newinstall-run` prompts you to install Miniconda, type ``no``. -Finally, install components of the LSST Science Pipelines that are relevant for your work. -A simple way to ensure that you have a fairly complete set of packages for this need is to install ``lsst_distrib``: +Be aware of these caveats when using your own Python installation: + +- You are responsible for installing Python package dependencies. + See :ref:`python-deps`. + +- Prebuilt binaries will not be available. + :command:`eups distrib install` will always install from source. + +.. _newinstall-binary-packages: + +About EUPS tarball packages +--------------------------- + +EUPS distrib binary (tarball) packages significantly speed up your installation. +Rather than compiling the LSST Science Pipelines from source, EUPS tarballs are prebuilt packages made specifically for supported platforms. + +Platforms are defined by four factors: + +1. Operating system. +2. Compiler. +3. Miniconda_ (Python) version. +4. lsstsw_ version (Git ref). + +EUPS distrib binary packages are currently being built for these platform combinations: + +.. csv-table:: EUPS distrib binary flavors + :header: "OS","Compiler","Python" + + "macOS ``osx/10.9``", "``clang-800.0.42.1``", "``miniconda3-4.2.12`` (Python 3)" + "macOS ``osx/10.9``", "``clang-800.0.42.1``", "``miniconda2-4.2.12`` (Python 2)" + "Redhat ``redhat/el7``", "``gcc-system``", "``miniconda3-4.2.12`` (Python 3)" + "Redhat ``redhat/el7``", "``gcc-system``", "``miniconda2-4.2.12`` (Python 2)" + "Redhat ``redhat/el6``", "``devtoolset-3``", "``miniconda2-4.2.12`` (Python 2)" + +When you run :command:`newinstall.sh`, it looks at your system to identify your operating system and compiler. +The version of :command:`newinstall.sh` you run also determines the Miniconda_ version and the lsstsw_ build system versions. + +Together, these four factors define the URL prefix (called an *EUPS package root*) that :command:`eups distrib install` looks for binary packages from. +If binary tarballs are unavailable for that EUPS package root, :command:`eups distrib install` automatically falls back to compiling LSST Science Pipelines packages from source. + +You can see the active EUPS package roots on your system by running: .. code-block:: bash - eups distrib install -t v13_0 lsst_distrib - setup lsst_distrib + eups distrib path + +Here is an example of the output: + +.. code-block:: text -After this initial setup, it is a good idea to test the installation. -See :ref:`source-install-testing-your-installation`. + https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67 + https://eups.lsst.codes/stack/src -.. _install-from-source-loadlsst: +Based on this example, :command:`eups distrib install` will preferentially install EUPS distrib binary packages for the macOS 10.9 system, ``clang-800.0.42.1`` compiler, and ``miniconda3-4.2.12-7c8e67`` Python and lsstsw combination. +If :command:`eups distrib install` cannot find packages at that EUPS package root it will look in the second EUPS package root (https://eups.lsst.codes/stack/src), which provides source packages. -6. Source the LSST environment in each shell session -==================================================== +.. _newinstall-binary-compatibility: -Whenever you want to run the installed LSST Science Pipelines in a new terminal session, be sure to :command:`source` the appropriate :file:`loadLSST.bash`, :file:`loadLSST.csh`, :file:`loadLSST.ksh` or :file:`loadLSST.zsh}` script. +EUPS tarball packages and compiler compatibility +------------------------------------------------ -Then setup the EUPS packages you need, typically: +EUPS binary tarball packages are prebuilt on LSST's continuous integration servers for a specific combination of operating system, compilers, Python, and Python dependencies. +If you are developing packages alongside this installation, you might encounter application binary interface (ABI) incompatibilities if are using a different compiler version or a different Python environment. + +In this case, the more reliable solution is to revert to a source installation. +To do this, repeat the installation but run :command:`newinstall.sh` *without* the :option:`-t ` argument: .. code-block:: bash - setup lsst_distrib + bash newinstall.sh -c -.. _source-install-testing-your-installation: +Without the :option:`-t ` argument to :command:`newinstall.sh`, :command:`eups distrib install` will always build and install packages from source, ensuring ABI compatibility. + +.. _newinstall-other-tags: + +Installing other releases (including daily and weekly tags) +----------------------------------------------------------- + +The instructions on this page guide you through installing the current release of the LSST Science Pipelines corresponding to this documentation. +You can, however, install other releases by running :command:`eups distrib install` with a different tag. + +The common types of tags are: + +- **Major releases,** tagged as ``v_`` (for example, ``v14_0``). +- **Weekly builds,** tagged as ``w__`` (for example, ``w_2017_33`` is the 33rd weekly build in 2017). +- **Daily builds,** tagged as ``d___`` (for example, ``d_2017_09_01`` is the daily build for September 1, 2017). + +There are also tags pointing to the most recent releases: + +- **Current major release,** tagged as ``current``. +- **Current weekly build,** tagged as ``w_latest``. +- **Current daily build,** tagged as ``d_latest``. + +You can see all available tags at https://eups.lsst.codes/stack/src/tags (each tag has a ``.list`` file). + +.. note:: + + Binary installations may not be available for all tags. + From https://eups.lsst.codes/stack, browse subdirectories corresponding to your platform and look for ``.list`` files of available tags. + :command:`eups distrib install` automatically falls back to a source build if binaries are not available. + +.. warning:: + + You need to ensure that the Python environment created by :command:`newinstall.sh` (see step :ref:`newinstall-run`) is compatible with the tagged software. + + For example, if you are installing a recent weekly you may need to download and run :command:`newinstall.sh` from master: + + .. code-block:: bash + + curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh + bash newinstall.sh -ct + + See https://github.com/lsst/lsst/tags for available tagged versions of :command:`newinstall.sh`. + +.. _newinstall-reference: + +newinstall.sh argument reference +-------------------------------- + +.. program:: newinstall.sh + +.. code-block:: text + + usage: newinstall.sh [-b] [-f] [-h] [-n] [-3|-2] [-t|-T] [-s|-S] [-P ] + +.. option:: -b + + Run in batch mode. Don't ask any questions and install all extra packages. + +.. option:: -c + + Attempt to continue a previously failed install. + +.. option:: -n + + No-op. Go through the motions but echo commands instead of running them. + +.. option:: -P + + Use a specific python interpreter for EUPS. + +.. option:: -2 + + Use Python 2 if the script is installing its own Python. + +.. option:: -3 + + Use Python 3 if the script is installing its own Python. (**default**) + +.. option:: -t + + Use pre-compiled EUPS "tarball" packages, if available. + +.. option:: -T + + **Do not** use pre-compiled EUPS "tarball" packages. (**default**) + +.. option:: -s + + Use EUPS source "eupspkg" packages, if available (compile from source). + +.. option:: -S + + **Do not** use EUPS source "eupspkg" packages (do not compile from source). + +.. option:: -h + + Display this help message. -7. Testing Your Installation -============================ -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. +.. _Miniconda: https://conda.io/miniconda.html +.. _Anaconda: https://docs.anaconda.com +.. _lsstsw: https://github.com/lsst/lsstsw +.. _EUPS: https://github.com/RobertLuptonTheGood/eups From d4617c98b87636065b214a7c494e303df4a48259 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Wed, 13 Sep 2017 10:58:17 -0700 Subject: [PATCH 08/35] Add page on top-level packages This explains and mentions that lsst_apps and lsst_distrib both exist. The page will get better when we have packages in the pipelines.lsst.io docs and they automatically report their package dependencies. --- index.rst | 1 + install/index.rst | 1 + install/top-level-packages.rst | 48 ++++++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) create mode 100644 install/top-level-packages.rst diff --git a/index.rst b/index.rst index f12dca3e..142946f6 100644 --- a/index.rst +++ b/index.rst @@ -15,6 +15,7 @@ Installation - :doc:`Overview of installation methods ` - :doc:`Prerequisites ` - :doc:`Installing with newinstall.sh ` (recommended) +- :doc:`install/top-level-packages` - :doc:`Installing from source with lsstsw ` (for developers) .. toctree:: diff --git a/install/index.rst b/install/index.rst index cfcd4263..8d11f3d1 100644 --- a/install/index.rst +++ b/install/index.rst @@ -7,6 +7,7 @@ Installing the LSST Science Pipelines prereqs/index newinstall + top-level-packages lsstsw demo diff --git a/install/top-level-packages.rst b/install/top-level-packages.rst new file mode 100644 index 00000000..ae33acc7 --- /dev/null +++ b/install/top-level-packages.rst @@ -0,0 +1,48 @@ +############################# +Top-level packages to install +############################# + +The LSST Science Pipelines are part of LSST's EUPS_ package stack. +This means that the Science Pipelines software is actually a collection of packages that you install and setup together. +By specifying different **top-level packages** to the :ref:`eups distrib install ` and :doc:`setup ` commands, you can control the size of the software installation or add new capabilities. + +This page describes the common top-level packages that make up the LSST Science Pipelines and related EUPS stacks. + +lsst\_apps +========== + +``lsst_apps`` provides the core packages that LSST will use in operations. + +Example installation (:ref:`more info `): + +.. code-block:: bash + + eups distrib install lsst_apps -t + setup lsst_apps + +lsst\_distrib +============= + +``lsst_distrib`` provides all the packages from ``lsst_apps`` along with these additional ones: + +.. https://github.com/lsst/lsst_distrib/blob/master/ups/lsst_distrib.table + +- `ctrl\_execute `_ +- `ctrl\_platform_lsstvc `_ +- `datarel `_ +- `meas\_extensions_convolved `_ +- `meas\_extensions_shapeHSM `_ +- `meas\_extensions_photometryKron `_ +- `pipe\_drivers `_ +- `lsst\_obs `_ +- `jointcal `_ +- `verify `_ + +Example installation (:ref:`more info `): + +.. code-block:: bash + + eups distrib install lsst_distrib -t + setup lsst_distrib + +.. _EUPS: https://github.com/RobertLuptonTheGood/eups From 0708d98b7294bf3757edece5e9ca460cb2180394 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Wed, 13 Sep 2017 13:37:01 -0700 Subject: [PATCH 09/35] Add topic on setting up the stack This is a standalone from the installation instructions to make it easier to understanding what steps need to be done in in each new shell, compared to a new installation. Also lets us add additional background on `setup` itself. --- index.rst | 1 + install/index.rst | 1 + install/setup.rst | 56 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 58 insertions(+) create mode 100644 install/setup.rst diff --git a/index.rst b/index.rst index 142946f6..a510f880 100644 --- a/index.rst +++ b/index.rst @@ -15,6 +15,7 @@ Installation - :doc:`Overview of installation methods ` - :doc:`Prerequisites ` - :doc:`Installing with newinstall.sh ` (recommended) +- :doc:`install/setup` - :doc:`install/top-level-packages` - :doc:`Installing from source with lsstsw ` (for developers) diff --git a/install/index.rst b/install/index.rst index 8d11f3d1..58d4246a 100644 --- a/install/index.rst +++ b/install/index.rst @@ -7,6 +7,7 @@ Installing the LSST Science Pipelines prereqs/index newinstall + setup top-level-packages lsstsw demo diff --git a/install/setup.rst b/install/setup.rst new file mode 100644 index 00000000..df54ca1f --- /dev/null +++ b/install/setup.rst @@ -0,0 +1,56 @@ +.. _setup: + +########################################### +Setting up installed LSST Science Pipelines +########################################### + +Whenever you start a new command-line shell you need to set up the LSST Science Pipelines software before you can use it. + +.. _setup-howto: + +Setting up +========== + +Setting the LSST Science Pipelines in a shell is a two-step process: + +1. Load the LSST environment by sourcing the ``loadLSST`` script in your installation directory: + + .. TODO Use sphinx-tabs here? + + .. code-block:: bash + + source loadLSST.bash # for bash + source loadLSST.csh # for csh + source loadLSST.ksh # for ksh + source loadLSST.zsh # for zsh + + .. note:: + + These installation are for :doc:`newinstall.sh `-based installations. + For ``lsstsw``, follow :ref:`these instructions ` instead. + +2. Set up a top-level package: + + .. code-block:: bash + + setup + + For example, ``setup lsst_apps`` or ``setup lsst_distrib``. + See :doc:`top-level-packages` for more about LSST's top-level packages. + +.. _setup-list: + +Listing what packages are set up +================================ + +To see what packages (and their versions) are currently set up: + +.. code-block:: bash + + setup list -s + +To see all packages that are installed, even if not currently set up, run: + +.. code-block:: bash + + setup list From d3ef922c2294d75d7fee6101757115d149901ffe Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:13:06 -0700 Subject: [PATCH 10/35] Add new topic on running from Docker This covers a basic quick start and how to find tags on Docker Hub. Future commits can add details on attaching volumes, developing against a containerized stack, and creating new images with Dockerfiles. --- index.rst | 1 + install/docker.rst | 77 ++++++++++++++++++++++++++++++++++++++++++++++ install/index.rst | 1 + 3 files changed, 79 insertions(+) create mode 100644 install/docker.rst diff --git a/index.rst b/index.rst index a510f880..dc5193ac 100644 --- a/index.rst +++ b/index.rst @@ -17,6 +17,7 @@ Installation - :doc:`Installing with newinstall.sh ` (recommended) - :doc:`install/setup` - :doc:`install/top-level-packages` +- :doc:`install/docker` - :doc:`Installing from source with lsstsw ` (for developers) .. toctree:: diff --git a/install/docker.rst b/install/docker.rst new file mode 100644 index 00000000..c68436c5 --- /dev/null +++ b/install/docker.rst @@ -0,0 +1,77 @@ +.. _docker: + +################### +Running with Docker +################### + +LSST provides versioned Docker images containing the Science Pipelines software. +With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. + +If you have issues, reach out on the `LSST Community support forum `_. + +.. _docker-prereqs: + +Prerequisites +============= + +To download Docker images and run containers, you need Docker's software. +The `Docker Community Edition `_ is freely available for most platforms, including macOS, Linux, and Windows. + +If you haven't used Docker before, you might want to learn more about Docker, images, and containers. +Docker's `Getting Started `_ documentation is a good resource. + +.. _docker-quick-start: + +Quick start +=========== + +This command downloads a weekly build of the LSST Science Pipelines, starts a container, and opens a prompt: + +.. code-block:: bash + + docker run -ti lsstsqre/centos:7-stack-lsst_distrib-w_2017_35 + +Then in the container's shell, load the LSST environment and set up a :doc:`top-level package ` (``lsst_distrib`` in this case): + +.. code-block:: bash + + source /opt/lsst/software/stack/loadLSST.bash + setup lsst_distrib + +This step is equivalent to the :doc:`setup instructions ` for a :doc:`newinstall.sh `\ -based installation. +In fact, the images are internally based on :command:`newinstall.sh`. + +Next, learn more with these topics: + +- :ref:`docker-tags` + +.. _docker-tags: + +Finding images for different LSST Science Pipelines releases +============================================================ + +LSST Science Pipelines Docker images are published as `lsstsqre/centos`_ on Docker Hub. +These images are based on the CentOS base image. + +Docker images are versioned with tags, allowing you to run any release of the LSST Science Pipelines software. +The schema of these tags is: + +.. code-block:: text + + -stack-- + +For example: + +.. code-block:: text + + 7-stack-lsst_distrib-w_2017_35 + +This tag corresponds to: + +- CentOS 7 operating system. +- ``lsst_distrib`` :doc:`top-level package `. +- ``w_2017_35`` EUPS tag. See :ref:`newinstall-other-tags` for an overview of LSST's EUPS tag schema. + +You can see what tags are available by browsing `lsstsqre/centos on Docker Hub `_. + +.. _`lsstsqre/centos`: https://hub.docker.com/r/lsstsqre/centos/ diff --git a/install/index.rst b/install/index.rst index 58d4246a..abd2df8b 100644 --- a/install/index.rst +++ b/install/index.rst @@ -9,6 +9,7 @@ Installing the LSST Science Pipelines newinstall setup top-level-packages + docker lsstsw demo From 5305126ebbda08b846f805c243f5c020f060425d Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:24:41 -0700 Subject: [PATCH 11/35] Update demo for v14.0 release --- install/demo.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/install/demo.rst b/install/demo.rst index 713c9604..fdfb610c 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -28,8 +28,8 @@ Then download the demo's data (if you aren't running the current stable release, .. code-block:: bash - curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/13.0.tar.gz | tar xvzf - - cd lsst_dm_stack_demo-13.0 + curl -L https://github.com/lsst/lsst_dm_stack_demo/archive/14.0.tar.gz | tar xvzf - + cd lsst_dm_stack_demo-14.0 .. note:: From 2a28db08968842b7126782b4a3cb787afd9dcde8 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:25:08 -0700 Subject: [PATCH 12/35] Demo page edits - Switch to sentence case headers. - Improve some of the technical writing style. --- install/demo.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/install/demo.rst b/install/demo.rst index fdfb610c..b5df28ac 100644 --- a/install/demo.rst +++ b/install/demo.rst @@ -1,5 +1,5 @@ ###################################################### -Testing the Science Pipelines Installation with a Demo +Testing the Science Pipelines installation with a demo ###################################################### This demo will allow you to quickly test your LSST Science Pipelines installation, :doc:`regardless of your installation method `. @@ -13,11 +13,11 @@ The method depends on how the Science Pipelines were installed: - :doc:`newinstall.sh ` - :ref:`lsstsw ` -2. Download the Demo Project +2. Download the demo project ============================ Choose a directory to run the demo in. -We'll call this directory :file:`demo_data`. +For example: .. code-block:: bash @@ -34,7 +34,7 @@ Then download the demo's data (if you aren't running the current stable release, .. note:: The demo's version should match your installed software. - If you installed from source (with :doc:`lsstsw `), clone the demo repository instead of downloading a release: + If you installed from source (with :doc:`lsstsw `) or with a :ref`newer tag `, clone the demo repository instead of downloading a release: .. code-block:: bash @@ -53,7 +53,7 @@ run camcol field filters *Filters in parentheses are not processed if run with the* ``--small`` *option, see below* -3. Run the Demo +3. Run the demo =============== Now setup the processing package and run the demo: @@ -65,11 +65,12 @@ Now setup the processing package and run the demo: For each input image the script performs the following operations: -* generate a subset of basic image characterization (e.g., determine photometric zero-point, detect sources, and measures positions, shapes, brightness with a variety of techniques), -* creates a :command:`./output` subdirectory containing subdirectories of configuration files, processing metadata, calibrated images, FITS tables of detected sources. These "raw" outputs are readable by other parts of the LSST pipeline, and -* generates a master comparison catalog in the working directory from the band-specific source catalogs in the ``output/sci-results/`` subdirectories. +- Generates a subset of basic image characterizations (such photometric zero-point estimation, source detection, and measurement). +- Creates a :file:`./output` directory containing subdirectories of configuration files, processing metadata, calibrated images, and FITS tables of detected sources. + These ``raw`` outputs are readable by other parts of the LSST pipeline. +- Generates a master comparison catalog in the working directory from the band-specific source catalogs in the ``output/sci-results/`` subdirectories. -4. Check the Demo Results +4. Check the demo results ========================= The demo will take a minute or two to execute (depending upon your machine), and will generate a large number of status messages. From 48e2d7a2ffe9b2a3a622252fd851ddc83af23f5d Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 11:44:20 -0700 Subject: [PATCH 13/35] Bump CURRENT_RELEASE to 14.0 --- conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf.py b/conf.py index 88df4bcc..eb8ad508 100644 --- a/conf.py +++ b/conf.py @@ -23,7 +23,7 @@ sys.setrecursionlimit(2000) # master variable for current Stack release -CURRENT_RELEASE = '13.0' +CURRENT_RELEASE = '14.0' # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the From 8020236d1c31088711e9f3ecdbb2a3eae83877ec Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 14:34:40 -0700 Subject: [PATCH 14/35] Update lsstsw-based install docs - Update for 14.0 - Move advanced how-to and reference material from the basic installation steps. - Add overviews of deploy and rebuild, along with command references. --- install/lsstsw.rst | 240 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 193 insertions(+), 47 deletions(-) diff --git a/install/lsstsw.rst b/install/lsstsw.rst index c4328b8a..1741b085 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -19,7 +19,7 @@ If you have difficulty installing LSST software: 1. Prerequisites ================ -You need to install some prerequisites to build the LSST Stack from source: +Before you begin: - `Install and configure Git LFS `_ for LSST DM's data servers. - Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. @@ -27,45 +27,30 @@ You need to install some prerequisites to build the LSST Stack from source: .. _lsstsw-deploy: -2. Obtaining a Development Stack with lsstsw -============================================ +2. Deploy lsstsw +================ -Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). -`lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. +.. Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). +.. `lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. Begin by choosing a working directory, then deploy ``lsstsw`` into it: .. code-block:: bash - :linenos: git clone https://github.com/lsst/lsstsw.git cd lsstsw ./bin/deploy - . bin/setup.sh - -The ``deploy`` script automates several things for you: + source bin/setup.sh -1. installs a miniconda_ Python environment specific to this lsstsw workspace, -2. installs EUPS_ in :file:`eups/current/`, -3. clones `lsst-build`_, which will run the build process for us, -4. clones versiondb_, a robot-made Git repository of package dependency information, and -5. creates an empty Stack *installation* directory, :file:`stack/`. - -``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. -Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. +For more information about the :command:`deploy` command, such as using Python 2.7 instead of 3, see :ref:`lsstsw-about-deploy`. .. The ``setup.sh`` step enables EUPS_, the package manager used by LSST. .. **Whenever you open a new terminal session, you need to run '. bin/setup.sh' to activate your lsstsw environment.** -.. _lsst-build: https://github.com/lsst/lsst_build -.. _versiondb: https://github.com/lsst/versiondb -.. _EUPS: https://github.com/RobertLuptonTheGood/eups -.. _miniconda: http://conda.pydata.org/miniconda.html - .. _lsstsw-rebuild: -3. Build Science Pipelines -========================== +3. Build the Science Pipelines packages +======================================= From the :file:`lsstsw` directory, run: @@ -75,54 +60,215 @@ From the :file:`lsstsw` directory, run: Once the ``rebuild`` step finishes, note the build number printed on screen. It is formatted as "``bNNNN``." -Tell EUPS this is the current build by making a clone of the build's EUPS tag and calling it "``current``:" +Tag this build as ``current`` so that EUPS can set it up by default: .. code-block:: bash eups tags --clone bNNNN current -The ``rebuild`` command accomplishes the following: - -1. Clones all Science Pipelines packages from `github.com/lsst `__. - A high-bandwidth connection is helpful since the stack contains a non-trivial amount of code and test data. -2. Runs the Scons-based build process to compile C++, make Swig bindings, and ultimately create the :lmod:`lsst` Python package. - The Stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. - Finally, set up the packages with EUPS: .. code-block:: bash setup lsst_distrib +See :doc:`setup` for more information. + +.. _lsstsw-testing-your-installation: + +4. Testing Your installation (optional) +======================================= + +Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. +This demo processes a small amount of SDSS data. + .. _lsstsw-setup: -4. Sourcing the Pipelines in a New Shell -======================================== +Sourcing the Pipelines in a new shell +===================================== + +In every new shell session you will need to set up the Science Pipelines environment and EUPS package stack. + +Run these two steps: + +1. Activate the lsstsw software environment by sourcing the :file:`setup.sh` script in lsstsw's :file:`bin` directory: + + .. code-block:: bash + + source bin/setup.sh + + If you are running in a :command:`csh` or :command:`tcsh`, run this set up script instead: + + .. code-block:: bash + + source bin/setup.csh + +2. Set up a :doc:`top-level package `: + + .. code-block:: bash + + setup lsst_distrib + + Instead of ``lsst_distrib``, you can set up a different top-level package like ``lsst_apps`` or any individual EUPS package. + See :doc:`top-level-packages`. + +.. _lsstsw-next: + +Next steps and advanced topics +============================== + +- :ref:`lsstsw-about-deploy`. +- :ref:`lsstsw-py2`. +- :ref:`lsstsw-about-rebuild`. +- :ref:`lsstsw-branches`. +- :ref:`lsstsw-deploy-ref`. +- :ref:`lsstsw-rebuild-ref`. -In every new shell session you will need to 'setup' the Science Pipelines. -Do this by running the ``setup.sh`` from the ``lsstsw/`` directory: +.. _lsstsw-about-deploy: + +About the lsstsw deploy script +------------------------------ + +The ``deploy`` script automates several things to prepare an LSST development environment: + +1. Installs Git. +2. Installs Git LFS (*you* are still responsible for `configuring it `_). +3. Installs a Miniconda_ Python environment specific to this lsstsw workspace. + The default Python is Python 3, but you can switch to Python 2.7 if necessary. + See :ref:`lsstsw-py2`. +4. Installs EUPS_ into :file:`eups/current/`. +5. Clones `lsst-build`_, which runs the build process. +6. Clones versiondb_, a robot-managed Git repository of package dependency information. +7. Creates an empty stack *installation* directory, :file:`stack/`. + +This environment, including the EUPS, Miniconda, Git, and Git LFS software, is only activated when you source the :file:`bin/setup.sh` or :file:`bin/setup.csh` scripts in a shell. +Otherwise, lsstsw does not affect the software installed on your computer. + +See also: :ref:`lsstsw-deploy-ref`. + +.. _lsstsw-py2: + +How to use Python 2.7 +--------------------- + +The lsstsw :command:`deploy` script installs Miniconda_ as self-contained Python environment. +By default, :command:`deploy` installs a Python 3 version of Miniconda_. +For testing Python 2.7 compatibility, you can create an lsstsw deployment with a Python 2.7 version of Miniconda using the :option:`deploy -2` argument: .. code-block:: bash - . bin/setup.sh - setup lsst_distrib # or an alternative top-level package + ./bin/deploy -2 -.. note:: +.. _lsstsw-about-rebuild: - If you are using a tcsh shell, run ``. bin/setup.csh`` instead (note ``csh`` extension). +About the lsstsw rebuild command +-------------------------------- -.. _lsstsw-testing-your-installation: +The :command:`rebuild` command accomplishes the following: -5. Testing Your Installation -============================ +1. Clones all Science Pipelines packages from `github.com/lsst `__. + :command:`rebuild` -Once the LSST Science Pipelines are installed, you can verify that it works by :doc:`running a demo project `. -This demo processes a small amount of SDSS data. +2. Runs the Scons-based build process to compile C++, make Pybind11 bindings, and ultimately create the :lmod:`lsst` Python package. + The stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. + +``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. +Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. + +See also: :ref:`lsstsw-rebuild-ref`. + +.. _lsstsw-branches: + +Building from branches +---------------------- + +lsstsw's :command:`rebuild` enables you to clone and build development branches. + +To build ``lsst_distrb``, but use the Git branch ``my-feature`` when it's available in a package's repository: + +.. code-block:: bash + + rebuild -r my-feature lsst_distrib + +Multiple ticket branches across multiple products can be built in order of priority: + +.. code-block:: bash + + rebuild -r feature-1 -r feature-2 lsst_distrib + +In this example, a ``feature-1`` branch will be used in any package's Git repository. +A ``feature-2`` branch will be used secondarily in repositories where ``feature-1`` doesn't exist. +Finally, ``lsstsw`` falls back to using the ``master`` branch for repositories that lack both ``feature-1`` and ``feature-2``. + +.. _lsstsw-deploy-ref: + +lsstsw deploy command reference +------------------------------- + +.. program:: deploy + +.. code-block:: text + + usage: deploy.sh [-2|-3] [-b] [-h] + +.. option:: -2 + + Install a Python 2-based Miniconda_. + +.. option:: -3 + + Use a Python 3-based Miniconda_ (default). + +.. option:: -b + + Use bleeding-edge conda packages. + +.. option:: -h + + Print the help message. + +.. _lsstsw-rebuild-ref: + +lsstsw rebuild command reference +-------------------------------- + +.. program:: rebuild + +.. code-block:: text + + rebuild [-p] [-n] [-u] [-r [-r [...]]] [-t ] [product1 [product2 [...]]] + +.. option:: -p + + Prep only. + +.. option:: -n + + Do not run :command:`git fetch` in already-downloaded repositories. + +.. option:: -u + + Update the :file:`repos.yaml` package index to the ``master`` branch on GitHub of https://github.com/lsst/repos. + +.. option:: -r + + Rebuild using the Git ref. + A Git ref can be a branch name, tag, or commit SHA. + Multiple ``-r`` arguments can be given, in order or priority. + +.. option:: -t + + EUPS tag. + +.. _lsst-build: https://github.com/lsst/lsst_build +.. _versiondb: https://github.com/lsst/versiondb +.. _EUPS: https://github.com/RobertLuptonTheGood/eups +.. _Miniconda: http://conda.pydata.org/miniconda.html .. _lsstsw-development: -6. Bonus: Developing a Package -============================== +Developing a package +-------------------- An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. From 9c2998caf8f4470339665ee5e9e523df3c323c23 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 15:11:23 -0700 Subject: [PATCH 15/35] Move package development docs from lsstsw The same docs should apply to all base installations, where it's newinstall, lsstsw, or Docker. This commit moves the original content to an orphan page to give us time to make that change. --- install/lsstsw.rst | 65 -------------------------------- install/package-development.rst | 67 +++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 65 deletions(-) create mode 100644 install/package-development.rst diff --git a/install/lsstsw.rst b/install/lsstsw.rst index 1741b085..d4219db9 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -264,68 +264,3 @@ lsstsw rebuild command reference .. _versiondb: https://github.com/lsst/versiondb .. _EUPS: https://github.com/RobertLuptonTheGood/eups .. _Miniconda: http://conda.pydata.org/miniconda.html - -.. _lsstsw-development: - -Developing a package --------------------- - -An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. -The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. - -1. Stack packages are found in the :file:`lsstsw/build/` directory. - -2. Create a new branch in a package's Git repository, - - .. code-block:: bash - - git checkout -b {{ticket-name}} - - Then declare this package for EUPS and set it up: - - .. code-block:: bash - - eups declare -r . -t $USER {{package_name}} git - setup -r . -t $USER - - Unpacking the ``eups declare`` arguments: - - - ``-r .`` is the path to the package's repository, which is the current working directory. - You don't *need* to be in the repository's directory if you provide the path appropriately. - - ``-t $USER`` sets the EUPS *tag*. - We use this because your username (``$USER``) is an allowed EUPS tag. - - ``git`` is used as an EUPS *version*. - Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. - - In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." - In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. - If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. - This is why we initially declared the entire lsstsw repository to have the version ``current``. - -3. Build the package with Scons: - - .. code-block:: bash - - scons -Q -j 6 opt=3 - - These flags tell SCons to build with flags: - - - ``-Q``: reduce logging to the terminal, - - ``-j 6``: build in parallel (e.g., with '6' CPUs), - - ``opt=3``: build with level 3 optimization. - - This ``scons`` command will run several targets by default, in sequence: - - 1. ``lib``: build the C++ code and SWIG interface layer - 2. ``python``: install the Python code - 3. ``tests``: run the test suite - 4. ``example``: compile the examples, - 5. ``doc``: compile Doxygen-based documentation, and - 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). - - You can build a subset of these targets by specifying one explicitly. - To simply compile C++, SWIG, build the Python package and run tests: - - .. code-block:: bash - - scons -q -j 6 opt=3 tests diff --git a/install/package-development.rst b/install/package-development.rst new file mode 100644 index 00000000..873379f3 --- /dev/null +++ b/install/package-development.rst @@ -0,0 +1,67 @@ +:orphan: 1 + +.. NOTE: this page is not currently part of the built site, though it is intended for re-integration. + +############################################################### +Developing a package with the installed Science Pipelines stack +############################################################### + +An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. +The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. + +1. Stack packages are found in the :file:`lsstsw/build/` directory. + +2. Create a new branch in a package's Git repository, + + .. code-block:: bash + + git checkout -b {{ticket-name}} + + Then declare this package for EUPS and set it up: + + .. code-block:: bash + + eups declare -r . -t $USER {{package_name}} git + setup -r . -t $USER + + Unpacking the ``eups declare`` arguments: + + - ``-r .`` is the path to the package's repository, which is the current working directory. + You don't *need* to be in the repository's directory if you provide the path appropriately. + - ``-t $USER`` sets the EUPS *tag*. + We use this because your username (``$USER``) is an allowed EUPS tag. + - ``git`` is used as an EUPS *version*. + Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. + + In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." + In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. + If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. + This is why we initially declared the entire lsstsw repository to have the version ``current``. + +3. Build the package with Scons: + + .. code-block:: bash + + scons -Q -j 6 opt=3 + + These flags tell SCons to build with flags: + + - ``-Q``: reduce logging to the terminal, + - ``-j 6``: build in parallel (e.g., with '6' CPUs), + - ``opt=3``: build with level 3 optimization. + + This ``scons`` command will run several targets by default, in sequence: + + 1. ``lib``: build the C++ code and SWIG interface layer + 2. ``python``: install the Python code + 3. ``tests``: run the test suite + 4. ``example``: compile the examples, + 5. ``doc``: compile Doxygen-based documentation, and + 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). + + You can build a subset of these targets by specifying one explicitly. + To simply compile C++, SWIG, build the Python package and run tests: + + .. code-block:: bash + + scons -q -j 6 opt=3 tests From 14e0cc098b8eac70b0f2c430b97104a55dabf4d3 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 15:41:44 -0700 Subject: [PATCH 16/35] Make homepage the curated install entrypoint Before, both the homepage and install/index.rst served as entrypoints for the installation docs. Now I'm making the homepage the curated documentation entrypoint, but install/index.rst has the toctree so that a user landing at https://pipelines.lsst.io/install can still find all the topics. --- index.rst | 16 ++++++++++++---- install/index.rst | 33 +++------------------------------ 2 files changed, 15 insertions(+), 34 deletions(-) diff --git a/index.rst b/index.rst index dc5193ac..c48f27b4 100644 --- a/index.rst +++ b/index.rst @@ -12,13 +12,21 @@ The latest release is |current-release|: :doc:`learn what's new Installation ============ -- :doc:`Overview of installation methods ` -- :doc:`Prerequisites ` -- :doc:`Installing with newinstall.sh ` (recommended) +Recommended installation path: + +- :doc:`Installing with newinstall.sh ` - :doc:`install/setup` - :doc:`install/top-level-packages` + +Alternative distributions and installation methods: + - :doc:`install/docker` -- :doc:`Installing from source with lsstsw ` (for developers) +- :doc:`Installing from source with lsstsw ` +- `CernVM FS `_ (contributed by IN2P3) + +To install the LSST Simulation software, such as MAF, follow the `LSST Simulations documentation `_. + +.. This toctree is hidden to let us curate the section above, but still add the install/ pages to the Sphinx toctree .. toctree:: :hidden: diff --git a/install/index.rst b/install/index.rst index abd2df8b..ccb2894f 100644 --- a/install/index.rst +++ b/install/index.rst @@ -2,8 +2,10 @@ Installing the LSST Science Pipelines ##################################### +.. This page acts as an index.html for the install/ path, but the currated contents are on the homepage. + .. toctree:: - :hidden: + :maxdepth: 1 prereqs/index newinstall @@ -12,32 +14,3 @@ Installing the LSST Science Pipelines docker lsstsw demo - -We offer a few ways of installing the LSST Science Pipelines. -Choose an option below to get started. - -.. note:: - - To install the LSST Simulation software, such as MAF, please follow the `LSST Simulations documentation `_ instead of these guides. - -:doc:`Source installation (newinstall.sh) ` - ``newinstall.sh`` allows you to build and install the LSST Science Pipelines from source. - You can use your existing Python 2.7, or opt to use a built-in `Miniconda `__. - -`CernVM FS (external link) `__ - CernVM FS is a virtual machine that makes it easy to run the LSST Science Pipelines without compiling code. - This distribution is supported by Fabio Hernandez of IN2P3. - -`Docker Containers (external link) `__ - These images are convenient for using the LSST Science Pipelines in distributed or cloud-based computing workflows, or for one-off installations without modifying your environment. You can get them from our `DockerHub repo `__ - -:doc:`lsstsw installation ` - ``lsstsw`` (and ``lsst-build``) are the tools we use internally to build and test the LSST Science Pipelines. - Use this installation option if you're interested in developing the Science Pipelines since ``lsstsw`` presents the Pipelines as a directory of cloned repositories from `github.com/lsst `__. - -Next, :doc:`try out your Science Pipelines installation by running a demo `. - -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. From 38218835bcf09dcb5f5447626be8fb57c22c5cb4 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 15 Sep 2017 15:50:33 -0700 Subject: [PATCH 17/35] Change prereq messaging about Miniconda This wording encourages users to use the provided Python instead of using their own. --- install/prereqs/index.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/install/prereqs/index.rst b/install/prereqs/index.rst index b0f2fcba..c0717d7a 100644 --- a/install/prereqs/index.rst +++ b/install/prereqs/index.rst @@ -21,8 +21,7 @@ System prerequisites Python dependencies =================== -You can use your own Python 2.7.\* install or let :command:`newinstall.sh` install `Miniconda `__ in your local directory. - +Both the :doc:`newinstall.sh <../newinstall>` and :doc:`lsstsw <../lsstsw>`\ -based installation methods provide dedicated Miniconda environments pre-loaded with Python dependencies. If you opt to use your own Python, you can re-create the default Python environment made by :command:`newinstall.sh` and ``lsstsw`` with these Conda environments: - `macOS and Python 2.7 `_. From 59fdc3bfbce773ae645c84e3c49e88b496474269 Mon Sep 17 00:00:00 2001 From: Tim Jenness Date: Sat, 16 Sep 2017 08:19:26 -0700 Subject: [PATCH 18/35] Point to Docushare test reports The versions here were migrated to DMTRs on Docushare to place them with other DM test reports. --- index.rst | 2 +- metrics.rst | 11 +++ metrics/index.rst | 12 ---- metrics/v12_0.rst | 112 ----------------------------- metrics/v13_0.rst | 128 --------------------------------- metrics/v14_0.rst | 128 --------------------------------- releases/README.md | 2 +- releases/note-source/v11_0.rst | 1 + releases/note-source/v12_0.rst | 2 +- releases/note-source/v13_0.rst | 2 +- releases/v12_0.rst | 3 +- releases/v13_0.rst | 2 +- 12 files changed, 18 insertions(+), 387 deletions(-) create mode 100644 metrics.rst delete mode 100644 metrics/index.rst delete mode 100644 metrics/v12_0.rst delete mode 100644 metrics/v13_0.rst delete mode 100644 metrics/v14_0.rst diff --git a/index.rst b/index.rst index c48f27b4..c38e7f2c 100644 --- a/index.rst +++ b/index.rst @@ -42,7 +42,7 @@ To install the LSST Simulation software, such as MAF, follow the `LSST Simulatio releases/notes known-issues - metrics/index + metrics .. toctree:: :hidden: diff --git a/metrics.rst b/metrics.rst new file mode 100644 index 00000000..27841fab --- /dev/null +++ b/metrics.rst @@ -0,0 +1,11 @@ +############################### +Characterization Metric Reports +############################### + +Starting from Summer 2015, administrative ("cycle") releases are accompanied by a measurements report characterizing the current performance. +Metrics included in these reports are expected to increase in number and sophistication at subsequent releases. + +- `V11.0 (DMTR-11) `_ +- `V12.0 (DMTR-14) `_ +- `V13.0 (DMTR-15) `_ +- `V14.0 (DMTR-41) `_ diff --git a/metrics/index.rst b/metrics/index.rst deleted file mode 100644 index 83952536..00000000 --- a/metrics/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -############################## -Characterization Metric Report -############################## - -Starting from Summer 2015, administrative ("cycle") releases are accompanied by a measurements report characterizing the current performance. -Metrics included in these reports are expected to increase in number and sophistication at subsequent releases. - -.. toctree:: - - v14_0 - v13_0 - v12_0 diff --git a/metrics/v12_0.rst b/metrics/v12_0.rst deleted file mode 100644 index 78bca0df..00000000 --- a/metrics/v12_0.rst +++ /dev/null @@ -1,112 +0,0 @@ -.. _metrics-v12-0: - -############################################################## -Characterization Metric Report: Science Pipelines Version 12.0 -############################################################## - -Photometric Repeatability Measurements -====================================== - -*Submitted by Jim Bosch* - -This dataset is a selection of *i*-band HyperSuprime-Cam engineering data -taken in the SDSS Stripe 82 region. This dataset consists of 30s -exposures, so it is somewhat similar to projected LSST data in depth. -Our current calibration approach has many limitations relative to what -we ultimately plan to implement for LSST: - -- There's currently no relative calibration being run at all. -- We have only limited correction for chromatic effects. -- There's currently no allowance for zeropoint variations smaller - than the scale of a CCD. -- We also use a much simpler sample selection than that proposed by the - SRD. - -A Jupyter notebook to compute the metrics can be found at -https://github.com/lsst/afw/blob/tickets/DM-3896/examples/repeatability.ipynb. - -+---------------------------+------------+----------------------+-----------+---------------------+ -| Metric Characterized | Metric Ref | Target | Measured | Measurement | -| | | | Value | Method | -+===========================+============+======================+===========+=====================+ -| Photometric repeatability | `DLP-307`_ | :math:`\leq 13` mmag | 10.6 mmag | `DM-3338`_ (i band) | -| (procCalRep) | | | | | -+---------------------------+------------+----------------------+-----------+---------------------+ -| Photometric repeatability | `DLP-315`_ | :math:`\leq 13` mmag | 10.6 mmag | `DM-3338`_ | -| (PA1gri) | | | | | -+---------------------------+------------+----------------------+-----------+---------------------+ -| Photometric repeatability | `DLP-316`_ | :math:`\leq 13` mmag | 10.6 mmag | `DM-3338`_ (i band) | -| (PA1uzy) | | | | | -+---------------------------+------------+----------------------+-----------+---------------------+ - -.. _DLP-307: https://jira.lsstcorp.org/browser/DLP-307 -.. _DLP-315: https://jira.lsstcorp.org/browser/DLP-315 -.. _DLP-316: https://jira.lsstcorp.org/browser/DLP-316 -.. _DM-3338: https://jira.lsstcorp.org/browse/DM-3338 - -Algorithmic Performance Measurements -==================================== - -*Submitted by John Swinbank* - -The *i*-band HSC engineering data (described above) was used where -possible and the same caveats apply. Consult the tickets in the -Measurement Method column for more details. - -+---------------------------------------------+------------+------------------------------+-------------------------+-------------+ -| Metric Characterized | Metric Ref | Target | Measured Value | Measurement | -+=============================================+============+==============================+=========================+=============+ -| Residual PSF Ellipticity Correlations (TE1) | `DLP-290`_ | :math:`\leq 5\times 10^{-3}` | :math:`6\times 10^{-5}` | `DM-3040`_ | -+---------------------------------------------+------------+------------------------------+-------------------------+-------------+ -| Residual PSF Ellipticity Correlations (TE2) | `DLP-290`_ | :math:`\leq 5\times 10^{-3}` | :math:`2\times 10^{-5}` | `DM-3047`_ | -+---------------------------------------------+------------+------------------------------+-------------------------+-------------+ -| Relative Astrometry (AM1) | `DLP-310`_ | :math:`< 60` mas | 12.49 mas | `DM-3057`_ | -+---------------------------------------------+------------+------------------------------+-------------------------+-------------+ -| Relative Astrometry (AM2) | `DLP-311`_ | :math:`< 60` mas | 12.19 mas | `DM-3064`_ | -+---------------------------------------------+------------+------------------------------+-------------------------+-------------+ - -.. _DLP-290: https://jira.lsstcorp.org/browse/DLP-290 -.. _DLP-310: https://jira.lsstcorp.org/browse/DLP-310 -.. _DLP-311: https://jira.lsstcorp.org/browse/DLP-311 -.. _DM-3040: https://jira.lsstcorp.org/browse/DM-3040 -.. _DM-3047: https://jira.lsstcorp.org/browse/DM-3047 -.. _DM-3057: https://jira.lsstcorp.org/browse/DM-3057 -.. _DM-3064: https://jira.lsstcorp.org/browse/DM-3064 - -Summary of Computational Performance Measurements -================================================= - -*Submitted by John Swinbank and Simon Krughoff* - -At this point of Construction, the computational performance -measurements are a combination of precursor data processing and -extrapolation from R&D assumptions. - -DECam/HITS data was used for the OTT1 estimate and for the diffim and -single-frame measurement of the Alert Production Computational Budget in -combination with data from the `3rd Data -Challenge `_. - -For the Data Release Production of the computational budget, we used -DECam/HITS data for estimating diffim performance, HSC-I for assembling -and measuring coadds and for forced measurement, estimates from FDR for -multifit, and data from the 3rd Data Challenge for SDQA. Calculations -for the DRP computational budget used `this iPython -notebook `__. - -+--------------------------+------------+-------------------------+----------------+--------------------+ -| Metric Characterized | Metric Ref | Target | Measured Value | Measurement Method | -+==========================+============+=========================+================+====================+ -| OTT1 | `DLP-328`_ | :math:`\leq 240` sec | 200-250 sec | `DM-3724`_ | -+--------------------------+------------+-------------------------+----------------+--------------------+ -| AP Computational Budget | `DLP-329`_ | :math:`\leq 231` TFLOPS | 34-39 TFLOPS | `DM-3267`_ | -+--------------------------+------------+-------------------------+----------------+--------------------+ -| DRP Computational Budget | `DLP-314`_ | :math:`\leq 645` TFLOPS | 318 TFLOPS | `DM-3083`_ | -+--------------------------+------------+-------------------------+----------------+--------------------+ - -.. _DLP-328: https://jira.lsstcorp.org/browse/DLP-328 -.. _DLP-329: https://jira.lsstcorp.org/browse/DLP-329 -.. _DLP-314: https://jira.lsstcorp.org/browse/DLP-314 -.. _DM-3724: https://jira.lsstcorp.org/browse/DM-3724 -.. _DM-3267: https://jira.lsstcorp.org/browse/DM-3267 -.. _DM-3083: https://jira.lsstcorp.org/browse/DM-3083 diff --git a/metrics/v13_0.rst b/metrics/v13_0.rst deleted file mode 100644 index 65cc9752..00000000 --- a/metrics/v13_0.rst +++ /dev/null @@ -1,128 +0,0 @@ -.. _metrics-v13-0: - -############################################################## -Characterization Metric Report: Science Pipelines Version 13.0 -############################################################## - -Measured using `validation_data_hsc`_, which consists of 8 HSC engineering images: 4 *r*-band, 2 *i*'-band, and 2 *y*-band. -Measurements were made on individual, separately-processed, single-frame images: Jointcal and/or meas_mosaic were not run. -For comparison, we provide the SRD required “minimum” value of each metric as defined in the Science Requirements Document (`LPM-17`_), and, where available, the target for this release as defined in the Data Management Development Milestone Roadmap (`LDM-240`_). -All values were computed using the :file:`examples/runHscTest.sh` script in the `validate_drp`_ package. - -Photometric Performance -======================= - -*Submitted by Michael Wood-Vasey* - -procCalRep corresponds to requirement OSS-REQ-0275 (defined in `LSE-30`_). -All other photometric performance metrics follow LSS-REQ-0093 (`LSE-29`_) and `LPM-17`_ table 14. - -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| Metric | Unit | SRD Requirement | Release 13 Target | Value | Comments | -+================+======+===================+===================+==============+====================+ -| procCalRep | mmag | :math:`\leq 13` | 10.0 | --- | Need simulations | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*g* band) | mmag | :math:`\leq 8` | 12.0 | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*r* band) | mmag | :math:`\leq 8` | 12.0 | 14.3 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*i* band) | mmag | :math:`\leq 8` | 12.0 | 12.0 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*g* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*r* band) | % | :math:`\leq 20` | --- | 30.6 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*i* band) | % | :math:`\leq 20` | --- | 18.2 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*g* band) | mmag | :math:`\leq 15` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*r* band) | mmag | :math:`\leq 15` | --- | 19.6 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*i* band) | mmag | :math:`\leq 15` | --- | 18.2 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*u* band) | mmag | :math:`\leq 12` | 13.0 | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*z* band) | mmag | :math:`\leq 12` | 13.0 | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*y* band) | mmag | :math:`\leq 12` | 13.0 | 24.3 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*u* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*z* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*y* band) | % | :math:`\leq 20` | --- | 36.9 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*u* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*z* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*y* band) | mmag | :math:`\leq 22.5` | --- | 33.6 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ - -Astrometric Performance -======================= - -*Submitted by Michael Wood-Vasey* - -The following metrics are defined following LSR-REQ-0094 (`LSE-29`_) and `LPM-17`_ table 18. - -+----------------+------+-----------------+-------------------+--------------+ -| Metric | Unit | SRD Requirement | Release 13 Target | Value | -+================+======+=================+===================+==============+ -| AM1 (*r*-band) | mas | :math:`\leq 20` | 50.0 | 6.5 | -+----------------+------+-----------------+-------------------+--------------+ -| AM1 (*i*-band) | mas | :math:`\leq 20` | 50.0 | 10.5 | -+----------------+------+-----------------+-------------------+--------------+ -| AF1 (*r*-band) | % | :math:`\leq 20` | --- | 0.27 | -+----------------+------+-----------------+-------------------+--------------+ -| AF1 (*i*-band) | % | :math:`\leq 20` | --- | 0.69 | -+----------------+------+-----------------+-------------------+--------------+ -| AD1 (*r*-band) | mas | :math:`\leq 40` | --- | 6.5 | -+----------------+------+-----------------+-------------------+--------------+ -| AD1 (*i*-band) | mas | :math:`\leq 40` | --- | 7.5 | -+----------------+------+-----------------+-------------------+--------------+ -| AM2 (*r*-band) | mas | :math:`\leq 20` | 50.0 | 6.5 | -+----------------+------+-----------------+-------------------+--------------+ -| AM2 (*i*-band) | mas | :math:`\leq 20` | 50.0 | 10.5 | -+----------------+------+-----------------+-------------------+--------------+ -| AF2 (*r*-band) | % | :math:`\leq 20` | --- | 0.47 | -+----------------+------+-----------------+-------------------+--------------+ -| AF2 (*i*-band) | % | :math:`\leq 20` | --- | 0.67 | -+----------------+------+-----------------+-------------------+--------------+ -| AD2 (*r*-band) | mas | :math:`\leq 40` | --- | 6.6 | -+----------------+------+-----------------+-------------------+--------------+ -| AD2 (*i*-band) | mas | :math:`\leq 40` | --- | 7.3 | -+----------------+------+-----------------+-------------------+--------------+ - -Ellipticity Correlations -======================== - -*Submitted by Michael Wood-Vasey* - -The following metrics are defined following LSR-REQ-0097 (`LSE-29`_) and `LPM-17`_ table 27. - -+----------------+------+------------------------------+--------------------------+----------------------------+ -| Metric | Unit | SRD Requirement | Release 13 Target | Value | -+================+======+==============================+==========================+============================+ -| TE1 (*r*-band) | --- | :math:`\leq 3\times 10^{-5}` | :math:`1 \times 10^{-3}` | :math:`7.9 \times 10^{-7}` | -+----------------+------+------------------------------+--------------------------+----------------------------+ -| TE1 (*i*-band) | --- | :math:`\leq 3\times 10^{-5}` | :math:`1 \times 10^{-3}` | :math:`1.9 \times 10^{-6}` | -+----------------+------+------------------------------+--------------------------+----------------------------+ -| TE2 (*r*-band) | --- | :math:`\leq 3\times 10^{-7}` | :math:`1 \times 10^{-3}` | :math:`1.8 \times 10^{-6}` | -+----------------+------+------------------------------+--------------------------+----------------------------+ -| TE2 (*i*-band) | --- | :math:`\leq 3\times 10^{-7}` | :math:`1 \times 10^{-3}` | :math:`3.0 \times 10^{-7}` | -+----------------+------+------------------------------+--------------------------+----------------------------+ - -Computational Performance -========================= - -Computational performance metrics were not re-measured for this release. -We expect no significant changes relative to the :ref:`report on version 12 `. - -.. _validation_data_hsc: https://github.com/lsst/validation_data_hsc -.. _validate_drp: https://github.com/lsst/validate_drp -.. _LDM-240: http://ls.st/ldm-240 -.. _LPM-17: http://ls.st/lpm-17 -.. _LSE-29: http://ls.st/lse-29 -.. _LSE-30: http://ls.st/lse-30 - diff --git a/metrics/v14_0.rst b/metrics/v14_0.rst deleted file mode 100644 index 9267f6b3..00000000 --- a/metrics/v14_0.rst +++ /dev/null @@ -1,128 +0,0 @@ -.. _metrics-v14-0: - -############################################################## -Characterization Metric Report: Science Pipelines Version 14.0 -############################################################## - -Measured using `validation_data_hsc`_, which consists of 8 HSC engineering images: 4 *r*-band, 2 *i*'-band, and 2 *y*-band. -Measurements were made on individual, separately-processed, single-frame images: Jointcal and/or meas_mosaic were not run. -For comparison, we provide the SRD required “minimum” value of each metric as defined in the Science Requirements Document -(`LPM-17`_), and, where available, the target for this release as defined in the Data Management Development Milestone Roadmap (`LDM-240`_). -All values were computed using the :file:`examples/runHscTest.sh` script in the `validate_drp`_ package. - -Photometric Performance -======================= - -*Submitted by Michael Wood-Vasey* - -procCalRep corresponds to requirement OSS-REQ-0275 (defined in `LSE-30`_). -All other photometric performance metrics follow LSS-REQ-0093 (`LSE-29`_) and `LPM-17`_ table 14. - -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| Metric | Unit | SRD Requirement | Release 14 Target | Value | Comments | -+================+======+===================+===================+==============+====================+ -| procCalRep | mmag | :math:`\leq 13` | --- | --- | Need simulations | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*u* band) | mmag | :math:`\leq 12` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*g* band) | mmag | :math:`\leq 8` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*r* band) | mmag | :math:`\leq 5` | 8.0 | 13.97 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*i* band) | mmag | :math:`\leq 5` | 8.0 | 11.8 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*z* band) | mmag | :math:`\leq 12` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA1 (*y* band) | mmag | :math:`\leq 12` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*u* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*g* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*r* band) | % | :math:`\leq 10` | 10.0 | 30.39 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*i* band) | % | :math:`\leq 10` | 10.0 | 25.83 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*z* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PF1 (*y* band) | % | :math:`\leq 20` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*u* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*g* band) | mmag | :math:`\leq 15` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*r* band) | mmag | :math:`\leq 15` | 22.5 | 26.65 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*i* band) | mmag | :math:`\leq 15` | 22.5 | 25.12 | | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*z* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ -| PA2 (*y* band) | mmag | :math:`\leq 22.5` | --- | --- | No data | -+----------------+------+-------------------+-------------------+--------------+--------------------+ - -Astrometric Performance -======================= - -*Submitted by Michael Wood-Vasey* - -The following metrics are defined following LSR-REQ-0094 (`LSE-29`_) and `LPM-17`_ table 18. - -+----------------+------+-----------------+-------------------+--------------+ -| Metric | Unit | SRD Requirement | Release 14 Target | Value | -+================+======+=================+===================+==============+ -| AM1 (*r*-band) | mas | :math:`\leq 20` | 30.0 | 6.2 | -+----------------+------+-----------------+-------------------+--------------+ -| AM1 (*i*-band) | mas | :math:`\leq 20` | 30.0 | 10.01 | -+----------------+------+-----------------+-------------------+--------------+ -| AF1 (*r*-band) | % | :math:`\leq 20` | 10.0 | 1.86 | -+----------------+------+-----------------+-------------------+--------------+ -| AF1 (*i*-band) | % | :math:`\leq 20` | 10.0 | 3.12 | -+----------------+------+-----------------+-------------------+--------------+ -| AD1 (*r*-band) | mas | :math:`\leq 20` | 20.0 | 10.42 | -+----------------+------+-----------------+-------------------+--------------+ -| AD1 (*i*-band) | mas | :math:`\leq 20` | 20.0 | 11.75 | -+----------------+------+-----------------+-------------------+--------------+ -| AM2 (*r*-band) | mas | :math:`\leq 10` | 50.0 | 6.13 | -+----------------+------+-----------------+-------------------+--------------+ -| AM2 (*i*-band) | mas | :math:`\leq 10` | 50.0 | 10.23 | -+----------------+------+-----------------+-------------------+--------------+ -| AF2 (*r*-band) | % | :math:`\leq 10` | 10.0 | 1.19 | -+----------------+------+-----------------+-------------------+--------------+ -| AF2 (*i*-band) | % | :math:`\leq 10` | 10.0 | 2.53 | -+----------------+------+-----------------+-------------------+--------------+ -| AD2 (*r*-band) | mas | :math:`\leq 20` | 20.0 | 10.26 | -+----------------+------+-----------------+-------------------+--------------+ -| AD2 (*i*-band) | mas | :math:`\leq 20` | 20.0 | 11.34 | -+----------------+------+-----------------+-------------------+--------------+ - -Ellipticity Correlations -======================== - -*Submitted by Michael Wood-Vasey* - -The following metrics are defined following LSR-REQ-0097 (`LSE-29`_) and `LPM-17`_ table 27. - -+----------------+------+------------------------------+--------------------------+----------------------------+ -| Metric | Unit | SRD Requirement | Release 14 Target | Value | -+================+======+==============================+==========================+============================+ -| TE1 (*r*-band) | --- | :math:`\leq 3\times 10^{-5}` | --- | --- | -+----------------+------+------------------------------+--------------------------+----------------------------+ -| TE1 (*i*-band) | --- | :math:`\leq 3\times 10^{-5}` | --- | --- | -+----------------+------+------------------------------+--------------------------+----------------------------+ -| TE2 (*r*-band) | --- | :math:`\leq 3\times 10^{-7}` | --- | --- | -+----------------+------+------------------------------+--------------------------+----------------------------+ -| TE2 (*i*-band) | --- | :math:`\leq 3\times 10^{-7}` | --- | --- | -+----------------+------+------------------------------+--------------------------+----------------------------+ - -Computational Performance -========================= - -Computational performance metrics were not re-measured for this release. -We expect no significant changes relative to the :ref:`report on version 12 `. - -.. _validation_data_hsc: https://github.com/lsst/validation_data_hsc -.. _validate_drp: https://github.com/lsst/validate_drp -.. _LDM-240: http://ls.st/ldm-240 -.. _LPM-17: http://ls.st/lpm-17 -.. _LSE-29: http://ls.st/lse-29 -.. _LSE-30: http://ls.st/lse-30 diff --git a/releases/README.md b/releases/README.md index 00b420f8..9649b5ee 100644 --- a/releases/README.md +++ b/releases/README.md @@ -34,7 +34,7 @@ Cycle 20XY Release (vNN_N) - :doc:`Installation instructions ` - :doc:`Known issues ` -- :doc:`Measurements & Characterization ` +- `Characterization Metric Report (DMTR-NN) `_ .. _release-vNN-N-major-changes: diff --git a/releases/note-source/v11_0.rst b/releases/note-source/v11_0.rst index b0945d71..6b1b751e 100644 --- a/releases/note-source/v11_0.rst +++ b/releases/note-source/v11_0.rst @@ -19,6 +19,7 @@ Winter 2015 release. - `Qserv release notes `_ - `Webserv release notes `_ - `Science User Interface release notes `_ +- `Characterization Metric Report (DMTR-11) `_ .. _release-11-0-major-changes: diff --git a/releases/note-source/v12_0.rst b/releases/note-source/v12_0.rst index 671da033..b7aae110 100644 --- a/releases/note-source/v12_0.rst +++ b/releases/note-source/v12_0.rst @@ -22,7 +22,7 @@ Winter 2016 & X2016 Release (v12_0) - :doc:`Installation instructions <../install/index>` - :doc:`Known issues ` -- :doc:`Measurements & Characterization ` +- `Characterization Metric Report (DMTR-14) `_ .. _release-v12-0-major-changes: diff --git a/releases/note-source/v13_0.rst b/releases/note-source/v13_0.rst index dc8f960e..302c1f66 100644 --- a/releases/note-source/v13_0.rst +++ b/releases/note-source/v13_0.rst @@ -22,7 +22,7 @@ Fall 2016 Release (v13_0) - :doc:`Installation instructions <../install/index>` - :doc:`Known issues ` -- :doc:`Measurements & Characterization ` +- `Characterization Metric Report (DMTR-15) `_ .. _release-v13-0-major-changes: diff --git a/releases/v12_0.rst b/releases/v12_0.rst index 154c665e..3df8a2de 100644 --- a/releases/v12_0.rst +++ b/releases/v12_0.rst @@ -17,8 +17,7 @@ LSST-driven developments. Even :ref:`Astropy views of LSST catalogs objects ` are available. Full details can be found in the :ref:`extensive release notes `. -We have also published a :ref:`Characterization Metric Report -`. +We have also published a `Characterization Metric Report `_. New: Install with Anaconda ========================== diff --git a/releases/v13_0.rst b/releases/v13_0.rst index e16ddd01..9584604f 100644 --- a/releases/v13_0.rst +++ b/releases/v13_0.rst @@ -9,7 +9,7 @@ LSST Pipelines Full details of the LSST Science Pipelines 13.0 release can be found in the :ref:`extensive release notes `. We have also -published a :ref:`Characterization Metric Report `. +published a `Characterization Metric Report `_. Qserv and Data Access ===================== From 6a7f09ac9b9b2e162ea7b2182be5204eac0fac58 Mon Sep 17 00:00:00 2001 From: John Swinbank Date: Wed, 27 Sep 2017 10:29:04 -0400 Subject: [PATCH 19/35] Removed unused section of v13 release notes. There were no documentation improvements listed. --- releases/note-source/v13_0.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/releases/note-source/v13_0.rst b/releases/note-source/v13_0.rst index 302c1f66..cf754681 100644 --- a/releases/note-source/v13_0.rst +++ b/releases/note-source/v13_0.rst @@ -16,7 +16,6 @@ Fall 2016 Release (v13_0) - :ref:`release-v13-0-internal-improvements` - :ref:`release-v13-0-sui` - :ref:`Qserv and Data Access ` -- :ref:`release-v13-0-doc` *See also:* @@ -790,8 +789,3 @@ The master is able to reduce data which has been returned from the slaves withou This is more efficient than the “map” functionality which was previously being used for many of the same tasks. :jirab:`DM-5989` - -.. _release-v13-0-doc: - -Documentation improvements --------------------------- From 37af95f9478bb10bcfb5b54cca613a6f587d668f Mon Sep 17 00:00:00 2001 From: John Swinbank Date: Thu, 28 Sep 2017 08:45:52 -0400 Subject: [PATCH 20/35] Remove 'new' label from CLO. --- releases/v13_0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/v13_0.rst b/releases/v13_0.rst index 9584604f..f5853b48 100644 --- a/releases/v13_0.rst +++ b/releases/v13_0.rst @@ -25,4 +25,4 @@ Join Us on the Community Forum ============================== If you have questions or comments or comments about the LSST Science Pipelines, -visit our new community forum for advice: https://community.lsst.org/. +visit our community forum for advice: https://community.lsst.org/. From c79d37ab803fc6b1e2cdf3de2bb5210e27ca4cb7 Mon Sep 17 00:00:00 2001 From: John Swinbank Date: Thu, 28 Sep 2017 11:08:50 -0400 Subject: [PATCH 21/35] Add Pipelines notes for v14.0. --- releases/index.rst | 2 + releases/note-source/v14_0.rst | 373 +++++++++++++++++++++++++++++++ releases/notes.rst | 5 +- releases/v14_0.rst | 28 +++ releases/v14_0_tickets.rst | 393 +++++++++++++++++++++++++++++++++ 5 files changed, 800 insertions(+), 1 deletion(-) create mode 100644 releases/note-source/v14_0.rst create mode 100644 releases/v14_0.rst create mode 100644 releases/v14_0_tickets.rst diff --git a/releases/index.rst b/releases/index.rst index d54517bf..2f96222b 100644 --- a/releases/index.rst +++ b/releases/index.rst @@ -4,6 +4,7 @@ Release History .. toctree:: + v14_0 v13_0 v12_0 @@ -11,3 +12,4 @@ Release History :hidden: notes + v14_0_tickets diff --git a/releases/note-source/v14_0.rst b/releases/note-source/v14_0.rst new file mode 100644 index 00000000..3310c80e --- /dev/null +++ b/releases/note-source/v14_0.rst @@ -0,0 +1,373 @@ +.. _release-v14-0: + +Spring 2017 Release (v14_0) +=========================== + ++-------------------------------------------+------------+ +| Source | Identifier | ++===========================================+============+ +| Git tag | 14.0 | ++-------------------------------------------+------------+ +| :doc:`EUPS distrib ` | v14\_0 | ++-------------------------------------------+------------+ + +- :ref:`release-v14-0-major-changes` +- :ref:`release-v14-0-bug-fixes` +- :ref:`release-v14-0-internal-improvements` + +.. - :ref:`release-v14-0-sui` +.. - :ref:`Qserv and Data Access ` + +*See also:* + +- :doc:`Installation instructions <../install/index>` +- :doc:`Known issues ` +- `Characterization Metric Report (DMTR-41) `_ + +These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. +A complete list of issues addressed in this release is available :ref:`here `. + +.. _release-v14-0-major-changes: + +Major Functionality and Interface Changes +----------------------------------------- + +- :ref:`release-v14-0-flags-neg` +- :ref:`release-v14-0-footprints` +- :ref:`release-v14-0-sfm-args` +- :ref:`release-v14-0-coadd-stat` +- :ref:`release-v14-0-fake-source` +- :ref:`release-v14-0-32-bit-masks` +- :ref:`release-v14-0-deblender-smoothing` +- :ref:`release-v14-0-matcher` +- :ref:`release-v14-0-ast-wcs` +- :ref:`release-v14-0-pybind` +- :ref:`release-v14-0-psf-coadd` +- :ref:`release-v14-0-cal-flag` +- :ref:`release-v14-0-meas-ext-convolved` +- :ref:`release-v14-0-pupil` +- :ref:`release-v14-0-jointcal` +- :ref:`release-v14-0-fits-index` +- :ref:`release-v14-0-blended-default` +- :ref:`release-v14-0-a&l-decorr` +- :ref:`release-v14-0-afw-table-string` + +.. _release-v14-0-flags-neg: + +Detection: only add ``flags.negative`` to catalog schema if needed +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Rather than always including this flag in the results of source measurement, it is now only provided if the ``thresholdPolarity`` argument is set to ``both``. +Otherwise, this flag is omitted altogether. + +:jirab:`DM-903` + +.. _release-v14-0-footprints: + +New ``Footprint`` system +^^^^^^^^^^^^^^^^^^^^^^^^ + +``Footprint``\s are used to represent contiguous areas of pixels on an image, corresponding e.g. to source detections. +The ``Footprint`` system has been comprehensively rewritten for performance, flexibility and ease of use in this release. +This change involves significant changes to the ``Footprint`` API: see `community.lsst.org `_ for a description of the changes and a migration guide. + +:jirab:`DM-3559` + +.. _release-v14-0-sfm-args: + +Change of argument order in single frame measurement +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For consistency with other code and documentation, the order of the arguments of the ``run()`` method of ``SingleFrameMeasurementTask`` has been reversed. + +:jirab:`RFC-273, DM-3804` + +.. _release-v14-0-coadd-stat: + +More statistical operations possible for assembling coadds +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Rather than simply taking the (possibly clipped) mean, it is now possible to specify other operations, such as the median, to be used for combining pixel data when assembling coadds. + +:jirab:`DM-4158` + +.. _release-v14-0-fake-source: + +Hooks for injecting simulated sources +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Hooks are now available in ``ProcessCcdTask`` for inserting simulated sources into a processing run. +These hooks are designed to enable the easy integration of external packages supporting this functionality (e.g. `SynPipe `_). + +:jirab:`DM-5310, DM-9998` + +.. _release-v14-0-32-bit-masks: + +Increased mask plane depth +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is now possible to include 32 separate mask planes with each image. + +:jirab:`DM-7477` + +.. _release-v14-0-deblender-smoothing: + +Option to disable template smoothing in deblender +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Under some circumstances, smoothing of deblend templates can have a detrimental effect on output quality. +This smoothing is now optional. + +:jirab:`DM-8059, DM-9856` + +.. _release-v14-0-matcher: + +New matcher algorithm +^^^^^^^^^^^^^^^^^^^^^ + +A new algorithm — “match pessimistic B” — for matching sources to reference catalog objects, typically used when performing photometry or astrometry, has been added to the release. +This matcher is not currently enabled by default: it should be regarded at this stage as a technology preview. + +:jirab:`DM-8110, DM-8111, DM-8112, DM-8113, DM-8645, DM-9623, DM-9751` + +.. _release-v14-0-ast-wcs: + +AST-backed WCS class +^^^^^^^^^^^^^^^^^^^^ + +A new representation of the celestial world coordinate system based on the `Starlink AST library `_ has been introduced. +This will ultimately replace our previous WCS implementation; the transition is not yet complete at the time of this release. + +:jirab:`DM-8440` + +.. _release-v14-0-pybind: + +C++ code is now exposed to Python using Pybind11 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The technology used to wrap our low-level C++ functionality into Python has been switched from `Swig `_ to `Pybind11 `_. +Swig provides a more convenient environment for developers and enables several API improvements in our Python code. +`A summary of the changes is available community.lsst.org `_ + +:jirab:`DM-8467` + +.. _release-v14-0-psf-coadd: + +PSF-matched coadds are an independent data product +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +That is, rather than generating a single “coadd” data product which may be configured to be either direct or PSF-matched, it is now possible to generate *both* PSF-matched and direct coadds in a single pipeline execution. + +:jirab:`DM-8491, DM-10004` + +.. _release-v14-0-cal-flag: + +Flag sources used in calibration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We now store flags indicating which sources were used in astrometric and/or photometric calibration. +This is analogous to the flags used to indicate sources used in PSF modeling. + +:jirab:`DM-9050` + +.. _release-v14-0-meas-ext-convolved: + +Convolved flux measurement +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The meas_extensions_convolved package has been added to the distribution. +This convolves images to a specified PSF before measuring aperture fluxes. +This may be useful when, for example: + +#. Measuring the flux that would be obtained with a fiber spectrograph, by using an aperture equal to the size of the spectrograph fiber and measuring with multiple target seeings to allow interpolation of the flux at the seeing of the spectroscopic observation. +#. Measuring the color of galaxies in variable seeing, by using a common aperture and target seeing in each band. + +:jirab:`RFC-294, DM-9394` + +.. _release-v14-0-pupil: + +Include pupil information in camera geometry +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``afw.cameraGeom.Camera`` object now provides an API for accessing a model of the pupil obscuration. + +:jirab:`RFC-320, DM-9572` + +.. _release-v14-0-jointcal: + +Joint calibration +^^^^^^^^^^^^^^^^^ + +The Jointcal package, which provides for simultaneous photometric and astrometric calibration over multiple exposures, has been added to the distribution. +While we believe that Jointcal works well, note that it is still under heavy development and is not currently run as part of regular processing by the LSST development team: regard this package as a technology preview. + +:jirab:`DM-9752` + +.. _release-v14-0-fits-index: + +FITS HDUs are now 0-indexed +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The previous LSST convention was to use 1-based indexing for HDUs. +That is, *previously*, the first HDU in the file was referred to as HDU ``1``, with ``0`` being a special value taken to mean “the first non-empty HDU”. +*Now* the first HDU in the file is HDU ``0``; the special value ``INT_MIN`` is used to refer to the first non-empty HDU. + +:jirab:`RFC-304, DM-9952` + +.. _release-v14-0-blended-default: + +Blendedness calculation is run by default +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default configuration has been updated so that “blendedness” is always calculated unless it has been specifically disabled by the user. + +:jirab:`DM-10506` + +.. _release-v14-0-a&l-decorr: + +Spatially-varying Alard & Lupton image decorrelation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is now possible to perform spatially-varying decorrelation when performing `Alard & Lupton (1998) `_ style image differencing. +See `DMTN-021 `_ for details. + +:jirab:`DM-10804` + +.. _release-v14-0-afw-table-string: + +Useful string representation of tabular data +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``afw::table``\s and the ``record``\s contained within them now provide a useful string representation. +This is particularly useful when, for example, working in an interactive Python environment. + +:jirab:`DM-6718` + +.. _release-v14-0-bug-fixes: + +Bug Fixes +--------- + +- :ref:`release-v14-0-return-code` +- :ref:`release-v14-0-version-checking` +- :ref:`release-v14-0-cmodel-prior` +- :ref:`release-v14-0-blended-debias` +- :ref:`release-v14-0-temp-local-bg` +- :ref:`release-v14-0-peak-cull` + +.. _release-v14-0-return-code: + +Command line tasks now provide standard Unix return codes +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When a command line task fails, it will now return a non-0 exit code to the calling shell, in line with standard Unix conventions. + +:jirab:`DM-4141` + +.. _release-v14-0-version-checking: + +Deterministic package version checking +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The codebase keeps track of which software versions were used to calculate a particular set of outputs as a means of tracking data provenance. +However, in some circumstances this list was being stored inconsistently, leading to version mismatches being reported where non existed. +This has now been resolved. + +:jirab:`DM-9439` + +.. _release-v14-0-cmodel-prior: + +Correct CModel priors +^^^^^^^^^^^^^^^^^^^^^ + +CModel priors are now weighted correctly relative to the likelihood. + +:jirab:`DM-9795` + +.. _release-v14-0-blended-debias: + +Correct blendedness debiasing calculation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Resolved a mathematical error. + +:jirab:`DM-10237` + +.. _release-v14-0-temp-local-bg: + +Fixed order of operations when using temporary local backgrounds in detection +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Resolved an issue which could result in missed isolated-object detections and incorrect ``Footprint``\s for large objects. + +:jirab:`DM-10271` + +.. _release-v14-0-peak-cull: + +Correct number of bands used for peak culling +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +During multi-band coadd processing, peaks may be rejected (or “culled”) if they have been detected in insufficiently many bands. +A book-keeping error led to the number of bands in which a peak had been detected to be recorded incorrectly, and hence to the wrong peaks being culled. +This has now been corrected. + +:jirab:`DM-11625` + +.. _release-v14-0-internal-improvements: + +Build and Code Improvements +--------------------------- + +- :ref:`release-v14-0-yaml-policy` +- :ref:`release-v14-0-py3` +- :ref:`release-v14-0-flaghandler` +- :ref:`release-v14-0-numpy` +- :ref:`release-v14-0-auto-bt` + +.. _release-v14-0-yaml-policy: + +YAML-based policy +^^^^^^^^^^^^^^^^^ + +The “policy” files associated with standard stack cameras, which define where and how datasets are persisted by the Butler, were updated to use a new `YAML `_-based syntax. + +:jirab:`DM-7363` + +.. _release-v14-0-py3: + +Full support for Python 3.5 +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +All of the Science Pipelines code included in this release is tested with both Python 2.7 and Python 3.5. +It should also support Python 3.6, although that has not been rigorously tested. + +:jirab:`DM-7756` + +.. _release-v14-0-flaghandler: + +More Robust Handling of Flagged Measurements +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``FlagHandler`` mechanism for setting flags indicating the status of particular measurements was enhanced to improve its robustness and ensure that flags are always set consistently. + +:jirab:`DM-9249` + +.. _release-v14-0-numpy: + +Support for recent NumPy +^^^^^^^^^^^^^^^^^^^^^^^^ + +Recent versions of NumPy have introduced a number of changes which were incompatible with usage in the Science Pipelines code. +The codebase has now been adapted to work around these changes. + +:jirab:`DM-9316, DM-10926` + +.. _release-v14-0-auto-bt: + +Automatic printing of C++ backtrace on segmentation faults +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In the unfortunate event of a segmentation fault in pipelines code, a backtrace will now automatically be printed. +This can be forwarded to the developers in the event of a problem. + +:jirab:`DM-10846` diff --git a/releases/notes.rst b/releases/notes.rst index 4b9f9d8d..aa256c86 100644 --- a/releases/notes.rst +++ b/releases/notes.rst @@ -4,10 +4,13 @@ Release Notes ############# -- :ref:`release-v13-0` --- current +- :ref:`release-v14-0` --- current +- :ref:`release-v13-0` - :ref:`release-v12-0` - :ref:`release-v11-0` +.. include:: note-source/v14_0.rst + .. include:: note-source/v13_0.rst .. include:: note-source/v12_0.rst diff --git a/releases/v14_0.rst b/releases/v14_0.rst new file mode 100644 index 00000000..78a4ffdb --- /dev/null +++ b/releases/v14_0.rst @@ -0,0 +1,28 @@ +##################################### +LSST Stack 14.0 / Spring 2017 Release +##################################### + +Version 14.0 of the LSST Science Pipelines (aka S17) is now available. + +LSST Pipelines +============== + +Full details of the LSST Science Pipelines 14.0 release can be found +in the :ref:`extensive release notes `. We have also +published a `Characterization Metric Report `_. + +Qserv and Data Access +===================== + +The database team is on a separate release cadence. + +Science User Interface +====================== + +The Science User Interface (SUI) team releases their codebase independently. + +Join Us on the Community Forum +============================== + +If you have questions or comments or comments about the LSST Science Pipelines, +visit our community forum for advice: https://community.lsst.org/. diff --git a/releases/v14_0_tickets.rst b/releases/v14_0_tickets.rst new file mode 100644 index 00000000..3fe094ad --- /dev/null +++ b/releases/v14_0_tickets.rst @@ -0,0 +1,393 @@ +.. _release-v14-0-tickets: + +Tickets Addressed in Spring 2017 Release (v14_0) +================================================ + +- `DM-430 `_: Control log levels on a per-plugin basis [meas_base] +- `DM-903 `_: SourceDetectionTask should only add flags.negative if config.thresholdParity == "both" [meas_algorithms] +- `DM-2009 `_: Please add cbegin and cend to afw tables [afw] +- `DM-2186 `_: Move astrometry_net wrapper code from meas_astrom to a new package [jointcal, meas_astrom, obs_monocam, pipe_tasks] +- `DM-2840 `_: Add support for listing (async) queries [qserv] +- `DM-3245 `_: Add support for SUBMIT query parsing to czar [qserv] +- `DM-3804 `_: Fix order of arguments - run method of meas_base SingleFrameMeasurementTask [meas_algorithms, meas_astrom, meas_base, meas_extensions_simpleShape, pipe_tasks] +- `DM-4043 `_: update memory management in jointcal [jointcal] +- `DM-4141 `_: cmdLineTasks should provide proper unix return codes [pipe_base, pipe_drivers] +- `DM-4158 `_: Allow configuring more statistical options for assembleCoadds.py [pipe_tasks] +- `DM-4232 `_: Variance is set after dark subtraction [ip_isr, obs_cfht, obs_decam, obs_subaru] +- `DM-4639 `_: modernize afw code and reduce doxygen errors [afw, ip_diffim, meas_algorithms, meas_astrom, meas_base, meas_deblender] +- `DM-4953 `_: Please eliminate misleading warning when running scons on OSX with SIP [base] +- `DM-5126 `_: newinstall.sh should directly install miniconda [lsst] +- `DM-5310 `_: Re-add fake source handling to ProcessCcdTask [pipe_tasks] +- `DM-5466 `_: obs_decam calibration ingest uses fragile relative paths [obs_decam, obs_subaru, pipe_tasks, testdata_decam] +- `DM-5637 `_: Add automated linting for GitHub pull requests [utils] +- `DM-5883 `_: Include more information in DECam registry [obs_decam] +- `DM-6626 `_: jointcal fails MemoryTestCase [jointcal] +- `DM-6718 `_: afw table and record should have useful str() and repr() [afw] +- `DM-7070 `_: Move consts from top of Associations.cc into JointcalConfig [jointcal] +- `DM-7177 `_: Document interface for new Footprints class [afw] +- `DM-7199 `_: afwTable's .getX()/.getY() do not appear in dir() [afw] +- `DM-7237 `_: Review cat package [cat, db] +- `DM-7363 `_: convert stack packages from paf policy to yaml policy [daf_persistence, obs_cfht, obs_decam, obs_lsstSim, obs_sdss, obs_subaru, obs_test] +- `DM-7457 `_: daf_persistence 3e6e15d fails build [daf_persistence] +- `DM-7468 `_: Work on Swift Butler Storage Trial [afw, daf_persistence, obs_base] +- `DM-7477 `_: Increase mask plane size to 32 bits [afw, coadd_utils, ip_diffim, meas_algorithms, meas_base, meas_deblender, meas_extensions_shapeHSM, obs_decam, obs_sdss, obs_subaru, pipe_drivers, pipe_tasks, utils] +- `DM-7611 `_: Implement spatially varying decorrelation kernel in ip_diffim [ip_diffim] +- `DM-7615 `_: Update tests in shapelet to support pytest [shapelet] +- `DM-7756 `_: Port pipe_drivers to Python 3 [pipe_drivers] +- `DM-7894 `_: mapper and butler queryMetadata method badly documented [daf_persistence] +- `DM-7913 `_: Implement RFC-240: make Angle's named methods const... [afw, obs_sdss] +- `DM-8032 `_: Tighten testProcessCcd thresholds once background model is fixed [pipe_tasks] +- `DM-8105 `_: Missing test case for SpherePoint [afw] +- `DM-8106 `_: SpherePoint does not have move constructors/assignment [afw] +- `DM-8108 `_: Update stack code to use new Footprint API [afw, ip_diffim, ip_isr, meas_algorithms, meas_base, meas_deblender, meas_extensions_photometryKron, meas_extensions_shapeHSM, meas_modelfit, pipe_tasks] +- `DM-8145 `_: Implement spatially varying ZOGY [ip_diffim] +- `DM-8169 `_: Use -isystem (rather than -I) for include files from external packages [sconsUtils] +- `DM-8294 `_: Perform cleanup on /qserv partition on IN2P3 cluster [qserv] +- `DM-8355 `_: AssembleCcdTask failure with Python 2 [ip_isr] +- `DM-8439 `_: Add wrapper on astshim to take point lists [afw] +- `DM-8440 `_: Create new Wcs class [afw, astshim] +- `DM-8467 `_: Wrap lsst_distrib with pybind11 [afw, base, coadd_chisquared, coadd_utils, ctrl_orca, daf_base, daf_persistence, display_ds9, ip_diffim, ip_isr, log, lsst_apps, meas_algorithms, meas_astrom, meas_base, meas_deblender, meas_extensions_photometryKron, meas_extensions_psfex, meas_extensions_shapeHSM, meas_extensions_simpleShape, meas_modelfit, obs_base, obs_cfht, obs_decam, obs_lsstSim, obs_sdss, obs_subaru, obs_test, pex_config, pex_exceptions, pex_policy, pipe_drivers, pipe_tasks, sconsUtils, shapelet, skymap, utils] +- `DM-8491 `_: Add Psf-matched CTEs and Coadds as independent data products in DRP [ip_diffim, obs_base, obs_cfht, obs_decam, obs_lsstSim, obs_sdss, obs_subaru, pipe_drivers, pipe_tasks] +- `DM-8520 `_: Create single frame spatial grid mapper processing task [ip_diffim] +- `DM-8560 `_: Update Scons package to v3.0 [qserv, sconsUtils] +- `DM-8606 `_: Improve call signature for makeCameraPoint [afw] +- `DM-8650 `_: Add static code analysis for python [qserv] +- `DM-8686 `_: Change Child Repo Access to Parent Registries [daf_persistence, obs_base, obs_decam, obs_sdss, obs_subaru, obs_test, pipe_base, pipe_tasks] +- `DM-8688 `_: testPsfSelectTest fails when run with via ``pytest *.py`` [meas_algorithms] +- `DM-8750 `_: eliminate jointcal compile warnings [jointcal] +- `DM-8823 `_: Create Pegasus plugin for Orca [ctrl_execute, ctrl_orca] +- `DM-8830 `_: Fix accounting for fraction of successful measurements [validate_drp] +- `DM-8842 `_: LeastSqFitter1d(..., unsigned int order) should be signed [meas_astrom] +- `DM-8914 `_: Improve container build in Jenkins [qserv] +- `DM-9019 `_: Advanced study on k8s (kubernetes) [qserv] +- `DM-9039 `_: reorder butler init to be more incremental when building Repositories [daf_persistence] +- `DM-9045 `_: Remove or revive bitrotted code in meas_modelfit [meas_modelfit] +- `DM-9049 `_: Enable autolinking in Doxygen [base] +- `DM-9050 `_: Add flags for sources used in astrometric and photometric calibration [meas_astrom, pipe_tasks] +- `DM-9060 `_: Add metadata access to get wcs, visitInfo, and calib from a calexp dataset [obs_base, obs_test] +- `DM-9111 `_: Add rotator information to HSC VisitInfo [afw, obs_lsstSim, obs_subaru, obs_test] +- `DM-9135 `_: bulk rename of jointcal variables [jointcal] +- `DM-9140 `_: Ensure __str__ and __repr__ copied from Swig [afw, meas_modelfit] +- `DM-9153 `_: Add metadata access to get filter, bbox, and detector [obs_base] +- `DM-9190 `_: Cleanup pybind11 remaining code [base, coadd_chisquared, coadd_utils, display_ds9, ip_diffim, ip_isr, meas_base, meas_extensions_photometryKron, meas_extensions_psfex, meas_extensions_shapeHSM, meas_extensions_simpleShape, obs_subaru, shapelet] +- `DM-9192 `_: Implement new spatially-variable PhotoCalib model [afw, meas_algorithms] +- `DM-9229 `_: Pad Psfs before PSF-matching [ip_diffim] +- `DM-9249 `_: Modify FlagHandler C++ and flagDecorator.py to make flag identification robust [ip_diffim, meas_algorithms, meas_base, meas_extensions_photometryKron, meas_extensions_shapeHSM, meas_extensions_simpleShape, meas_modelfit] +- `DM-9261 `_: Update git-lfs repositories to address deprecations. [afwdata, testdata_cfht, testdata_decam, testdata_subaru] +- `DM-9281 `_: Wrap sphgeom with pybind11 instead of swig [sphgeom] +- `DM-9297 `_: Get jointcal to work with pytest [jointcal, obs_lsstSim] +- `DM-9315 `_: MarkCompleted threw an exception when a Tracked job could not be located. [qserv] +- `DM-9316 `_: meas_base tests broken with NumPy 1.12 [ip_diffim, meas_base, meas_deblender, meas_modelfit] +- `DM-9321 `_: Change qserv to use xrootd features to improve performance on interactive queries. [qserv] +- `DM-9353 `_: Update configuration for HSC calib construction [obs_subaru] +- `DM-9364 `_: wcs creation is mandatory [obs_base, obs_cfht, obs_decam, obs_lsstSim, obs_sdss] +- `DM-9378 `_: remove and replace ctrl_events package dependencies from ctrl_orca [ctrl_execute, ctrl_orca] +- `DM-9394 `_: Add meas_extensions_convolved to lsst_distrib [lsst_distrib] +- `DM-9412 `_: Diffim datasets should specify a "level" [obs_base] +- `DM-9419 `_: lsst.log documentation should cite other lsst.log documentation [log] +- `DM-9423 `_: Port HSC patch to allow multiple filters in mosaic [obs_subaru] +- `DM-9431 `_: Please do not use Exception.message [obs_cfht] +- `DM-9433 `_: ds9.py error code not working as intended [afw] +- `DM-9434 `_: Fix database creation error in testTimeFuncs.py [cat] +- `DM-9438 `_: Switch default reference catalog for HSC to PS1 in LSST format [obs_subaru] +- `DM-9439 `_: Package version checking is non-deterministic [base] +- `DM-9476 `_: ISR fails in overscan for HSC visit=90738 ccd=33 [obs_subaru] +- `DM-9495 `_: Fix all jointcal header multiple-inclusion #defines [jointcal] +- `DM-9502 `_: SpherePoint throws wrong exception for invalid arguments [afw] +- `DM-9504 `_: lsst_py3 CI failure due to meas_extensions_ngmix [afw] +- `DM-9506 `_: Make astrometry distortion model configurable [jointcal] +- `DM-9520 `_: Remove calibrateTask.py example from Stack (for now) [pipe_tasks] +- `DM-9521 `_: Remove runIsrTask.py example from Stack (for now) [ip_isr] +- `DM-9523 `_: update lsstsw/newinstall EUPS version [lsst] +- `DM-9526 `_: add tarball production to weekly tag/release jenkins' job [lsst] +- `DM-9531 `_: Fix override warnings in afw [afw] +- `DM-9541 `_: Bug related to MPI pickling when running coaddDriver [ctrl_pool] +- `DM-9553 `_: Investigate the best algorithm to compute derivatives for the Brighter-Fatter correction [ip_isr] +- `DM-9556 `_: All NaNs in coord_ra and coord_dec columns in deepCoadd forced src tables [meas_base, meas_extensions_photometryKron, meas_modelfit, pipe_tasks] +- `DM-9564 `_: Set assembled Coadd Psf to modelPsf with auto-computed dimensions [ip_diffim, pipe_tasks] +- `DM-9567 `_: Create Donut Fit Pipeline [afw, obs_base, obs_subaru] +- `DM-9572 `_: Include instrument model in cameraGeom [afw, obs_base, obs_subaru] +- `DM-9574 `_: Log all measurement exceptions at debug level [meas_base] +- `DM-9577 `_: Ignore focus CCDs in singleFrameDriver [obs_subaru] +- `DM-9595 `_: Allow Transform to return its inverse [afw, astshim] +- `DM-9598 `_: Allow Transform to return a matrix of derivatives [afw] +- `DM-9599 `_: Support concatenation of Transforms [afw, astshim] +- `DM-9629 `_: Research: how to implement spatially varying PSF for spatially-varying ZOGY and AL(decorrelated) [ip_diffim] +- `DM-9669 `_: Butler(root="foo") should not warn about mapper class instance [daf_persistence] +- `DM-9670 `_: Reduce the time it takes to send a user query to workers. [qserv] +- `DM-9672 `_: Query deployment to the cluster hangs with very large number of chunks. [qserv] +- `DM-9706 `_: WISE object ids are strings and this is causing a problem with qserv [qserv] +- `DM-9747 `_: Fix build error in obs_lsstSim [pipe_tasks] +- `DM-9750 `_: Clean up and stackify matchPessimisticB code [meas_astrom] +- `DM-9751 `_: Verify the performance of new matchPessimisticB code on selected test fields [meas_algorithms, meas_astrom] +- `DM-9752 `_: Add jointcal to lsst_distrib [lsst_distrib] +- `DM-9757 `_: Add stat table usage options to mysql config file [qserv] +- `DM-9764 `_: SOURCE_IO_NO_FOOTPRINTS and related enums should be properly wrapped in pybind11 [afw, meas_base] +- `DM-9765 `_: Suspicious numerical precision code in Angle [afw] +- `DM-9776 `_: Install Kubernetes at CC-IN2P3 [qserv] +- `DM-9783 `_: Improve documentation of processing example data sets with validate_drp [validate_drp] +- `DM-9794 `_: Pass both LSST_LIBRARY_PATH and DYLD_LIBRARY_PATH in scons on Mac OS [sconsUtils] +- `DM-9795 `_: CModel priors are weighted incorrectly relative to likelihood [meas_modelfit, obs_subaru] +- `DM-9806 `_: update lsstsw/newinstall to EUPS 2.1.2 [lsst] +- `DM-9810 `_: Make PSFEx oversampling configurable [meas_extensions_psfex, obs_subaru] +- `DM-9811 `_: Add 1.3 arcsec target seeing for convolved flux measurement [obs_subaru] +- `DM-9812 `_: Clean up outputs from CharacterizeImageTask and CalibrateTask [meas_astrom, obs_base, obs_subaru, pipe_tasks] +- `DM-9828 `_: Enable rectangular binSizes in SubtractBackgroundTask [meas_algorithms] +- `DM-9829 `_: validate_drp broken on decam dataset as of 03/15 [validate_drp] +- `DM-9846 `_: Improve handling of error messages [astshim] +- `DM-9848 `_: obs_subaru test failures possibly related to daf_persistence [daf_persistence, obs_base] +- `DM-9853 `_: Clean up makeVisitInfo docstring [afw] +- `DM-9855 `_: Select images for coadd based on PSF quality. [obs_subaru, pipe_tasks] +- `DM-9856 `_: Add option to deblender to turn off smoothing. [meas_deblender] +- `DM-9862 `_: Update meas_mosaic's wcs/fcr output files to reflect LSST coordinate system [afw, meas_astrom] +- `DM-9863 `_: Replace use of makeVisitInfo(... with VisitInfo(...) [ip_isr, obs_base, obs_cfht, obs_decam, obs_lsstSim, obs_subaru, obs_test, pipe_tasks] +- `DM-9866 `_: Make change to remove flagDecorator (RFC-302) [meas_base] +- `DM-9871 `_: Move wcs and fcr datasets out of {pointing} directory in obs_subaru [obs_subaru] +- `DM-9872 `_: Validation of obs_comcam [obs_base] +- `DM-9874 `_: Increase matcher.numBrightStars [obs_subaru] +- `DM-9882 `_: Add integrate interface to BoundedField/ChebyshevBoundedField [afw] +- `DM-9885 `_: Rename deepCoadd_srcMatch as deepCoadd_measMatch [obs_base, obs_cfht, obs_decam, obs_lsstSim, obs_sdss, obs_subaru, pipe_tasks] +- `DM-9895 `_: FrameSet frames not preserved by Transform(frameSet) constructor [afw] +- `DM-9897 `_: conda channel errors causing lsstsw/bin/deploy to fail [lsst] +- `DM-9899 `_: FrameSet.permAxes will fail if nIn != nOut [astshim] +- `DM-9905 `_: Change WARN to DEBUG for raw data without a WCS [ip_isr] +- `DM-9909 `_: hsc cat format change breaks processing of validation_data_hsc [validate_drp] +- `DM-9917 `_: Add a callback to cameraGeom.showCamera [afw] +- `DM-9918 `_: Robustify sqlite use [daf_persistence] +- `DM-9925 `_: PolyTran should not provide an iterative inverse by default [astshim] +- `DM-9939 `_: Stop transposing data in Mapping.tranForward and tranInverse [afw, astshim] +- `DM-9946 `_: Remove debugging example refcount.cc [astshim] +- `DM-9952 `_: Change our interpretation of FITS HDUs to be 0-indexed [afw, obs_base] +- `DM-9961 `_: Add treecorr to validate_drp and lsst_ci dependencies [validate_drp] +- `DM-9967 `_: ctrl_pool should not accept a default for --time on real batch systems [ctrl_pool] +- `DM-9974 `_: Correct scheduling for certain interactive queries [qserv] +- `DM-9985 `_: Reduce Wcs pickle size [afw] +- `DM-9986 `_: Add a persistable for ``PropertyList`` [afw] +- `DM-9990 `_: Update Detector information as needed [afw, ip_isr] +- `DM-9998 `_: Enable fake sources on coadds [pipe_drivers, pipe_tasks] +- `DM-10000 `_: Create/revive CModel model data residuals display [meas_modelfit] +- `DM-10004 `_: Write brief guide to building PSF-matched coadds [ip_diffim, pipe_tasks] +- `DM-10006 `_: Don't overwrite Filter/Detector in _standardizeExposure [obs_base] +- `DM-10008 `_: MapBox.maxOutCoord not set to nout if specified as 0 during construction [astshim] +- `DM-10009 `_: Ensure masks are valid from ImageMapReduceTask [ip_diffim] +- `DM-10039 `_: Incorrect docs for CatalogCalculationConfig [meas_base] +- `DM-10042 `_: Update mariadb statistics on 35TB dataset [qserv] +- `DM-10069 `_: Remove boost_thread Qserv dependency [qserv] +- `DM-10091 `_: Fix problems left over from DM-9952 [afw, daf_persistence, obs_base] +- `DM-10093 `_: Revert disabling of meas_modelfit dependency in lsst_apps [lsst_apps] +- `DM-10096 `_: Add unit test asserts for SpherePoint, SpherePointList and PointList [afw, ip_diffim, ip_isr, meas_algorithms, meas_astrom, meas_base, meas_extensions_simpleShape, obs_base, obs_cfht, obs_decam, obs_lsstSim, obs_test, pipe_tasks, shapelet, skymap] +- `DM-10100 `_: Add qhttp C++ HTTP server library to qserv builds [qserv] +- `DM-10101 `_: bad exception handling in afw for python3 [afw] +- `DM-10105 `_: Inconsistency in meas/forced wcs leads to CModel failure [afw] +- `DM-10108 `_: Stop wrapping __div__ for BoundedField [afw] +- `DM-10116 `_: Add ChebyMap to astshim [astshim] +- `DM-10146 `_: Fix minor doc typos [astshim] +- `DM-10193 `_: Remove subaruLib import from crosstalkYagi [obs_subaru] +- `DM-10206 `_: Fix obs_decam compatibility with 0-indexed HDUs [obs_decam] +- `DM-10212 `_: Check memory locking in containers [qserv] +- `DM-10221 `_: Allow --id to use any key in the registry [pipe_base] +- `DM-10226 `_: Fix code layout in jointcal [jointcal] +- `DM-10229 `_: pipe_base tests try to write to obs_test [pipe_base] +- `DM-10231 `_: FileForWriteOnceCompareSame does not respect umask [daf_persistence] +- `DM-10233 `_: getInfoFromMetadata() throws away errors without warning. [pipe_tasks] +- `DM-10235 `_: Bug in coaddDriver when selecting images by PSF quality. [obs_subaru] +- `DM-10237 `_: Error in blendedness debiasing calculation [meas_base] +- `DM-10252 `_: getOutputId() assumes keys will exist, and doesn't use butler to retrieve them [pipe_drivers] +- `DM-10253 `_: Construct master calibs for obs_ctio0m9 [pipe_drivers] +- `DM-10257 `_: Fix compiler warnings in afw [afw] +- `DM-10263 `_: Add libcurl to qserv eups table file [qserv] +- `DM-10265 `_: Include table persistence docs in Doxygen listing for afw [afw] +- `DM-10267 `_: Port HSC support for PostgreSQL registries to LSST [daf_persistence, obs_base, pipe_tasks] +- `DM-10268 `_: Butler cannot read a repo using the realpath when it was created with a link [daf_persistence] +- `DM-10270 `_: isrTask does not provide config option for defects [ip_isr] +- `DM-10271 `_: Fix order of operations when using temporary local backgrounds in detection [afw, meas_algorithms] +- `DM-10274 `_: Transform is not assignable [afw] +- `DM-10281 `_: compiler warnings in astshim [astshim] +- `DM-10283 `_: Add ability to join different databases. [qserv] +- `DM-10286 `_: Update cmake in qserv base containers to latest from jessie-backports [qserv] +- `DM-10287 `_: Add measurement plugin to store footprint area [meas_base] +- `DM-10288 `_: afwImage.TanWcs.cast() not supported anymore in jointcalCoadd [jointcal] +- `DM-10289 `_: record.setValidPolygon(xxx) does not accept None as a valid input anymore [afw] +- `DM-10292 `_: The FrameSet returned by Transform.getFrameSet can change the contained FrameSet in Python [afw] +- `DM-10297 `_: Fix "double treerings" [obs_decam, pipe_tasks] +- `DM-10308 `_: Add support for box regions and magnitudes in brightObjectMasks [pipe_tasks] +- `DM-10314 `_: Uprade to kubernetes/kubeadm v1.6.1 on openstack [qserv] +- `DM-10315 `_: Mask bad channels in ccd=33 [obs_subaru] +- `DM-10328 `_: len(propertySet) throws an exception in Python [daf_base] +- `DM-10336 `_: DM-10271 seems to have broken afw [afw] +- `DM-10338 `_: Mix of tabs and spaces in breaks meas_base builds [meas_base] +- `DM-10340 `_: parent repository properties are dropped when loaded via child repositories. [daf_persistence, obs_test] +- `DM-10347 `_: Please fix the typo in afw/display/ds9.py [afw] +- `DM-10359 `_: Reconstruct measurement pixels from a SourceRecord [meas_base] +- `DM-10374 `_: Recover and load SDSS PDAC metadata [cat] +- `DM-10381 `_: Enhance test for meas_deblender's clipFootprintToNonzeroImpl [meas_deblender] +- `DM-10386 `_: Add Constructor documentation to Footprints [afw] +- `DM-10392 `_: Upgrade kubernetes/docker on cc-in2p3 cluster [qserv] +- `DM-10393 `_: correct variable name in sites.xml template [ctrl_platform_lsstvc] +- `DM-10394 `_: astshim attribute page titles have Doxygen commands [astshim] +- `DM-10401 `_: getPackageDir raises RuntimeError instead of pex::exceptions::NotFoundError [utils] +- `DM-10410 `_: Update starlink_ast [astshim] +- `DM-10426 `_: Identify stable version of kubernetes and docker on openstack [qserv] +- `DM-10429 `_: Write a version of the warper that uses SkyWcs and compare performance [afw] +- `DM-10430 `_: Add time stamps to the standard outputs to BatchCmdLineTask [ctrl_pool] +- `DM-10438 `_: Add DCR model data types [obs_base, obs_test] +- `DM-10448 `_: Use 79 character line length in utils [sconsUtils] +- `DM-10449 `_: Restore bad pixel masks [obs_subaru] +- `DM-10451 `_: Activate CModel prior fix [obs_subaru] +- `DM-10452 `_: Create bboxFromIraf function in obs_base utils [obs_base] +- `DM-10453 `_: Fix bugs in matchPessimisticB [meas_astrom] +- `DM-10469 `_: Add support for pipe_analysis in obs_subaru [obs_subaru] +- `DM-10485 `_: Protected methods should not use private method naming convention [astshim] +- `DM-10487 `_: obs_cfht needs to be updated in order to be compatible with latest CFHT data [obs_cfht] +- `DM-10490 `_: Cache camera in HscMapper [obs_subaru] +- `DM-10496 `_: test_chebyMap.py sometimes segfaults [astshim] +- `DM-10497 `_: Fix minor typos in main.dox [pipe_base] +- `DM-10504 `_: Make more stringent cuts to input visits for HSC [pipe_tasks] +- `DM-10505 `_: Robustify validate_drp fitting and catching errors. [validate_drp] +- `DM-10506 `_: Run blendedness by default [meas_base] +- `DM-10508 `_: Remove writing of warped template added in DM-8145 [ip_diffim] +- `DM-10510 `_: Squash some compiler warnings in afw [afw] +- `DM-10514 `_: Check qserv/qserv:dev works correctly [qserv] +- `DM-10521 `_: Create script to produce release performance table [validate_drp] +- `DM-10525 `_: Fix xrootd random crash [qserv] +- `DM-10526 `_: newinstall.sh should not default to using tarball binary packages [lsst] +- `DM-10530 `_: don't set filter if the filter ID is not UNKNOWN (instead of testing if filter is None) [obs_base] +- `DM-10535 `_: Incomplete subchunk list causing query to fail. [sphgeom] +- `DM-10541 `_: Add properties to image classes [afw, obs_base] +- `DM-10542 `_: Replace XYTransform::linearizeTransform [afw] +- `DM-10552 `_: Upgrade display_firefly to work with more servers [afw] +- `DM-10558 `_: disable or remove butler caching [daf_persistence, obs_base] +- `DM-10559 `_: afw.image.makeWcs() returns null pointer without warning [afw] +- `DM-10561 `_: Switch to LSST capitalization and properties in Python [afw, astshim] +- `DM-10562 `_: Provide POC for bundling xrootd as a microservice [qserv] +- `DM-10574 `_: Hit AssertionError in deblender [meas_deblender] +- `DM-10575 `_: obs_decam build takes 10 minutes [obs_decam] +- `DM-10587 `_: newinstall creates faulty loadLSST.csh file [lsst] +- `DM-10598 `_: Include pixels flags in forced measurement [meas_base] +- `DM-10602 `_: Cancel a user query when the result becomes too large. [qserv] +- `DM-10608 `_: Disable Config history [log, pex_config] +- `DM-10610 `_: Optimise SpanSet components used by CModel [afw] +- `DM-10621 `_: ArgumentParser's butler doesn't output calibs in the calib storage [pipe_base] +- `DM-10623 `_: Mismatching dataId in logger output [log, pipe_base] +- `DM-10633 `_: Increase CmdLineTask multiprocessing timeout [pipe_base] +- `DM-10686 `_: RingsSkyMap.findAllTracts() behaves oddly at poles [skymap] +- `DM-10728 `_: Near-term jointcal acceptance: make jointcal and meas_mosaic use the same output formats [afw, daf_persistence, obs_base, obs_subaru] +- `DM-10729 `_: Near-term jointcal acceptance: make validate_drp use meas_mosaic outputs [afw, validate_drp] +- `DM-10734 `_: Fix InfileMerger queryIdStr [qserv] +- `DM-10737 `_: Make meas_mosaic use new reference catalogs by default [obs_subaru] +- `DM-10749 `_: New filters for HSC [obs_subaru] +- `DM-10751 `_: Fix module versions [base, ctrl_pool, meas_base, meas_deblender, meas_extensions_psfex, obs_decam, obs_subaru, pipe_base, pipe_drivers, pipe_tasks, shapelet, skymap] +- `DM-10760 `_: Switch warpType from enum to strings [pipe_tasks] +- `DM-10764 `_: Rename Transform::of and Mapping::of to ::then [afw, astshim] +- `DM-10767 `_: fix py3 build after tickets/DM-7363 [daf_persistence] +- `DM-10777 `_: Create TransformBoundedField [afw, astshim] +- `DM-10778 `_: Add metadata access for Filter [obs_base, obs_test] +- `DM-10785 `_: setBrightObjectMasks does not properly construct footprint. [pipe_tasks] +- `DM-10799 `_: Rename Transform::tranForward to applyForward [afw, astshim] +- `DM-10803 `_: Refactor grid generation in ImageMapReduceTask [ip_diffim] +- `DM-10804 `_: Spatially-varying A&L decorrelation option [ip_diffim] +- `DM-10808 `_: Add defect on ccd=43 [obs_subaru] +- `DM-10818 `_: Fix array continuity for ndims > 2 in FITS writing [afw] +- `DM-10834 `_: CmdLineTask -j multiprocessing hangs with long data ID lists [afw] +- `DM-10835 `_: Add color terms for NB filters [obs_subaru] +- `DM-10837 `_: Fix eimageIsr import [obs_lsstSim] +- `DM-10846 `_: Enable automatic printing of C++ backtrace on segfault [sconsUtils, utils] +- `DM-10871 `_: Add unit test for MDC overwriting [log] +- `DM-10901 `_: Support not needing to specify all fields when creating calibrations. [pipe_drivers] +- `DM-10905 `_: Add jobId and retry number to result tables. [qserv] +- `DM-10924 `_: Add verify (and verify_metrics) to lsst_distrib [lsst_distrib] +- `DM-10926 `_: Incompatibility with NumPy 1.13 [afw] +- `DM-10930 `_: Robustify tests/repository.py [daf_persistence] +- `DM-10931 `_: Fix variable name bug and remove print statements in matchPessimisticB. [meas_astrom] +- `DM-10939 `_: Fix macOS compilation problems with Qserv [log, qserv] +- `DM-10942 `_: MatrixMap with all zeros cannot be simplified [astshim] +- `DM-10943 `_: References to TranForward and TranInverse are confusing [astshim] +- `DM-10945 `_: update eups to 2.1.3 to improve tarball package installation [lsst] +- `DM-10946 `_: Non-square MatrixMap composed with a ShiftMap cannot be simplified [astshim] +- `DM-10947 `_: Allow linearizeTransform and affineTransform to simplify their mappings [afw] +- `DM-10953 `_: Give ModelPsfMatchTask ablilty to match to all PSF types [afw, ip_diffim, meas_algorithms, meas_extensions_psfex] +- `DM-10965 `_: FootprintSet setter unable to accept results from getter [afw] +- `DM-10970 `_: Remove rows from cancelled jobs from the result table [qserv] +- `DM-10971 `_: Switch to using jemalloc. [qserv] +- `DM-10973 `_: Make SkyWcs transform to IcrsCoord instead of SpherePoint [afw] +- `DM-10997 `_: remove dependencies on lsst package [lsst_distrib] +- `DM-11008 `_: Unable to construct CoaddBoundedFieldElement without a validPolygon [meas_algorithms] +- `DM-11009 `_: Automatic backtrace printing is unhelpful [utils] +- `DM-11010 `_: Footprint.transform may be transforming the wrong position [afw] +- `DM-11022 `_: Create HSC calibs for 2016 September - 2017 April [pipe_tasks] +- `DM-11026 `_: Remove use of PsfAttributes from image subtraction tasks [datarel, ip_diffim, pipe_tasks] +- `DM-11031 `_: Expand defect mask on HSC ccd=43 [obs_subaru] +- `DM-11033 `_: Problem with exists/getStorage() with composite datatypes [daf_persistence, obs_base] +- `DM-11051 `_: Change logger level for wcs warning [afw] +- `DM-11052 `_: Enhance afw table to support variable-length string data [afw] +- `DM-11090 `_: Identify regression in HSC astrometric matching success rate between w_2017_17 and w_2017_25 [obs_subaru] +- `DM-11091 `_: Fix multibandDriver attempting to run detection when no data present [pipe_drivers] +- `DM-11095 `_: Add reference catalog flux support to photometry fitter [jointcal, testdata_jointcal] +- `DM-11136 `_: Fine-tune calib construction mechanics [obs_base, pipe_drivers, pipe_tasks] +- `DM-11137 `_: testSafeFileIO fails with too many open files on Python 3 and Terminal [daf_persistence] +- `DM-11159 `_: Push to docker-hub containers produced by Travis continuous integration [qserv] +- `DM-11163 `_: Always create VisitInfo from metadata when available [obs_base] +- `DM-11164 `_: Write suitable metadata for VisitInfo when writing calibrations [pipe_drivers] +- `DM-11166 `_: Fix typo in log message [obs_base] +- `DM-11172 `_: Test cancellation code for failed worker or network connection. [qserv] +- `DM-11190 `_: Immediate return for qserv disconnected queries [qserv] +- `DM-11191 `_: Result retrieval for qserv disconnected queries [qserv] +- `DM-11196 `_: Move yaml camera model to obs_base [obs_base] +- `DM-11215 `_: Enable validateDrp.py to run from JSON file. [validate_drp] +- `DM-11217 `_: Butler+CmdLineTask cannot output to a (non-repo) folder where some files already exist [daf_persistence, obs_decam] +- `DM-11218 `_: ingestCalibs.py should put the registry in the output directory [pipe_tasks] +- `DM-11219 `_: Please add an option --ignore-ingested to ingestCalibs.py [pipe_tasks] +- `DM-11220 `_: ingestImage.py should not stop after failing to ingest a file [pipe_tasks] +- `DM-11224 `_: Problems setting noclaim shutdown time via allocateNodes.py [ctrl_platform_lsstvc] +- `DM-11236 `_: Image constructors do not obey INHERIT [afw, obs_decam] +- `DM-11237 `_: Please turn down verbosity of sipterms.cc test [afw] +- `DM-11243 `_: Refine PSF-based image selection [pipe_tasks] +- `DM-11251 `_: add support for the standardize function for composite datasets [daf_persistence, obs_base] +- `DM-11267 `_: Add an option to not interpolate over saturated pixels [ip_isr] +- `DM-11269 `_: Please move "Empty WCS extension, using FITS header" from INFO to DEBUG [afw] +- `DM-11277 `_: Use posix interpretation of empty globs when ingesting [pipe_tasks] +- `DM-11280 `_: Update configs missed in DM-10469 [obs_subaru] +- `DM-11284 `_: Butler writes many copies of identical entries _parents to repositoryCfg.yaml [daf_persistence, obs_base] +- `DM-11289 `_: RepositoryArgs needs to check that mapperArgs is dict-like [daf_persistence] +- `DM-11296 `_: Butler loading registry messages are at inconsistent logging levels [obs_base] +- `DM-11300 `_: Fix kwargs passing (outputPrefix is None error) in validate_drp [validate_drp] +- `DM-11302 `_: Include DATE-OBS in calibration products headers [pipe_drivers] +- `DM-11306 `_: Remove pre-pybind11 hack from CalibTask.updateMetadata [pipe_drivers] +- `DM-11326 `_: Add _bbox dataset [afw, obs_base] +- `DM-11335 `_: obs_cfht tests fall over due to -9999.9 for TELEAZ for calibs [obs_cfht] +- `DM-11338 `_: Make choice of interpolation kernel for CoaddPsf configurable [meas_algorithms, pipe_tasks] +- `DM-11339 `_: Fix weirdly formatted check_astronomy plots [validate_drp] +- `DM-11351 `_: Fix display's writeBasicFits for MaskX change (used by e.g. display_ds9) [afw] +- `DM-11352 `_: Display a Wcs for the 0-indexed pixel coordinates when no other Wcs is present [afw] +- `DM-11353 `_: afw.cameraGeom.makeUpdatedDetector sometimes offsets amplifier bboxes when flipping [afw] +- `DM-11354 `_: Make afwDisplay callback message cut-and-pastable [afw] +- `DM-11360 `_: CatalogCalculation plugins can get added in non deterministic order [meas_base] +- `DM-11369 `_: ValueError from SafeClipAssembleCoaddTask.detectClip [pipe_tasks] +- `DM-11378 `_: Remove config option to make PSF-matched warps with old and wrong order of operations [pipe_tasks] +- `DM-11388 `_: Update hscConfig.py in validate_drp/config to get new location LoadAstrometryNetTask [validate_drp] +- `DM-11403 `_: Adapt to new exit feature in pipe_base [ctrl_pool] +- `DM-11410 `_: validate_drp incorrectly outputs filenames as '_.json' [validate_drp] +- `DM-11411 `_: On python3 sys.platform is linux and not linux2 [qserv] +- `DM-11421 `_: Fix unicode support in obs_lsstSim [obs_lsstSim] +- `DM-11429 `_: tests/testPhotoCal.py fails on 2017-07-31 "master" [afw] +- `DM-11442 `_: afw fails to install via eups if afwData not installed [afw] +- `DM-11454 `_: Modify UnitNormMap to round trip zero-length vectors [astshim] +- `DM-11463 `_: validate_drp plotPhotErrModelFit gives wrong units for sigmaSys [validate_drp] +- `DM-11467 `_: Fix docker image name generated with travis-ci [qserv] +- `DM-11487 `_: Fix OSX flakiness in qhttp unit test [qserv] +- `DM-11514 `_: Modify sconsUtils to use pytest for test execution [afw, jointcal, meas_base, meas_extensions_photometryKron, pex_config, verify] +- `DM-11514 `_: Modify sconsUtils to use pytest for test execution [afw] +- `DM-11518 `_: Modify db tests to support pytest [db] +- `DM-11538 `_: fix a few C++ compiler warns [qserv] +- `DM-11539 `_: meas_base test_PluginLogs.py depends on global state [meas_base] +- `DM-11574 `_: Make testDistortion test the distortion [obs_subaru] +- `DM-11579 `_: Support LOCAL and PARENT in bounding box syntactic sugar [afw] +- `DM-11585 `_: Add pytest support to packages [coadd_chisquared, coadd_utils, ctrl_execute, ctrl_orca, meas_extensions_simpleShape, meas_modelfit, pex_exceptions, pipe_base, skymap, utils, validate_drp] +- `DM-11589 `_: pipe_tasks unit test fails under pytest [pipe_tasks] +- `DM-11593 `_: Migrate qserv to pybind11 [qserv] +- `DM-11594 `_: test_distortion fails when run from pytest [obs_subaru] +- `DM-11595 `_: daf_persistence tests fail with pytest-xdist [daf_persistence] +- `DM-11599 `_: Migrate qserv LUA wrapper from SWIG to native C API [qserv] +- `DM-11607 `_: obs_base fails with pytest-xdist [obs_base] +- `DM-11625 `_: Bug in calculation of number-of-detected-in-bands in peak culling [pipe_tasks] +- `DM-11628 `_: Fix minor bugs in peak culling [pipe_tasks] From c0601214aa196d7f1a20cfa13b0d1636aabd4182 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 18 Sep 2017 13:50:45 -0700 Subject: [PATCH 22/35] Add OS prereq statement to Prerequisites page The Prerequisites page is likely the best page to state this information so that it can apply to both the newinstall.sh and lsstsw-based installation pathways. The reference platform statement is based on what we use at lsst-dev right now https://github.com/lsst/pipelines_lsst_io/pull/55#issuecomment-330275935 --- install/prereqs/index.rst | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/install/prereqs/index.rst b/install/prereqs/index.rst index c0717d7a..005351ef 100644 --- a/install/prereqs/index.rst +++ b/install/prereqs/index.rst @@ -4,6 +4,17 @@ Prerequisites This page lists software needed to install and use the LSST Science Pipelines. +.. _prereq-platforms: + +Platform compatibility +====================== + +The LSST Data Management reference platform is CentOS 7-1611. +This is the platform we officially develop, test, and operate with. + +Besides the reference platform, we generally expect the Pipelines to also compile and run under CentOS 6, Debian Linux (including Ubuntu), and macOS. +See `LSST Stack Testing Status `_ reports of building LSST software on various platforms. + .. _system-prereqs: System prerequisites From c0f3f1e41e04df83340a1c7b89d186201c10d89f Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 18 Sep 2017 13:53:36 -0700 Subject: [PATCH 23/35] Update install pages to point to common OS topic Both newinstall.sh and lsstsw-based installation pathways reference the same common topic on OS compatibility in the Prerequisites page. --- install/lsstsw.rst | 5 ++++- install/newinstall.rst | 5 +++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/install/lsstsw.rst b/install/lsstsw.rst index d4219db9..8bf80aeb 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -19,10 +19,13 @@ If you have difficulty installing LSST software: 1. Prerequisites ================ +The LSST Science Pipelines can generally be compiled on CentOS, Debian, and macOS platforms. +See :ref:`prereq-platforms` for information about LSST's official reference platform and build reports with other platforms. + Before you begin: - `Install and configure Git LFS `_ for LSST DM's data servers. -- Install prerequisites for your platform: :doc:`macOS `, :doc:`Debian / Ubuntu `, or :doc:`Centos / RedHat `. +- Install prerequisites for your platform: :doc:`CentOS / RedHat `, :doc:`Debian / Ubuntu `, or :doc:`macOS `. - If you opt not to use ``lsstsw`` \’s default Python environment you need to :ref:`install these Python dependencies `. .. _lsstsw-deploy: diff --git a/install/newinstall.rst b/install/newinstall.rst index 87083f7c..1a2daeed 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -15,12 +15,13 @@ If you have installation issues, here are some ways to get help: 1. Prerequisites ================ -You can install the LSST Science Pipelines on CentOS 7 (LSST's reference platform) as well as other Linux distributions and macOS (see the `LSST Stack Testing Status `_ for information on the platforms we have tested with). +The LSST Science Pipelines can be compiled and run on CentOS, Debian, and macOS. +See :ref:`prereq-platforms` for information about LSST's official reference platform and build reports with other platforms. Before you begin, install prerequisite software for your platform: - :doc:`macOS ` -- :doc:`Centos / RedHat ` +- :doc:`CentOS / RedHat ` - :doc:`Debian / Ubuntu ` .. From e16c7d4d817c8934d8d022cc60a5a91c816b854e Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 25 Sep 2017 12:11:43 -0700 Subject: [PATCH 24/35] Copy edit newinstall page --- install/newinstall.rst | 45 ++++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/install/newinstall.rst b/install/newinstall.rst index 1a2daeed..d5fe6ddf 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -5,10 +5,10 @@ Install with newinstall.sh and eups distrib This page guides you through installing the LSST Science Pipelines software. This installation method is recommended for anyone who uses or develops the Pipelines software. -If you have installation issues, here are some ways to get help: +If you have issues with the installation, here are two ways to get help: - Review the :ref:`known installation issues `. -- Ask a question in the `LSST Community support forum `_. +- Ask a question on the `LSST Community support forum `_. .. _newinstall-prereqs: @@ -32,8 +32,8 @@ Before you begin, install prerequisite software for your platform: 2. Make an installation directory ================================= -First, choose where you want to install the LSST Science Pipelines. -Create and change into that directory: +Create a directory where you want to install the LSST Science Pipelines into. +For example: .. code-block:: bash @@ -58,11 +58,11 @@ For most use cases we recommend downloading and running :command:`newinstall.sh` curl -OL https://raw.githubusercontent.com/lsst/lsst/14.0/scripts/newinstall.sh bash newinstall.sh -ct -Always execute `newinstall.sh` with bash, as shown, regardless of what shell you are in. +Always execute :command:`newinstall.sh` with :command:`bash`, as shown, regardless of what shell you're in. We recommend that you opt into the provided Miniconda Python environment (see the links below for more information). -Then load this environment into your shell: +Then load the LSST software environment into your shell: .. TODO Use sphinx-tabs here? @@ -80,7 +80,7 @@ Then load this environment into your shell: - The default Python environment is Python 3.5. **Python 2.7 users,** see :ref:`newinstall-py2`. - :ref:`newinstall-user-python`. - - The recommended installation uses precompiled binary tarballs if available for your platform (and falls back to a source build). + - The recommended installation uses precompiled binary tarballs if they're available for your platform (and falls back to a source build). See :ref:`newinstall-binary-packages`. If you will be compiling and linking C++ code against this installation you'll need to ensure your compilers match the distribution's. **Developers should review** :ref:`newinstall-binary-compatibility`. @@ -96,9 +96,9 @@ Then load this environment into your shell: 4. Install Science Pipelines packages ===================================== -Install the LSST Science Pipelines packages by running :command:`eups distrib install` for a top-level package and a tagged version. +Install the LSST Science Pipelines packages by running :command:`eups distrib install` for a :doc:`top-level package ` and a tagged version. -This example installs the ``v14_0_rc1`` tagged version (current release) of the ``lsst_distrib`` top-level package: +This example installs the ``v14_0_rc1`` tagged version (current release) of the ``lsst_distrib`` :doc:`top-level package `: .. code-block:: bash @@ -106,8 +106,10 @@ This example installs the ``v14_0_rc1`` tagged version (current release) of the curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python setup lsst_distrib -If pre-built binaries are available for your platform (and you ran :command:`newinstall.sh` with the :option:`-t ` argument) the installation should take roughly 10 minutes. -Otherwise, installation falls back to a source build which can take two hours, depending on the top-level package and your machine's performance. +If prebuilt binaries are available for your platform (and you ran :command:`newinstall.sh` with the :option:`-t ` argument) the installation should take roughly 10 minutes. +Otherwise, the installation falls back to a source build that can take two hours, depending on the top-level package and your machine's performance. + +.. TK add mention of how-to for debugging binary package root issues. The last command, :command:`setup`, activates the installed packages in your shell environment. You'll need to run :command:`setup` in each shell session you'll use the LSST Science Pipelines in. @@ -115,7 +117,7 @@ See :doc:`setup` for more information. .. note:: - - ``lsst_distrib`` is a top-level package that brings-in most LSST Data Management pipelines software, but other top-level packages may be more applicable for your work, such as ``lsst_apps`` or ``lsst_sims``. + - ``lsst_distrib`` is a top-level package that provides most LSST Data Management pipelines software, but other top-level packages may be more applicable for your work, such as ``lsst_apps`` or ``lsst_sims``. See :doc:`top-level-packages` for more information. - ``v14_0_rc1`` is the current release. @@ -168,7 +170,7 @@ These topics provide additional information about the installation and ways to c Setting unix permissions for shared installations ------------------------------------------------- -You can make the LSST Science Pipelines installation accessible to multiple users on the same machine. +You can make a single LSST Science Pipelines installation accessible to multiple users on the same machine. First, create a separate unix group (called ``lsst``, for example) with a ``umask`` of ``002`` (all access permissions for the group and allow other users to read/execute). @@ -198,8 +200,8 @@ You activate this environment in a shell by sourcing the :command:`loadLSST` scr Here is how :command:`newinstall.sh` prepares the environment: -- Identifies your operating system and compilers to determine what EUPS binary packages should be installed (the EUPS package root, see :ref:`newinstall-binary-packages`). -- Installs a specific version of Python, through Miniconda, that is compatible with EUPS binary packages (see :ref:`newinstall-miniconda`). +- Identifies your operating system and compilers to determine what EUPS binary packages should be installed (the *EUPS package root,* see :ref:`newinstall-binary-packages`). +- Installs a specific version of Python, through Miniconda_, that is compatible with EUPS binary packages (see :ref:`newinstall-miniconda`). - Installs Conda packages that the LSST Science Pipelines depends on (see :ref:`python-deps`). - Checks for :command:`git` on your systems and offers to install it if necessary. - Installs EUPS_, the package manager used by the LSST software stack. @@ -255,8 +257,8 @@ How to use your own Python with newinstall.sh :command:`newinstall.sh` creates a new Python environment by default (pre-configured with Python dependencies). If necessary, you can use your own pre-existing Python environment. -To do so, run :command:`newinstall.sh` (see :command:`newinstall-run` for details and command arguments). -When :command:`newinstall-run` prompts you to install Miniconda, type ``no``. +To do so, run :command:`newinstall.sh` (see :ref:`newinstall-run` for details and command arguments). +When :command:`newinstall.sh` prompts you to install Miniconda, type ``no``. Be aware of these caveats when using your own Python installation: @@ -409,23 +411,24 @@ newinstall.sh argument reference .. option:: -t - Use pre-compiled EUPS "tarball" packages, if available. + Allows :command:`eups distrib install` to install prebuilt binary (tarball) packages, where available .. option:: -T - **Do not** use pre-compiled EUPS "tarball" packages. (**default**) + Prevents :command:`eups distrib install` from installing prebuilt binary (tarball) packages. (**default**) .. option:: -s - Use EUPS source "eupspkg" packages, if available (compile from source). + Allows :command:`eups distrib install` to compile and install packages from source (eupspkg). (**default**) .. option:: -S **Do not** use EUPS source "eupspkg" packages (do not compile from source). + Prevents :command:`eups distrib install` from compiling and installing packages from source (eupspkg). .. option:: -h - Display this help message. + Display a help message. .. _Miniconda: https://conda.io/miniconda.html From fd77cafe6e42aaa8d555a949aea7a2c5a946aca4 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 25 Sep 2017 13:20:15 -0700 Subject: [PATCH 25/35] Installation guide copy edits - index - docker installation - lsstsw installation - set up topic - top-level package topic --- index.rst | 2 +- install/docker.rst | 16 ++++++++++--- install/lsstsw.rst | 41 ++++++++++++++++++---------------- install/setup.rst | 2 +- install/top-level-packages.rst | 2 +- 5 files changed, 38 insertions(+), 25 deletions(-) diff --git a/index.rst b/index.rst index c38e7f2c..46ec2683 100644 --- a/index.rst +++ b/index.rst @@ -22,7 +22,7 @@ Alternative distributions and installation methods: - :doc:`install/docker` - :doc:`Installing from source with lsstsw ` -- `CernVM FS `_ (contributed by IN2P3) +- `CernVM FS `_ (contributed by CC-IN2P3) To install the LSST Simulation software, such as MAF, follow the `LSST Simulations documentation `_. diff --git a/install/docker.rst b/install/docker.rst index c68436c5..394fc073 100644 --- a/install/docker.rst +++ b/install/docker.rst @@ -6,8 +6,9 @@ Running with Docker LSST provides versioned Docker images containing the Science Pipelines software. With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. +Docker is an effective and reliable alternative to the :doc:`newinstall.sh ` and :doc:`lsstsw`\ -based methods that install LSST software directly on your system. -If you have issues, reach out on the `LSST Community support forum `_. +If you have issues using the LSST Docker images, reach out on the `LSST Community support forum `_. .. _docker-prereqs: @@ -38,9 +39,17 @@ Then in the container's shell, load the LSST environment and set up a :doc:`top- source /opt/lsst/software/stack/loadLSST.bash setup lsst_distrib -This step is equivalent to the :doc:`setup instructions ` for a :doc:`newinstall.sh `\ -based installation. +This step is equivalent to the :doc:`set up instructions ` for a :doc:`newinstall.sh `\ -based installation. In fact, the images are internally based on :command:`newinstall.sh`. +When you're done with the container, exit from the container's shell: + +.. code-block:: bash + + exit + +This will return you to the original shell on your host system. + Next, learn more with these topics: - :ref:`docker-tags` @@ -51,7 +60,7 @@ Finding images for different LSST Science Pipelines releases ============================================================ LSST Science Pipelines Docker images are published as `lsstsqre/centos`_ on Docker Hub. -These images are based on the CentOS base image. +These images are based on a CentOS_ base image. Docker images are versioned with tags, allowing you to run any release of the LSST Science Pipelines software. The schema of these tags is: @@ -75,3 +84,4 @@ This tag corresponds to: You can see what tags are available by browsing `lsstsqre/centos on Docker Hub `_. .. _`lsstsqre/centos`: https://hub.docker.com/r/lsstsqre/centos/ +.. _CentOS: https://www.centos.org diff --git a/install/lsstsw.rst b/install/lsstsw.rst index 8bf80aeb..7dc5ef3c 100644 --- a/install/lsstsw.rst +++ b/install/lsstsw.rst @@ -2,17 +2,16 @@ Installation with lsstsw and lsst-build ####################################### -This page will guide you through installing the LSST Science Pipelines from source with lsstsw and lsst-build. +This page guides you through installing the LSST Science Pipelines from source with lsstsw and lsst-build. These are the same tools LSST Data Management uses to build and test the Science Pipelines. -Since lsstsw presents the Science Pipelines as a directory of Git repositories clone from `github.com/lsst `__, this installation method can be very convenient for developing Science Pipelines code. +Since lsstsw presents the Science Pipelines as a directory of Git repositories cloned from `github.com/lsst `__, this installation method can be very convenient for developing Science Pipelines code. +Other methods of installing LSST Science Pipelines software are :doc:`newinstall.sh ` and :doc:`Docker `. -Science Pipelines developers should also consult the `DM Developer Guide `_, and the `Workflow `_ page in particular. +If you have issues using lsstsw, here are two ways to get help: -If you have difficulty installing LSST software: - -- review the :ref:`known installation issues for your platform `. -- reach out on the `Support forum at community.lsst.org `_. +- Review the :ref:`known installation issues `. +- Ask a question on the `LSST Community support forum `_. .. _lsstsw-prerequisites: @@ -26,16 +25,13 @@ Before you begin: - `Install and configure Git LFS `_ for LSST DM's data servers. - Install prerequisites for your platform: :doc:`CentOS / RedHat `, :doc:`Debian / Ubuntu `, or :doc:`macOS `. -- If you opt not to use ``lsstsw`` \’s default Python environment you need to :ref:`install these Python dependencies `. +- If you opt not to use the default Python environment provided by ``lsstsw`` you will need to :ref:`install these Python dependencies `. .. _lsstsw-deploy: 2. Deploy lsstsw ================ -.. Code for the LSST Stack is distributed across many Git repositories (see `github.com/lsst `_). -.. `lsstsw `_ is a tool that helps you manage the codebase by automating the process of cloning all of these repositories and building that development Stack for testing. - Begin by choosing a working directory, then deploy ``lsstsw`` into it: .. code-block:: bash @@ -45,10 +41,7 @@ Begin by choosing a working directory, then deploy ``lsstsw`` into it: ./bin/deploy source bin/setup.sh -For more information about the :command:`deploy` command, such as using Python 2.7 instead of 3, see :ref:`lsstsw-about-deploy`. - -.. The ``setup.sh`` step enables EUPS_, the package manager used by LSST. -.. **Whenever you open a new terminal session, you need to run '. bin/setup.sh' to activate your lsstsw environment.** +For more information about the :command:`deploy` command, such as using Python 2.7 instead of Python 3, see :ref:`lsstsw-about-deploy`. .. _lsstsw-rebuild: @@ -77,6 +70,15 @@ Finally, set up the packages with EUPS: See :doc:`setup` for more information. +.. note:: + + You can do more with the :command:`build` command, including building from branches of GitHub repositories. + For more information: + + - :ref:`lsstsw-about-rebuild`. + - :ref:`lsstsw-branches`. + - :ref:`lsstsw-rebuild-ref`. + .. _lsstsw-testing-your-installation: 4. Testing Your installation (optional) @@ -112,7 +114,7 @@ Run these two steps: setup lsst_distrib - Instead of ``lsst_distrib``, you can set up a different top-level package like ``lsst_apps`` or any individual EUPS package. + Instead of ``lsst_distrib``, you can set up a different top-level package like ``lsst_apps`` or any individual EUPS package you previously installed. See :doc:`top-level-packages`. .. _lsstsw-next: @@ -170,12 +172,12 @@ About the lsstsw rebuild command The :command:`rebuild` command accomplishes the following: 1. Clones all Science Pipelines packages from `github.com/lsst `__. - :command:`rebuild` + The `repos.yaml`_ file in the https://github.com/lsst/repos repository maps package names to GitHub repositories. 2. Runs the Scons-based build process to compile C++, make Pybind11 bindings, and ultimately create the :lmod:`lsst` Python package. The stack is built and installed into the :file:`stack/` directory inside your :file:`lsstsw/` work directory. -``lsstsw`` clones repositories using HTTPS (`see repos.yaml `_. +lsstsw clones repositories using HTTPS (`see repos.yaml `_). Our guide to `Setting up a Git credential helper `_ will allow you to push new commits up to GitHub without repeatedly entering your GitHub credentials. See also: :ref:`lsstsw-rebuild-ref`. @@ -185,7 +187,7 @@ See also: :ref:`lsstsw-rebuild-ref`. Building from branches ---------------------- -lsstsw's :command:`rebuild` enables you to clone and build development branches. +lsstsw's :command:`rebuild` command enables you to clone and build development branches. To build ``lsst_distrb``, but use the Git branch ``my-feature`` when it's available in a package's repository: @@ -267,3 +269,4 @@ lsstsw rebuild command reference .. _versiondb: https://github.com/lsst/versiondb .. _EUPS: https://github.com/RobertLuptonTheGood/eups .. _Miniconda: http://conda.pydata.org/miniconda.html +.. _`repos.yaml`: https://github.com/lsst/repos/blob/master/etc/repos.yaml diff --git a/install/setup.rst b/install/setup.rst index df54ca1f..c9f3814a 100644 --- a/install/setup.rst +++ b/install/setup.rst @@ -4,7 +4,7 @@ Setting up installed LSST Science Pipelines ########################################### -Whenever you start a new command-line shell you need to set up the LSST Science Pipelines software before you can use it. +Whenever you start a new command-line shell, you need to set up the LSST Science Pipelines software before you can use it. .. _setup-howto: diff --git a/install/top-level-packages.rst b/install/top-level-packages.rst index ae33acc7..4ee68ab0 100644 --- a/install/top-level-packages.rst +++ b/install/top-level-packages.rst @@ -3,7 +3,7 @@ Top-level packages to install ############################# The LSST Science Pipelines are part of LSST's EUPS_ package stack. -This means that the Science Pipelines software is actually a collection of packages that you install and setup together. +This means that the Science Pipelines software is actually a collection of packages that you install and set up together. By specifying different **top-level packages** to the :ref:`eups distrib install ` and :doc:`setup ` commands, you can control the size of the software installation or add new capabilities. This page describes the common top-level packages that make up the LSST Science Pipelines and related EUPS stacks. From e16d5ed2b5705ddd3ceab3843e23f046d98f2d7b Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 25 Sep 2017 14:24:00 -0700 Subject: [PATCH 26/35] Add topic on debugging binary package availability We're seeing users struggle with knowing why binaries aren't being used. The goal of this topic is to give users some strategies for determining why they aren't getting binaries. --- install/newinstall.rst | 46 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/install/newinstall.rst b/install/newinstall.rst index d5fe6ddf..869d5705 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -108,6 +108,7 @@ This example installs the ``v14_0_rc1`` tagged version (current release) of the If prebuilt binaries are available for your platform (and you ran :command:`newinstall.sh` with the :option:`-t ` argument) the installation should take roughly 10 minutes. Otherwise, the installation falls back to a source build that can take two hours, depending on the top-level package and your machine's performance. +See :ref:`newinstall-find-binaries`. .. TK add mention of how-to for debugging binary package root issues. @@ -161,6 +162,7 @@ These topics provide additional information about the installation and ways to c - :ref:`newinstall-py2`. - :ref:`newinstall-user-python`. - :ref:`newinstall-binary-packages`. +- :ref:`newinstall-find-binaries`. - :ref:`newinstall-binary-compatibility`. - :ref:`newinstall-other-tags`. - :ref:`newinstall-reference`. @@ -316,6 +318,50 @@ Here is an example of the output: Based on this example, :command:`eups distrib install` will preferentially install EUPS distrib binary packages for the macOS 10.9 system, ``clang-800.0.42.1`` compiler, and ``miniconda3-4.2.12-7c8e67`` Python and lsstsw combination. If :command:`eups distrib install` cannot find packages at that EUPS package root it will look in the second EUPS package root (https://eups.lsst.codes/stack/src), which provides source packages. +**See also:** + +- :ref:`newinstall-find-binaries` +- :ref:`newinstall-binary-compatibility` + +.. _newinstall-find-binaries: + +How to determine if tarball packages are available for your platform +-------------------------------------------------------------------- + +When you run :ref:`eups distrib install `, it will attempt to install prebuilt binary packages first and fall back to compiling the Science Pipelines if binary packages aren't available for your platform (by default). +This fallback is automatic. +You'll know packages are being compiled from source if you see compiler processes (like :command:`gcc` or :command:`clang`) load your machine. + +The instructions in this section will help you diagnose *why* :command:`eups distrib install` is falling back to a source installation. + +First, get your EUPS package root URLs: + +.. code-block:: bash + + eups distrib path + +If the only URL listed is https://eups.lsst.codes/stack/src, it means that :command:`newinstall.sh` configured your environment to not use binary packages. +Try re-running :command:`newinstall.sh` (see :ref:`newinstall-run`) with the :option:`-t ` argument. +Also, ensure that you accept the default Miniconda Python environment. + +If :command:`eups distrib path` includes an additional URL that doesn't end with ``/src`` (for example, ``https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67``), it means :command:`newinstall.sh` has configured a binary package root. +The construction of the binary package root URL is based on your OS, compilers, and Python environment (see :ref:`newinstall-binary-packages`). + +:command:`eups distrib install` will only install binary packages if they exist on the binary package root. +To check this, open the binary package root URL in a web browser. +If the binary package root URL does not load in a browser it means LSST does not publish prebuilt binaries for your platform. +Either continue the installation from source or consider using the :doc:`LSST Docker images `. + +If the URL does open, though, search for files with a ``.list`` extension. +A ``.list`` file is created for each release that has binary packages. +The name of the ``.list`` file matches the release tag (for example, ``w_2017_33.list``). +See :ref:`newinstall-other-tags` for more information about tags. + +For example, if the binary package root is ``https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67`` and you wish to install the ``w_2017_33`` tag, the file ``https://eups.lsst.codes/stack/osx/10.9/clang-800.0.42.1/miniconda3-4.2.12-7c8e67/w_2017_33.list`` must exist for a binary installation. + +If the ``.list`` file does not exist for the tag you want to install, but does exist for other tags in that EUPS package root, it may be due to an issue with the LSST binary package publishing system. +You can either continue with an installation from source, consider switching to a tag that is known to have binary packages, or consider using :ref:`LSST's Docker images `. + .. _newinstall-binary-compatibility: EUPS tarball packages and compiler compatibility From 57417142c55d4065baf494ada0d5a09e25ca4684 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 25 Sep 2017 17:27:08 -0700 Subject: [PATCH 27/35] Add topic on how to develop a package This covers the mechanics of setting up cloned packages and building them. It isn't intended to cover all development topics---just topics related to package installation. --- index.rst | 4 + install/index.rst | 1 + install/package-development.rst | 228 ++++++++++++++++++++++---------- 3 files changed, 166 insertions(+), 67 deletions(-) diff --git a/index.rst b/index.rst index 46ec2683..d2721d94 100644 --- a/index.rst +++ b/index.rst @@ -24,6 +24,10 @@ Alternative distributions and installation methods: - :doc:`Installing from source with lsstsw ` - `CernVM FS `_ (contributed by CC-IN2P3) +Related topics: + +- :doc:`install/package-development` + To install the LSST Simulation software, such as MAF, follow the `LSST Simulations documentation `_. .. This toctree is hidden to let us curate the section above, but still add the install/ pages to the Sphinx toctree diff --git a/install/index.rst b/install/index.rst index ccb2894f..11753506 100644 --- a/install/index.rst +++ b/install/index.rst @@ -14,3 +14,4 @@ Installing the LSST Science Pipelines docker lsstsw demo + package-development diff --git a/install/package-development.rst b/install/package-development.rst index 873379f3..a646b10d 100644 --- a/install/package-development.rst +++ b/install/package-development.rst @@ -1,67 +1,161 @@ -:orphan: 1 - -.. NOTE: this page is not currently part of the built site, though it is intended for re-integration. - -############################################################### -Developing a package with the installed Science Pipelines stack -############################################################### - -An lsstsw-based installation is great for developing packages against the LSST Science Pipelines stack. -The `Developer Guide describes Data Management's workflow `__, but this section will get your started with the basics related to lsstsw and EUPS. - -1. Stack packages are found in the :file:`lsstsw/build/` directory. - -2. Create a new branch in a package's Git repository, - - .. code-block:: bash - - git checkout -b {{ticket-name}} - - Then declare this package for EUPS and set it up: - - .. code-block:: bash - - eups declare -r . -t $USER {{package_name}} git - setup -r . -t $USER - - Unpacking the ``eups declare`` arguments: - - - ``-r .`` is the path to the package's repository, which is the current working directory. - You don't *need* to be in the repository's directory if you provide the path appropriately. - - ``-t $USER`` sets the EUPS *tag*. - We use this because your username (``$USER``) is an allowed EUPS tag. - - ``git`` is used as an EUPS *version*. - Semantically we default to calling the version "``git``" to indicate this package's version is the HEAD of a Git development branch. - - In the above ``eups declare`` command we associated the package version "``git``" with the tag "``$USER``." - In running ``setup``, we told EUPS to setup the package *and its dependencies* with the version associated to the ``$USER`` tag. - If the ``$USER`` tag isn't found for dependencies, EUPS will revert to using versions of dependencies linked to the ``current`` tag. - This is why we initially declared the entire lsstsw repository to have the version ``current``. - -3. Build the package with Scons: - - .. code-block:: bash - - scons -Q -j 6 opt=3 - - These flags tell SCons to build with flags: - - - ``-Q``: reduce logging to the terminal, - - ``-j 6``: build in parallel (e.g., with '6' CPUs), - - ``opt=3``: build with level 3 optimization. - - This ``scons`` command will run several targets by default, in sequence: - - 1. ``lib``: build the C++ code and SWIG interface layer - 2. ``python``: install the Python code - 3. ``tests``: run the test suite - 4. ``example``: compile the examples, - 5. ``doc``: compile Doxygen-based documentation, and - 6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). - - You can build a subset of these targets by specifying one explicitly. - To simply compile C++, SWIG, build the Python package and run tests: - - .. code-block:: bash - - scons -q -j 6 opt=3 tests +############################################################# +Building a package with the installed Science Pipelines stack +############################################################# + +You can build packages alongside an LSST Science Pipelines stack that you have installed with :doc:`newinstall.sh `, :doc:`lsstsw `, or :doc:`Docker `. +This pages describes how to build and set up packages cloned directly from GitHub. + +.. _package-dev-prereq: + +0. Prerequisites +================ + +Before developing an individual package, you need an installed Science Pipelines stack. +You can install the Pipelines through any of these methods: :doc:`newinstall.sh `, :doc:`lsstsw `, or :doc:`Docker `. + +Then set up the Pipelines software in a shell. +See :doc:`setup` for more information. + +.. warning:: + + **newinstall.sh users:** read :ref:`newinstall-binary-compatibility` if the installed Pipelines stack uses prebuilt binary packages. + +.. _package-dev-clone: + +1. Clone a package +================== + +Use Git to clone the package you want to work on. +Most LSST packages are available in the LSST's GitHub organization (https://github.com/lsst). +The `repos.yaml file in the 'repos' repository `_ also maps package names to repository URLs. + +For example: + +.. code-block:: bash + + git clone https://github.com/lsst/pipe_tasks + cd pipe_tasks + +You can also create a new package repository, though this is beyond this document's scope. + +You will work from a Git branch when developing a package. +The `DM Developer Guide `_ describes the branching workflow that LSST staff use. + +.. note:: + + **Docker users:** Clone the package onto your host filesystem rather then directly into the container by mounting a host directory in the container. + See :ref:`docker-develop`. + +.. warning:: + + **lsstsw users:** The :file:`lsstsw/build` directory already includes clones of Git repositories. + These repositories are reset when you run :ref:`rebuild `, though, so you can potentially lose local changes. + It's usually better to clone and work with Git repositories outside of the :file:`lsstsw` directory. + +.. _package-dev-setup: + +2. Set up the package +===================== + +From the package's directory, set up the package itself in the EUPS stack: + +.. code-block:: bash + + setup -r . -t $USER + +.. _package-dev-scons: + +3. Build the package with Scons +=============================== + +.. code-block:: bash + + scons -Q -j 6 opt=3 + +These flags configure Scons: + +- ``-Q``: reduce logging to the terminal. +- ``-j 6``: build in parallel (for example, with '6' CPUs). +- ``opt=3``: build with level 3 optimization. + Use ``opt=0`` (or ``opt=g`` with gcc compilers) for debugging. + +This ``scons`` command will run several targets by default, in sequence: + +1. ``lib``: build the C++ code and Pybind11 interface layer. +2. ``python``: install the Python code. +3. ``tests``: run the unit tests. +4. ``example``: compile the examples. +5. ``doc``: compile Doxygen-based documentation. +6. ``shebang``: convert the ``#!/usr/bin/env`` line in scripts for OS X compatibility (see `DMTN-001 `_). + +You can build a subset of these targets by specifying one explicitly. +For example, to compile C++, build the Python package and run tests: + +.. code-block:: bash + + scons -Q -j 6 opt=3 tests + +.. _package-dev-next-steps: + +Next steps +========== + +By following these steps, you have built a package from source alongside an installed Science Pipelines software stack. +Now when you run the Science Pipelines, your new package will be used instead of the equivalent package provided by the Science Pipelines installation. +Here are some tasks related to maintaining this development software stack: + +- :ref:`package-dev-eups-list`. +- :ref:`package-dev-setup-shell`. +- :ref:`package-dev-unsetup`. + +.. _package-dev-eups-list: + +Reviewing set up packages +------------------------- + +Packages that are *set up* are part of the active Science Pipelines software stack. +You can see what packages are currently set up by running: + +.. code-block:: bash + + eups list -s + +You can also review what version of a single package is set up by running: + +.. code-block:: bash + + eups list + +.. _package-dev-setup-shell: + +Setting up in a new shell +------------------------- + +Whenever you open a new shell you need to set up both the LSST software environment and the LSST software stack. +See :doc:`setup` for the basic procedure. + +In addition to setting up the installed Science Pipelines software, you separately need to set up the development package itself. +You can do this following the instruction in step :ref:`package-dev-setup`. + +.. _package-dev-unsetup: + +Un-set up the development package +--------------------------------- + +You can un-set up a development package to revert to the installed LSST Science Pipelines distribution. + +To switch from a development package to the released package: + +.. code-block:: bash + + setup -j -t current + +``current`` is the default tag normally used for the installed LSST Science Pipelines software stack. + +To un-set up a development package without replacing it: + +.. code-block:: bash + + unsetup -j -t $USER + +This is useful if you are developing a new package that is not part of the installed LSST Science Pipelines software stack. From ac04c1a35174e9074b5fbb9ba436237e6e50e196 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 26 Sep 2017 13:24:24 -0700 Subject: [PATCH 28/35] Document Docker development workflow This documents how to develop packages within docker containers. Includes section on mounting volumes and running containers in the background, as well, which are critical patterns for containerized development. --- install/docker.rst | 177 +++++++++++++++++++++++++++++++- install/package-development.rst | 5 + 2 files changed, 179 insertions(+), 3 deletions(-) diff --git a/install/docker.rst b/install/docker.rst index 394fc073..3c0f1ad1 100644 --- a/install/docker.rst +++ b/install/docker.rst @@ -6,7 +6,7 @@ Running with Docker LSST provides versioned Docker images containing the Science Pipelines software. With Docker, you can quickly install download and run the LSST Science Pipelines on any platform without compiling from source. -Docker is an effective and reliable alternative to the :doc:`newinstall.sh ` and :doc:`lsstsw`\ -based methods that install LSST software directly on your system. +Docker is an effective and reliable alternative to the :doc:`newinstall.sh ` and :doc:`lsstsw `\ -based methods that install LSST software directly on your system. If you have issues using the LSST Docker images, reach out on the `LSST Community support forum `_. @@ -26,7 +26,7 @@ Docker's `Getting Started `_ documentation Quick start =========== -This command downloads a weekly build of the LSST Science Pipelines, starts a container, and opens a prompt: +This command downloads a :ref:`weekly build ` of the LSST Science Pipelines, starts a container, and opens a prompt: .. code-block:: bash @@ -48,12 +48,183 @@ When you're done with the container, exit from the container's shell: exit -This will return you to the original shell on your host system. +This returns you to the original shell on your host system. Next, learn more with these topics: +- :ref:`docker-mount` +- :ref:`docker-detached` +- :ref:`docker-develop` - :ref:`docker-tags` +.. _docker-mount: + +How to mount a host directory into a container +============================================== + +When you run a Docker container, you're working inside a system that is isolated from your host machine. +The container's filesystem is distinct from your host machine's. + +You can mount a host directory into the container, however. +When you mount a host directory to a container, the data and code that resides on your host filesystem is accessible to the container's filesystem. +This is useful for processing data with the LSST Science Pipelines and even developing packages for the Science Pipelines. + +To mount a local directory, add a ``-v /`` argument to the :command:`docker run` command. +For example: + +.. code-block:: bash + + docker run -it -v `pwd`:/home/vagrant/mnt lsstsqre/centos:7-stack-lsst_distrib-w_2017_35 + +The example mounts the current working directory (```pwd```) to the ``/home/vagrant/mnt`` directory in the container. +If you run :command:`ls` from the container's prompt you should see all files in the current working directory of the host filesystem: + +.. code-block:: bash + + ls mnt + +Note that ``/home/vagrant`` is the default home directory for LSST Science Pipelines Docker images. + +As usual with interactive mode (``docker run -it``), you can ``exit`` from the container's shell to stop the container and return to the host shell: + +.. code-block:: bash + + exit + +.. _docker-detached: + +How to run a container in the background and attach to it +========================================================= + +The :ref:`docker-quick-start` showed you how to run a container in interactive mode. +In this mode, Docker immediately opens a shell in the new container. +When you ``exit`` from the shell, the container stops. + +An alternative is to run a container in a detached state. +With a detached container, the container won't stop until you specify it. + +To get started, run the container with the ``-d`` flag (**detached**): + +.. code-block:: bash + + docker run -itd --name lsst lsstsqre/centos:7-stack-lsst_distrib-w_2017_35 + +You still use the ``-it`` arguments to put the container in interactive mode, even though Docker doesn't immediately open a container prompt for you. + +The ``--name lsst`` argument gives the new container a name. +You can choose whatever name makes sense for your work. +This example uses the name "``lsst``." + +Next, from a shell on your host system (the same shell as before, or even a new shell) open a shell in the container with the :command:`docker exec` command: + +.. code-block:: bash + + docker exec -it lsst /bin/bash + +Your prompt is now a prompt in the container. + +You can repeat this process, attaching to the container multiple times, to open multiple container shells. + +To close a container shell, type ``exit``. + +Finally, to stop the container entirely, run this command from your host's shell: + +.. code-block:: bash + + docker stop lsst + +And delete the container: + +.. code-block:: bash + + docker rm lsst + +.. _docker-develop: + +How to develop packages inside Docker containers +================================================ + +You can develop code, including LSST Science Pipelines packages, with the LSST Science Pipelines Docker images. +This section summarizes the containerized development workflow. +Refer to :doc:`package-development` for general information. + +Basic set up +------------ + +These steps show how to run a container and build a LSST Science Pipelines package in it: + +1. **From the host shell,** clone packages into the current working directory. + For example: + + .. code-block:: bash + + git clone https://github.com/lsst/pipe_tasks + + Any datasets you're working with should be in the current working directory as well. + +2. **From the host shell,** start the container with the current working directory mounted: + + .. code-block:: bash + + docker run -itd -v `pwd`:/home/vagrant/mnt --name lsst lsstsqre/centos:7-stack-lsst_distrib-w_2017_35 + + This starts the container in a detached mode so you can open and exit multiple container shells. + Follow the steps in :ref:`docker-detached` to open a shell in the container. + +3. **From the container's shell,** activate the LSST environment and setup the top-level package: + + .. code-block:: bash + + source /opt/lsst/software/stack/loadLSST.bash + setup lsst_distrib + +4. **From the container's shell,** change into the directory of the package you cloned and set it up. + For example: + + .. code-block:: bash + + cd mnt/pipe_tasks + setup -r . + + .. note:: + + Compared to the :ref:`typical development work `, the :command:`setup` command shown here does not include the ``-t $USER`` argument to tag the development package. + This is because the Docker container doesn't have a ``$USER`` environment variable set by default. + You can still set up and develop the package this way, it just won't be tagged by EUPS. + +5. **From the container's shell,** build the package. + For example: + + .. code-block:: bash + + scons -Q -j 6 opt=3 + +The containerized development workflow +-------------------------------------- + +To develop packages with Docker containers you will use a combination of shells and applications on both the host system and inside the Docker container. + +**On the host system** you will run your own code editors and :command:`git` to develop the package. +This way you don't have to configure an editor of :command:`git` inside the container. +This is why we mount a local directory with the code and data in it. + +**In container shells** you run commands to set up packages (:command:`setup`), compile code (:command:`scons`), test code (:command:`pytest`), and run the Pipelines on data (:command:`processCcd.py`, for example). +Use :command:`docker exec` to open multiple shells in the container (see :ref:`docker-detached`). + +Cleaning up the development container +------------------------------------- + +You can stop and delete the container at any time: + +.. code-block:: bash + + docker stop + docker rm + +In this example, the container is named ``lsst``. + +Stopping and deleting a container doesn't affect the data in the local directory you mounted into that container. + .. _docker-tags: Finding images for different LSST Science Pipelines releases diff --git a/install/package-development.rst b/install/package-development.rst index a646b10d..02919c4f 100644 --- a/install/package-development.rst +++ b/install/package-development.rst @@ -5,6 +5,11 @@ Building a package with the installed Science Pipelines stack You can build packages alongside an LSST Science Pipelines stack that you have installed with :doc:`newinstall.sh `, :doc:`lsstsw `, or :doc:`Docker `. This pages describes how to build and set up packages cloned directly from GitHub. +.. note:: + + If you are developing with the LSST Docker images, refer to :ref:`docker-develop`. + That page describes Docker-specific patterns and complements the general information on this page. + .. _package-dev-prereq: 0. Prerequisites From 34bc854092891ac27454a177e3fc58957eda4d62 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 26 Sep 2017 13:29:50 -0700 Subject: [PATCH 29/35] Make v14_0 the tagged version in newinstall docs --- install/newinstall.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/install/newinstall.rst b/install/newinstall.rst index 869d5705..f2a45ca9 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -98,11 +98,11 @@ Then load the LSST software environment into your shell: Install the LSST Science Pipelines packages by running :command:`eups distrib install` for a :doc:`top-level package ` and a tagged version. -This example installs the ``v14_0_rc1`` tagged version (current release) of the ``lsst_distrib`` :doc:`top-level package `: +This example installs the ``v14_0`` tagged version (current release) of the ``lsst_distrib`` :doc:`top-level package `: .. code-block:: bash - eups distrib install lsst_distrib -t v14_0_rc1 lsst_distrib + eups distrib install lsst_distrib -t v14_0 lsst_distrib curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python setup lsst_distrib @@ -121,7 +121,7 @@ See :doc:`setup` for more information. - ``lsst_distrib`` is a top-level package that provides most LSST Data Management pipelines software, but other top-level packages may be more applicable for your work, such as ``lsst_apps`` or ``lsst_sims``. See :doc:`top-level-packages` for more information. - - ``v14_0_rc1`` is the current release. + - ``v14_0`` is the current release. You can install other tagged versions of the LSST Science Pipelines, though. See :ref:`newinstall-other-tags`. From b2238c268a669c0421179215d5ef43605f5aaf23 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Tue, 26 Sep 2017 13:54:13 -0700 Subject: [PATCH 30/35] Add next steps section to newinstall Eventually we should link to the "Getting started" tutorial series from here rather than only linking to more installation-related topics. --- install/newinstall.rst | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/install/newinstall.rst b/install/newinstall.rst index f2a45ca9..b3f3c326 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -142,12 +142,11 @@ See :doc:`demo` for instructions. Next steps ========== -Add links to topics the reader should go to next. -Examples: +Now that you have a working LSST Science Pipelines installation, these topics will help you learn and do more: -- A topic reminding a user to setup the stack in a new shell. -- Topic explaining top-level packages. -- Topic on compiling a package alongside an existing stack. +- :doc:`setup`. +- :doc:`top-level-packages`. +- :doc:`package-development`. Advanced installation topics ============================ From 8aef9c315626c479721e6b56743d3f8b8708e1ec Mon Sep 17 00:00:00 2001 From: Frossie Date: Fri, 20 Oct 2017 11:41:17 -0700 Subject: [PATCH 31/35] Remove seasonal reference --- releases/v14_0.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/releases/v14_0.rst b/releases/v14_0.rst index 78a4ffdb..04376d3e 100644 --- a/releases/v14_0.rst +++ b/releases/v14_0.rst @@ -1,8 +1,8 @@ -##################################### -LSST Stack 14.0 / Spring 2017 Release -##################################### +####################### +LSST Stack 14.0 Release +####################### -Version 14.0 of the LSST Science Pipelines (aka S17) is now available. +Version 14.0 of the LSST Science Pipelines is now available. LSST Pipelines ============== From 001aa6adb7eb8cdf0016809dd7677d195e8e7527 Mon Sep 17 00:00:00 2001 From: Frossie Date: Fri, 20 Oct 2017 11:43:35 -0700 Subject: [PATCH 32/35] Add weekly provenance --- releases/v14_0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/v14_0.rst b/releases/v14_0.rst index 04376d3e..23b670af 100644 --- a/releases/v14_0.rst +++ b/releases/v14_0.rst @@ -2,7 +2,7 @@ LSST Stack 14.0 Release ####################### -Version 14.0 of the LSST Science Pipelines is now available. +Version 14.0 of the LSST Science Pipelines is now available. It is based on 2017 weekly 33. LSST Pipelines ============== From 490a1fbd7622ade7d5078669a9ad084d41684cce Mon Sep 17 00:00:00 2001 From: John Swinbank Date: Fri, 20 Oct 2017 15:21:03 -0700 Subject: [PATCH 33/35] Tweak release note wording. --- releases/note-source/v14_0.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/note-source/v14_0.rst b/releases/note-source/v14_0.rst index 3310c80e..4c2133f2 100644 --- a/releases/note-source/v14_0.rst +++ b/releases/note-source/v14_0.rst @@ -25,7 +25,7 @@ Spring 2017 Release (v14_0) - `Characterization Metric Report (DMTR-41) `_ These release notes highlight significant changes to the Science Pipelines codebase which are likely to be of wide interest. -A complete list of issues addressed in this release is available :ref:`here `. +For a complete list of changes, see :ref:`release-v14-0-tickets`. .. _release-v14-0-major-changes: @@ -175,7 +175,7 @@ This is analogous to the flags used to indicate sources used in PSF modeling. Convolved flux measurement ^^^^^^^^^^^^^^^^^^^^^^^^^^ -The meas_extensions_convolved package has been added to the distribution. +The meas_extensions_convolved package has been added to the lsst\_distrib distribution. This convolves images to a specified PSF before measuring aperture fluxes. This may be useful when, for example: @@ -198,7 +198,7 @@ The ``afw.cameraGeom.Camera`` object now provides an API for accessing a model o Joint calibration ^^^^^^^^^^^^^^^^^ -The Jointcal package, which provides for simultaneous photometric and astrometric calibration over multiple exposures, has been added to the distribution. +The Jointcal package, which provides for simultaneous photometric and astrometric calibration over multiple exposures, has been added to the lsst\_distrib distribution. While we believe that Jointcal works well, note that it is still under heavy development and is not currently run as part of regular processing by the LSST development team: regard this package as a technology preview. :jirab:`DM-9752` From 715c94b317e730cc4fc1e8284f26e8763085a868 Mon Sep 17 00:00:00 2001 From: Frossie Date: Tue, 24 Oct 2017 12:30:37 -0700 Subject: [PATCH 34/35] Fixed lsst_distrib install line --- install/newinstall.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/install/newinstall.rst b/install/newinstall.rst index b3f3c326..cd9ade6d 100644 --- a/install/newinstall.rst +++ b/install/newinstall.rst @@ -102,7 +102,7 @@ This example installs the ``v14_0`` tagged version (current release) of the ``ls .. code-block:: bash - eups distrib install lsst_distrib -t v14_0 lsst_distrib + eups distrib install -t v14_0 lsst_distrib curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python setup lsst_distrib From 728ea17199e7e9e7aad744fa13fd974717138ce4 Mon Sep 17 00:00:00 2001 From: Frossie Date: Tue, 24 Oct 2017 15:55:20 -0700 Subject: [PATCH 35/35] Additional ubuntu instructions --- install/prereqs/debian.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/install/prereqs/debian.rst b/install/prereqs/debian.rst index dbc39b67..2ffff966 100644 --- a/install/prereqs/debian.rst +++ b/install/prereqs/debian.rst @@ -19,6 +19,14 @@ Debian or Ubuntu systems require the following packages: Prefix the :command:`apt-get` command with :command:`sudo` if necessary. +If you get an error run: + +.. code-block:: bash + + apt-get update --fix-missing + +and try again. + .. note:: **New since 11.0**: The minimum gcc version required to compile the LSST Science Pipelines is **gcc 4.8.**