Building a package with the installed Science Pipelines stack
You can build packages alongside an LSST Science Pipelines stack that you have installed with :doc:`newinstall.sh <newinstall>`, :doc:`lsstsw <lsstsw>`, or :doc:`Docker <docker>`. This pages describes how to build and set up packages cloned directly from GitHub.
If you are developing with the LSST Docker images, refer to :ref:`docker-develop`. That page describes Docker-specific patterns and complements the general information on this page.
Before developing an individual package, you need an installed Science Pipelines stack. You can install the Pipelines through any of these methods: :doc:`newinstall.sh <newinstall>`, :doc:`lsstsw <lsstsw>`, or :doc:`Docker <docker>`.
Then set up the Pipelines software in a shell. See :doc:`setup` for more information.
newinstall.sh users: read :ref:`newinstall-binary-compatibility` if the installed Pipelines stack uses prebuilt binary packages.
1. Clone a package
Use Git to clone the package you want to work on. Most LSST packages are available in the LSST's GitHub organization (https://github.com/lsst). The repos.yaml file in the 'repos' repository also maps package names to repository URLs.
git clone https://github.com/lsst/pipe_tasks cd pipe_tasks
You can also create a new package repository, though this is beyond this document's scope.
You will work from a Git branch when developing a package. The DM Developer Guide describes the branching workflow that LSST staff use.
Docker users: Clone the package onto your host filesystem rather then directly into the container by mounting a host directory in the container. See :ref:`docker-develop`.
lsstsw users: The :file:`lsstsw/build` directory already includes clones of Git repositories. These repositories are reset when you run :ref:`rebuild <lsstsw-rebuild>`, though, so you can potentially lose local changes. It's usually better to clone and work with Git repositories outside of the :file:`lsstsw` directory.
2. Set up the package
From the package's directory, set up the package itself in the EUPS stack:
setup -r . -t $USER
3. Build the package with Scons
scons -Q -j 6 opt=3
These flags configure Scons:
-Q: reduce logging to the terminal.
-j 6: build in parallel (for example, with '6' CPUs).
opt=3: build with level 3 optimization. Use
opt=gwith gcc compilers) for debugging.
scons command will run several targets by default, in sequence:
lib: build the C++ code and Pybind11 interface layer.
python: install the Python code.
tests: run the unit tests.
example: compile the examples.
doc: compile Doxygen-based documentation.
shebang: convert the
#!/usr/bin/envline in scripts for OS X compatibility (see DMTN-001).
You can build a subset of these targets by specifying one explicitly. For example, to compile C++, build the Python package and run tests:
scons -Q -j 6 opt=3 tests
By following these steps, you have built a package from source alongside an installed Science Pipelines software stack. Now when you run the Science Pipelines, your new package will be used instead of the equivalent package provided by the Science Pipelines installation. Here are some tasks related to maintaining this development software stack:
Reviewing set up packages
Packages that are set up are part of the active Science Pipelines software stack. You can see what packages are currently set up by running:
eups list -s
You can also review what version of a single package is set up by running:
eups list <package name>
Setting up in a new shell
Whenever you open a new shell you need to set up both the LSST software environment and the LSST software stack. See :doc:`setup` for the basic procedure.
In addition to setting up the installed Science Pipelines software, you separately need to set up the development package itself. You can do this following the instruction in step :ref:`package-dev-setup`.
Un-set up the development package
You can un-set up a development package to revert to the installed LSST Science Pipelines distribution.
To switch from a development package to the released package:
setup -j <package name> -t current
current is the default tag normally used for the installed LSST Science Pipelines software stack.
To un-set up a development package without replacing it:
unsetup -j <package name> -t $USER
This is useful if you are developing a new package that is not part of the installed LSST Science Pipelines software stack.