-
Notifications
You must be signed in to change notification settings - Fork 3
Home
dist is the collection of Seattle Testbed's tools for building and packaging installers, including platform-specific scripts for installation, startup, and shutdown.
This page discusses future build and packaging options required (and desired) following our move from a self-hosted SVN to GitHub.
This is a collection of pointers to possible tools and techniques as well as requirements, architecture drafts, and discussion about the future build system for Seattle. It also touches on questions of source code management, contributions, etc. We have different hierarchical/degree-of-integration components as outcomes of the processes to be designed here; the system is useful (and used) by different audiences on either of these components.
There are multiple options to tackle the packaging task:
- To make use of Python Packaging Index (PyPI) and PIP to package and install python scripts (as done in TUF project).
- Another option is to use Python Eggs to package the dependencies.
- Third one is to use Virtual Python Environment builder (virtualenv) for installing python packages.
- Last option is to resist from using any of these tools and to stick to manually customising the new building and packaging scripts from scratch (as it was done until now).
As per suggestion given by @vladimir-v-diaz of @theupdateframework, the procedure followed by other big projects in building and packaging stuff will also be looked into to gain inspiration from and to see if any of those ideas can be implemented in Seattle or not!
Package and environment management systems to explore:
- Buildout: http://www.buildout.org/en/latest/
- Conda: http://conda.pydata.org/docs/build.html, http://www.continuum.io/blog/conda
- Difference between pip and conda: http://stackoverflow.com/questions/20994716/what-is-the-difference-between-pip-and-conda
The desired outcome of a build process, such as a ready-to-use component of the system, or a ready-to-install package. Created by configuring and putting required objects into predetermined places, optionally packaging them. Existing build targets are
- local runtime: Yields an environment to develop Repy programs on your machine, including the Seattle libraries.
- unit tests: Run tests against the full platform (runtime and libraries).
- base installers: Used by the custom installer builder / clearinghouse. Includes platform-specific install/start/stop scripts. Outputs platform-specific archive files (.zip, .tar.gz)
- demokit: Includes seash and a couple of example programs people can run following our tutorials.
Designated build targets are
- barebones seash: As the name implies, demokit without the example code.
- experiment manager: Scriptable environment to deploy programs on Seattle, download files from nodes etc. without interaction with seash.
- local setup: Make the local node run Seattle from the current build directory using locally stored pubkeys so you can interface it using seash. (nodemanger/nminint.py does something like that on a preparetest'ed directory.)
- clearinghouse / custom installer builder runtime: Put together the Seattle components needed for running the backend of a CH/CIB web site. Maybe identical to local runtime.
- clearinghouse backend: The (mainly Django/HTML/JS/CSS, less Repy) components required to run a clearinghouse, except for the software (Apache, Django, MySQL). Currently exists in the form of a directory you should copy to a specific location on the CH machine.
- custom installer builder backend: As with clearinghouse backend.
Build scripts need to take into account the required repositories for the desired target (so it’s easy to build locally), and should provide a comprehensible way to build off non-standard repos/branches. ("non-standard" to us is their version of the code for others!)
They should work across the major platforms (Linux, Windows, Mac OS X) with a minimal set of external tools (git).
A single-file representation of a built component, e.g. a tarball, or a Python Wheel file, etc.
Different package formats bring different possible installation procedures: unzip and run installer manually, automatically run predefined setup.py script, install into system locations rather than user’s home dir, etc.
Structure that marks (source or built) components as belonging together, or loosely combines them so that the original structure of parts is essentially unchanged from the non-bundled form. Forms include a tarball of (part of) the source tree, a Git commit, Git submodules/submodule meta-data files.
Who will use the packages/processes described? Which targets in which form are they interested in? What are their expectations? Note: We should always expect and not treat differently developers from outside of the project. We should strive to make our projects and sub-projects accessible, reusable, and meta-reusable (as architectural blueprints for other projects).
People deploying existing code on Seattle’s (or derived testbeds’) VMs, but don’t write their own. This includes most of the other groups below (testing!!!).
Example: Students doing the take-home assignment
Targets: seash, demokit
Requirements: A stable (but possibly updateable!) system, super-easy download/install.
People who write code for Seattle VMs but presumably not contribute back.
Example: students taking a course
Targets: Local runtime, very likely seash
Requirements: Like end users
Similar to program developers, but with an inclination to contribute back.
Example: “Newcomers” (new contributors to Seattle)
Targets: Local runtime sources, unit tests, scripts to build the local runtime for testing
Requirements: easy to check out, fork, keep synced with upstream
Developers targeting the inner workings of the Repy runtime, nodemanager, softwareupdater etc.
Targets: Like library developers
Requirements: Like library developers
People setting up clearinghouses and adapting our code for that.
Targets: clearinghouse runtime and backend
Requirements: As with end users above
The current build scripts used by our clearinghouses essentially build an installer (copy stuff together from different repos), possibly add session-specific content like the vesselinfo file, finally package into a platform-specific archive format like .zip or .tar.gz. This archive is installed by the user in a two-step process by first unzipping it, then running our custom Python-based Seattle installer.
The current build system for developers is based on preparetest.py, see below. This script takes care of “copying stuff together” into a local directory, optionally including unit tests.
For end users we have the demokit that includes seash and example programs.
Build scripts that we have:
- https://github.com/SeattleTestbed/dist/blob/git-aware-buildscripts/preparetest.py for local runtime and unit tests, with nminit.py also for local setup
- https://github.com/SeattleTestbed/dist/blob/git-aware-buildscripts/make_base_installers.py
- https://github.com/SeattleTestbed/dist/blob/git-aware-buildscripts/rebuild_base_installers_for_seattlegeni.sh
- https://github.com/SeattleTestbed/dist/blob/git-aware-buildscripts/build_demokit.py
Inherent sync issue: Build a target, develop there, then copy back your changes to your working copy to track them. (Hasn’t been different in SVN, difficult to overcome as you won’t preparetest into a working copy, will you?)
Desired outcomes that require tools to be written.
Derive these from the different audiences’ targets. Describe how which packaging/bundling process supports the audiences’ requirements.