1st EasyBuild hackathon meeting minutes day 1

Kenneth Hoste edited this page Oct 16, 2012 · 1 revision
Clone this wiki locally

(Thursday Aug. 16th 2012, 10am-7pm)

During the first day of the 1st EasyBuild hackathon, mostly discussions were held. These notes were taken by Kenneth, and edited by Fotis.

Attendees

introduction round:

  • Stijn De Weirdt: HPC-UGent team lead, started EasyBuild
  • Fotis Georgatos:
    • stumbled onto EasyBuild via HEPIX'2012 agenda, in a period he was looking for a tool like this (but didn't expect to find it in HEPIX!)
    • Cedric (clueful hpc user at Fotis' site) has been using it to build over 20 bioinformatics software packages, without much trouble
  • Jens Timmerman: has been extending EasyBuild in the last couple of months
  • Toon Willems: summer intern for EasyBuild, has been working on features and redesign for last month
  • Kenneth Hoste: lead EasyBuild developer during last year

Discussion topics

Development timeline

Brief overview of development timeline:

  • v0.9 by end of Aug. 2012
  • v1.0 by end of Sept. 2012
  • v1.1 by end of 2012
  • long-term v2.0 and v3.0 milestones already defined, no date set yet

Open bug tickets

Went through open bug tickets (in internal Trac bug tracker), discussed some of them:

  • several ones could be closed already, e.g. dumping dependency graph, failing to build particular software packages, ... (action point KH)
  • discuss security issues w.r.t. exec'ing easyconfig files (see #129)
    • (FG) not really a security issues per se, there are others ways to do nasty stuff (e.g. in easyblock, or through patch files)
    • (TW) avoid being able to use import statements by emptying __builtins__ (see internal Trac ticket nr. 212)

Discuss changes in v0.9

  • discuss more generic way of handling os dependencies (see #174)

    • package names differ between OSs
    • now support for running commands to check whether functionality is available is already there
    • (SDW) should not go outside of package manager, ask package manager for checking commands, command version, files being there, ...
    • support different package managers in EasyBuild (rpm/yum, dpkg, ...) to be implemented
    • or through meta modules
    • (FG): ie. meta-modules may correspond to policies, as seen here:
    • http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration and here:
    • http://www.ccac.hpc.mil/consolidated/bc/policy.php
    • example: "tcsh" can be provided via "LS2_05-06" or "FY05-06"; names are fictional they can be anything; they could also correspond to rpm/deb packages
  • easyconfig repository

    • (FG) make EasyBuild check the repository to look for easyconfig files (see #207)
    • (FG) encourage people to share easyconfig files by adding some incentive if they contribute easyconfigs that worked back to the community
    • (FG) will create an easy-going easybuild.experimental repository for people to play with ports and beyond; idea: give incentives with stickers/t-shirts/towels/whatever makes you comfortable with, for 1/10/100/1000 .eb contributions
    • (JT) automatically create a pull request after a worked build (see #208)
  • devel modules (see #175)

    • (FG) support installing devel modules in a dedicated devel-modules MODULEPATH place, or both (also in install path)
  • checking for loaded modules created by EasyBuild by looking for (SOFT | EB)ROOT env vars (see #153)

    • (FG) add EasyBuild option to just ignore that step, if desired for special reasons (eg. massive build of packages where some may work anyhow)
  • (FG) specify the exact policy on branch names (e.g. _) ; hopefully something that gives the headline information

  • (FG) make toolkit support more modular (see #176)

    • create classes for compilers, MPI libs, ...

Experience report by FG

  • EasyBuild could be very useful for users as well
    • build (new versions of) software themselves, test it thoroughly before it gets installed system-wide by sysadmins or, group-managers (eg. per scientific community)
  • overall fairly positive experience so far
  • easy to get started with EasyBuild based on "demo for the impatient"

Public bug tracker

  • GitHub issue tracker
    • lacks bug hierarchy, blocking marks, ...
    • let's just try it
    • seems to be feature-rich enough (milestones, assignees, labels, reference to issue in commit messages, ...)
    • migrate all open Trac tickets to hpcugent GitHub issue trackers (action point KH)
    • see https://github.com/trustmaster/trac2github/
    • FIXED: cleaned up tickets, all open tickets moved to GitHub issue tracker (manually)

FGs wishlist

action point KH and FG: issues need to be opened for these, and mentioned here

  • need to create EasyBuild config option to specify dir for temporary log dir (see #83, #84)
  • consider a standardized way of handle dependencies (eg. CUDF) so that the resolver can be an external tool (see #210)
    • support something like a CUDF function to output easyconfig in CUDF format; 1 idea is to embed it in the easyconfig source
    • allow to override current dependency solver with something different (e.g. CUDF based)
  • support more freedom in having alternative namespace(s) for modules; this is badly needed for most big operating HPC sites (see #173)
    • (FG) killer feature for v1.0 so that existing documentation and user conventions does not have to be broken; eg.: lower-case module namespace
    • create a symlink farm with given alternative names to modules files created by EasyBuild
    • allow sysadmins to define function (or Python module) to map easyconfig parameters to this alternative namespace
    • try and come up with documentation that shows how to implement different namespaces
    • issues with dependency resolving by modules, conflicts between modules can be fixed by making modules more complex (if-else-if constructs for loading dependencies)
    • (TW) or maybe my 'alias' feature offered by environment modules?
    • also release tool to merge from one namespace to another
  • categories of modules (see #209)
    • already supported by moduleclass, except:
    • list of valid module classes is now hard coded, should be able to extend these in easybuild_config.py
    • specifying a list of module classes should also be supported
    • also support subcategories, e.g. bioinformatics/tools, bioinformatics/apps, bioinformatics/datasets
  • make a-z easyblocks hierarchy customizable (see #87)
    • we should define a single namespace easyblocks, and then others can extend that namespace in any way they want (action point JT)
  • add support to robot for multiple paths to look for easyconfigs (see #211)
    • and use regex to match .eb file names, together with os.walk
  • handling of non-.eb files in a modular way. eg: .rpm, .srpm, .spec, .deb, .py etc (see #212)
  • add support for generating RPMs with EasyBuild (see #200)
  • create a 'default' module that installs software that's not available yet ;-)
    • "module load software/not_yet" => "Installing software version not_yet, please hold..."
  • promote EasyBuild in BioInformatics community (see #120)
    • big need and good fit because of requirement for reproducibility of experiments (software versions, etc.) AND multiple relatively small packages
  • next project: easydoc, to organize documentation on scientific software
    • e.g. make it produce HTML pages like http://eniac.cyi.ac.cy/display/UserDoc/GROMACS
    • format to write documentation in: MarkDownor so; _(JT)_PanDoc can help here for generality in output
    • idea is to allow community to organize documentation about scientific software centrally
  • add support for skipping particular steps in the build process (see #213)
    • i.e. add support for a skipsteps=[..., ...] easyconfig parameter
    • so that implementing easyblocks just to skip steps can be avoided completely
    • and that preconfigopts="/bin/true &&" hack is not needed any more
  • add support for checking checksum of source files (option for source) (see #213)
    • make checksum part of dumped easyconfig file in repository; eg. source_checksum=('sha1','2fd4e1c6 7a2d28fc ed849ee1 bb76e739 1b93eb12') or source_checksum='sha1:2fd4e1c6 7a2d28fc ed849ee1 bb76e739 1b93eb12'
  • arrows in dependency graphs should be reversed (KH): FIXED
  • rename some functions, e.g. make to build, source to fetch, ... (see #99)
    • figure out community consensus (see e.g. MacPorts)
    • FG will create table to see if there's any kind of consensus on this
    • should make changes soon, as this will break existing easyblocks
  • make it possible to select how to configure, install, ... from within an easyconfig file (see #215)
    • e.g. easyblock = {'configure': "git_fetch", 'install': "binary"}
    • pull these from the framework part of EasyBuild, figure out a way to make it extensible
  • see http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration for inspiration of list of software to support in EasyBuild
    • and for policies for HPC sites to comply to w.r.t. software/support
  • discussion about coming up with an encoding of package name to module/class name that avoids clashing (see #86)
    • module names should be case-insensitive, to comply with PEP008 and support OSs like Windows and case-insensitive OS X HFS filesystems
    • class names should be unique (so case-sensitive)
    • encoding should only be used in case of clashes
    • we should have an academic answer to the academic question
    • action point SDW: come up with a documented encoding approach to be used in case of clashes
  • allow to specify default versions of selected software packages in EasyBuild config file (see #219)
    • e.g. for Python, add an entry that Python/2.7.2 should be the default (until we change it there)
  • add extra installtarget easyconfig option, to specify a different make install target (see #220)
    • default value should be "install"
  • support specifying conflicts in easyconfig files (see #221)
    • with version numbers, e.g. openssl v11 conflicts with "ssleay < v1"
    • idea: employ CUDF format style directly for this
  • create an easybuild-experimental repository to house generated easyconfig files (and maybe even easyblocks) (see #222)
    • from pkgsrc/ports/portage
    • FG will set this up, will be linked from EasyBuild main repo
  • specify that public contributions should be made under an OSI-approved license (requirement: open-source, redistributable) (see #223)
    • adjust LICENSE file as needed to clarify that different code files may have different licenses
    • action point KH