forked from easybuilders/easybuild-framework
-
Notifications
You must be signed in to change notification settings - Fork 0
Wishlist
fgeorgatos edited this page Aug 17, 2012
·
22 revisions
- auto-gen TOC: {:toc}
Here is a list of features that are desired to have in next versions of EasyBuild: (importance of them varies according to who you ask ;-)
- Add possibility to save the logfile somewhere even when failures happen (/tmp is no good for queue submitted jobs)
- interim solution is to use -ld; OK.
- ANSWER: "/tmp" is hardcoded now, so an issue needs to be opened for this; See #84
- Consider support of "formal" language to describe package info & dependencies, eg. CUDF syntax
- http://blog.mancoosi.org
- http://www.mancoosi.org/papers/cbse11.pdf "MPM: a modular package manager"
- http://arxiv.org/abs/1109.0456 "Aligning component upgrades"
- ANSWER: may consider; TBD
- Consider category-oriented module sets (eg.
module load bioinformatics-tools) - may be done outside of EB logic, IFF the meaning is to mangle MODULEPATH? "moduleclass" can help
- ANSWER: either as said OR, produce an easyconfig with multiple dependencies; TBD
- Allow for custom-definition of easyblock paths (instead of enforcing any "a-z/" dir structure
- ANSWER: it seems to be OK w. PYTHON_PATH? nope, the a-z dirs need to have "reverse" mapping (ref. ".."); TBD
- How to organize a collection of .eb files coming from different sources? (eg. distinct git repos)
- Seen at http://joeyh.name/code/mr/ : "When updating a git repository, pull from two different upstreams and merge the two together."
- ie. can EasyBuild --robot dependency resolution support multiple paths?
- ANSWER: will support multiple directories; prefer to use many indeed
- Consider integrated support for relocatable .rpms (with --prefix), .srpms & RPM's .spec files
- in effect, let's create a handler for non "*.eb" suffixes to be managed externally; that would allow to create from eg. a .spec file (for RPMs) the equivalent .eb, then parse that from "stdio"
- ANSWER: it's already supported, but in previous versions; to be cleanup
- extreme thought: yes, this is an extreme thought: consider module search/install extensions!
- ANSWER: or try it with "module load" instead ;-)
- promote easybuild within bioinformatics community, esp. for "clinical or translational workflows", as described below: http://www.bioinfo-core.org/index.php/11th_Discussion-7_November_2011#Topic_2:_Managing_and_Tracking_Software_Updates_.28Led_by_Brent_Richter.29
- ANSWER: makes sense to approach communities; TBD
- support cmake-like builds (see example below)
- ANSWER: for the default case an easyblock is upcoming; TBD
- how to override certain steps ("/bin/true" is considered a hack as of now)
- ANSWER: to be discussed; first create wiki page with all the [different ports conventions]
- Consider the bigger picture as seen at http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration
- ANSWER: discussed; interesting for Flamish supercomputing activities
- BENELUX collab?
- ANSWER: discussed; TBD
- Define the namespace "by a standard"; eg. like [https://github.com/fgeorgatos/HPC-RFC/blob/master/0001/0001.md]
- Give freedom to define the namespace format (eg. original Package names OR the lower case version)
- allow to export namespace in both UpperCase and LowerCase conventions # a post-install hook? TBD
- Support more configurable formats of version strings (check below, how other HPC sites do it)
- nested levels (ie. split version string in sub sections)
- provide hooks for gnu/gcc/intel/pgi etc strings
- IDEA: For now, supply a "custom_namespace_function" variable, to define eg. modulename.lower();
- Provide some smart way to handle osdependencies like "tcsh" which may have no meaning on Debian # a particularly important case are the boost-devel vs boost-dev differences (rpm vs deb conventions)
- and for now allow escaping with --ignore-os-dependencies
# GROMACS-GPU uses a cmake build-generator and makefiles on unix.
# To avoid confusion build in separate folder
export OPENMM_ROOT_DIR=path_to_custom_openmm_installation
mkdir build_gromacs_gpu
cd build_gromacs_gpu
cmake PATH_TO_SOURCE_DIRECTORY -DGMX_OPENMM=ON -DGMX_THREADS=OFF
make mdrun
make install-mdrun
---------------------------------------------------------- --------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Example name WHO Reference URL & Comments
---------------------------------------------------------- --------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
netcdf/3.6.2 TACC [[http://services.tacc.utexas.edu/index.php/using-modules][]] \\ Lmod: automatic reloading of an entire module hierarchy when a single module anywhere in the hierarchy is changed.
hpc/netcdf-3.6.3_intel-11.0.083 HARVARD [[http://rc.fas.harvard.edu/faq/modulelist][]] \\ very long and complete list
misc-libs/netcdf/3.6.3_intel CLUMEQ [[https://www.clumeq.ca/wiki/index.php/ModulesDisponiblesSurColosse][]] \\ nested layout
netcdf/intel/64/4.0 SARA [[http://www.sara.nl/systems/shared/modules/][]] \\ [[http://www.sara.nl/systems/lisa/news/modules-ng][]] \\ [[http://www.sara.nl/systems/lisa/news/amd64-phase1][]] \\ [[http://www.sara.nl/systems/lisa/news/Modules-deprecated-2011-04-04][]]
[[https://www.clumeq.ca/wiki/index.php/ModulesDisponiblesSurColosse][]] \\ nested layout
netcdf/gnu/64/4.0 SARA [[http://www.sara.nl/systems/shared/modules/][]] \\ [[http://www.sara.nl/systems/lisa/news/modules-ng][]] \\ [[http://www.sara.nl/systems/lisa/news/amd64-phase1][]] \\ [[http://www.sara.nl/systems/lisa/news/Modules-deprecated-2011-04-04][]]
netcdf/4.0.1 UIBK [[http://www.uibk.ac.at/zid/systeme/hpc-systeme/common/tutorials/modules-howto.html][]] \\ PREFERRED_MC
netcdf/X.Y.Z/gnu-4.1.2 UIBK, too [[http://www.uibk.ac.at/th-physik/howto/hpc/modules.html][]]
netcdf/X.Y.Z-gnu CSC [[http://www.csc.fi/english/pages/hippu_guide/using_hippu/modules/index_html][]]
ofed/qlogic/gcc/64/1.2.7 UCL [[http://www.ucl.ac.uk/isd/common/research-computing/services/legion-upgrade/userguide/userenvironment][]]
netcdf/3.6.2 MIT [[http://www.darwinproject.mit.edu/wiki/index.php/Compute_cluster_hardware/software_overview][]]
netcdf/X.Y.Z MIT, too [[http://coyote.mit.edu/mediawiki/index.php/Modules][]]
netcdf/3.6.2 CAM [[http://www.hpc.cam.ac.uk/user/software.html][]]
netcdf/4.0.1_nc3 UTORONTO [[https://support.scinet.utoronto.ca/wiki/index.php/Software_and_Libraries][]]
blas/? VLSCI@AU [[http://www.vlsci.org.au/documentation/software-applications][]]
fluent/? EPFL [[http://pleiades.epfl.ch/index.php?option=com_content&task=view&id=15&Itemid=30][]]
netcdf/4.0.0.3.1jg64 CSCS [[http://user.cscs.ch/software_and_programming_environment/compilers_and_programming/rosa_cray_xt5/modules_framework/index.html][]]
netcdf/4.1.1/pgi/10.3/64 TAMU [[http://brazos.tamu.edu/software/modules.html][]]
sles10.1_gnu4.1.2_shared UTENNESSEE [[http://www.nics.tennessee.edu/computing-resources/kraken/software][]]
netcdf/4.0-pgi, /4.0-gcc , /4.1.1 UMICH [[http://cac.engin.umich.edu/resources/software/][]]
netcdf-4.0.1 ARSC [[http://www.arsc.edu/support/news/systemnews/][]]
NetCDF/4.1.3-gnu Griffith University [[http://confluence.rcs.griffith.edu.au:8080/display/GHPC/netcdf][]]
HMMER iCER/HPCC [[https://wiki.hpcc.msu.edu/display/Bioinfo/Module+Files][]]
HMMER UPPNEX [[https://www.uppnex.uu.se/installed-software][]]
uberftp-client-2.6 NCSA [[http://www.ncsa.illinois.edu/UserInfo/Resources/Hardware/CommonDoc/module.html][]]
netcdf LRZ [[http://www.lrz.de/services/software/utilities/modules/deisa_details.html][]] \\ Ref. on DEISA setup: [[http://www.lrz.de/services/software/utilities/modules/deisa_details.html][]]
globus PRACE sites [[http://www.prace-ri.eu/Interactive-Access-to-HPC][]]
globus/5.0.4 & GLOBUS-5.0 TACC [[http://www.underworldproject.org/BuildRecipes/ranger.rst][]] See Teragrid below
netcdf/4.1.2-gnu, /4.1.2-intel, /3.6.3-gnu, /3.6.3-intel cyi/euclid@ls2 [[http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration]]
netcdf/3.6.3-gcc, netcdf-4/4.0.1-gcclf cyi/planck [[http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration]]
netcdf/3.6.2-intel, /3.6.2-gnu ba@ls2 [[http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration]]
netcdf/4.1.3-gcc(default) cytera@ls2 [[http://eniac.cyi.ac.cy/display/UserDoc/HPC+Baseline+Configuration]]
netcdf/1.4.3-mpich2-intel-64bit FZK [[http://www.google.com]]
---------------------------------------------------------- --------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
package: m4
version: 3
depends: libc6 >= 8
package: openssl
version: 11
depends: libc6 >= 18, libssl0.9.8 >= 8, zlib1g >= 1
conflicts: ssleay < 1
package: gcc
version: 4.4.4
provides: compiler
conflicts: compiler
package: icc
version: 10.2
provides: compiler
conflicts: compiler
| name | what the function is about |
|---|---|
| fetch | Fetch the ${distfiles} from ${master_sites} and place it in ${prefix}/var/macports/distfiles/${name} |
| checksum | Compare ${checksums} specified in a Portfile to the checksums of the fetched ${distfiles}. |
| extract | Unzip and untar the ${distfiles} into the path ${prefix}/var/macports/build/..../work |
| patch | Apply optional patch files specified in ${patchfiles} to modify a port's source code file(s). |
| configure | Execute the command configure in ${worksrcpath}. |
| build | Execute ${build.cmd} in ${worksrcpath}. |
| test | Execute commands to run test suites bundled with a port. |
| destroot | Execute the command make install DESTDIR=${destroot}in ${worksrcpath}. |
Ref. http://guide.macports.org/#reference.phases -> 5.3. Port Phases
| name | what the function is about |
|---|---|
| pkg_setup | Use this function to perform any miscellaneous prerequisite tasks. This might include checking for an existing configuration file. |
| pkg_nofetch | Inform the user about required actions if for some reason (such as licensing issues) the sources may not be downloaded by Portage automatically. Use this in conjunction with RESTRICT="fetch". You only should display messages in this function, never call die. |
| src_unpack | |
| src_compile | Use this function to configure and build the package. The initial working directory is S. |
| src_install | Use this function to install the package to an image in D. If your package uses automake, you can do this simply with emake DESTDIR="${D}" install. Make sure your package installs all its files using D as the root! The initial working directory is S. |
| src_test | Executed only when FEATURES="test" is set and RESTRICT="test" is unset, the default of this function executes an available testing function from any Makefiles in the ${S} directory, running either "make test" or "make check" depending on what is provided. It can be overriden to create a custom test setup. |
| pkg_preinst | The commands in this function are run just prior to merging a package image into the file system. |
| pkg_postinst | The commands in this function are run just following merging a package image into the file system. |
| pkg_prerm | The commands in this function are run just prior to unmerging a package image from the file system. |
| pkg_postrm | The commands in this function are run just following unmerging a package image from the file system. |
| pkg_config | You use this function to set up an initial configuration for the package after it's installed. All paths in this function should be prefixed with ROOT which points to user-specified install root which may not happen to be /. This function is only executed if and when the user runs: emerge --config =${PF}. |
Ref: http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=1#doc_chap2 -> Functions
| name | what the function is about |
|---|---|
| depends | to build and install dependencies |
| fetch | to fetch distribution file(s) |
| checksum | to fetch and check distribution file(s) |
| extract | to look at unmodified source |
| patch | to look at initial source |
| configure | to stop after configure stage |
| all or build | to stop after build stage |
| stage-install | to install under stage directory |
| test | to run package's self-tests, if any exist and supported |
| package | to create binary package before installing it |
| replace | to change (upgrade, downgrade, or just replace) installed package in-place |
| deinstall | to deinstall previous package |
| package-install | to install package and build binary package |
| install | to install package |
| bin-install | to attempt to skip building from source and use pre-built binary package |
| show-depends | print dependencies for building |
| show-options | print available options from options.mk |
Ref: http://wiki.netbsd.org/pkgsrc/targets/
| name | what the function is about |
|---|---|
| config | Configure OPTIONS for this port using dialog(1). |
| fetch | Fetch all of the files needed to build this port from the sites listed in MASTER_SITES and PATCH_SITES. See FETCH_CMD, MASTER_SITE_OVERRIDE and MASTER_SITE_BACKUP. |
| checksum | Verify that the fetched distfile's checksum matches the one the port was tested against. Defining NO_CHECKSUM will skip this step. |
| depends | Install (or compile if only compilation is necessary) any dependencies of the current port. When called by the extract or fetch targets, this is run in piecemeal as fetch-depends, build-depends, etc. Defining NO_DEPENDS will skip this step. |
| extract | Expand the distfile into a work directory. |
| patch | Apply any patches that are necessary for the port. |
| configure | Configure the port. Some ports will ask you questions during this stage. See INTERACTIVE and BATCH. |
| build | Build the port. This is the same as calling the all target. |
| install | Install the port and register it with the package system. This is all you really need to do. |
| name | from | to | what the function is about |
|---|---|---|---|
| %prep | %_sourcedir | %_builddir | This reads the sources and patches in the source directory %_sourcedir. It unpackages the sources to a subdirectory underneath the build directory %_builddir and applies the patches. |
| %build | %_builddir | %_builddir | This compiles the files underneath the build directory %_builddir. This is often implemented by running some variation of "./configure && make". |
| %check | %_builddir | %_builddir | Check that the software works properly. This is often implemented by running some variation of "make test". Many packages don't implement this stage. |
| %install | %_builddir | %_buildrootdir | This reads the files underneath the build directory %_builddir and writes to a directory underneath the build root directory %_buildrootdir. The files that are written are the files that are supposed to be installed when the binary package is installed by an end-user. Beware of the weird terminology: The build root directory is not the same as the build directory. This is often implemented by running "make install". |
| bin | %_buildrootdir | %_rpmdir | This reads the files underneath the build root directory %_buildrootdir to create binary RPM packages underneath the RPM directory %_rpmdir. Inside the RPM directory is a directory for each architecture, and a "noarch" directory for packages that apply to any architecture. These RPM files are the packages for users to install. |
| src | %_sourcedir | %_srcrpmdir | This creates a source RPM package (.src.rpm) inside the source RPM directory %_srcrpmdir. These files are needed for reviewing and updating packages. |
Apparently, the pkgsrc seems to be the most factorized build process. It is worthy to check though if the naming convention suits EasyBuild purposes.