cache result of 'module avail' calls #1742

boegel · 2016-04-27T20:57:25Z

~~partial~~ fix for #1739, implements caching for module avail (without any arguments) and show

boegel · 2016-04-27T20:57:28Z

@ocaisa can you give this a try? should make quite a bit of difference for your use case...

hpcugentbot · 2016-04-27T21:23:02Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2973/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2016-04-28T06:24:53Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2974/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2016-04-28T06:58:04Z

@wpoely86 thoughts?

ocaisa · 2016-04-28T07:19:31Z

Just took it for a test drive:

swmanage@jrl08:/homeb/zam/swmanage/software/easybuild-framework> time eb /usr/local/software/jureca/Stage1/eb_repo/Python/Python-2.7.11-intel-2016a.eb -D -d --disable-cleanup-tmpdir
== temporary log file in case of crash /tmp/eb-erROoI/easybuild-yhsGUf.log
Dry run: printing build status of easyconfigs and dependencies
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/zlib-1.2.8.eb (module: Core | zlib/.1.2.8)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/M4-1.4.17.eb (module: Core | M4/.1.4.17)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Bison-3.0.4.eb (module: Core | Bison/.3.0.4)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/flex-2.6.0.eb (module: Core | flex/.2.6.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/binutils-2.25.eb (module: Core | binutils/.2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/GCCcore-4.9.3.eb (module: Core | GCCcore/.4.9.3)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/zlib-1.2.8-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | zlib/.1.2.8)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/M4-1.4.17-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | M4/.1.4.17)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Bison-3.0.4-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Bison/.3.0.4)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/flex-2.6.0-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | flex/.2.6.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/binutils-2.25-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | binutils/.2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/ncurses-6.0-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | ncurses/.6.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/icc-2016.2.181-GCC-4.9.3-2.25.eb (module: Core | icc/.2016.2.181-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/ifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Core | ifort/.2016.2.181-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Tcl-8.6.4-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Tcl/.8.6.4)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/bzip2-1.0.6-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | bzip2/.1.0.6)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/libreadline-6.3-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | libreadline/.6.3)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Core | Intel/2016.2.181-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/SQLite-3.9.2-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | SQLite/.3.9.2)
 * [ ] /usr/local/software/jureca/Stage1/eb_repo/Tk/Tk-8.6.4-GCCcore-4.9.3-no-X11.eb (module: Compiler/GCCcore/4.9.3 | Tk/.8.6.4-no-X11)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/zlib-1.2.8-iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Compiler/intel/2016.2.181-GCC-4.9.3-2.25 | zlib/.1.2.8)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/impi-5.1.3.181-iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Compiler/mpi/intel/2016.2.181-GCC-4.9.3-2.25 | IntelMPI/5.1.3.181)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Autoconf-2.69-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Autoconf/.2.69)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Automake-1.15-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Automake/.1.15)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/libtool-2.4.6-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | libtool/.2.4.6)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/iimpi-8.2.5-GCC-4.9.3-2.25.eb (module: Core | iimpi/8.2.5-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Autotools-20150215-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Autotools/20150215)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/imkl-11.3.2.181-iimpi-8.2.5-GCC-4.9.3-2.25.eb (module: MPI/intel/2016.2.181-GCC-4.9.3-2.25/impi/5.1.3.181 | imkl/11.3.2.181)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/GMP-6.1.0-iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Compiler/intel/2016.2.181-GCC-4.9.3-2.25 | GMP/6.1.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/intel-2016a.eb (module: Core | intel/2016a)
 * [ ] /usr/local/software/jureca/Stage1/eb_repo/Python/Python-2.7.11-intel-2016a.eb (module: MPI/intel/2016.2.181-GCC-4.9.3-2.25/impi/5.1.3.181 | Python/2.7.11)
== Keeping temporary log file(s) /tmp/eb-erROoI/easybuild-yhsGUf.log* and directory /tmp/eb-erROoI.

real    0m57.717s
user    0m41.416s
sys 0m13.012s

Which is compared to 4.5m before any caching and 2.5m with module use caching

hpcugentbot · 2016-04-28T08:18:02Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2975/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

hpcugentbot · 2016-04-28T08:36:26Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2976/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

wpoely86 · 2016-04-28T08:50:31Z

easybuild/tools/modules.py

@@ -115,9 +115,31 @@
 _log = fancylogger.getLogger('modules', fname=False)


+MODULE_AVAIL_CACHE = {}
+MODULE_SHOW_CACHE = {}


it would help it you would put a comment about the format of this cache

…endency

wpoely86 · 2016-04-28T09:02:08Z

easybuild/tools/modules.py

-        ans = nub([mod['mod_name'] for mod in mods])
+        # cache 'avail' calls without an argument, since these are particularly expensive...
+        key = ('MODULEPATH=%s' % os.environ.get('MODULEPATH', ''), ';'.join(extra_args))
+        if not mod_name and key in MODULE_AVAIL_CACHE:


so we only cache a full ml av?

yes, that's also by far the most expensive and common call

also caching argument-less avail calls would need more care w.r.t. invalidating the cache, etc.

ocaisa · 2016-04-28T09:02:17Z

Now 10X on the original 4m30...that's an order of magnitude :)

swmanage@jrl08:/homeb/zam/swmanage/software/easybuild-framework> time eb /usr/local/software/jureca/Stage1/eb_repo/Python/Python-2.7.11-intel-2016a.eb -D -d --disable-cleanup-tmpdir
== temporary log file in case of crash /tmp/eb-W1LesH/easybuild-20rPhe.log
Dry run: printing build status of easyconfigs and dependencies
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/zlib-1.2.8.eb (module: Core | zlib/.1.2.8)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/M4-1.4.17.eb (module: Core | M4/.1.4.17)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Bison-3.0.4.eb (module: Core | Bison/.3.0.4)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/flex-2.6.0.eb (module: Core | flex/.2.6.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/binutils-2.25.eb (module: Core | binutils/.2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/GCCcore-4.9.3.eb (module: Core | GCCcore/.4.9.3)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/zlib-1.2.8-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | zlib/.1.2.8)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/M4-1.4.17-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | M4/.1.4.17)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Bison-3.0.4-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Bison/.3.0.4)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/flex-2.6.0-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | flex/.2.6.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/binutils-2.25-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | binutils/.2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/ncurses-6.0-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | ncurses/.6.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/icc-2016.2.181-GCC-4.9.3-2.25.eb (module: Core | icc/.2016.2.181-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/ifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Core | ifort/.2016.2.181-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Tcl-8.6.4-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Tcl/.8.6.4)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/bzip2-1.0.6-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | bzip2/.1.0.6)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/libreadline-6.3-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | libreadline/.6.3)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Core | Intel/2016.2.181-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/SQLite-3.9.2-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | SQLite/.3.9.2)
 * [ ] /usr/local/software/jureca/Stage1/eb_repo/Tk/Tk-8.6.4-GCCcore-4.9.3-no-X11.eb (module: Compiler/GCCcore/4.9.3 | Tk/.8.6.4-no-X11)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/zlib-1.2.8-iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Compiler/intel/2016.2.181-GCC-4.9.3-2.25 | zlib/.1.2.8)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/impi-5.1.3.181-iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Compiler/mpi/intel/2016.2.181-GCC-4.9.3-2.25 | IntelMPI/5.1.3.181)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Autoconf-2.69-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Autoconf/.2.69)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Automake-1.15-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Automake/.1.15)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/libtool-2.4.6-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | libtool/.2.4.6)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/iimpi-8.2.5-GCC-4.9.3-2.25.eb (module: Core | iimpi/8.2.5-GCC-4.9.3-2.25)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/Autotools-20150215-GCCcore-4.9.3.eb (module: Compiler/GCCcore/4.9.3 | Autotools/20150215)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/imkl-11.3.2.181-iimpi-8.2.5-GCC-4.9.3-2.25.eb (module: MPI/intel/2016.2.181-GCC-4.9.3-2.25/impi/5.1.3.181 | imkl/11.3.2.181)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/GMP-6.1.0-iccifort-2016.2.181-GCC-4.9.3-2.25.eb (module: Compiler/intel/2016.2.181-GCC-4.9.3-2.25 | GMP/6.1.0)
 * [x] /work/zam/swmanage/EasyBuild/Golden_Repo/Stage1/intel-2016a.eb (module: Core | intel/2016a)
 * [ ] /usr/local/software/jureca/Stage1/eb_repo/Python/Python-2.7.11-intel-2016a.eb (module: MPI/intel/2016.2.181-GCC-4.9.3-2.25/impi/5.1.3.181 | Python/2.7.11)
== Keeping temporary log file(s) /tmp/eb-W1LesH/easybuild-20rPhe.log* and directory /tmp/eb-W1LesH.

real    0m22.205s
user    0m15.952s
sys 0m4.581s

wpoely86 · 2016-04-28T09:03:10Z

easybuild/tools/modules.py

@@ -490,7 +526,16 @@ def show(self, mod_name):
        """
        Run 'module show' for the specified module.
        """
-        return self.run_module('show', mod_name, return_output=True)
+        key = ('MODULEPATH=%s' % os.environ.get('MODULEPATH', ''), mod_name)


if you do this in several place, why not put it in a function?

very valid remark, thanks

hpcugentbot · 2016-04-28T09:14:02Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2977/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

boegel · 2016-04-28T10:12:24Z

easybuild/tools/modules.py

+            paths_in_key = '='.join(key[0].split('=')[1:]).split(os.pathsep)
+            _log.debug("Paths for 'module %s' key '%s': %s", subcmd, key, paths_in_key)
+            for path_in_key in paths_in_key:
+                if os.path.samefile(path, path_in_key):


hmm, may need to be more careful here, samefile throws an error if either of these paths don't exist

ocaisa · 2016-04-28T10:12:27Z

Just got an error when using this for an installation:

  File "/homeb/zam/swmanage/software/easybuild-framework/easybuild/tools/modules.py", line 137, in invalidate_module_caches_for
    if os.path.samefile(path, path_in_key):
  File "/usr/lib64/python2.7/posixpath.py", line 162, in samefile
    s1 = os.stat(f1)
OSError: [Errno 2] No such file or directory: '/tmp/eb-7sOJmy/tmpbb_qV6/all/MPI/intel/2016.2.181-GCC-4.9.3-2.25/impi/5.1.3.181/LAMM'

hpcugentbot · 2016-04-28T10:43:11Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2980/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

…to invalidate module caches with (i.e. take into account possible .lua extension)

…es_for function

boegel · 2016-04-28T10:52:12Z

@ocaisa the error you ran into should be fixed, please verify?

hpcugentbot · 2016-04-28T11:12:03Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2981/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

hpcugentbot · 2016-04-28T12:03:42Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2982/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

ocaisa · 2016-04-28T12:26:42Z

@boegel Error is fixed

boegel · 2016-04-28T12:31:46Z

@ocaisa Great, thanks for the confirmation!

@wpoely86 Remarks fixed, please rereview?

wpoely86 · 2016-04-28T14:46:49Z

Just tested Perl: instead of 4-5 hours, it now builds in 37m 😃

wpoely86 · 2016-04-28T14:58:02Z

👍

hpcugentbot · 2016-04-29T12:09:12Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2983/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2016-04-29T13:32:06Z

Alright, going in, thanks for the feedback @wpoely86 and @ocaisa!

boegel added 4 commits April 27, 2016 18:29

cache 'module avail' results, using $MODULEPATH as key

b3f561c

global avail cache

8c130d1

reset avail cache for every test

dacf259

add test to check on two builds performed in a single go

5ccebc3

boegel added this to the v2.8.0 milestone Apr 27, 2016

boegel changed the title ~~Cache avail~~ cache result of 'module avail' calls Apr 27, 2016

boegel closed this Apr 28, 2016

boegel reopened this Apr 28, 2016

boegel added 3 commits April 28, 2016 09:50

optimize for single module case in ModulesTool.exist

feb2b1f

invalidate cache entries after installing a new module file

996dfad

also cache 'module show' commands

ce0e379

fix method signature for 'available' in MockModule

b48f8aa

wpoely86 reviewed Apr 28, 2016
View reviewed changes

invalidate module caches in test_robot_find_minimal_toolchain_for_dep…

c95fee9

…endency

wpoely86 reviewed Apr 28, 2016
View reviewed changes

boegel mentioned this pull request Apr 28, 2016

unset $PYTHONPATH in before tested bootstrapped EasyBuild module #1743

Merged

Merge branch 'develop' into cache_avail

87a6353

boegel reviewed Apr 28, 2016
View reviewed changes

boegel added 2 commits April 28, 2016 12:51

use module path + short/long module name rather than module filepath …

211f15e

…to invalidate module caches with (i.e. take into account possible .lua extension)

be more careful when checking path equality in invalidate_module_cach…

1e06007

…es_for function

fix remarks, add tests on module caching

1baf462

fix log msg for cached show

66d8e82

boegel merged commit 61b6fbb into easybuilders:develop Apr 29, 2016

boegel deleted the cache_avail branch April 29, 2016 13:32

boegel mentioned this pull request Apr 29, 2016

Cache-able calls to "module show ..." and "module avail" #1739

Closed

boegel mentioned this pull request May 21, 2016

fatal error Invalidating the module cache with 2.8 #1775

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cache result of 'module avail' calls #1742

cache result of 'module avail' calls #1742

boegel commented Apr 27, 2016 •

edited

Loading

boegel commented Apr 27, 2016

hpcugentbot commented Apr 27, 2016

hpcugentbot commented Apr 28, 2016

boegel commented Apr 28, 2016

ocaisa commented Apr 28, 2016 •

edited

Loading

hpcugentbot commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

wpoely86 Apr 28, 2016

wpoely86 Apr 28, 2016

boegel Apr 28, 2016

ocaisa commented Apr 28, 2016 •

edited

Loading

wpoely86 Apr 28, 2016

boegel Apr 28, 2016

hpcugentbot commented Apr 28, 2016

boegel Apr 28, 2016

ocaisa commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

boegel commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

ocaisa commented Apr 28, 2016

boegel commented Apr 28, 2016

wpoely86 commented Apr 28, 2016

wpoely86 commented Apr 28, 2016

hpcugentbot commented Apr 29, 2016

boegel commented Apr 29, 2016

cache result of 'module avail' calls #1742

cache result of 'module avail' calls #1742

Conversation

boegel commented Apr 27, 2016 • edited Loading

boegel commented Apr 27, 2016

hpcugentbot commented Apr 27, 2016

hpcugentbot commented Apr 28, 2016

boegel commented Apr 28, 2016

ocaisa commented Apr 28, 2016 • edited Loading

hpcugentbot commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

wpoely86 Apr 28, 2016

Choose a reason for hiding this comment

wpoely86 Apr 28, 2016

Choose a reason for hiding this comment

boegel Apr 28, 2016

Choose a reason for hiding this comment

ocaisa commented Apr 28, 2016 • edited Loading

wpoely86 Apr 28, 2016

Choose a reason for hiding this comment

boegel Apr 28, 2016

Choose a reason for hiding this comment

hpcugentbot commented Apr 28, 2016

boegel Apr 28, 2016

Choose a reason for hiding this comment

ocaisa commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

boegel commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

hpcugentbot commented Apr 28, 2016

ocaisa commented Apr 28, 2016

boegel commented Apr 28, 2016

wpoely86 commented Apr 28, 2016

wpoely86 commented Apr 28, 2016

hpcugentbot commented Apr 29, 2016

boegel commented Apr 29, 2016

boegel commented Apr 27, 2016 •

edited

Loading

ocaisa commented Apr 28, 2016 •

edited

Loading

ocaisa commented Apr 28, 2016 •

edited

Loading