modules-4.4.0 :: big files modulepath causes slowdown or crashes when module load #375
Description
Hello
a user have a private module tree that it enable via module use command
problem is that his tree contains singularity containers in the same subtree as modulesfiles
eg: assuming he have modulefiles for 2 pacakges foo and bar
modules/foo/V1
/V2
singularity/container1.simg
singulartiy/container2.sigm
/bar/V1
/V2
singularity/container1.simg
singulartiy/container2.sigmg
after tracing the moduletcl call we noticed that modulecmd read the whole content of the singularity containers while searching for the modulefiles.
see:
1886288 stat("/pasteur/sonic/scratch/users/edeveaud/modules3/rnadiff/prod", {st_mode=S_IFREG|0644, st_size=428, ...}) = 0
1886288 openat(AT_FDCWD, "/pasteur/sonic/scratch/users/edeveaud/modules3/rnadiff/prod", O_RDONLY) = 5
1886288 read(5, "#%Module1.0\nproc ModulesHelp { }"..., 4096) = 428
1886288 read(5, "", 4096) = 0
1886288 close(5) = 0
1886288 stat("/pasteur/sonic/scratch/users/edeveaud/modules3/rnadiff/singularity/rnadiff_4feb2019.img", {st_mode=S_IFREG|0755, st_size=988377088, ...}) = 0
1886288 openat(AT_FDCWD, "/pasteur/sonic/scratch/users/edeveaud/modules3/rnadiff/singularity/rnadiff_4feb2019.img", O_RDONLY) = 5
1886288 read(5, "#!/usr/bin/env run-singularity\n\0"..., 4096) = 4096
1886288 read(5, "\1@\0\0\1\1\0\0\0\1\0\0\360\0\0\0\0\0\200\0\0\0\0\0\0\362\3\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
1886288 read(5, "Bootstrap: localimage\nFrom: R.im"..., 4096) = 4096
1886288 read(5, "{\"Env\":[\"PATH=/usr/local/sbin:/u"..., 4096) = 4096
1886288 read(5, "hsqs\204\262\1\0eF9^\0\0\2\0\307\33\0\0\1\0\21\0\300\0\20\0\4\0\0\0"..., 4096) = 4096
1886288 read(5, "\266BH\26\200\234\6HG\335\32\340L\224\263\200\327\241\234\v\234'R\347\365\220\336\4\330\f\330\2"..., 4096) = 4096
1886288 read(5, "\324,\334\232\376'\372)\377\274\211\353\234\216^\277\302+'\233%F|I\237r\327\322q\364D\227"..., 4096) = 4096
1886288 read(5, "\222\177\177\250\372\351\322\332\346\203\307$B\264\222B\206\245\236\2341\311l\277\330\273\254u\363\17\210\263"..., 4096) = 4096
1886288 read(5, "\\1\30\333\255\231,g)\266\333\37\364\317\274v\272\235\257\300u\23\3673=^=\215\353\342\306\32"..., 4096) = 4096
1886288 brk(NULL) = 0x2109000
1886288 brk(0x2146000) = 0x2146000
1886288 brk(NULL) = 0x2146000
1886288 brk(NULL) = 0x2146000
1886288 brk(0x2138000) = 0x2138000
1886288 brk(NULL) = 0x2138000
1886288 read(5, "\227\301\336\17p\244\313a7\r=\271\300\263w\266\307yl|_\306\363W\335w\320[\177p\336\33"..., 4096) = 4096
that produce to a big slowdonw and worst, crashed modulecmd in one case where the subtree contains lots of huge sinngularity containers with following error message
max size for a Tcl value (2147483647 bytes) exceeded
as far as I undertand at search time modulecmd should only read the first line of the flies to check the module magic cookie
NB by curiosity we traced the same module load command using module C installation on another machine and it was
just checking the magic cookie.
see:
57131 getdents(4, /* 3 entries */, 32768) = 88
57131 open("/pasteur/projets/policy01/Matrix/modules2/rnadiff/singularity/rnadiff_4feb2019.img", O_RDONLY) = 5
57131 read(5, "#!/usr/b", 8) = 8
57131 close(5) = 0
NB we taught the user some good practices about file organisation ;-)
regards
Eric
edit "formatage"