Overwriting TMP environment variable breaks AMD ROCm in Linux

[ahesford]

In src/folder.c, a comment notes that a good workaround for problematic handling of -I parameters is to set the TMP environment variable before loading the runtime. This is not the case for the AMD ROCm runtime in Linux, at least with the latest version (3.3.0), unless hashcat is run as the superuser.

The ROCm Code Object Manager and ROCm OpenCL Runtime use the value of the TMP environment variable to determine where compiled objects should be placed. Lines 465 and 467 of src/folder.c set this value to ${PREFIX}/share/hashcat/OpenCL, which is not typically writeable by ordinary users. Thus, OpenCL kernel compilation fails when the build products can not be stored.

While I'm not sure about other OpenCL runtimes (it certainly doesn't seem to hurt ocl-icd with the amdgpu-pro OpenCL ICD), the proper behavior for ROCm is either to leave TMP alone or explicitly set it to somewhere like /tmp that can be written by any user running hashcat.

[jsteube]

That's not the case for me. You are right that TMP is set by hashcat as a workaround for buggy OpenCL runtimes, but this has no effect on ROCM.

I've just tested this locally with a normal (not superuser) user.

$ strace -f -v -o x hashcat -b -m 0
...
$ grep local/share x

Output:

12800 execve("/usr/local/bin/hashcat", ["hashcat", "-b", "-m", "0"], ["LS_COLORS=rs=0:di=01;34:ln=01;36"..., "LESSCLOSE=/usr/bin/lesspipe %s %"..., "LANG=en_US.UTF-8", "USER=atom", "PWD=/home/atom", "HOME=/home/atom", "XDG_DATA_DIRS=/usr/local/share:/"..., "MAIL=/var/mail/atom", "SHELL=/bin/bash", "TERM=linux", "SHLVL=1", "LOGNAME=atom", "PATH=/usr/local/sbin:/usr/local/"..., "LESSOPEN=| /usr/bin/lesspipe %s", "_=/usr/bin/strace", "OLDPWD=/opt/rocm/hsa-amd-aqlprof"...]) = 0
12800 lstat("/usr/local/share", {st_dev=makedev(259, 2), st_ino=919323, st_mode=S_IFDIR|0755, st_nlink=8, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1555923680 /* 2019-04-22T11:01:20.762108155+0200 */, st_atime_nsec=762108155, st_mtime=1588835031 /* 2020-05-07T09:03:51.323902392+0200 */, st_mtime_nsec=323902392, st_ctime=1588835031 /* 2020-05-07T09:03:51.323902392+0200 */, st_ctime_nsec=323902392}) = 0
12800 lstat("/usr/local/share/hashcat", {st_dev=makedev(259, 2), st_ino=1311915, st_mode=S_IFDIR|0755, st_nlink=4, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1588835031 /* 2020-05-07T09:03:51.323902392+0200 */, st_atime_nsec=323902392, st_mtime=1588835031 /* 2020-05-07T09:03:51.327902380+0200 */, st_mtime_nsec=327902380, st_ctime=1588835031 /* 2020-05-07T09:03:51.327902380+0200 */, st_ctime_nsec=327902380}) = 0
12800 lstat("/usr/local/share/hashcat/OpenCL", {st_dev=makedev(259, 2), st_ino=1447965, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=120, st_size=57344, st_atime=1588835083 /* 2020-05-07T09:04:43.507795274+0200 */, st_atime_nsec=507795274, st_mtime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_mtime_nsec=911897844, st_ctime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_ctime_nsec=911897844}) = 0
12800 write(1, "TMP=/usr/local/share/hashcat/Ope"..., 36) = 36
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/hashcat.hctune", O_RDONLY) = 3
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/hashcat.hctune", O_RDONLY) = 3
12800 access("/usr/local/share/hashcat", R_OK) = 0
12800 access("/usr/local/share/hashcat/OpenCL", R_OK) = 0
12800 access("/usr/local/share/hashcat/modules/module_00000.so", F_OK) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/modules/module_00000.so", O_RDONLY|O_CLOEXEC) = 3
12800 access("/usr/local/share/hashcat/OpenCL/m00000_a3-pure.cl", R_OK) = 0
12800 access("/usr/local/share/hashcat/OpenCL/m00000_a3-optimized.cl", R_OK) = 0
12800 access("/usr/local/share/hashcat/OpenCL/m00400-pure.cl", R_OK) = 0
12800 chdir("/usr/local/share/hashcat/OpenCL") = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/modules/module_00000.so", O_RDONLY|O_CLOEXEC) = 3
12800 access("/usr/local/share/hashcat/OpenCL/m00000_a3-pure.cl", R_OK) = 0
12800 access("/usr/local/share/hashcat/OpenCL/m00000_a3-optimized.cl", R_OK) = 0
12800 stat("/usr/local/share/hashcat/hashcat.hcstat2", {st_dev=makedev(259, 2), st_ino=1312368, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=472, st_size=240526, st_atime=1588835039 /* 2020-05-07T09:03:59.467884119+0200 */, st_atime_nsec=467884119, st_mtime=1588835031 /* 2020-05-07T09:03:51.327902380+0200 */, st_mtime_nsec=327902380, st_ctime=1588835031 /* 2020-05-07T09:03:51.327902380+0200 */, st_ctime_nsec=327902380}) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/hashcat.hcstat2", O_RDONLY) = 3
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/hashcat.hcstat2", O_RDONLY) = 3
12800 access("/usr/local/share/hashcat/OpenCL/shared.cl", R_OK <unfinished ...>
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/shared.cl", O_RDONLY <unfinished ...>
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/shared.cl", O_RDONLY) = 3
12800 stat("/usr/local/share/hashcat/OpenCL/shared.cl", {st_dev=makedev(259, 2), st_ino=1449867, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=3715, st_atime=1588835040 /* 2020-05-07T09:04:00.299882494+0200 */, st_atime_nsec=299882494, st_mtime=1588835031 /* 2020-05-07T09:03:51.715901268+0200 */, st_mtime_nsec=715901268, st_ctime=1588835031 /* 2020-05-07T09:03:51.715901268+0200 */, st_ctime_nsec=715901268}) = 0
12800 stat("/usr/local/share/hashcat/OpenCL", {st_dev=makedev(259, 2), st_ino=1447965, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=120, st_size=57344, st_atime=1588835083 /* 2020-05-07T09:04:43.507795274+0200 */, st_atime_nsec=507795274, st_mtime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_mtime_nsec=911897844, st_ctime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_ctime_nsec=911897844}) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_vendor.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 44
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_types.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 43
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_platform.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 47
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_platform.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 46
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_common.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 45
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_common.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 44
12800 access("/usr/local/share/hashcat/OpenCL/m00000_a3-optimized.cl", R_OK) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/m00000_a3-optimized.cl", O_RDONLY) = 3
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/m00000_a3-optimized.cl", O_RDONLY) = 3
12800 stat("/usr/local/share/hashcat/OpenCL/m00000_a3-optimized.cl", {st_dev=makedev(259, 2), st_ino=1450651, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=48, st_size=22429, st_atime=1588835040 /* 2020-05-07T09:04:00.759881594+0200 */, st_atime_nsec=759881594, st_mtime=1588835032 /* 2020-05-07T09:03:52.631898645+0200 */, st_mtime_nsec=631898645, st_ctime=1588835032 /* 2020-05-07T09:03:52.631898645+0200 */, st_ctime_nsec=631898645}) = 0
12800 stat("/usr/local/share/hashcat/OpenCL", {st_dev=makedev(259, 2), st_ino=1447965, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=120, st_size=57344, st_atime=1588835083 /* 2020-05-07T09:04:43.507795274+0200 */, st_atime_nsec=507795274, st_mtime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_mtime_nsec=911897844, st_ctime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_ctime_nsec=911897844}) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_vendor.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 44
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_types.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 43
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_platform.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 47
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_platform.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 46
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_common.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 45
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_common.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 44
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_simd.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 43
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_simd.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 42
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_hash_md5.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 47
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_hash_md5.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 46
12800 access("/usr/local/share/hashcat/OpenCL/markov_le.cl", R_OK) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/markov_le.cl", O_RDONLY) = 3
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/markov_le.cl", O_RDONLY) = 3
12800 stat("/usr/local/share/hashcat/OpenCL/markov_le.cl", {st_dev=makedev(259, 2), st_ino=1450780, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=5525, st_atime=1588835042 /* 2020-05-07T09:04:02.667877857+0200 */, st_atime_nsec=667877857, st_mtime=1588835032 /* 2020-05-07T09:03:52.847898027+0200 */, st_mtime_nsec=847898027, st_ctime=1588835032 /* 2020-05-07T09:03:52.847898027+0200 */, st_ctime_nsec=847898027}) = 0
12800 stat("/usr/local/share/hashcat/OpenCL", {st_dev=makedev(259, 2), st_ino=1447965, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=120, st_size=57344, st_atime=1588835083 /* 2020-05-07T09:04:43.507795274+0200 */, st_atime_nsec=507795274, st_mtime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_mtime_nsec=911897844, st_ctime=1588835032 /* 2020-05-07T09:03:52.911897844+0200 */, st_ctime_nsec=911897844}) = 0
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_vendor.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 44
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_types.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 43
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_platform.cl", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 47
12800 openat(AT_FDCWD, "/usr/local/share/hashcat/OpenCL/inc_platform.h", O_RDONLY|O_CLOEXEC) = 3
12800 readlink("/proc/self/fd/3", "/usr/local/share/hashcat/OpenCL/"..., 4096) = 46

As you can see, hashcat is not trying to write to any file in that folder and there's not a single permission denied. Just to make sure my user is not allowed to write to that folder:

atom@ro:~$ touch /usr/local/share/hashcat/OpenCL/x
touch: cannot touch '/usr/local/share/hashcat/OpenCL/x': Permission denied

[ahesford]

What version of ROCm are you running, and do you have TMPDIR set in your environment? Grepping through the ROCm fork of LLVM suggests that the preferred order of environment variables to identify temporary directories is TMPDIR, TMP, TEMP, TEMPDIR. It turns out I can avoid the issue with ROCm 3.3.0 if I set TMPDIR=/tmp, but setting TEMP=/tmp does not avoid the issue.

The whole point of temporary directories is to provide "scratch" space, so redirecting to a non-writeable directory seems like bad practice in general. If this is absolutely necessary, maybe the solution is to set TMPDIR (if it is not already set) to the current contents of TMP (if it is already set) before overwriting TMP for the workaround. If TMP is not set prior to the workaround, maybe setting TMPDIR=/tmp on non-Windows systems would be a good solution. After all, TMPDIR is the canonical variable in POSIX.

[jsteube]

While I like your PR and the way you solved it, I was thinking maybe we can get rid of this workaround in general. It goes back to a runtime which had this bug years ago and my hopes are that it is fixed for now.

I have pushed commit dc9f4e9 with the idea to let it there for a while and give hashcat users some time to test and and see if there's any negative response to it. There's simply to many different opencl runtimes in different version on different platforms for me to test all of them. If it turns out to break something I will revert my test commit and use your PR.

It would be nice if you can also do some testing as it seems your installation is somewhat different than all of my test systems.

[ahesford]

I'd be happy to just drop the TMP hack. My ROCm runtime was built for Void Linux according to PR #21153 and uses the AMD comgr library to compile for GPU targets. I'm not positive, but it looks like the official AMD packages for Ubuntu and others still used clang-ocl as of version 3.3.0, even though it depends on the depreacted hcc. Another possibility is that AMD uses some patches when building packages that didn't make it into the open-source repos.

AMD has recently released ROCm 3.5.0, but I haven't been terribly motivated to try packaging the new version. I don't know whether that release is subject to the same issue.

[jsteube]

Handled in #2387