Successful compile doesn't work macosx-x86-64-opencl #50

mubix · 2012-07-03T21:05:03Z

Downloaded from the OpenWall site and making the unsigned int / uint changes I was able to compile with the following errors:

md5u_fmt_plug.o salted_sha1_fmt_plug.o sapB_fmt_plug.o sapG_fmt_plug.o sha1_gen_fmt_plug.o sip_fmt_plug.o vnc_fmt_plug.o wbb3_fmt_plug.o rawSHA1_ng_fmt.o plugin.o dummy.o batch.o bench.o charset.o common.o compiler.o config.o cracker.o crc32.o external.o formats.o getopt.o idle.o inc.o john.o list.o loader.o logger.o math.o memory.o misc.o options.o params.o path.o recovery.o rpp.o rules.o signals.o single.o status.o tty.o wordlist.o mkv.o mkvlib.o fake_salts.o win32_memmap.o unicode.o unshadow.o unafs.o undrop.o unique.o common-opencl.o common_opencl_pbkdf2.o opencl_mysqlsha1_fmt.o opencl_cryptmd5_fmt.o opencl_phpass_fmt.o opencl_rawsha1_fmt.o opencl_nt_fmt.o opencl_rawmd5_fmt.o opencl_nsldaps_fmt.o opencl_cryptsha512_fmt.o opencl_mscash2_fmt.o opencl_wpapsk_fmt.o opencl_xsha512_fmt.o opencl_rawsha512_fmt.o opencl_bf_std.o opencl_bf_fmt.o opencl_pwsafe_fmt.o x86-64.o sse-intrinsics.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/john
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/unshadow
ln -s john ../run/unshadow
rm -f ../run/unafs
ln -s john ../run/unafs
rm -f ../run/unique
ln -s john ../run/unique
rm -f ../run/undrop
ln -s john ../run/undrop
rm -f ../run/ssh2john
ln -s john ../run/ssh2john
rm -f ../run/pdf2john
ln -s john ../run/pdf2john
rm -f ../run/rar2john
ln -s john ../run/rar2john
rm -f ../run/zip2john
ln -s john ../run/zip2john
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64 -I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0 -Wno-deprecated-declarations -funroll-loops genmkvpwd.c
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64 -I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0 -Wno-deprecated-declarations -funroll-loops -D_JOHN_MISC_NO_LOG misc.c -o miscnl.o
gcc genmkvpwd.o mkvlib.o memory.o miscnl.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/genmkvpwd
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64 -I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0 -Wno-deprecated-declarations -funroll-loops mkvcalcproba.c
gcc mkvcalcproba.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/mkvcalcproba
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64 -I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0 -Wno-deprecated-declarations -funroll-loops calc_stat.c
gcc calc_stat.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/calc_stat
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64 -I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0 -Wno-deprecated-declarations -funroll-loops tgtsnarf.c
gcc tgtsnarf.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/tgtsnarf
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/racf2john
ln -s john ../run/racf2john
make[1]: Nothing to be done for `../run/mozilla2john'.
rm -f ../run/hccap2john
ln -s john ../run/hccap2john
rm -f ../run/pwsafe2john
ln -s john ../run/pwsafe2john
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64 -I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0 -Wno-deprecated-declarations -funroll-loops raw2dyna.c
gcc raw2dyna.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/raw2dyna
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/keepass2john
ln -s john ../run/keepass2john
[ -f ../run/john.local.conf ] || touch ../run/john.local.conf
cp opencl/*.cl ../run/
cp opencl_*.h ../run/

which resuts in:

/opt/john-1.7.9-jumbo-6/run $ ./john md5test.txt --format=raw-md5-opencl
OpenCL platform 0: Apple, 2 device(s).
Using device 0: Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
Max local work size 1024 Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Optimal local work size 1
(to avoid this test on next run do export LWS=1)
Local work size (LWS) 1, Global work size (GWS) 2097152
Loaded 1 password hash (Raw MD5 [OpenCL])
guesses: 0  time: 0:00:00:05 99.90% (1) (ETA: Tue Jul  3 16:34:44 2012)  c/s: 0.00
guesses: 0  time: 0:00:00:10 98.24% (2) (ETA: Tue Jul  3 16:34:49 2012)  c/s: 76.88  trying: 123456 - Sssing
Session aborted
/opt/john-1.7.9-jumbo-6/run $ ./john md5test.txt --format=raw-md5-opencl --device=1
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: Error getting function data from server
OpenCL error (CL_INVALID_PROGRAM_EXECUTABLE) in file (opencl_rawmd5_fmt.c) at line (218) - (Error creating kernel. Double-check kernel name?)

And if this helps here is the device list:

/opt/john-1.7.9-jumbo-6/run$ ./john --list=opencl-devices
Platform #0 name: Apple
Platform version: OpenCL 1.1 (Apr  9 2012 19:41:45)
    Device #0 name:     Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
    Device vendor:      Intel
    Device type:        CPU (LE)
    Device version:     OpenCL 1.1 
    Driver version:     1.1
    Global Memory:      8192 MB
    Global Memory Cache:    64 bytes
    Local Memory:       32 KB (Global)
    Max clock (MHz) :   2660
    Max Work Group Size:    1024
    Parallel compute cores: 4

    Device #1 name:     GeForce GT 330M
    Device vendor:      NVIDIA
    Device type:        GPU (LE)
    Device version:     OpenCL 1.0 
    Driver version:     CLH 1.0
    Global Memory:      512 MB
    Global Memory Cache:    0 bytes
    Local Memory:       16 KB (Local)
    Max clock (MHz) :   1100
    Max Work Group Size:    512
    Parallel compute cores: 6
    Stream processors:  48  (6 x 8)

The text was updated successfully, but these errors were encountered:

magnumripper · 2012-07-03T21:18:39Z

I am forwarding this to the john-dev mailing list. To me it seems you
need to fix your OpenCL paths in Makefile.

magnum

-------- Original Message --------
Subject: [magnum-jumbo] Successful compile doesn't work
macosx-x86-64-opencl (#50)
Date: Tue, 3 Jul 2012 14:05:06 -0700
From: Rob Fuller
reply@reply.github.com
To: magnum john.magnum@hushmail.com

Downloaded from the OpenWall site and making the unsigned int / uint
changes I was able to compile with the following errors:

md5u_fmt_plug.o salted_sha1_fmt_plug.o sapB_fmt_plug.o sapG_fmt_plug.o
sha1_gen_fmt_plug.o sip_fmt_plug.o vnc_fmt_plug.o wbb3_fmt_plug.o
rawSHA1_ng_fmt.o plugin.o dummy.o batch.o bench.o charset.o common.o
compiler.o config.o cracker.o crc32.o external.o formats.o getopt.o
idle.o inc.o john.o list.o loader.o logger.o math.o memory.o misc.o
options.o params.o path.o recovery.o rpp.o rules.o signals.o single.o
status.o tty.o wordlist.o mkv.o mkvlib.o fake_salts.o win32_memmap.o
unicode.o unshadow.o unafs.o undrop.o unique.o common-opencl.o
common_opencl_pbkdf2.o opencl_mysqlsha1_fmt.o opencl_cryptmd5_fmt.o
opencl_phpass_fmt.o opencl_rawsha1_fmt.o opencl_nt_fmt.o
opencl_rawmd5_fmt.o opencl_nsldaps_fmt.o opencl_cryptsha512_fmt.o
opencl_mscash2_fmt.o opencl_wpapsk_fmt.o opencl_xsha512_fmt.o
opencl_rawsha512_fmt.o opencl_bf_std.o opencl_bf_fmt.o
opencl_pwsafe_fmt.o x86-64.o sse-intrinsics.o -s -L/usr/local/lib
-L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/li
 b/x86_64 -L/lib64 -framework OpenCL -o ../run/john
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/unshadow
ln -s john ../run/unshadow
rm -f ../run/unafs
ln -s john ../run/unafs
rm -f ../run/unique
ln -s john ../run/unique
rm -f ../run/undrop
ln -s john ../run/undrop
rm -f ../run/ssh2john
ln -s john ../run/ssh2john
rm -f ../run/pdf2john
ln -s john ../run/pdf2john
rm -f ../run/rar2john
ln -s john ../run/rar2john
rm -f ../run/zip2john
ln -s john ../run/zip2john
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops genmkvpwd.c
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops -D_JOHN_MISC_NO_LOG misc.c
-o miscnl.o
gcc genmkvpwd.o mkvlib.o memory.o miscnl.o -s -L/usr/local/lib
-L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64
-L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/genmkvpwd
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops mkvcalcproba.c
gcc mkvcalcproba.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl
-lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64
-framework OpenCL -o ../run/mkvcalcproba
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops calc_stat.c
gcc calc_stat.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto
-lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL
-o ../run/calc_stat
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops tgtsnarf.c
gcc tgtsnarf.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto
-lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL
-o ../run/tgtsnarf
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/racf2john
ln -s john ../run/racf2john
make[1]: Nothing to be done for `../run/mozilla2john'.
rm -f ../run/hccap2john
ln -s john ../run/hccap2john
rm -f ../run/pwsafe2john
ln -s john ../run/pwsafe2john
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops raw2dyna.c
gcc raw2dyna.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto
-lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL
-o ../run/raw2dyna
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/keepass2john
ln -s john ../run/keepass2john
[ -f ../run/john.local.conf ] || touch ../run/john.local.conf
cp opencl/*.cl ../run/
cp opencl_*.h ../run/

which resuts in:

/opt/john-1.7.9-jumbo-6/run $ ./john md5test.txt --format=raw-md5-opencl
OpenCL platform 0: Apple, 2 device(s).
Using device 0: Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
Max local work size 1024 Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Optimal local work size 1
(to avoid this test on next run do export LWS=1)
Local work size (LWS) 1, Global work size (GWS) 2097152
Loaded 1 password hash (Raw MD5 [OpenCL])
guesses: 0  time: 0:00:00:05 99.90% (1) (ETA: Tue Jul  3 16:34:44 2012)
 c/s: 0.00
guesses: 0  time: 0:00:00:10 98.24% (2) (ETA: Tue Jul  3 16:34:49 2012)
 c/s: 76.88  trying: 123456 - Sssing
Session aborted
/opt/john-1.7.9-jumbo-6/run $ ./john md5test.txt --format=raw-md5-opencl
--device=1
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: Error getting function data from server
OpenCL error (CL_INVALID_PROGRAM_EXECUTABLE) in file
(opencl_rawmd5_fmt.c) at line (218) - (Error creating kernel.
Double-check kernel name?)

And if this helps here is the device list:

/opt/john-1.7.9-jumbo-6/run$ ./john --list=opencl-devices
Platform #0 name: Apple
Platform version: OpenCL 1.1 (Apr  9 2012 19:41:45)
    Device #0 name:     Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
    Device vendor:      Intel
    Device type:        CPU (LE)
    Device version:     OpenCL 1.1
    Driver version:     1.1
    Global Memory:      8192 MB
    Global Memory Cache:    64 bytes
    Local Memory:       32 KB (Global)
    Max clock (MHz) :   2660
    Max Work Group Size:    1024
    Parallel compute cores: 4

    Device #1 name:     GeForce GT 330M
    Device vendor:      NVIDIA
    Device type:        GPU (LE)
    Device version:     OpenCL 1.0
    Driver version:     CLH 1.0
    Global Memory:      512 MB
    Global Memory Cache:    0 bytes
    Local Memory:       16 KB (Local)
    Max clock (MHz) :   1100
    Max Work Group Size:    512
    Parallel compute cores: 6
    Stream processors:  48  (6 x 8)

Reply to this email directly or view it on GitHub:
https://github.com/magnumripper/magnum-jumbo/issues/50

mubix · 2012-07-03T21:56:01Z

modding the paths removed the errors about path, but still getting the 'results in:' stuff. so I'm guessing there are two unrelated issues.

magnumripper · 2012-07-03T22:34:52Z

OK so the path issue is fixed and does not need fixing in our tree (I'm
pretty sure it's documented).

The "Error -54" is a bug: It should give you a human-readable error.
This is fixed now and soon committed. The actual error is
"CL_INVALID_WORK_GROUP_SIZE"

The root cause has been known since April but no-one has apparently done
anything about it yet, likely because people develop on heavier GPU cards:
http://www.openwall.com/lists/john-dev/2012/04/10/4

Hopefully, it will be fixed before Jumbo-7 (within weeks). Thank you for
reporting!

magnum

This answer arrived on github only:
modding the paths removed the errors about path, but still getting the 'results in:' stuff. so I'm guessing there are two unrelated issues.

Reply to this email directly or view it on GitHub:
https://github.com/magnumripper/magnum-jumbo/issues/50#issuecomment-6747688

On 2012-07-03 23:18, magnum wrote:

I am forwarding this to the john-dev mailing list. To me it seems you
need to fix your OpenCL paths in Makefile.

magnum

-------- Original Message --------
Subject: [magnum-jumbo] Successful compile doesn't work
macosx-x86-64-opencl (#50)
Date: Tue, 3 Jul 2012 14:05:06 -0700
From: Rob Fuller
reply@reply.github.com
To: magnum john.magnum@hushmail.com

Downloaded from the OpenWall site and making the unsigned int / uint
changes I was able to compile with the following errors:

md5u_fmt_plug.o salted_sha1_fmt_plug.o sapB_fmt_plug.o sapG_fmt_plug.o
sha1_gen_fmt_plug.o sip_fmt_plug.o vnc_fmt_plug.o wbb3_fmt_plug.o
rawSHA1_ng_fmt.o plugin.o dummy.o batch.o bench.o charset.o common.o
compiler.o config.o cracker.o crc32.o external.o formats.o getopt.o
idle.o inc.o john.o list.o loader.o logger.o math.o memory.o misc.o
options.o params.o path.o recovery.o rpp.o rules.o signals.o single.o
status.o tty.o wordlist.o mkv.o mkvlib.o fake_salts.o win32_memmap.o
unicode.o unshadow.o unafs.o undrop.o unique.o common-opencl.o
common_opencl_pbkdf2.o opencl_mysqlsha1_fmt.o opencl_cryptmd5_fmt.o
opencl_phpass_fmt.o opencl_rawsha1_fmt.o opencl_nt_fmt.o
opencl_rawmd5_fmt.o opencl_nsldaps_fmt.o opencl_cryptsha512_fmt.o
opencl_mscash2_fmt.o opencl_wpapsk_fmt.o opencl_xsha512_fmt.o
opencl_rawsha512_fmt.o opencl_bf_std.o opencl_bf_fmt.o
opencl_pwsafe_fmt.o x86-64.o sse-intrinsics.o -s -L/usr/local/lib
-L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64 -L/usr/local/cuda/li
 b/x86_64 -L/lib64 -framework OpenCL -o ../run/john
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/unshadow
ln -s john ../run/unshadow
rm -f ../run/unafs
ln -s john ../run/unafs
rm -f ../run/unique
ln -s john ../run/unique
rm -f ../run/undrop
ln -s john ../run/undrop
rm -f ../run/ssh2john
ln -s john ../run/ssh2john
rm -f ../run/pdf2john
ln -s john ../run/pdf2john
rm -f ../run/rar2john
ln -s john ../run/rar2john
rm -f ../run/zip2john
ln -s john ../run/zip2john
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops genmkvpwd.c
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops -D_JOHN_MISC_NO_LOG misc.c
-o miscnl.o
gcc genmkvpwd.o mkvlib.o memory.o miscnl.o -s -L/usr/local/lib
-L/usr/local/ssl/lib -lssl -lcrypto -lm -lz    -m64
-L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL -o ../run/genmkvpwd
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops mkvcalcproba.c
gcc mkvcalcproba.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl
-lcrypto -lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64
-framework OpenCL -o ../run/mkvcalcproba
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops calc_stat.c
gcc calc_stat.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto
-lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL
-o ../run/calc_stat
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops tgtsnarf.c
gcc tgtsnarf.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto
-lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL
-o ../run/tgtsnarf
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/racf2john
ln -s john ../run/racf2john
make[1]: Nothing to be done for `../run/mozilla2john'.
rm -f ../run/hccap2john
ln -s john ../run/hccap2john
rm -f ../run/pwsafe2john
ln -s john ../run/pwsafe2john
gcc -c -Wall -O2 -fomit-frame-pointer -I/usr/local/include     -m64
-I/usr/local/cuda/include -DBSD -DCL_VERSION_1_0
-Wno-deprecated-declarations -funroll-loops raw2dyna.c
gcc raw2dyna.o -s -L/usr/local/lib -L/usr/local/ssl/lib -lssl -lcrypto
-lm -lz    -m64 -L/usr/local/cuda/lib/x86_64 -L/lib64 -framework OpenCL
-o ../run/raw2dyna
ld: warning: directory not found for option '-L/usr/local/ssl/lib'
ld: warning: directory not found for option '-L/usr/local/cuda/lib/x86_64'
ld: warning: directory not found for option '-L/lib64'
ld: warning: option -s is obsolete and being ignored
rm -f ../run/keepass2john
ln -s john ../run/keepass2john
[ -f ../run/john.local.conf ] || touch ../run/john.local.conf
cp opencl/*.cl ../run/
cp opencl_*.h ../run/

which resuts in:

/opt/john-1.7.9-jumbo-6/run $ ./john md5test.txt --format=raw-md5-opencl
OpenCL platform 0: Apple, 2 device(s).
Using device 0: Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
Max local work size 1024 Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Error -54
Optimal local work size 1
(to avoid this test on next run do export LWS=1)
Local work size (LWS) 1, Global work size (GWS) 2097152
Loaded 1 password hash (Raw MD5 [OpenCL])
guesses: 0  time: 0:00:00:05 99.90% (1) (ETA: Tue Jul  3 16:34:44 2012)
 c/s: 0.00
guesses: 0  time: 0:00:00:10 98.24% (2) (ETA: Tue Jul  3 16:34:49 2012)
 c/s: 76.88  trying: 123456 - Sssing
Session aborted
/opt/john-1.7.9-jumbo-6/run $ ./john md5test.txt --format=raw-md5-opencl
--device=1
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: Error getting function data from server
OpenCL error (CL_INVALID_PROGRAM_EXECUTABLE) in file
(opencl_rawmd5_fmt.c) at line (218) - (Error creating kernel.
Double-check kernel name?)

And if this helps here is the device list:

/opt/john-1.7.9-jumbo-6/run$ ./john --list=opencl-devices
Platform #0 name: Apple
Platform version: OpenCL 1.1 (Apr  9 2012 19:41:45)
 Device #0 name:     Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
 Device vendor:      Intel
 Device type:        CPU (LE)
 Device version:     OpenCL 1.1
 Driver version:     1.1
 Global Memory:      8192 MB
 Global Memory Cache:    64 bytes
 Local Memory:       32 KB (Global)
 Max clock (MHz) :   2660
 Max Work Group Size:    1024
 Parallel compute cores: 4

 Device #1 name:     GeForce GT 330M
 Device vendor:      NVIDIA
 Device type:        GPU (LE)
 Device version:     OpenCL 1.0
 Driver version:     CLH 1.0
 Global Memory:      512 MB
 Global Memory Cache:    0 bytes
 Local Memory:       16 KB (Local)
 Max clock (MHz) :   1100
 Max Work Group Size:    512
 Parallel compute cores: 6
 Stream processors:  48  (6 x 8)

Reply to this email directly or view it on GitHub:
https://github.com/magnumripper/magnum-jumbo/issues/50

magnumripper · 2012-07-03T22:54:38Z

Please try cloning the new branch "1.7.9-jumbo-6-fixes" and see if it solves your problems, and report back!

Thanks,
magnum

mubix · 2012-07-03T23:22:07Z

I didn't make any mods to the Makefile (not sure if you wanted me to), so this is straight from the pull (still had the path issues but it built):

/opt/magnum-jumbo/run(1.7.9-jumbo-6-fixes)$ ./john rawmd5test.txt --format=raw-md5-opencl
OpenCL platform 0: Apple, 2 device(s).
Using device 0: Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
Compilation log: <program source>:40:16: warning: comparison of integers of different signs: 'int' and 'unsigned int'
        for (i = 0; i != (KEY_LENGTH / 4) && keys[base + i]; i++)
                    ~ ^  ~~~~~~~~~~~~~~~~
<program source>:40:16: warning: comparison of integers of different signs: 'int' and 'unsigned int'
        for (i = 0; i != (KEY_LENGTH / 4) && keys[base + i]; i++)
                    ~ ^  ~~~~~~~~~~~~~~~~

Max local work size 1 Optimal local work size 1
(to avoid this test on next run do export LWS=1)
Local work size (LWS) 1, Global work size (GWS) 2097152
Loaded 1 password hash (Raw MD5 [OpenCL])
guesses: 0  time: 0:00:00:09 98.24% (2) (ETA: Tue Jul  3 19:12:13 2012)  c/s: 85.68  trying: 123456 - Sssing
Session aborted
/opt/magnum-jumbo/run(1.7.9-jumbo-6-fixes)$ 
/opt/magnum-jumbo/run(1.7.9-jumbo-6-fixes)$ ./john rawmd5test.txt --format=raw-md5-opencl --device=1
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: Error getting function data from server
OpenCL error (CL_INVALID_PROGRAM_EXECUTABLE) in file (opencl_rawmd5_fmt.c) at line (219) - (Error creating kernel. Double-check kernel name?)
/opt/magnum-jumbo/run(1.7.9-jumbo-6-fixes)$

magnumripper · 2012-07-03T23:31:22Z

Thank you. You are [currently] supposed to edit the Makefile so it reflects the OpenCL paths your system is using.
I'm not sure about the other problems: Please report this to the john-dev mailing list instead, you will probably get follow-up questions from developers that are not using github.

magnum

magnumripper · 2012-07-04T00:49:01Z

I suspect this format needs OpenCL 1.1 conformance (or at least support for byte_addressable_store) as of now, and it seems your GPU only supports OpenCL 1.0. If this is it, you will unfortunately have to wait longer for a fix.

magnumripper · 2012-07-04T18:28:30Z

Claudio posted a fix. Please try latest commit from "1.7.9-jumbo-6-fixes", hopefully it will work better

mubix · 2012-07-05T03:58:54Z

Definitely fixed for my CPU using OpenCL 1.1, I think I'm SOL for my GPU though (as you say wait longer for a fix)

magnumripper · 2012-07-05T13:43:48Z

After Claudio's fixes I believe we can fix it sooner rather than later. Can you please do this:

pull latest "1.7.9-jumbo-6-fixes"
fix the OpenCL paths in Makefile
un-comment line 25 in common-opencl.h so we define REPORT_OPENCL_WARNINGS
(make clean ; make macosx-x86-64-opencl) >/dev/null
Test it and report output (including any errors or warnings from the build in Magnum jumbo #4)

mubix · 2012-07-05T16:16:58Z

Now I'm getting errors for both CPU and GPU :(
CPU:

./john ../../rawmd5test.txt --format=raw-md5-opencl --device=0
OpenCL platform 0: Apple, 2 device(s).
Using device 0: Intel(R) Core(TM) i7 CPU       M 620  @ 2.67GHz
Max local work size 1
Optimal local work size = 8
Local work size (LWS) 8, Global work size (GWS) 2097152
Loaded 1 password hash (Raw MD5 [OpenCL])
OpenCL error (CL_INVALID_WORK_GROUP_SIZE) in file (opencl_rawmd5_fmt.c) at line (292) - (failed in clEnqueueNDRangeKernel)

GPU:

./john ../../rawmd5test.txt --format=raw-md5-opencl --device=1
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
 Compilation log: Error getting function data from server
OpenCL error (CL_INVALID_PROGRAM_EXECUTABLE) in file (opencl_rawmd5_fmt.c) at line (168) - (Error creating kernel. Double-check kernel name?)

I'm on the John-dev list now and will post this same reply there.

kost · 2012-08-12T14:56:40Z

Mubix - I have hit the same issue. OpenCL error (CL_INVALID_PROGRAM_EXECUTABLE) you're getting is probably because the build options are unsupported for your OpenCL setup. Because of the error in OpenCL error handling, you're getting wrong OpenCL error. I've managed to track my problem to the build options (both specified are unsupported on my platform).

Take a look at the pull requests I've made (76 - fix error handling, 77 - fix build options):
https://github.com/magnumripper/magnum-jumbo/pull/76
https://github.com/magnumripper/magnum-jumbo/pull/77

In any case, I would recommend trying out my fork (with these two fixed implemented):
https://github.com/kost/magnum-jumbo

If it will not fix your problems, at least you will have more info about the error you're getting. So, pinpointing the problem would be easier.

mubix · 2012-08-12T15:14:22Z

Here's the output:

./john --test=3 --format=nt-opencl --device=1           
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: 
Error building kernel. Returned build code: -11. DEVICE_INFO=130
OpenCL error (CL_BUILD_PROGRAM_FAILURE) in file (common-opencl.c) at line (137) - (clBuildProgram failed.)

kost · 2012-08-12T17:29:57Z

As I thought. It was not problem in executing nt_kernel (CL_INVALID_PROGRAM_EXECUTABLE), but in the process of building nt_kernel.cl (CL_BUILD_PROGRAM_FAILURE).

On the other hand, other part of the problem is not the same as mine. From what I see, your problem is in clBuildProgram call (build_kernel function in common-opencl.c). As I don't have same card as you it's hard for me to play with it.

But, if you have some spare time. I would suggest following:

trying some other opencl based formats (raw-md5-opencl, raw-sha1-opencl, ...) and reporting if you have same errors
removing comments from common-opencl.c in function include_source:
//fprintf(stderr, "Options used: %s\n", include);
report that line here.
upgrading OpenCL implementation (CL_BUILD_PROGRAM_FAILURE with empty Compilation log usually means bug in CL compiler), i.e. try with MacOSX10.7.sdk if you have some older version currently
intelligently commenting out parts of nt_kernel.cl and finding out offending line/statement

Hope it helps!

mubix · 2012-08-12T21:55:49Z

trying some other opencl based formats (raw-md5-opencl, raw-sha1-opencl, ...) and reporting if you have same errors

$ ./john --test=3 --device=1 --format=raw-md5-opencl

OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: 
Error building kernel. Returned build code: -11. DEVICE_INFO=130
OpenCL error (CL_BUILD_PROGRAM_FAILURE) in file (common-opencl.c) at line (137) - (clBuildProgram failed.)

$ ./john --test=3 --device=1 --format=raw-sha1-opencl
OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Compilation log: <program source>:49:18: warning: comparison of integers of different signs: 'int' and '__attribute__((address_space(1))) uint' (aka '__attribute__((address_space(1))) unsigned int')
    for(i = 0; i < data_info[0] && ((uchar) plain_key[msg_pad + i]) != 0x0 ; i++){
               ~ ^ ~~~~~~~~~~~~

Error building kernel. Returned build code: -11. DEVICE_INFO=130
OpenCL error (CL_BUILD_PROGRAM_FAILURE) in file (common-opencl.c) at line (137) - (clBuildProgram failed.)

removing comments from common-opencl.c in function include_source:
//fprintf(stderr, "Options used: %s\n", include);
report that line here.

$ ./john --test=3 --device=1 --format=raw-sha1-opencl

OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Options used: -I ./  -DDEVICE_INFO=130 -cl-nv-verbose

upgrading OpenCL implementation (CL_BUILD_PROGRAM_FAILURE with empty Compilation log usually means bug in CL compiler), i.e. try with MacOSX10.7.sdk if you have some older version currently

currently using: http://developer.download.nvidia.com/compute/cuda/4_2/rel/sdk/gpucomputingsdk_4.2.9_macos.pkg

intelligently commenting out parts of nt_kernel.cl and finding out offending line/statement

You are going to have to help with the 'intelligently' part, I'm not unfamiliar with coding but I need a guided hand

kost · 2012-08-14T08:53:45Z

Now, just please another output:

CL_LOG_ERRORS=stderr ./john --test=3 --format=nt-opencl --device=1

mubix · 2012-08-15T03:12:19Z

$  CL_LOG_ERRORS=stderr ./john --test=3 --format=nt-opencl --device=1

OpenCL platform 0: Apple, 2 device(s).
Using device 1: GeForce GT 330M
Options used: -I ./  -DDEVICE_INFO=130 -cl-nv-verbose 
OpenCL Warning : clBuildProgram failed: could not build program for 0x1022600 (GeForce GT 330M) (err:-44)
Break on OpenCLWarningBreak to debug.
Compilation log: 
Error building kernel. Returned build code: -11. DEVICE_INFO=130
OpenCL error (CL_BUILD_PROGRAM_FAILURE) in file (common-opencl.c) at line (137) - (clBuildProgram failed.)
[CL_INVALID_MEM_OBJECT] : OpenCL Error : clEnqueueUnmapMemObject failed: invalid cl_mem object 0x0.
Break on OpenCLErrorBreak to debug.
[CL_INVALID_MEM_OBJECT] : OpenCL Error : clEnqueueUnmapMemObject failed: invalid cl_mem object 0x0.
Break on OpenCLErrorBreak to debug.
[CL_INVALID_KERNEL] : OpenCL Error : Failed to release kernel! Kernel given is invalid!
Break on OpenCLErrorBreak to debug.

magnumripper · 2012-08-17T18:19:51Z

Can you build any other GPU stuff? Like some kind of OpenCL hello-world?

mubix · 2012-08-17T18:47:32Z

This: http://developer.apple.com/library/mac/#samplecode/OpenCL_Hello_World_Example/Introduction/Intro.html

compiles and worked perfectly. would you like me to try another?

magnumripper · 2012-08-17T19:05:57Z

No, that's fine. Just so we don't look in the wrong place for problems. I will likely get myself a Macbook Pro within weeks - with some luck this will help solving a number of OSX problems. Though normally I will likely run Linux on it.

kost · 2012-08-21T12:26:18Z

From my limited experience with OpenCL, we're down to the issue:

you've hit the bug in OpenCL installation/compiler (it's not that unusual!)

From my experience, there's lot of problems with double/float handling, but in given .cl files there's no such things.

Maybe you can try commenting out various code in offending .cl file in order to find offending line or function.
or try to put simple .cl file(one that works for you from example) instead of original one. Just to narrow down the problem.

Sorry, I'm really out of ideas :-/

magnumripper · 2012-08-27T22:21:26Z

Wow. Just wow. I just tried all OpenCL formats under OSX 10.8.1. Not a single one works on GPU. Every single one fails just like the one discussed here. Apple's driver/framework apparently does not conform to the same ideas that the rest of the world. Using the CPU device, most or all seems fine.

The good news is I will fix this sooner or later, now that I can see for myself!

mubix · 2012-08-27T22:29:36Z

Welcome to the MacFanboy-nation

kost · 2012-08-29T15:33:37Z

Probably, it depends on graphic card. As I have been successful in running john opencl stuff on MacBook Pro with AMD Radeon HD 6750M. I guess, it is nvidia in question for both of you?

magnumripper · 2012-08-29T15:46:52Z

Yes, nvidia. Like mubix, I can build and run a hello world opencl program just fine. I strongly suspect the problem is some peculiarity with Apple+nvidia that we need to take account for / work around in common-opencl.c. The question is what...

kost · 2012-08-29T16:03:26Z

Huh. Then, I can't help too much, as I don't have access to any nvidia based. BTW could be something stupid like this:

https://github.com/kost/magnum-jumbo/commit/cd9856c2ae412e871a2aacc130b37e6ffb138a62

magnumripper · 2012-08-30T00:27:14Z

Problem partly solved. A bunch of fixes is committed. Also, on my laptop I need to run gfxCardStatus and force discrete GPU (that would be a bug in OSX). All CUDA formats and maybe half of the OpenCL ones work fine now. I'll try to fix the rest.

mubix · 2012-08-30T04:56:40Z

@magnumripper where can I get said magic?

kost · 2012-08-30T07:05:02Z

I've tested it (bleeding jumbo) and it works with ATI. So, nothing broken on that side with those bunch of fixes. On other hand, what makes me worry is that opencl is usually slower (on Mac and Linux):

$ run/john --format=nt-opencl --test
OpenCL platform 0: Apple, 2 device(s).
Using device 0: Intel(R) Core(TM) i7-2675QM CPU @ 2.20GHz
Benchmarking: NT MD4 [OpenCL]... DONE
Raw: 23068K c/s real, 4908K c/s virtual

$ run/john --format=nt --test
Benchmarking: NT MD4 [128/128 X2 SSE2-16]... DONE
Raw: 37944K c/s real, 37944K c/s virtual

$ run/john --format=nt-opencl --device=1 --test
OpenCL platform 0: Apple, 2 device(s).
Using device 1: ATI Radeon HD 6750M
Benchmarking: NT MD4 [OpenCL]... ^[DONE
Raw: 22616K c/s real, 44362K c/s virtual

I get similar discrepancy for raw-md5 as well :-/

magnumripper · 2012-08-30T09:05:25Z

@mubix if you mean gfxCardStatus, it's here: http://codykrieger.com/gfxCardStatus.

@kost this applies to the fast raw formats, where the JtR "api" has bottlenecks. Work in progress. Slow formats like phpass or rar are much faster on GPU. We have the same problem with OMP - the raw formats does not even support it because it does not make them faster.

magnumripper · 2012-09-10T01:21:15Z

I have a very odd clue: Since NT and raw-MD4 are very similar, I tried swapping their kernels to see if the problem moved. It did not. So apparently the problem is not in the kernel code. But I can't see what it would be then. Not much happen in the formats before the kernel is built.

magnumripper · 2012-09-10T01:26:58Z

Scrap that. I'm too tired. The problem did move. So the problems are in the kernels.

mubix · 2012-09-12T14:03:08Z

Is there anything I can do to help or test?

magnumripper · 2012-09-12T16:59:38Z

Yeah, complain to Apple :) Seriously, I tried commenting out a super-trivial no-magics part of the NT kernel, and then it compiles fine (although obviously fails to crack hashes). To me this proves there isn't anything we can do about it other than wait for Apple to fix their über-crappy OpenCL :(

In the meantime, download CUDA for OSX and run a CUDA build of JtR. All CUDA formats works fine (as long as you do not rely on dynamic GPU switching).

magnumripper · 2012-10-09T17:11:08Z

CUDA 5 fixes the problem with kernel hangs in dynamic mode.

I'm closing this now. All is not fine but many things are better now. Please open new issues for specific problems (other than that "Error getting function data from server" which is clearly due to Apple bugs).

magnumripper closed this as completed Oct 9, 2012

shivanraptor mentioned this issue Jul 18, 2018

OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error when using format=7z-opencl #3308

Closed

Successful compile doesn't work macosx-x86-64-opencl #50

Successful compile doesn't work macosx-x86-64-opencl #50

Comments

mubix commented Jul 3, 2012

magnumripper commented Jul 3, 2012

mubix commented Jul 3, 2012

magnumripper commented Jul 3, 2012

magnumripper commented Jul 3, 2012

mubix commented Jul 3, 2012

magnumripper commented Jul 3, 2012

magnumripper commented Jul 4, 2012

magnumripper commented Jul 4, 2012

mubix commented Jul 5, 2012

magnumripper commented Jul 5, 2012

mubix commented Jul 5, 2012

kost commented Aug 12, 2012

mubix commented Aug 12, 2012

kost commented Aug 12, 2012

mubix commented Aug 12, 2012

kost commented Aug 14, 2012

mubix commented Aug 15, 2012

magnumripper commented Aug 17, 2012

mubix commented Aug 17, 2012

magnumripper commented Aug 17, 2012

kost commented Aug 21, 2012

magnumripper commented Aug 27, 2012

mubix commented Aug 27, 2012

kost commented Aug 29, 2012

magnumripper commented Aug 29, 2012

kost commented Aug 29, 2012

magnumripper commented Aug 30, 2012

mubix commented Aug 30, 2012

kost commented Aug 30, 2012

magnumripper commented Aug 30, 2012

magnumripper commented Sep 10, 2012

magnumripper commented Sep 10, 2012

mubix commented Sep 12, 2012

magnumripper commented Sep 12, 2012

magnumripper commented Oct 9, 2012