OpenCL CL_INVALID_DEVICE (-33) error in opencl_common.c:452 - Error querying PLATFORM_NAME

[madtomic]

Environment Setup

OS: Ubuntu 16.04.5
JohnTheRipper
https://github.com/magnumripper/JohnTheRipper/blob/bleeding-jumbo/doc/INSTALL-UBUNTU
***configure and make using this install guide.
What was used:
sudo apt-get install build-essential libssl-dev git zlib1g-dev
sudo apt-get install yasm libgmp-dev libpcap-dev pkg-config libbz2-dev
sudo apt-get install nvidia-opencl-dev
sudo apt-get install libopenmpi-dev openmpi-bin
./configure --enable-mpi
make -s clean && make -sj4

Sample test file:

./tezos2john.py 'put guide flat machine express cave hello connect stay local spike ski romance express brass' 'jbzbdybr.vpbdbxnn@tezos.example.org' 'tz1eTjPtwYjdcBMStwVdEcwY2YE3th1bXyMR' > tezos

Running John

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --devices=gpu --fork=2 --format=tezos-opencl --session=tezos tezos
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 4 OpenMP threads per process (8 total across 2 processes)
Node numbers 1-2 of 2 (fork)
OpenCL CL_INVALID_DEVICE (-33) error in opencl_common.c:452 - Error querying PLATFORM_NAME
Device 0@tezos-Desktop: GeForce GTX 1070 Ti
Press 'q' or Ctrl-C to abort, almost any other key for status
1 0g 0:00:00:04 3/3 0g/s 38678p/s 38678c/s 38678C/s GPU:61°C clmom..mia196
1 0g 0:00:00:09 3/3 0g/s 51592p/s 51592c/s 51592C/s GPU:61°C 196642..biliyou
1 0g 0:00:00:11 3/3 0g/s 54131p/s 54131c/s 54131C/s GPU:60°C 115la..0998am
Waiting for 1 child to terminate
Session aborted

BUG/Issues: It can't run across multiple GPUs

System Information:

tezos@tezos-Desktop:~/JohnTheRipper/run$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 21
Model: 2
Model name: AMD FX(tm)-8350 Eight-Core Processor
Stepping: 0
CPU MHz: 1411.029
CPU max MHz: 4000.0000
CPU min MHz: 1400.0000
BogoMIPS: 8037.07
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall bmi1 arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --list=build-info
Version: 1.8.0.13-jumbo-1-bleeding-9c715c6 2018-11-04 18:00:16 +0530
Build: linux-gnu 64-bit x86_64 XOP AC MPI + OMP
SIMD: XOP, interleaving: MD4:2 MD5:2 SHA1:1 SHA256:2 SHA512:1
CPU tests: XOP
$JOHN is ./
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 5.4.0
GNU libc version: 2.23 (loaded: 2.23)
OpenCL headers version: 2.0
Crypto library: OpenSSL
OpenSSL library version: 01000207f
OpenSSL 1.0.2g 1 Mar 2016
GMP library version: 6.1.0
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --list=opencl-devices
Platform #0 name: NVIDIA CUDA, version: OpenCL 1.2 CUDA 10.0.185

Device #0 (0) name:     GeForce GTX 1070 Ti
Device vendor:          NVIDIA Corporation
Device type:            GPU (LE)
Device version:         OpenCL 1.2 CUDA
Driver version:         410.73 [recommended]
Native vector widths:   char 1, short 1, int 1, long 1
Preferred vector width: char 1, short 1, int 1, long 1
Global Memory:          7.9 GB
Global Memory Cache:    304 KB
Local Memory:           48 KB (Local)
Constant Buffer size:   64 KB
Max memory alloc. size: 2 GB
Max clock (MHz):        1683
Profiling timer res.:   1000 ns
Max Work Group Size:    1024
Parallel compute cores: 19
CUDA cores:             2432  (19 x 128)
Speed index:            4093056
Warp size:              32
Max. GPRs/work-group:   65536
Compute capability:     6.1 (sm_61)
Kernel exec. timeout:   yes
NVML id:                0
PCI device topology:    01:00.0
PCI lanes:              16/16
Fan speed:              22%
Temperature:            59°C
Utilization:            4%

Device #1 (1) name:     GeForce GTX 1070 Ti
Device vendor:          NVIDIA Corporation
Device type:            GPU (LE)
Device version:         OpenCL 1.2 CUDA
Driver version:         410.73 [recommended]
Native vector widths:   char 1, short 1, int 1, long 1
Preferred vector width: char 1, short 1, int 1, long 1
Global Memory:          8 GB
Global Memory Cache:    304 KB
Local Memory:           48 KB (Local)
Constant Buffer size:   64 KB
Max memory alloc. size: 2 GB
Max clock (MHz):        1683
Profiling timer res.:   1000 ns
Max Work Group Size:    1024
Parallel compute cores: 19
CUDA cores:             2432  (19 x 128)
Speed index:            4093056
Warp size:              32
Max. GPRs/work-group:   65536
Compute capability:     6.1 (sm_61)
Kernel exec. timeout:   no
NVML id:                1
PCI device topology:    05:00.0
PCI lanes:              4/16
Fan speed:              0%
Temperature:            34°C
Utilization:            9%

[magnumripper]

https://www.openwall.com/lists/john-users/2018/11/06/2

Does it always fail like this? I have seen some strange race condition problem with later nvidia drivers and I've been testing a work-around hack for some time. Perhaps it's time to merge that to the repo.

You could always try running two sessions (in two different terminal sessions) without fork, like this:

OMP_NUM_THREADS=4 ./john -dev=0 -node=1/2 -form=tezos-opencl -ses=tezos1 tezos -inc

OMP_NUM_THREADS=4 ./john -dev=1 -node=2/2 -form=tezos-opencl -ses=tezos2 tezos -inc

Please test that and report whether it works or not.

[magnumripper]

FWIW, it works fine on my Macbook as well as on Openwall's test rig (which has an older driver, 361.28):

$ ../run/john --devices=6,7 --fork=2 --format=tezos-opencl --session=tezos tezos.in -w -ru
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 16 OpenMP threads per process (32 total across 2 processes)
Node numbers 1-2 of 2 (fork)
Device 7: GeForce GTX TITAN
Device 6: GeForce GTX TITAN X
Each node loaded 1/2 of wordfile to memory (about 12 KB/node)
Press 'q' or Ctrl-C to abort, almost any other key for status
1 0g 0:00:00:01 DONE (2018-11-07 00:40) 0g/s 43217p/s 43217c/s 43217C/s GPU:44C 12345..Sssing
Waiting for 1 child to terminate
2 0g 0:00:00:03 DONE (2018-11-07 00:40) 0g/s 20395p/s 20395c/s 20395C/s GPU:45C 7network..Notuseding
Session completed

I remember this (-33) error. When that happened, it was not related to fork. An interesting test is to run JtR in all devices (up to an "invalid device" error). For example (basically the same magnum did above without node):

./john --devices=0 --format=tezos-opencl --session=tezos0 --max-run=30 tezos
./john --devices=1 --format=tezos-opencl --session=tezos1 --max-run=30 tezos
./john --devices=2 --format=tezos-opencl --session=tezos2 --max-run=30 tezos
./john --devices=3 --format=tezos-opencl --session=tezos3 --max-run=30 tezos

To confirm it is related to fork (or not).

[madtomic]

@magnumripper

It always fails when I tried like this. I am using a slight older driver 384.130 now and I am still getting the same error: OpenCL CL_INVALID_DEVICE (-33) error in opencl_common.c:452 - Error querying PLATFORM_NAME

Side notes: when using Ubuntu 14.04.5 with 2 AMD RX 480. It seems to be working as expected.

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --devices=0,1 --fork=2 --format=tezos-opencl --session=tezos tezos -w -ru
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 4 OpenMP threads per process (8 total across 2 processes)
Node numbers 1-2 of 2 (fork)
OpenCL CL_INVALID_DEVICE (-33) error in opencl_common.c:452 - Error querying PLATFORM_NAME
Device 0@tezos-Desktop: GeForce GTX 1070 Ti
Each node loaded 1/2 of wordfile to memory (about 12 KB/node)
Press 'q' or Ctrl-C to abort, almost any other key for status
1 0g 0:00:00:01 DONE (2018-11-06 20:36) 0g/s 45227p/s 45227c/s 45227C/s GPU:58°C 12345..Sssing
Waiting for 1 child to terminate
Main process session completed, but some child processes failed

Running in two different terminal sessions without fork seems to work but this raise another question when doing it this way. Question: Would this be doing the same work twice if doing it this way?

Eventually, the end game setup is running this across 6-12 GeForce GTX 1070 Ti GPUs with AMD Threadripper 2950x as the CPU. I am only doing some preliminary system tests before I commit to this project.

tezos@tezos-Desktop:~/JohnTheRipper/run$ OMP_NUM_THREADS=4 ./john -dev=0 -node=1/2 -form=tezos-opencl -ses=tezos1 tezos -inc
Device 0@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 4 OpenMP threads
Node number 1 of 2
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:55 0g/s 57326p/s 57326c/s 57326C/s GPU:65°C sulsio..amr3mk
0g 0:00:01:26 0g/s 57634p/s 57634c/s 57634C/s GPU:66°C phyrero..178jkl
0g 0:00:01:54 0g/s 57970p/s 57970c/s 57970C/s GPU:66°C sanno181..pandysum
0g 0:00:03:15 0g/s 58256p/s 58256c/s 58256C/s GPU:67°C legl7w..dm120*
Session aborted

tezos@tezos-Desktop:~/JohnTheRipper/run$ OMP_NUM_THREADS=4 ./john -dev=1 -node=2/2 -form=tezos-opencl -ses=tezos2 tezos -inc
Device 1@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 4 OpenMP threads
Node number 2 of 2
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:40 0g/s 56295p/s 56295c/s 56295C/s GPU:53°C 019m5n..chybam
0g 0:00:01:12 0g/s 57255p/s 57255c/s 57255C/s GPU:57°C abi00..tt29c
0g 0:00:01:42 0g/s 57963p/s 57963c/s 57963C/s GPU:60°C muradogi..cliser22
0g 0:00:02:57 0g/s 58405p/s 58405c/s 58405C/s GPU:64°C sucefass..prylos10
Session aborted

[madtomic]

@claudioandre-br

Interesting test, as it uses 8 OpenMP threads instead of 4 OpenMP

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --devices=0 --format=tezos-opencl --session=tezos0 --max-run=30 tezos
Device 0@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:30 3/3 0g/s 51864p/s 51864c/s 51864C/s GPU:65°C hyelu..ccmcsc
0g 0:00:00:30 3/3 0g/s 53041p/s 53041c/s 53041C/s GPU:62°C ccmcso..02428a
Session stopped (max run-time reached)

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --devices=1 --format=tezos-opencl --session=tezos1 --max-run=30 tezos
Device 1@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:26 3/3 0g/s 50845p/s 50845c/s 50845C/s GPU:54°C 067644..rjbbs
0g 0:00:00:30 3/3 0g/s 54363p/s 54363c/s 54363C/s GPU:52°C ccmcso..02428a
Session stopped (max run-time reached)

[madtomic]

@magnumripper

I decided to let it run awhile and it would randomly abort the session if it is in separate terminal session. So this is would not a viable solution.

Edit: I think, I might have aborted it myself. Doing more testing to confirm this.

tezos@tezos-Desktop:~/JohnTheRipper/run$ OMP_NUM_THREADS=4 ./john -dev=0 -node=1/2 -form=tezos-opencl -ses=tezos1 tezos -inc
Device 0@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 4 OpenMP threads
Node number 1 of 2
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:34 0g/s 54886p/s 54886c/s 54886C/s GPU:61°C ppawei..mampse10
0g 0:00:01:21 0g/s 56658p/s 56658c/s 56658C/s GPU:66°C sustsot1..chuchk06
0g 0:00:02:38 0g/s 57349p/s 57349c/s 57349C/s GPU:67°C lyoni03..rogdis2
Session aborted

I said something wrong above. The -33 is related to --fork.

• OpenCL CL_INVALID_DEVICE error in common-opencl.c:435 - Error querying PLATFORM_NAME #2547 (comment)

As I wrote there, the error happens in some machines: yours is the 2nd. But, for example, JtR works in your Ubuntu 14 and our super machine. I still can't see why.

BTW:

Question: Would this be doing the same work twice if doing it this way?

No.

[madtomic]

@claudioandre-br

It seems like I am having the very same issue a year ago. I will try Ubuntu 14 with the same Nvidia cards to see it it would works like AMD cards.

Another unrelated issue is why it took a hit on performance? Run the card by itself it can hash over 70k and when fork between sessions, it would only top out around 60k.

Is the number of threads or memory affect how it would performance?
I did noticed an increase in performance when using a dual core cpu vs quad cpu.

Run only a session

tezos@tezos-Desktop:~/JohnTheRipper/run$ ./john --devices=0 --format=tezos-opencl --session=tezos1 tezos
Device 0@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 8 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:11 3/3 0g/s 56323p/s 56323c/s 56323C/s GPU:66°C sigl3r..meleron
0g 0:00:00:26 3/3 0g/s 65551p/s 65551c/s 65551C/s GPU:67°C 02429j..brannyah
0g 0:00:00:34 3/3 0g/s 66283p/s 66283c/s 66283C/s GPU:70°C dents2..djllle
0g 0:00:00:44 3/3 0g/s 68963p/s 68963c/s 68963C/s GPU:67°C mr2290..21lolo
0g 0:00:00:58 3/3 0g/s 69748p/s 69748c/s 69748C/s GPU:70°C mym1lp..jchsh7
0g 0:00:01:04 3/3 0g/s 70537p/s 70537c/s 70537C/s GPU:69°C 251se1..10298j
0g 0:00:01:10 3/3 0g/s 71141p/s 71141c/s 71141C/s GPU:68°C juichy1..muady1
0g 0:00:01:17 3/3 0g/s 71461p/s 71461c/s 71461C/s GPU:67°C pepppan..pr42485
0g 0:00:01:22 3/3 0g/s 71153p/s 71153c/s 71153C/s GPU:70°C 2h0m02..pcmana
0g 0:00:01:30 3/3 0g/s 71721p/s 71721c/s 71721C/s GPU:70°C mmdot2..byah30
0g 0:00:01:37 3/3 0g/s 72161p/s 72161c/s 72161C/s GPU:69°C tielix..calmuco
0g 0:00:01:52 3/3 0g/s 72802p/s 72802c/s 72802C/s GPU:67°C 0okorm..jlcc3l
0g 0:00:01:54 3/3 0g/s 72852p/s 72852c/s 72852C/s GPU:67°C lc102t..tmmsh8
Session aborted

One of two sessions

tezos@tezos-Desktop:~/JohnTheRipper/run$ OMP_NUM_THREADS=4 ./john -dev=1 -node=2/2 -form=tezos-opencl -ses=tezos2 tezos -inc
Device 1@tezos-Desktop: GeForce GTX 1070 Ti
Using default input encoding: UTF-8
Loaded 1 password hash (tezos-opencl, Tezos Key [PBKDF2-SHA512 OpenCL])
Cost 1 (iteration count) is 2048 for all loaded hashes
Will run 4 OpenMP threads
Node number 2 of 2
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:32 0g/s 55796p/s 55796c/s 55796C/s GPU:58°C lob16..adelyna
0g 0:00:01:19 0g/s 57669p/s 57669c/s 57669C/s GPU:61°C lmmmy16..dmum7
0g 0:00:03:04 0g/s 58759p/s 58759c/s 58759C/s GPU:65°C 0p-a-j..b11j70
0g 0:00:05:34 0g/s 60485p/s 60485c/s 60485C/s GPU:63°C nenziz..nebskk
0g 0:00:07:52 0g/s 59849p/s 59849c/s 59849C/s GPU:65°C abugh8..babol7
0g 0:00:09:02 0g/s 59716p/s 59716c/s 59716C/s GPU:63°C brijrh..bfg5fe
0g 0:00:12:02 0g/s 59481p/s 59481c/s 59481C/s GPU:63°C mr.karo..shr7831
0g 0:00:17:05 0g/s 59264p/s 59264c/s 59264C/s GPU:63°C tr39vm..ts63lb
0g 0:00:30:48 0g/s 59030p/s 59030c/s 59030C/s GPU:63°C ckfyl..ta7qm
0g 0:00:30:53 0g/s 59007p/s 59007c/s 59007C/s GPU:65°C ip5bv..vo5k7
0g 0:00:31:08 0g/s 59022p/s 59022c/s 59022C/s GPU:63°C myf43208..muleid29
Session aborted

[kholia]

Eventually, the end game setup is running this across 6-12 GeForce GTX 1070 Ti GPUs with AMD Threadripper 2950x as the CPU.

I did noticed an increase in performance when using a dual core cpu vs quad cpu.

This format is not a pure GPU format. It uses the CPUs heavily for doing asymmetric crypto stuff.

I am guessing that a single Threadripper 2950x CPU won't be enough when working with 6-12 fast GPUs.

Perhaps, the workload can be divided among multiple computers using something like OpenMP? CC @magnumripper.

[madtomic]

@kholia

This format is not a pure GPU format. It uses the CPUs heavily for doing asymmetric crypto stuff.
I am guessing that a single Threadripper 2950x CPU won't be enough when working with 6-12 fast GPUs.

Threadripper 2950x CPU has over 32 threads. If I pair up with just 6 GPUs. It would allocated 5-6 threads per GPU.

I think 2970X would do a better job as it has 48 threads or 8 OpenMP threads per card when pair up with 6 cards?

It seems like I am having the very same issue a year ago. I will try Ubuntu 14 with the same Nvidia cards to see it it would works like AMD cards.

If you can, please, do some trials in order to narrow the problem. In fact, since it works on super, I would start trying CUDA 7 (using whatever Ubuntu version you are happy with).

I decided to let it run awhile and it would randomly abort the session if it is in separate terminal session. So this is would not a viable solution.

This is scaring. It means you can't use 2 GPUs at the same time. Even in different and unrealated processes!?! Odd.

And please, run other tests. E.g.:

• 1 GPU, two processes

./john -form:cpu --list=format-tests 2> /dev/null | cut -f3 1> alltests.in
./john --fork=2 --devices=0 --format=raw-sha256-opencl alltests.in

• And, using separate terminals JtR randomly aborts the session

./john --devices=0 --format=raw-sha256-opencl alltests.in -session=1 # terminal 1
./john --devices=1 --format=raw-sha512-opencl alltests.in -session=2 # terminal 2

---

I'm running these tests now on super (on -dev 6 and 7).
On super everything seems ok. And it has CUDA 8 now (but it is hybrid AMD/NVIDIA env).

Platform #2 name: NVIDIA CUDA, version: OpenCL 1.2 CUDA 8.0.13
    Device version:         OpenCL 1.2 CUDA
    Driver version:         361.28 [recommended]

[magnumripper]

I will try to find my old patch for working around the race condition I saw so we can test if it helps here. Basically I was taking an exclusive lock on some temporary file so only one process could perform OpenCL init or run-time kernel build at a time.