Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hashcat v4.1 mode 11300 doesn't work with ROCm 1.8.1 #437

Closed
hashuser1 opened this issue Jun 18, 2018 · 34 comments
Closed

hashcat v4.1 mode 11300 doesn't work with ROCm 1.8.1 #437

hashuser1 opened this issue Jun 18, 2018 · 34 comments
Labels
Compiler Functional Bug

Comments

@hashuser1
Copy link

So I thought this issue was with hashcat but the admins on that project are stating the issue is with ROCm. Im going to post the issue I opened with hashcat and their response. I cant write code so please be gentle. How can I fix the JiT compiler issue?

Ive been doing some testing and hashcat v3.5 and 4.1 do not work correctly when using mode 11300 (bicoin wallet) with ROCm 1.8.1.

Im using xubuntu 16.04.4, 4.13.0-45-generic.

I created a test wallet.dat with a known password and used btcrecover scripts to extract the hash. With ROCm 1.8.1 installed, hashcat does not find the password with the password in a wordlist. I thought it could be the drivers so I did a clean install and tried the AMDGPU-PRO Beta Mining Driver 17.40 with optional rocm. With the AMDGPU driver installed v.3.5 does find the password in the wordlist, but v4.1 still does not.

The good thing is, hashcat informs you about the invalid kernel processing on startup:* Device #2: ATTENTION! OpenCL kernel self-test failed. However, it's not a hashcat problem. I've traced the intermediate keys up to the comparison kernel (where AES comes into play). Both the AES key and the IV is correctly computed on rocm.

The next step would be to initialize the AES key scheduler array and on here (using the same shared library code) rocm start computing invalid values. All other tested OpenCL runtimes (NVidia, Intel, pocl, etc) compute it correctly.

The key scheduler access a shared memory region of the GPU to make it faster. You can however disable this behavior and make it access constant memory region. If we do this you can see how rocm starts to crack without changing any of the code portion. You can reproduce locally by commenting out line 24 in inc_vendor.cl and remove the kernel/ cache folder entirely.

Note that the same shared AES function are used in many other kernels (truecrypt 6221 for example) and they work fine over there. This is not a problem in how shared memory is accessed.I was able to further track down the issue to this data copy from constant memory to shared memory in m11300_comp() function in m11300.cl:

s_td0[i] = td0[i];

So basically there's nothing we can do to make this more easy. If this doesn't work it's for sure a JiT compiler problem.
@b-sumner
Copy link
Collaborator

What is the hashcat issue number? Can you give a command line and .hash file contents along the lines of the existing example* in the hashcat repo that will definitely exercise mode 11300?

@hashuser1
Copy link
Author

hashuser1 commented Jun 19, 2018 •
edited
Loading

hashcat issue #1596.

The command I was using "./hashcat64.bin -m 11300 hash.txt. wordlist.txt". I would rather not give you the exact hash Im trying to crack. But you can download bitcoin-core and make a wallet using bitcoin-qt with your own known password and add it to a wordlist (most linux systems have a wordlist stored at /usr/share/dict/words or /usr/dict/words) . I used the attach script to extract the hash.
wallet_hash_extractor.txt

@b-sumner
Copy link
Collaborator

Would it be possible for you to post the hash to "xyzzy" here? It is already in example.dict

@hashuser1
Copy link
Author

hashuser1 commented Jun 19, 2018 via email

@b-sumner
Copy link
Collaborator

Thank you very much. How about "zzzzzzzzzz" (10 z's), also located in example.dict.

@hashuser1
Copy link
Author

As requested.
hash.txt

@b-sumner
Copy link
Collaborator

Thanks. I see that a debug build of the compiler is asserting. We'll follow up on this.

@hashuser1
Copy link
Author

hashuser1 commented Jun 19, 2018 via email

@gstoner
Copy link

gstoner commented Jun 19, 2018

no it is not known issue

@hashuser1
Copy link
Author

Any update on this? Additionally, is there any work around to get this to work? Like if I try CENTOS instead or try a newer/older kernel in Ubuntu. I don't understand if the problem is with Rocm and the way it can't perform math functions correctly or just a bug in version 1.8 and the version of Ubuntu I'm using? I guess I'm saying can you prove this ever worked with any version since 1.6?

@ghost
Copy link

ghost commented Jul 13, 2018

I just noticed the same behavior, can you try compiling hash cat with https://developer.amd.com/amd-aocc/ . Then try the amd sdk 2.6 .

@hashuser1
Copy link
Author

I will make an attempt. I would appreciate an answer though. Has this ever worked with any version of ROCm since 1.6? And please understand, Im not pointing any fingers (the problem still could be hashcat), I just want to know if Ive wasted 7 months of letting hashcat run under different settings just to find out it will never solve the problem.

@hashuser1
Copy link
Author

hashuser1 commented Jul 13, 2018 •
edited
Loading

Followed instructions listed here.
https://developer.amd.com/wordpress/media/2017/04/AOCC-1.2.1-Install-Guide.pdf

Hashcat is still giving the following error
"Device #1: ATTENTION! OpenCL kernel self-test failed."

Maybe I edited the make file incorrectly?
makefile_edited.txt
Makefile_original.txt
make_output.txt

@hashuser1
Copy link
Author

hashuser1 commented Jul 13, 2018 •
edited
Loading

Cant find the AMD SDK 2.6, its not listed at either of the following links.
https://developer.amd.com/tools-and-sdks/
http://developer.amd.com/sdks/AMDAPPSDK/downloads/Pages/default.aspx

@hashuser1
Copy link
Author

I noticed 1.8.2 was released. I tried using centOS 7.5 and hashcat is still giving the error "
"Device #1: ATTENTION! OpenCL kernel self-test failed." I tried compiling using devtoolset and another try using AOCC. Still broken.

Is this problem limited to Polaris? Or should I just give up and use NVIDIA hardware?

@ghost
Copy link

ghost commented Jul 27, 2018 via email

@hashuser1
Copy link
Author

hashuser1 commented Jul 27, 2018 via email

@hashuser1
Copy link
Author

Any update?

@b-sumner
Copy link
Collaborator

We've fixed the cause of the assertion that was being hit here and by other unrelated applications. Following that, hashcat 4.2 successfully finds the password for hash.txt. I'm not really clear on when those compiler fixes will appear in a release.

@hashuser1
Copy link
Author

hashuser1 commented Aug 21, 2018 via email

@hashuser1
Copy link
Author

hashuser1 commented Sep 11, 2018 via email

@hashuser1
Copy link
Author

hashuser1 commented Sep 11, 2018 via email

@b-sumner
Copy link
Collaborator

I'm running on gfx803, but I only have one installed. Running the hash you provided against example.dict:

hashcat (v4.2.1-51-gc8dbcf9) starting...

OpenCL Platform #1: Advanced Micro Devices, Inc.

Hashes: 1 digests; 1 unique digests, 1 unique salts
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes, 5/13 rotates
Rules: 1
...
Restore.Sub.#1...: Salt:0 Amplifier:0-1 Iteration:198432-198443
Candidates.#1....: 0 -> zzzzzzzzzzz
Hardware.Mon.#1..: Temp: 68c Fan: 27% Core: 874MHz Mem: 500MHz Bus:0

Started: Fri Sep 14 12:55:46 2018
Stopped: Fri Sep 14 13:00:23 2018

Can you post here what hashcat says for you? I see you have two gfx803. Does selftest fail on both devices or just one? If just one, does it run if you remove the other? Does it run with just the other?

@hashuser1
Copy link
Author

Are you using the AOCC compiled binary or the precompiled binary? The output below is from the precomiled binary (both devices fail self test). When I try to compile using AOCC it gives an error about missing <Alloc.h>. See hashcat issue# #1691.

./hashcat64.bin -b -m 11300
hashcat (v4.2.1) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

OpenCL Platform #1: Advanced Micro Devices, Inc.

Benchmark relevant options:

  • --optimized-kernel-enable

Hashmode: 11300 - Bitcoin/Litecoin wallet.dat (Iterations: 199999)

Your device driver installation is probably broken.
See also: https://hashcat.net/faq/wrongdriver

Your device driver installation is probably broken.
See also: https://hashcat.net/faq/wrongdriver

Speed.Dev.#1.....: 2557 H/s (72.67ms) @ Accel:128 Loops:32 Thr:256 Vec:1
Speed.Dev.#2.....: 2572 H/s (72.57ms) @ Accel:128 Loops:32 Thr:256 Vec:1
Speed.Dev.#*.....: 5129 H/s

Started: Fri Sep 14 13:21:23 2018
Stopped: Fri Sep 14 13:21:57 2018

@hashuser1
Copy link
Author

I also noticed your gfx803 has 64MCU while mine are showing 36MCU. Is my configuration messed up?

@b-sumner
Copy link
Collaborator

I'm not sure what you mean by "AOCC compiled binary or precompiled binary". I simply cloned the hascat repo, and followed the directions in BUILD.md, except I stopped before "make install".

I think gfx803 comes in different packages. I apparently have one with more CUs.

@hashuser1
Copy link
Author

Can I see the output from 'clang -v' on the machine you compiled it on? Or did you compile using gcc? In the ticket listed above (SR #{ticketno:[8200822583]}), I was told you were compiling using AMD AOCC.

https://developer.amd.com/amd-aocc/

@jlgreathouse
Copy link
Collaborator

A 36 CU gfx803 is likely Polaris 10. @b-sumner is likely using Fiji, which has 64 CUs.

@hashuser1
Copy link
Author

Ok, now we are getting somewhere. If I dont use AOCC and just follow the instructions listed in BUILD.md the selftest errors go away.

clang -v
clang version 4.0
Target: amdgcn--amdhsa
Thread model: posix
InstalledDir: /opt/rocm/opencl/bin/x86_64

./hashcat -b -m 11300
hashcat (v4.2.1-51-gc8dbcf9) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

OpenCL Platform #1: Advanced Micro Devices, Inc.

Benchmark relevant options:

  • --optimized-kernel-enable

Hashmode: 11300 - Bitcoin/Litecoin wallet.dat (Iterations: 199999)

Speed.#1.........: 2558 H/s (72.62ms) @ Accel:128 Loops:32 Thr:256 Vec:1
Speed.#2.........: 2575 H/s (72.47ms) @ Accel:128 Loops:32 Thr:256 Vec:1
Speed.#*.........: 5133 H/s

Started: Fri Sep 14 13:40:49 2018
Stopped: Fri Sep 14 13:41:20 2018

@b-sumner
Copy link
Collaborator

I used gcc 5.4.0 to build hashcat.

@hashuser1
Copy link
Author

Ok, even though the self test is passing using GCC. Its still not finding the correct answer. I made 2 new wallets and encrypted them with passphrases that are included in example.txt. The wallets unlock using the core wallet with the same password. But still no success finding the hash using hashcat. Can you test the following hashes with the included example.txt?

test.txt
test2.txt

example.txt

@b-sumner
Copy link
Collaborator

The fix may not have arrived in a released compiler (I'm using the tip which finds the answers in both cases).

You should be able to try the new compiler in the ROCM 1.9 release very soon.

@jlgreathouse
Copy link
Collaborator

ROCm 1.9.0 was released a few minutes ago. :)

@jlgreathouse
Copy link
Collaborator

I received an internal report that this was fixed with ROCm 1.9, so closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Compiler Functional Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gstoner @jlgreathouse @b-sumner @hashuser1