Create a Snappy Ubuntu package for JtR #2118

claudioandre-br · 2016-04-17T01:58:33Z

Snappy works on: Raspberry Pi 2, Intel NUC, Azure, Google Compute Engine cloud, Amazon Elastic Compute Cloud, Beagle, Duovero, ...
Snappy packages are going to be supported on Ubuntu 16.04 LTS [1].
It provides transactional updates with rigorous application isolation.
It has an improved security model.

So, when we have this available, users can install JtR via Ubuntu Store.

[1] Ubuntu 16.04 LTS introduces the snappy Ubuntu Core experience to the desktop by allowing you to create, install and distribute snaps (snappy apps).

claudioandre-br · 2016-04-17T02:01:52Z

Thinking loud:

one package can handle (JtR already does this) AVX, SSE, XOP, ...
not sure: one package with MPI and one without it?
improve OpenCL runtime detection to use only one package?

claudioandre-br · 2016-04-17T23:26:51Z

Ubuntu Core (150MB download) running inside kvm.

$ ./john -test=0
[...]
All 343 formats passed self-tests!

The basic stuff works fine, but the overall quality is low. Anyway, when the environment matures, it might be very useful.

$ kvm -m 512 -redir :8090::80 -redir :8022::22 ubuntu-snappy-amd64.img &
$ scp -CP 8022 JtR/*.snap ubuntu@localhost:~
$ ssh -p 8022 ubuntu@localhost
$ sudo snappy install john-the-ripper_1.8.0-Jumbo-2.dev_amd64.snap

claudioandre-br · 2016-04-18T15:10:35Z

$ john-the-ripper --list=build-info
Version: 1.8.0-11765-g9a09113+
Build: linux-gnu 64-bit SSE2-ac
SIMD: SSE2, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
System-wide exec: $JOHN/
System-wide home: $JOHN/
Private home is /home/ubuntu/snap/john-the-ripper/100001/.john
$JOHN is /snap/john-the-ripper/100001/run/
Format interface version: 13
Max. number of reported tunable costs: 3
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 5.3.1
GNU libc version: 2.23 (loaded: 2.23)
OpenCL headers version: 1.2
Crypto library: OpenSSL
OpenSSL library version: 01000207f
OpenSSL 1.0.2g-fips  1 Mar 2016
GMP library version: 6.1.0
Regex library version: 1.3  (loaded: 1.3.1)
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's

claudioandre-br · 2016-04-19T12:11:14Z

~$ john-the-ripper.john-the-ripper --stdout --regex='[0-2]password[A-C]'
Warning: regex mode currently can't be resumed if aborted
0passwordA
1passwordA
2passwordA
0passwordB
1passwordB
2passwordB
0passwordC
1passwordC
2passwordC
9p 0:00:00:00 0.00% 24.32p/s 2passwordC

claudioandre-br · 2016-04-19T12:12:21Z

JOHN_SYSTEMWIDE_EXEC and JOHN_SYSTEMWIDE_HOME hard coded are suitable for the new era?
I have to correct myself: the overall quality is low is a wrong statement. The real problems is the package system.

~$ john-the-ripper.john-the-ripper -form=raw-SHA512 $SNAP_USER_DATA/alltests.in --incremental -fork=2 --max-run=80
Created directory: /home/ubuntu/snaps/john-the-ripper.sideload/LWdCNNGPcKPf/.john
Using default input encoding: UTF-8
Loaded 20 password hashes with no different salts (Raw-SHA512 [SHA512 128/128 SSE2 2x])
Node numbers 1-2 of 2 (fork)
Press 'q' or Ctrl-C to abort, almost any other key for status
                 (?)
password         (?)
12345678         (?)
2 2g 0:00:01:18  0.02540g/s 366227p/s 366227c/s 6549KC/s l0431s..l0435p
1 1g 0:00:01:18  0.01270g/s 365427p/s 365427c/s 6943KC/s robchu1..robchuy
Waiting for 1 child to terminate
Warning: passwords printed above might not be all those cracked
Use the "--show" option to display all of the cracked passwords reliably
Session stopped (max run-time reached)

claudioandre-br · 2016-04-24T16:21:18Z

Working:

john-the-ripper.john-the-ripper --test=0;
Fallback for CPU and OMP;
regex and prince (libs are installed with/inside the package itself);
incremental, wordlist, single;
register the package and its name (see setpriority below);
setpriority with seccomp (release planned for a few weeks: 16.04.1(?)) - setpriority/nice disabled now;
python and ruby are not going to be installed with the package. Decide what to do with *2john files;
is experimental architectures usable? Find someone eager to try JtR on PI.

claudioandre-br · 2016-05-07T17:35:45Z

Just in case something unexpected appears on john-users:

There are people aware of the snap package:
1. http://blog.topspeedsnail.com/archives/4733
2. https://askubuntu.com/questions/763531/cant-install-snap-apps

The package itself works (real-world tests needed). No announcement done or planned for a while.

solardiz · 2016-05-08T13:21:26Z

Claudio, I just want to let you know that I appreciate your work on this. Thank you!

claudioandre-br · 2016-05-10T00:51:12Z

magnumripper · 2016-05-10T05:27:29Z

Thinking loud:

one package can handle (JtR already does this) AVX, SSE, XOP, ...

not sure: one package with MPI and one without it?

improve OpenCL runtime detection to use only one package?

Using dlopen for OpenCL would take a fair amount of work but is definitely doable (Hashkill does that).

Now that I think about it, doing the same for MPI should be much easier. I'll open issues for these.

claudioandre-br · 2016-05-10T15:23:23Z

Using dlopen for OpenCL would take a fair amount of work but is definitely doable (Hashkill does that).

This make sense and is going to be very useful. If you use a fallback executable, it can be much easier, no?
(I might be confused about the real challenge here).

if dlopen("opencl_lib")
   exec("/regular/opencl_build/JtR")
else
   'Go-on and use this non-opencl build

BTW:

To be a Snappy app, this might be not necessary, since I can assure Generic OpenCL ICD Loader is always available. So, using one executable, I can enabled OpenCL formats only if there is at least one device available.
I haven't tested the idea since I'm far from home and, even if I bring my computer, I can't use OpenCL on Ubuntu 16.04 (no AMD OpenCL for a while).

Keep this in mind to make your final decision.

magnumripper · 2016-05-10T16:42:36Z

if dlopen("opencl_lib")
   exec("/regular/opencl_build/JtR")
else
   'Go-on and use this non-opencl build

Wow that is a quick solution... and it should work just fine. I never thought of that!

magnumripper · 2016-05-10T16:47:14Z

On the other hand we've been discussing doing the opposite: Include code for SSE2/3/4.1, AVX, AVX2 and whatnot in a single binary, that picks code path at run time. That together with real use of dlopen() for OpenCL and MPI (and perhaps librexgen etc.) would provide a single binary capable of most anything. It would be a bit fatter though.

claudioandre-br · 2016-05-10T17:07:38Z

The ugly

after OpenCL, a lot of executable files (avx, avx_omp, avx_cl, avx_omp_cl, sse_cl, ...).

The other path (one binary for everything) has pros and cons. An important: Invasive. Could frighten people.

magnumripper · 2016-05-10T17:54:50Z

We currently have SSE2, SSSE3, SSE4.1, AVX and AVX2 on one hand, and OMP/non-OMP on the other. That's 2x5 == 10 combinations. Adding AVX512, OpenCL and MPI to the mix will mean 2x2x2x6 == 48 combinations... not to mention the time for building them.

magnumripper · 2016-05-10T17:56:41Z

But for this Snappy thing (which I have yet to get acquainted with) like you said you can just build with OpenCL and MPI support and ensure the run-time libs are included. The functionality will not hurt at all even if unused.

claudioandre-br · 2016-05-10T19:04:02Z

which I have yet to get acquainted with

If you need to try it yourself (disclaimer: Ubuntu Core 16.04 is under construction, you will notice that the welcome message is simply wrong).

On a deb base system
$ sudo apt-get install qemu-kvm

On Fedora (my guess)
# yum install kvm

On the fancy OS
no idea

Obtain Ubuntu Core image (200MB)

$ wget https://people.canonical.com/~mvo/all-snaps/amd64-all-snap.img.xz
$ unxz amd64-all-snap.img.xz

Launch the VM
$ kvm -m 512 -redir :8022::22 amd64-all-snap.img &

Access the VM (user and password are "ubuntu")
$ ssh -p 8022 ubuntu@localhost

Install JtR (70MB)
$ sudo snap install john-the-ripper

Test if John is working properly
$ john-the-ripper -test=0

$ export DATA=/home/ubuntu/snap/john-the-ripper/5/.john
$ john-the-ripper -list=format-tests | cut -f3 > $DATA/alltests.in
$ john-the-ripper -form=SHA512crypt $DATA/alltests.in

Obs: John the Ripper Jumbo with CPU and OpenMP fallback.

claudioandre-br · 2016-05-13T22:51:50Z

John the Ripper Jumbo with:

CPU and OpenMP fallback;
with OpenCL auto-detection.

$ john-the-ripper -test --form=raw-sha512-opencl
Unknown ciphertext format name requested

$ john-the-ripper --list=opencl-devices
No OpenCL devices found

$ john-the-ripper -test --form=raw-sha512
Benchmarking: Raw-SHA512 [SHA512 128/128 SSE2 2x]... DONE
Raw:    893578 c/s real, 893578 c/s virtual

frank-dittrich · 2016-05-14T09:07:00Z

What happens if you run --test=0?
Will john try to test opencl formats (and fail)?

Should OpenCL formats be listed only

if (get_number_of_devices_in_use())

I.e., do we need to adjust the output of --list=formats, --list=format-details, an may be the

--format=CLASS            valid classes: dynamic, cpu, gpu, opencl, omp

in the --list=hidden-options output?

This might be useful for bash completion, for johnny (the gui) or just for somehow letting the jtrTestSuite know that opencl formats can't be tested.

Time to add a --list=runtime-info to the hidden options?

claudioandre-br · 2016-05-14T16:37:31Z

What happens if you run --test=0?

It works. I already worked around it. But I guess the final solution will be to register (inside john_register_one) an OpenCL format only if (get_number_of_available_devices() > 0).

There are even more use cases requiring to change registering:

e.g., crypt(3) hash recommendations for a run without --format.
--list=format-tests
...

Edge cases to be solved otherwise

--format=CLASS

That said:

the OpenCL auto-detection should be merged on Jumbo because a regular user has to be able to remove his GPU(s) without the need to rebuild JtR.
once done, I need to check if any OpenCL device is available even for a non OpenCL run [1]. Solar once complained about this.
previous committed code calls OpenCL stuff only inside fmt_init().

[1] (Have to check) this won't happen for a -format=xxx-cpu-only run.

claudioandre-br · 2016-05-14T16:40:48Z

Sorry, read the previous comment on GitHub site.

claudioandre-br · 2016-05-20T00:47:03Z

Well, since I was unable to find a better approach, I intend to:

add (to the existing snap package) new command. It means.

CPU only (CPU fallback, OMP fallback, regex, etc)
$ john-the-ripper --list=build-info

CPU + OpenCL (CPU fallback, OMP fallback, regex, etc)
$ john-the-ripper.opencl --list=build-info

So, one snap will have all 16 john binaries.

john-the-ripper: is the application name;
opencl: is one of its commands (I can create cpu, mpi, whatever).

Of course, we can shrink the package creating one snap for OpenCL, one for MPI, ...

Comments are welcome.

claudioandre-br · 2016-06-12T19:28:31Z

Create a JtR package for Windows 10 (zip or Windows Store).

Testing Ubuntu binaries of John the Ripper Jumbo on Windows 10 64bits.

It's native Ubuntu binaries running directly in Windows.
Bit-for-bit, checksum-for-checksum Ubuntu ELF binaries running directly in Windows.

Notice:

It will allow us to create a zip file or publish on Windows Store the same binaries used in the SNAP package.
Using a VM, container, whatever, it will be possible to automate / isolate the building (I'm using a VM).

PS: not sure if one can publish using a free Vistual Studio.

Working

claudioandre-br · 2016-06-12T19:30:16Z

claudioandre-br · 2016-06-12T19:42:37Z

Tested so far:

$  export DATA=/home/winlix/snap/john-the-ripper
$ ./john-the-ripper -list=format-tests | cut -f3 > $DATA/alltests.in
$ ./john-the-ripper -form=raw-SHA512 $DATA/alltests.in --max-run=30
$ ./john-the-ripper -form=SHA512crypt $DATA/alltests.in
$ ./john-the-ripper -form=raw-SHA512 $DATA/alltests.in --incremental -fork=2 --max-run=80

$ ./john-the-ripper -form=SHA512crypt --test=2
Will run 6 OpenMP threads
Benchmarking: sha512crypt, crypt(3) $6$ (rounds=5000) [SHA512 128/128 SSE2 2x]... (6xOMP) DONE
Speed for cost 1 (iteration count) of 5000
Raw:    2767 c/s real, 467 c/s virtual

Format	Win c/s	Linux c/s [2]	Linux c/s [3]
raw-sha512	8933K	9172K	2641K
raw-sha256	15347K	14745K	5271K
sha512crypt	2792	2671	520
sha256crypt	3375	2871	653

[1] Using the same machine via dual booting. Ubuntu 16.04 LTS snap app.
[2] With OpenMP, 6 threads.
[3] Disable OpenMP via OMP_THREAD_LIMIT=1

claudioandre-br · 2016-06-13T18:57:53Z

Since we already have hundreds of installations, I intend to publish this on john-users:

Greetings

For some time now, we have a John the Ripper SNAP package available on Ubuntu Store. The plan is to release it as stable and tested, but offering recent bug fixes and improvements.

To install it, on Ubuntu 16.04 LTS, you only need to run a:
$ sudo snap install john-the-ripper

To test it, do:
$ john-the-ripper --list=build-info

John run confined under a restrictive security sandbox by default. For example, it won't be able to read or write any file outside its "box". So, word-lists, hash files, and so on, should be copied to the "box". Below, an example:
$ export DATA=/home/user/snap/john-the-ripper/5/.john
$ john-the-ripper -list=format-tests | cut -f3 > $DATA/alltests.in
$ john-the-ripper -form=SHA512crypt $DATA/alltests.in

The highlights:

fallback for CPU and OMP.
regex and prince modes available.

More information at #2118. Feedback is welcome.

Happy cracking

claudioandre-br · 2016-06-14T17:56:06Z

For the record. More info at: https://lists.ubuntu.com/archives/snapcraft/2016-June/000193.html

[...] snaps should now work, unmodified, on Arch, CentOS, Debian, Elementary, Fedora, Gentoo, RHEL, SUSE... and of course the whole family of *buntu's.
[...]

claudioandre-br · 2016-06-28T02:53:23Z

There is an armhf (arm7) version available on Ubuntu Store for a week or so. It has:

regex and prince modes available
fallback for OMP

claudioandre-br · 2016-07-13T14:20:32Z

Well, I dont have hardware to develop and test the changes needed in Ubuntu Core to allow OpenCL confined. I intend to create a workaround. Notice:

CPU OpenCL fails even in --devmode (a workaround is needed anyway).
Older AMD GPU (< GCN 1.1) is legacy hardware.

The procedure is:

$ export SNAP=/snap/john-the-ripper/10 #--10-- is the latest/desired installed version/revision 
$ /snap/john-the-ripper/10/john-the-ripper.opencl --list=build-info

claudioandre-br · 2016-07-29T14:20:11Z

Posting the message below to john-users and closing:

BTW: it is possible to run the JtR openCL binary using the workaround seen below (notice: 12 is the current/desired revision [1]):
~$ /snap/john-the-ripper/12/john-the-ripper.opencl -list=build-info
~$ /snap/john-the-ripper/12/john-the-ripper.opencl -list=opencl-devices

Cracking using the CPU and/or the OpenCL version:
~$ export DATA=$HOME/snap/john-the-ripper/12/.john
~$ john-the-ripper -list=format-tests | cut -f3 > $DATA/alltests.in
~$ /snap/john-the-ripper/12/john-the-ripper.opencl -form=SHA512crypt-opencl $DATA/alltests.in
~$ john-the-ripper -form=SHA512crypt $DATA/alltests.in

Just in case you have ARM hardware, there are armhf and arm64 JtR versions available on Ubuntu Store. They have:

regex and prince modes available
fallback for OMP

If your distro already accepts the snap format (without a store), it is possible to download the reviewed snap package from: https://uappexplorer.com/app/john-the-ripper.claudioandre-br

Claudio

[1] Rev 12 means commit: 36a7264 from July 29th. Execute a sudo snap refresh and/or snap list to assure you have the latest version installed.

claudioandre-br · 2016-12-05T02:47:49Z

I just created a readme file (since I started to wonder details about the package). Everything is there, even the new fancy stuff.

BTW: nice to see; strong foundations, high-quality software engineering.

claudioandre-br · 2018-03-02T19:27:54Z

More notes at:
https://snapcraft.io/john-the-ripper

claudioandre-br self-assigned this Apr 17, 2016

claudioandre-br added enhancement portability labels Apr 17, 2016

claudioandre-br added this to the 1.8.0-jumbo-2 milestone Apr 17, 2016

frank-dittrich mentioned this issue Apr 18, 2016

Always save the commit used to build JtR. #2124

Merged

This was referenced May 10, 2016

Use dlopen for OpenCL #2133

Open

Use dlopen for MPI #2134

Closed

claudioandre-br mentioned this issue May 16, 2016

OpenCL build works without any installed device (WIP). #2137

Closed

claudioandre-br closed this as completed Jul 29, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a Snappy Ubuntu package for JtR #2118

Create a Snappy Ubuntu package for JtR #2118

claudioandre-br commented Apr 17, 2016

claudioandre-br commented Apr 17, 2016

claudioandre-br commented Apr 17, 2016 •

edited

claudioandre-br commented Apr 18, 2016 •

edited

claudioandre-br commented Apr 19, 2016

claudioandre-br commented Apr 19, 2016 •

edited

claudioandre-br commented Apr 24, 2016 •

edited

claudioandre-br commented May 7, 2016

solardiz commented May 8, 2016

claudioandre-br commented May 10, 2016 •

edited

magnumripper commented May 10, 2016 •

edited

claudioandre-br commented May 10, 2016

magnumripper commented May 10, 2016

magnumripper commented May 10, 2016

claudioandre-br commented May 10, 2016

magnumripper commented May 10, 2016

magnumripper commented May 10, 2016

claudioandre-br commented May 10, 2016 •

edited

claudioandre-br commented May 13, 2016 •

edited

frank-dittrich commented May 14, 2016 •

edited

claudioandre-br commented May 14, 2016 •

edited

claudioandre-br commented May 14, 2016

claudioandre-br commented May 20, 2016

claudioandre-br commented Jun 12, 2016 •

edited

claudioandre-br commented Jun 12, 2016

claudioandre-br commented Jun 12, 2016 •

edited

claudioandre-br commented Jun 13, 2016 •

edited

claudioandre-br commented Jun 14, 2016

claudioandre-br commented Jun 28, 2016

claudioandre-br commented Jul 13, 2016 •

edited

claudioandre-br commented Jul 29, 2016 •

edited

claudioandre-br commented Dec 5, 2016 •

edited

claudioandre-br commented Mar 2, 2018

Create a Snappy Ubuntu package for JtR #2118

Create a Snappy Ubuntu package for JtR #2118

Comments

claudioandre-br commented Apr 17, 2016

claudioandre-br commented Apr 17, 2016

claudioandre-br commented Apr 17, 2016 • edited

claudioandre-br commented Apr 18, 2016 • edited

claudioandre-br commented Apr 19, 2016

claudioandre-br commented Apr 19, 2016 • edited

claudioandre-br commented Apr 24, 2016 • edited

claudioandre-br commented May 7, 2016

solardiz commented May 8, 2016

claudioandre-br commented May 10, 2016 • edited

magnumripper commented May 10, 2016 • edited

claudioandre-br commented May 10, 2016

magnumripper commented May 10, 2016

magnumripper commented May 10, 2016

claudioandre-br commented May 10, 2016

magnumripper commented May 10, 2016

magnumripper commented May 10, 2016

claudioandre-br commented May 10, 2016 • edited

claudioandre-br commented May 13, 2016 • edited

frank-dittrich commented May 14, 2016 • edited

claudioandre-br commented May 14, 2016 • edited

claudioandre-br commented May 14, 2016

claudioandre-br commented May 20, 2016

claudioandre-br commented Jun 12, 2016 • edited

Create a JtR package for Windows 10 (zip or Windows Store).

claudioandre-br commented Jun 12, 2016

claudioandre-br commented Jun 12, 2016 • edited

claudioandre-br commented Jun 13, 2016 • edited

claudioandre-br commented Jun 14, 2016

claudioandre-br commented Jun 28, 2016

claudioandre-br commented Jul 13, 2016 • edited

claudioandre-br commented Jul 29, 2016 • edited

Posting the message below to john-users and closing:

claudioandre-br commented Dec 5, 2016 • edited

claudioandre-br commented Mar 2, 2018

claudioandre-br commented Apr 17, 2016 •

edited

claudioandre-br commented Apr 18, 2016 •

edited

claudioandre-br commented Apr 19, 2016 •

edited

claudioandre-br commented Apr 24, 2016 •

edited

claudioandre-br commented May 10, 2016 •

edited

magnumripper commented May 10, 2016 •

edited

claudioandre-br commented May 10, 2016 •

edited

claudioandre-br commented May 13, 2016 •

edited

frank-dittrich commented May 14, 2016 •

edited

claudioandre-br commented May 14, 2016 •

edited

claudioandre-br commented Jun 12, 2016 •

edited

claudioandre-br commented Jun 12, 2016 •

edited

claudioandre-br commented Jun 13, 2016 •

edited

claudioandre-br commented Jul 13, 2016 •

edited

claudioandre-br commented Jul 29, 2016 •

edited

claudioandre-br commented Dec 5, 2016 •

edited