Make a new release soon as the previous one has Python 3 compatibility issues and known-fixed bugs

[noraj]

$ /usr/bin/env python --version
Python 3.9.1

$ ssh2john id_rsa
Traceback (most recent call last):
  File "/usr/bin/ssh2john", line 193, in <module>
    read_private_key(filename)
  File "/usr/bin/ssh2john", line 103, in read_private_key
    data = base64.decodestring(data)
AttributeError: module 'base64' has no attribute 'decodestring'

Same kind of error as for #3345.

$ grep -n decodestring /usr/bin/ssh2john
103:        data = base64.decodestring(data)

It is preset in 1.9.0-jumbo-1 git tag:

john/run/ssh2john.py

Line 103 in a16c8a7

data = base64.decodestring(data)

But it's fixed in bleeding-jumbo git tag:

john/run/ssh2john.py

Lines 107 to 114 in 07c0f3c

	# if we trudged to the end of the file, just try to cope.
	try:
	data = ''.join(lines[start:end]).encode()
	data = codecs.decode(data, 'base64_codec')
	except base64.binascii.Error:
	e = sys.exc_info()[1]
	raise Exception('base64 decoding error: ' + str(e))

tag 1.9.0-jumbo is from May 14, 2019.

So releasing a new version will help fix that bug in distro packages that are mostly using python 3.9 nowadays.

System configuration

$ john --list=build-info                                                                         
Version: 1.9.0-jumbo-1
Build: linux-gnu 64-bit x86_64 AVX AC MPI + OMP
SIMD: AVX, interleaving: MD4:3 MD5:3 SHA1:1 SHA256:1 SHA512:1
System-wide exec: /usr/lib/john
System-wide home: /usr/share/john
Private home: ~/.john
CPU tests: AVX
CPU fallback binary: john-non-avx
$JOHN is /usr/share/john/
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
gcc version: 10.2.0
GNU libc version: 2.32 (loaded: 2.32)
OpenCL headers version: 2.2
Crypto library: OpenSSL
OpenSSL library version: 01010108f      (loaded: 01010109f)
OpenSSL 1.1.1h  22 Sep 2020     (loaded: OpenSSL 1.1.1i  8 Dec 2020)
GMP library version: 6.2.0      (loaded: 6.2.1)
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's

Workaround for distro packager waiting for the new release

$ sed 's/decodestring/decodebytes/' /usr/bin/ssh2john | python3.9 - id_rsa

ArchLinux FS#69545

[anthraxx]

I would highly appreciate to get a guicy release :P

[solardiz]

@noraj Do you want to help us make that release sooner rather than later? We need to at least bring NEWS up to date (many important changes were unfortunately not reflected in there yet) and try and fix all of the issues under the "Definitely 1.9.0-jumbo-2" milestone. Want to help with these?

[noraj]

@solardiz You mean that some major changes where not documented into NEWS? How could I help?

[solardiz]

@noraj Right. The task is to go over the commits made since 1.9.0-jumbo-1 and add to NEWS non-trivial higher-level changes describing them in a suitable form (see existing entries), and send a pull request updating NEWS. Thank you!

[solardiz]

I've just removed most issues from the "Definitely 1.9.0-jumbo-2" milestone. Most of those went to the "Potentially 1.9.0-jumbo-2" milestone, some were removed from milestones altogether. Some of the issues only need more review to determine that we can close them or don't have to work on them before the release. Some need actual work.

[solardiz]

go over the commits made since 1.9.0-jumbo-1 and add to NEWS non-trivial higher-level changes

This task is now #4570.

We also had #3979 titled after for the upcoming release, but it has some "irrelevant" stuff said on it (on general and longer-term planning rather than on what really needs to be done for the release).

So I suggest we use this new issue for planning the release from now on.

[claudioandre-br]

Another example of a "bug" fixed in bleeding-jumbo:

• https://www.openwall.com/lists/john-users/2021/02/18/1

$ john teste.hash 
Using default input encoding: UTF-8
Loaded 1 password hash (HMAC-SHA256 [password is key, SHA256 256/256 AVX2 8x])
Will run 8 OpenMP threads
Proceeding with single, rules:Single
Press 'q' or Ctrl-C to abort, almost any other key for status
Almost done: Processing the remaining buffered candidate passwords, if any.
Proceeding with wordlist:/snap/john-the-ripper/current/run/password.lst
Proceeding with incremental:ASCII
0g 0:00:00:12  3/3 0g/s 2068Kp/s 2068Kc/s 2068KC/s licuato..lynns17
Session aborted

$ cat teste.hash 
john.txt
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa#26a8d8c86b34a358817ced22bfde859642ba4d13a30a0b814345cb3e878b8618

[solardiz]

@claudioandre-br No, this wasn't anything we fixed - rather, the line wrapping prevented you from testing the same thing that the question on john-users was about. Anyway, let's not collect supposedly fixed issues in here. (If you feel anything like this should be added to NEWS, we have a separate issue for updating that file.) I'd rather see us make progress towards making a new release. Thanks!

[D3vil0p3r]

Since on OpenWall website the latest john package is still the one with this issue, a new working official release with these fixes will be released or not? Or we should use the bleeding version forever? I see that the latest version on the website is on 2019...

[solardiz]

@D3vil0p3r A new release will be made. Hopefully soon, but no specific date has been set. We should be making releases more often. Unfortunately, releases were also infrequent before - e.g., 1.8.0-jumbo-1 in December 2014 followed by 1.9.0-jumbo-1 only in May 2019.

[D3vil0p3r]

Thank you for the answer @solardiz ! The important aspect for me, even if there is no a date, in the future could be a new release in order to fix the described issue with the latest "stable" official release. Thank you again ^^

[ryandesign]

Please make a new release. Your latest release does not compile on Apple Silicon Macs. You have already fixed this problem in the code years ago (#4585) but users continue to encounter it as they attempt to use your latest release.

[ryandesign]

Your latest release does not compile on Intel Macs either due to multiple definition of symbols, another problem you fixed in the repository years ago (#4258).

[noraj]

Another bug already been fixed on bleeding jumbo

➜ ssh2john ~/.ssh/id_ed25519_crack
Traceback (most recent call last):
  File "/usr/bin/ssh2john", line 194, in <module>
    read_private_key(filename)
  File "/usr/bin/ssh2john", line 154, in read_private_key
    saltstr = data[salt_offset:salt_offset+salt_length].encode("hex")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bytes' object has no attribute 'encode'. Did you mean: 'decode'?

➜ python --version
Python 3.11.6

[prbhtkumr]

Another bug already been fixed on bleeding jumbo

➜ ssh2john ~/.ssh/id_ed25519_crack
Traceback (most recent call last):
  File "/usr/bin/ssh2john", line 194, in <module>
    read_private_key(filename)
  File "/usr/bin/ssh2john", line 154, in read_private_key
    saltstr = data[salt_offset:salt_offset+salt_length].encode("hex")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bytes' object has no attribute 'encode'. Did you mean: 'decode'?

➜ python --version
Python 3.11.6

I've made a fix locally by using the hex() method instead of encode("hex"):

saltstr = data[salt_offset:salt_offset+salt_length].hex()

I believe this change should resolve the issue. However, I'm not sure where in the codebase this change should be made to reflect it correctly. Any guidance on this would be appreciated.

[solardiz]

@prbhtkumr My understanding is that all bugs discussed on this issue are already fixed in this repo. The only reason you ran into those bugs is you're using an older version. We appreciate your willingness to contribute fixes, but for that please start by using our latest code from here to find/fix bugs that still exist. Thank you!

[claudioandre-br]

Early and Often

Thinking out loud:

• we have seen fixes for nasty "typos" in the source code;
  • especially recently.

Therefore (or because of this):

• I still think that mixing the Openwall product with the releases from this repository is neither necessary nor useful.
  • The OpenWall product is based on this repository and the inclusion of extras as, .e.g., support.
• we could create a 1.10.0 version exclusively on GitHub (or something like 1.10.0-rc) [1], with some freedom without requiring the release process and quality involved in an Openwall version.

[1] rc or ce or ... Any "flag" indicating the final version is not ready yet.

[solardiz]

I don't mind us starting to create 1.9.1, etc. releases on GitHub while preparing for a 2.0.0 release from Openwall, but I also don't intend to spend my time on it currently. Claudio, is this something you'd like to start doing?

[claudioandre-br]

I don't mind doing that. But I simply don't have the power to do it. Using a GUI or CLI takes more than I can do.

The release commit itself is very simple, but I can create a PR. I won't dwell too much on release notes and things like that.

"This is a bug fix, renovate, and update release intended to provide a modernized version to end users and packagers. "Early and Often".

BTW: I'll pick a commit, change the release ID and such. And that's it. After that, I will go back to using the bleeding name. GitHub itself will be the advertising platform.

---

Just to be clear, only one thing bothers me: it is #3517. It seems like a change in behavior that is not clear whether it is right or wrong.