Random format errors reprocessing the .POT files in TS with -passthru=--fork=n" #1074
Comments
|
May be it is not related to dynamic, because just now I triggered the same problem for sha512crypt-new. |
frank-dittrich
changed the title
|
Could be an older problem (related to locking?) which is not yet fully fixed. |
|
I guess I am going to have to bite the bullet, and simply change code in jtrts.pl to strip out any -fork code for the .pot check, since you continue to post pretty much the same type bugs on this one. So if running in -fork mode, -fork will be used properly on the 'real' cracking session (which writes data to the .pot file). But -fork mode will not be used when reprocessing the .pot file, which in of itself has been changed to be more aggressive into finding cases where the password stored in the .pot file is not the correct one, BUT which has problems that can NOT be worked around fully due to expectations of screen IO and the problem interaction we get with -fork and running with memdbg. Now that you were also getting this when running in non -memdbg is a bit curious, but like you said, it may be OS related. Using -fork for the .pot re-run is not a great thing, and provides no additional benefit, but causes numerous bad things to happen, since we are reading a mix of writing to the stdout, and not all of it is from the password guessing code. On the initial run, all data comes from .pot file, and all of that is being done with flocking, and flushing of writes (or 'should' be). The main run is what is important anyway. That is what we really should be testing against. What you show for -jumbo-1 is expected. There were known problems, which have been fixed since that time, with the fflush(.potfile) since that time. |
|
With memdbg and asan disabled, I got |
|
In a repeated run, it got to dynamic_6 before failing: |
|
I guess it is a TS bug and your parsing logic is wrong. Apparently, these are always summary lines of forked processes where the last password of the last chunk of passwords ends with a |
|
It is not a really TS bug, it is a JtR 'bug' side effect of forking. It is where screen output gets smashed. But this is happening in the real run, and not the .pot recheck? It may be that the fork 'guesses' line outputs are smashing each other. I will not be chasing these 'phantom' bugs any more. -fork is not highly compatible with the TS. The TS is a tool to help make sure that jtr is finding all passwords. The last 10-15 'bugs' found with -fork have not been bugs. Yes, we found some -asan related things, that really needed fixed. But most of these are nothing more than side effects of garbled screen output, and how jtrts.pl was written to work. If you are going to continue to find pretty much the exact same 'bug', then simply continue to add to this bug. List exact build type, the exact jtrts.pl command line used, the OS, etc. I will look at possibly re-writing the inner logic of the cracking loop in jtrts.pl at some time, but not right now. |
|
The ones that 'appear' to be side effects of how the TS is coded, probably should only go on the TS. I really doubt if you ran all of these formats by hand, using the --fork=xx, that there would be any passwords 'missing'. That is the main point of what we should focus on with the testing within TS against JtR. IF we have problems where only 1456 are found, when 1500 should be, AND when we run the file by hand we also see that only 1456 ARE being found, then that certainly IS a bug in that format of JtR (or in JtR proper). But if JtR is finding the right data, and it is simply the TS complaining, then we should put it over in TS like you did with #45. |
|
Closing, because if there's anything wrong here it is the TS parsing logic, see openwall/john-tests#45 |
On my laptop (quad core with HT) I need a sufficiently high number of forked processes to trigger this.
On another quad core system without HT, I was able to trigger this with -passthru="--fork=2", but a higher number of forked processes helps to make such errors more likely (increase the number of errors from 1 or 2 with --fork=2 to about 15-30 for --fork=20).
The error is not related to memdbg or asan, since I was able to reproduce this with asan and memdbg disabled.
Which of these formats fail appears to be random, since for releated runs the result differs.
But it looks like only dynamic and thin formats fail in this way.
The text was updated successfully, but these errors were encountered: