Implement the Nihilist Cipher Auto-Solver

[toebes]

Provide an interactive guide to show coaches and participants how to approach and solve the Nihilist Substitution cipher.
This is dependent on #385

[Pojoto]

There's still a bit of work needed on the cryptanalysis solver, so this issue isn't fully resolved yet. Mostly aesthetic changes, but some bug fixes, are coming soon.

[Agilus]

I'm liking this so far, but I'm running into some areas you might want to look at if you haven't already:

• If my crib contains a space, the solver doesn't know what to do with it unless I include the space inside the crib. For example, if my crib is YOURDESIRE, I must write YOUR DESIRE where it says "crib text", or the solver will not use it.
• Also, the solver does not seem to be using my entire crib in Step 3 unless my block size is set at 0. With any other block size, it looks like it is choosing not to use parts of it.
  Quote: Never give up your desire to be what you want to be. Stay focused, persistent and relentless.
  Polybius Key: SIDEWALK
  Key: MOVIE
  Crib: YOUR DESIRE
  For example, Block size 3 acts strangely here in Step 3 and does not fill in letters that it does know. With some block sizes (2, 6, 8), it seems to be filling in (maybe it is deducing?) letters that are not in the crib but are adjacent to it.
  Definitely possible that I'm not understanding all its work (and I know this feature is incomplete), but I would expect to see the whole crib represented there regardless of block size, right?

[Pojoto]

I'm liking this so far, but I'm running into some areas you might want to look at if you haven't already:

• If my crib contains a space, the solver doesn't know what to do with it unless I include the space inside the crib. For example, if my crib is YOURDESIRE, I must write YOUR DESIRE where it says "crib text", or the solver will not use it.
• Also, the solver does not seem to be using my entire crib in Step 3 unless my block size is set at 0. With any other block size, it looks like it is choosing not to use parts of it.
  Quote: Never give up your desire to be what you want to be. Stay focused, persistent and relentless.
  Polybius Key: SIDEWALK
  Key: MOVIE
  Crib: YOUR DESIRE
  For example, Block size 3 acts strangely here in Step 3 and does not fill in letters that it does know. With some block sizes (2, 6, 8), it seems to be filling in (maybe it is deducing?) letters that are not in the crib but are adjacent to it.
  Definitely possible that I'm not understanding all its work (and I know this feature is incomplete), but I would expect to see the whole crib represented there regardless of block size, right?

Thanks for the feedback!

To address your first point, I personally think the space should be included when picking the crib, since the point of the crib is to match the plaintext exactly - you wouldn't be able to distinguish between certain preferred cribs otherwise. This is more of a personal preference though, other people can give input on that.

Your second point is definitely a bug though, and I'm glad you caught it. I just fixed the issue (#499) and pushed the new changes out to production. I tested it on your given input - and a few others - and it all seems to work now.

[Agilus]

The space in the crib is an issue I've struggled to figure out how to use properly throughout the tool. I used to always put it in (not in the Nihilist ciphers specifically, but anywhere I was using a crib with spaces), but then I felt like I was having some problems so I began removing it because I thought that was the expected behavior on my part. I think that was before I learned about github and now I'd be hard-pressed to remember which problems I was experiencing, and maybe they've been fixed in the meantime!

In this case, though, since I was trying to change my block size to remove/alter the spacing, I think I should be allowed to omit it without a penalty. With any block size other than 0, the Nihilist doesn't really have spaces in the plaintext. It would be kind of me to put it in, but I'm not always kind :)

[toebes]

I've spent some time building on the code and in the process bumped the scores as well as made the autosolver provide input into the scores. @Agilus @Pojoto it would be great to spend some time with the new code to see if there are more tweaks to be made.

[Agilus]

Ooooh, these are exciting changes. I'm looking forward to giving these to my students this week.

If I do this same quote, it tells me that the solver was unable to recover the keyword - but it definitely finds MOVIE towards the end. Does it give the same "unable to recover the keyword" error because it was unable to recover the Polybius key?

[toebes]

If it can't find the keyword, it tells you and then CHEATS and continues to solve with what it knows to be. You can look at how many keyword options and what it found to decide if you want to let it be a problem to solve.

[Agilus]

OK, I looked back at one we did last week.
Quote: I believe in the possibility of miracles but more to the point, I believe in our need for them.
PKey: SIGNIFYING
Key: HOPEFUL
Crib: BELIEVE

• I definitely remember that when I gave out this cipher on the 15th, it had made more progress on HOPEFUL in Step 3 - enough that I was optimistic that students would be able to recover it.
• So in Step 3, I think one deduction it could make given what it knows about the Es is it can confirm the tens digit from part of the crib, and the ones digit from the rest of the crib. I don't know whether the likelihood of this happening more in the future is worth writing code for it, but it's something that it would know.
• It's making some interesting choices in Step 4. First, I'm having trouble reading to the end of the sentence that begins with "Since K3 'P' could be any of..." because it looks like it terminates mid-thought towards the end of that line without finishing drawing its conclusion.
• In the table in Step 4, it's very conservative with its choices for understandable reasons, but at this point we have enough letters in sequential alphabetical order that a human solver is almost certainly ruling out Z and its friends in Rows 3&4, and have probably moved these rows to "alphabet" status.
• But under the table, it tells me the letters QRST fit into a 4-letter gap and can be filled in -- but the actuality is that S is in the PKey, and these blanks should be PQRT. This causes it to make off-by-one errors in at least the 2nd, 3rd, and 4th lines with it predicting plaintext QO... This could be related to it not finishing its thought about K3 'P'?

[toebes]

Ok this was definitely hard to fix, but I've gone through and done a bit of restructuring. In the process I've added code to analyze how much of the answer is revealed to give better feedback on the crib.

[toebes]

Fixed and pushed to production

[Agilus]

I'm still experiencing this same issue, where the AutoSolver doesn't finish its thought that begins with "...only the choice of 15+32 works which means that", without finishing the sentence, and then jumps right into "Since K1 'L' must be", involving an erroneous choice, and getting the wrong answer:
Quote: The difference between fiction and reality? Fiction has to make sense.
PKey: SCHOLARSHIPS
Keyword: LATENCY
Crib: BETWEEN
Block size 7

So it says, "Since K1 'L' could be any of '12, 13, 14, 15' and 'L' at position 14 can be any of '31, 32', but they also have to add up to be 47, only the choice of 15+32 works which means that Since K1 'L'..."
But if I review the problem, isn't it in this case, K1 'L' can be any of '12, 13, 14, 15' and 'E' at position 14 can be '31, 32', they add to 47, only 15+32 works which means that K1 'L' should be at 15 and 'E' should be in the PTable at 32? If that's the case, then that would tell it that 32 should be the E instead of L and it would put E into its plaintext where it erroneously has L.

It seems like what it's doing with the plaintext at the very bottom doesn't match the table - it clearly shows L in 15 in the PTable, which it should. I think it should have determined that E is 32 based on its reasoning above, but it leaves that spot in the table blank and assigns L to the value 32 when making decisions on the plaintext for all values of 32 except the ones in the crib.

Similarly, in this one:
Quote: Before you've practiced, the theory is useless. After you've practiced, the theory is obvious.
PKey: THEORY
Key: OBVIOUS
Crib: PRACTICED
Block size 10

It has the same "doesn't finish its thought" issue between the two PTables in Step 4. But, fascinatingly, in this case the last table clearly shows S in position 45, but when it assigns to the plaintext, it assigns R to the 45s. (And S to the 22s, which is the same as the case above where it should be known to be A.)

Edited to add: At the bottom of Step 2, it correctly determines the tens digit for K7 to be either 3 or 4 (and the ones digit is 5), but once it gets to Steps 3 and 4, it widens the possibility to 15, 25, 35, 45, 55. I'm very confused then in Step 3 where it states "Because we know that the ones digit of K7 at position 13 is 2 by subtraction from 67, we can eliminate all the values for A that don't have 2 in the ones position. This leaves only '12, 22, 32, 42, 52' as potential values for A and '55, 45, 35, 25, 15' as potential values for K7" - which contradicts its earlier knowledge about the ones digit (and the tens digit) of K7.

Hope this feedback is helpful.