Proof that the eyes cannot be a dihedral GAK cipher

This proof relies on a (reasonable) assumption that the main set of isomorphs in the first 3 messages comes from the same underlying plaintext.

Assume for the sake of contradiction that the eyes are a group autokey (GAK) cipher with dihedral state group $D_{166}$ and hidden subgroup $C_2$ (a reflection subgroup, it does not matter which).

Look at this small core subset of the main isomorphs in the first three messages:

OLPJ3P-O3QL
&-`=Q`_&Q?-
dN1D-15d-)N

The first isomorph contains Q and -, while the other two contain an alignment between Q and -. This sets up the conditions to see if a conflict is present or not (which tells us whether the elements corresponding to these pairs of isomorphs commute).

Looking at just these letters...

3-Q
Q_?
-5)

Call the contexts $a = 1 \rightarrow 2$, and $b = 1 \rightarrow 3$. (A "context" between isomorphs is the transformation between them, which corresponds to the cumulative product of all the plaintext letters between them if they are in the same message, or whatever sequence of state changes is necessary to transform between them if they are in different messages).

We have chain links:

$3 \xrightarrow{a} Q, \hspace{1em} 3 \xrightarrow{b} -, \hspace{1em} Q \xrightarrow{a} ?, \hspace{1em} Q \xrightarrow{b} ), \hspace{1em} - \xrightarrow{a} \_, \hspace{1em} - \xrightarrow{b} 5$

To see the commutativity conflict in these chains, look at $a$ and check whether it commutes with $b$.

$3 \xrightarrow{a} Q \xrightarrow{b} ), \hspace{1em} 3 \xrightarrow{b} - \xrightarrow{a} \_$

So starting at 3, $a * b$ and $b * a$ lead to different CT symbols. They do not commute.

This conflict on its own is not immediately an issue if the group is $D_{166}$, because the group contains plenty of pairs of elements that don't commute. The issue comes when looking at the possible orders for all elements, and looking at the other chains formed from these isomorphs. First, about the structure of $D_{166}$: Since 83 is prime, the cyclic rotation subgroup $C_{83}$ contains the identity (order 1) and 82 elements of order 83. The other 83 elements of $D_{166}$ correspond to reflections, so they are all order 2.

In the chaining graph with a hidden state, a group element's induced permutation (which we know is not the identity, since the isomorphs are different) can only form cycles of lengths that are factors of the order of the element. Since the orders here are both prime (2 and 83), this means we can only see 1-cycles, 2-cycles, and 83-cycles in the chaining graph for a cipher using group $D_{166}$.

In particular, this means that if we see a chain with length longer than 2, the context element must have order 83, and therefore must be in the cyclic $C_{83}$ subgroup. Note that all elements of this cyclic subgroup commute with each other.

For the $a = 1 \rightarrow 2$ context, we have a chain:

$L \rightarrow - \rightarrow \_$

This is longer than 2, so the $a$ context must be order 83.

For the $b = 1 \rightarrow 3$ context, we have a chain:

$3 \rightarrow - \rightarrow 5$

This is longer than 2, so the $b$ context must be order 83.

But the chaining conflict shows that these two contexts do not commute. This is a contradiction. So the state group cannot be $D_{166}$.

Aside from $C_{83}$ (which cannot have commutativity conflicts at all so was already ruled out), this rules out the simplest case of GAK which could have been applicable to the eyes, and the one that would have been most realistically solved with brute force. The remaining options have many more than 2 hidden states.

(It's possible that there could be something like a strategically placed typo that invalidates this proof assuming the underlying plaintext for these isomorphs is the same, but it seems pretty unlikely. Plus, this isn't the only commutativity conflict which could show the same result, it's just the most convenient one.)