Isomorphs (Gap Patterns)

Isomorphs are repeating patterns of repeated ciphertext symbols, where the symbols used can be different each time, but the relative positions of repeats are the same, so that all of the occurrences are substitutions of each other.

(I really need to think of a better one-sentence explanation that is accurate without being excessively long or confusing...)

Isomorphs can be identified by their gap patterns, showing the positions of repeated symbols within a segment of ciphertext. The format used for gap patterns is:

• . is used to represent non-repeated symbols. Note that every occurrence of . should be a distinct symbol.
• Letters A, B, C, ... are used to represent repeated symbols, in the order they occur. The first repeated symbol is replaced with A, the next repeated symbol which is different from A is replaced with B, etc.

For example, in the eyes, the most prominent isomorph which was first noticed was the following set of 5 sequences occurring in the first 3 messages (written in ASCII+32, though the form the symbols take does not matter, only which ones are the same, and writing it in a format where one trigram is one character makes it easier to see), with one of them occurring twice:

OLPJ3P-O3 (twice)
g+jX$j3g$
dN1D-15d-
&-`=Q`_&Q
IhY47YaI7

All of these sequences have the same gap pattern:

A.B.CB.AC

Additionally, all of these occur in similar places in the messages, with two per message. The exact positions they occur are:

• East 1: First at trigram 41, distance 28 between them
• West 1: First at trigram 41 (same position and CT as East 1), distance 30 between them
• East 2: First at trigram 46, distance 35 between them

The chance of seeing this pattern with 3 repeated characters in a sequence of 9 random characters is 1/83^3, or about 0.0002%. There are 1036 - 9 × 9 = 955 opportunities in the ciphertext for this pattern to occur, and it occurs 6 times. The chance of it occurring this many times is about 3 × 10^-20. This is extremely significant, and you would never expect to see this happen by chance.

Due to this identical pattern and similarity in placement within these messages, and the high statistical significance, it was conjectured that these could be an artifact of the cipher mechanism caused by the same plaintext word or phrase being repeated in these places. Then we found that there are many kinds of ciphers which produce these isomorphs, and they are produced exactly when a word or phrase is repeated in different cipher states, where the state changes in the same way across the course of the word/phrase.

Many other isomorphs exist in the eyes, of varying levels of confidence, and many of them are strong patterns with multiple repeated symbols with similar placement in multiple messages like this, so it seems nearly certain that the eyes are a cipher which produces isomorphs and these isomorphs come from repeated segments of plaintext.

It's important to note that the gap pattern of an isomorph is not the same as the gap pattern of the plaintext word/phrase used to produce it. Instead, ciphers which produce isomorphs generally produce the same gap pattern consistently for any particular word, but the positions of the repeated symbols in that pattern are not directly related to the repeated symbols in the plaintext at all. The exact details of the cipher mechanism lead to any given word forming a particular gap pattern in an unpredictable way, and not all words produce strong gap patterns with multiple repeats either.

It's possible to have isomorphs from repeated plaintext which don't end up being readily apparent in the ciphertext, because they have fewer repeated symbols, or none at all. It's only the most clearly visible isomorphs that we're able to see and work with.

The isomorphs are a major piece of the puzzle that is the eyes. They tell us that the cipher method is something which is capable of producing isomorphs, which allows us to say the eyes must be a polyalphabetic substitution cipher, and at the same time they show us directly some of the places that plaintext is being repeated, showing that the messages actually have quite a lot of repeating content, especially in sets of 3 messages (even more than the more immediately noticeable shared sections).

One caveat with gap patterns is that even in ciphers that produce isomorphs, it's possible to have the same gap pattern occur despite the plaintext being different, just from having two different words happen to produce the same pattern. This becomes much less likely the more repeated symbols the isomorph has in it, and seeing multiple occurrences of a pattern at similar positions in multiple messages also helps to confirm that it's not a coincidental repeat, but it is still possible for the same gap pattern to occur for different words by chance.

Some isomorphs listed may be false positives. Ones with fewer repeats are more likely. For example, there are 7 instances of the A.B..B.A pattern. 6 of these occur inside the A.B.CB.AC pattern which occurs in the first 3 messages, while the last one occurs toward the end of East 3. There are three possible explanations for this:

• Only part of the plaintext is repeated, with the last letter missing because the plaintext stops being shared. This could be caused by a typo, or from the plaintext being multiple words without all of them repeated, or even as simple as the same word lacking some suffix like "-s" for plural. That would naturally produce this kind of partial gap pattern.
• It could be a completely different plaintext sequence that happens to produce this partially matching gap pattern. This is somewhat unlikely, but since the pattern only has two repeated letters, the chance of it occurring randomly in any particular place is about 0.01%, and there are close to 1000 opportunities in the ciphertext. The chance of seeing at least one occurrence of this gap pattern in random data is about 13%. (This is why repeats of gap patterns are so important for concluding they are isomorphs).
• The cipher is not perfectly isomorphic, so repeats of the same word can sometimes produce slightly different gap patterns rather than perfect isomorphs, and this could be an instance of the same word with a partial gap pattern.

Research into which types of ciphers are capable of producing isomorphs resulted in the isomorphic cipher hierarchy.

Ciphers which consistently produce isomorphs can usually be solved with some form of alphabet chaining. However, for the more complicated ciphers, this stops being effective because we currently lack a GAK attack don't know any way to take deltas in group autokey ciphers with hidden states.

When text is not isomorphic, and the cipher mechanism is one that consistently produces isomorphs, we can say that the underlying plaintext definitely differs somewhere in that range. These non-isomorphic sections showing differing plaintext are called allomorphs.

For an interactive viewer of isomorphs in the eyes (or in ciphertext you copy in), Tomster12 made this useful web app: Isomorph Viewer.

The scores in this app are the negative log-probabilities of the gap pattern occurring in random data, based on the number of occurrences, number of internal repeats in the gap pattern, and how long the isomorph is (which determines how many opportunities it has to appear in this amount of ciphertext). For example, the top score of 28 means a 10^-28 probability that this could happen by chance.

Based on the number of opportunities, the isomorphs with very low scores are much more likely to just be coincidental. However, a few of these low score gap patterns become more likely to be real isomorphs in a way that is hard to quantify because they occur in similar positions in multiple messages.

Using 1000 opportunities as an approximation (since otherwise it depends slightly on the length of the isomorph):

• For isomorphs with 2 repeated symbols and 2 occurrences, the chance of this happening randomly is about 1%.
• For isomorphs with 2 repeated symbols and 3 occurrences, the chance of this happening randomly is about 0.05%.
• For isomorphs with 2 repeated symbols and 4 occurrences, the chance of this happening randomly is about 0.001%.
• For isomorphs with 3 repeated symbols and 2 occurrences, the chance of this happening randomly is about 0.0001%.
• For isomorphs with 3 repeated symbols and 3 occurrences, the chance of this happening randomly is about 0.0000001%.