Detect and ignore hashes

[epage]

For example, #269 has

"hash": "kqul2pqqzcpobrd6fh6ect7d3sxlvvks",

Though I wonder what other formats exist to detect

[vergoh]

Another example is pretty much any base64 encoded content used in files as variables. Such strings end up causing false positives rather easily. Simple way to test (in empty directory):

ls /bin | while read f ; do base64 /bin/$f >$f ; done
typos .

I suspect it could help to have some maximum length for strings that are interpreted as words that get checked.

[epage]

• uuid's should be lower-case hex xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx, see https://en.wikipedia.org/wiki/Universally_unique_identifier
• Git hashes can be ambiguously short to 40 characters long (or more with newhash), lower case hex
• base64 is multiples of 4, padded with =, and uses 0-9, a-z, A-Z, +, /
  • Variants don't require padding and use ,, -, _ as alternative symbols
  • ipynb is standard format, with padding

Challenges

• UUIDs could look like var1 - var2 - var3 - var4
• hashes could look like a variable name
• base64 could look like var1 + var2 / var3 =
  • This is a similar challenge to detecting emails since @ can be an operator in some languages
• Ignoring the separator can look indiscernible from multiple variables and it can subtly break, depending on the data (+ and / can easily change position, changing the length of "words")
• If we use length as a determining factor, we have to consider what the longest Java class name might be...
  • Most ridiculous class name lengths are in the mid 90s (InternalFrameInternalFrameTitlePaneInternalFrameTitlePaneMaximizeButtonPainter from the JDK and HasThisTypePatternTriedToSneakInSomeGenericOrParameterizedTypePatternMatchingStuffAnywhereVisitor from AspectJ)

hex values are least likely to contain words and base64 i most likely, so I assume that should be the priority.