Find some way to tolerate hexadecimal better

[nmathewson]

I have a program that uses a fair amount of hex strings in order to represent binary sequences in its tests. But this doesn't sit so well with typos, which gives lots of false positives like:

error: `ba` should be `by`, `be`
  --> ./tor-llcrypto/src/pk.rs:77:21
   |
77 |              0816ed13ba3303ac5deb911548908025"

It seems that this is a byproduct of treading any sequence of non-numeric wordlike characters as a possible word, even if they occur adjacent to numbers, not specified by spaces.

[epage]

0816ed13ba3303ac5deb911548908025"

Is there anything that can help identify these as hex?

For example, we exclude

• 0x<hexdigit>
• UUIDs
• Hashes between 40 (sha1) and 64 (sha256) characters
• Base64 that is at least 90 characters long and contain the appropriate padding

We could exclude number prefixed but that will only help in 10/16 cases.

It seems that this is a byproduct of treading any sequence of non-numeric wordlike characters as a possible word, even if they occur adjacent to numbers, not specified by spaces.

This is because we are focused on spell checking code and want to get all words within identifiers, regardless of whether they split on case (e.g. PascalCase) or not (e.g. snake_case)

[nmathewson]

Hashes between 40 (sha1) and 64 (sha256) characters

Thinking aloud:

How about "strings of at least 32 characters containing only hexadecimal characters"?

Alternatively, excluding r"xxx" strings would probably work for me, and might make regex-users happier. (Though it might make somebody else unhappy.)

Alternatively, "strings within hex!()" would work here, though that is probably less general.

In the most general case, it might be okay to have a way to annotate something as "not text".

These would all work for my use case; do you think any of them are clean enough to use in practice?

[epage]

How about "strings of at least 32 characters containing only hexadecimal characters"?

While its unlikely for someone to spell out an identifier 32 characters long with only hex digits, I don't think its impossible. I guess the question is how likely it would happen since we do err on the side of not spell checking vs reporting false positives.

In the most general case, it might be okay to have a way to annotate something as "not text".

See #316

[nmathewson]

While its unlikely for someone to spell out an identifier 32 characters long with only hex digits, I don't think its impossible. I guess the question is how likely it would happen since we do err on the side of not spell checking vs reporting false positives.

I think this is rare in practice. To try to measure this empirically, I used this ripgrep command pipe:

rg -I -o '[a-fA-F0-9]{32,}' | sort |uniq > all_hex_strings

I then ran this command against my ~/src directory (where I put software that I work on), and found 105204 distinct hex strings. That's too many to scan by hand, so I took a random sample of 1000. I then read all of those 1000 strings, and did not find any that were real text.

I reproduced this same result in my ~/build directory (where I put software that I build but do not work on) and got the same results. (Sampled 1000 strings out of 189683 and found none with real text.)

So I think that this rule is probably safe? You might want to double-check it on our own collection of software, though, to be sure.

[nmathewson]

Friendly ping. If you agree with my analysis above, I'm happy to try to write a patch here. Alternatively, if the analysis isn't convincing, I'd be happy to re-run a similar experiment with a corpus, or sample size of your choosing.

[epage]

I think restricting ourselves to 32 same-case hex characters would work.