Usage data sanitization

[curusarn]

My idea is to replace sensitive info with placeholders.

It's important to make sure that the same piece of information is always replaced with the same placeholder.

Replace sensitive info with hashes.

[curusarn]

paths -> replace dirs except for common ones
git url -> replace url, replace dirs of path

replace parts of arguments if possible:
scp user@server.dev:path/path -> scp HASH@HASH:HASH/HASH

replace whole arguments if they don't match any special form
git commit -m "message" -> git commit -m HASH

keep standard arguments and commands (?)

[curusarn]

Almost done: https://github.com/curusarn/resh/tree/dev_2

[curusarn]

I'm handling different types of data differently.

Types

Single value entries

e.g. username, hostname (usually sensitive information)

1. replace with its hash
   • no exceptions, no whitelist

Paths

1. split by /
2. repace each part by its hash
   • unless it's in the whitelist or it's only one character
3. append together

Git origin URL

1. parse the URL using this library https://github.com/whilp/git-urls
2. repace each part by its hash
   • unless it's in the whitelist or it's only one character long
3. get a string of sanitized URL

Command line

I need to replace the command and arguments separately so that I can analyze partial matches later.
However, I don't want to parse bash.
I'm doing the following:

1. split the line into consecutive strings of letters and/or digits (tokens)
   • command options are detected and left unhashed
2. replace each token with its hash
   • unless it's in the whitelist or it's only one character long
3. append together

Whitelisting

I created a whitelist containing various common strings.

• directories in /
• commands installed by default on Ubuntu, Debian or Fedora
• bash and zsh keywords and builtins
• file-extensions
• git subcommands
• some more stuff added by hand:
  • "com", "cz", ...
  • "vim", "emacs", ...
  • "Makefile", "Dockerfile", ...
  • ...

TL;DR

I pretty much hash everything except:

• Commandline options
• All non-alphanumeric chars
• Single-letter or single-digit strings
• Anything whitelisted (see above^)

[curusarn]

• TODO: add more file extensions to the whitelist

• TODO: show this to people

[curusarn]

I have shown this to 3 of my colleagues. Everyone was okay with the result. I got a suggestion that data is sanitized too much.

[curusarn]

I have found a couple of file extension databases. However, they don't seem very fond of other people using their data. I have asked FileInfo.com for permission to use their data.

• WAIT for fileinfo.com to message you back

[curusarn]

I have added a few of common TLDs to the list. Source: https://www.hayksaakian.com/most-popular-tlds/