Add Support for Token Dictionaries

[vfsrfs]

This PR adds support for dictionaries containing interesting keywords (tokens) that are useful for the mutation of inputs while fuzzing, particularly, when fuzzing syntax-aware programs (dvyukov#174). This modification allows to provide the -dict flag to go-fuzz, so that the user can provide a dictionary file with useful tokens for the fuzzing campaign. E.g.:

-dict /path/dictionary.dict

The tokens parsed from the dictionary are stored in ROData.strLits, as those are the string literals that are used by the mutator engine when generating new fuzzing inputs.

The dictionary format that is accepted by the -dict flag is the same that is used by AFL/Libfuzzer (see https://github.com/google/AFL/tree/master/dictionaries).

This dictionary format defines that there is one token per line. Every line consists of a name followed by an equal sign and the token in quotes (e.g. name=”token”). It is also possible to define binary sequences by providing the values in hex (e.g. \xNN) within the token. To insert a backslash or a double quote within the token, it has to be escaped using a backslash (e.g. \\ or \”). \n and \t are recognized as well, since they might be useful for text-based protocols. Other problematic characters can be added by providing its hex value.

To make this implementation fully compatible with AFL/Libfuzzer’s dictionaries, token levels are supported. A level can be appended to every token, by appending @<num> to the keyword, e.g.
keyword@1=”token”

These tokens will be loaded only, if the dictionary level is equal to or greater than the specified number. The default dictionary level is 0, but it can be increased by appending @<num> to the dictionary path. E.g.:

-dict /path/dictionary.dict@1

[CLAassistant]

All committers have signed the CLA.