pip install pnu-ngc
ngc - n-grams count
ngc [-b|--block] [-c|--convert ARGS] [-d|--discard ARGS] [-l|--length ARG] [-p|--partial ARG] [-q|--quiet] [-s|--summary] [-t|--text] [-w|--word] [--debug] [--help|-?] [--version] [--] [filename ...]
The ngc utility is used for counting the number of occurrences and computing the frequency of n-grams in cryptanalysis.
For n=1, the n-gram is simply a letter or character. For n=2, the n-gram is called a bigram or digraph. And so on...
The -l option is for setting the length of the n-gram (the default is 1 character), and the -b option for using a fixed-window instead of the default sliding one ("ABCD" giving "AB" and "CD" instead of "AB", "BC" and "CD").
The -c option is used to perform some prior conversions on your input data: Unicode characters removal (especially accented ones), upper to lower case conversions (or the reverse), extra spaces removal (this last one is performed after other conversions).
You can also use the -d option to discard selected categories of characters (for example if you only want to keep letters).
If you want to check your input data after these transformations, you can use the -t option to print it.
If you want to print only this, you can use the -q option.
And if you want to print some stats on the remaining characters, you can print a summary with the -s option. This summary also includes the coincidence index of your input text.
Finally, you can use the -w option to process your input word by word instead of line by line. If you selected the fixed-window -b option, you can decide what to do with partial blocks with the -p option: between keeping them as-is, discarding them, or filling them with spaces in order to have only n-grams of the same length.
The ngc utility processes all the indicated file names as one file. If none are provided, it processes the standard input, thus behaving as a filter.
| Options | Use |
|---|---|
| -b|--block | Use fixed- instead of sliding-windows blocks |
| -c|--convert ARGS | Convert text input. A combination of: |
| a / Unicode characters to ASCII (remove accents) | |
| l / Upper case letters to lower | |
| u / Lower case letters to upper | |
| s / Spaces-like characters to 1 space | |
| Warning: l and u can't be used at the same time | |
| -d|--discard ARGS | Discard characters. A combination of: |
| U / Unicode characters | |
| u / Upper case letters | |
| l / Lower case letters | |
| L / All letters | |
| c / Connection symbols (apostrophe and hyphen) | |
| d / Digits | |
| p / Punctuation (.,;:?!) | |
| o / Other printable symbols | |
| s / Spaces (space, tab, return, formfeed, vtab) | |
| n / Non printable control characters | |
| -l|--length ARG | Length of the n-gram. Defaults to 1 |
| -p|--partial ARGS | What to do with partial blocks? One among: |
| d / Discard | |
| k / Keep as-is (default) | |
| j / Keep but right-justify with spaces | |
| -q|--quiet | Don't show occurrences and frequency by n-gram |
| -s|--summary | Show a summary of what was processed |
| -t|--text | Show modified text input |
| -w|--word | Respect Word boundaries (delimited by spaces) |
| --debug | Enable debug mode |
| --help|-? | Print usage and a short help message and exit |
| --version | Print version and exit |
| -- | Options processing terminator |
The NGC_DEBUG environment variable can also be set to any value to enable debug mode.
The ngc utility exits 0 on success, and >0 if an error occurs.
wc(1), caesar(1), Frequency analysis, Index of coincidence
The ngc utility is not a standard UNIX/POSIX command.
It tries to follow the PEP 8 style guide for Python code.
This utility was made for The PNU project, while playing with a reimplementation of the caesar(1) utility.
This utility is available under the 3-clause BSD license.