lua-arbitrary-binary-string

A Lua tool/library to convert arbitrary binary data into a Lua string literal with various levels of printability.

As A Command Line Tool

Usage: lua main.lua [flags] [input_filepath] [output_filepath]

Arguments:
  input_filepath           if not specified, stdin is used
  output_filepath          if not specified, stdout is used

Flags:
  --help                   print this help screen and exit
  --escape-minimal         [default] escape only the chars necessary (leave most control chars unescaped)
  --escape-control-chars   escape all control chars (leave extended ascii unescaped)
  --escape-all-nonascii    escape all non-ascii chars (only leave printable ascii unescaped)

Example

For the purposes of this example, example.dat is a binary file with arbitrary data.

lua main.lua example.dat example.lua

example.lua will now be a file that can be loaded by Lua and will return the data contained in example.dat as a string:

-- load from the encoded string
local example_data_lua = require('example')

-- read from the original file
local example_data_dat = io.open('example.dat', 'rb'):read("*all")

-- they will match
assert(example_data_lua == example_data_dat)

Output / Options

Each 'level' of escaping presents a trade-off between printability (i.e. being able to embed the result in a .lua file without making text editors think its a binary file) and size (the more characters that are escaped, the larger the encoded string will be).

Here is a table showing an overview of the options:

Escape Option	Expected Encoded Size Increase	Printability
ESCAPE_MINIMAL (default)	~103%	Contains unescaped embedded NUL and control characters; almost guaranteed to make text editors treat the file as a binary file
ESCAPE_CONTROL_CHARS	~135%	Contains unescaped extended ASCII characters, which may have printable representations in text editors (but may look strange)
ESCAPE_ALL_NONASCII	~275%	Escapes everything except standard printable ASCII characters. Will be printable in all text editors, but greatly inflates the size of the embedded data.

As A Library

See arbitrary-binary-string.lua. I haven't documented it/made it easy to use because I don't think it has much utility as a library.

Running the tests

• To run a basic sanity check: lua test/test.lua
• To run the fuzz tester indefinitely (until a failing case is found): lua test/fuzz.lua
• To run the fuzz tester a certain number of iterations: lua test/fuzz.lua <limit> e.g. lua test/fuzz.lua 10000