spicy-build
is a shell frontend that compiles Spicy source code into a standalone executable by running spicyc
to generate the necessary C++ code, then spawning the system compiler to compile and link that.
usage-spicy-build
spicy-config
reports information about Spicy's build & installation options.
usage-spicy-config
spicyc
compiles Spicy code into C++ output, optionally also executing it directly through JIT.
usage-spicyc
spicyc
also supports the following environment variables to control the compilation process:
SPICY_PATH
Replaces the built-in search path for *.spicy source files.
SPICY_CACHE
Location for storing precompiled C++ headers. Default is
~/.cache/spicy/<VERSION>
.HILTI_CXX
Specifies the path to the C++ compiler to use.
HILTI_CXX_INCLUDE_DIRS
Specified additional, colon-separated C++ include directory to search for header files.
HILTI_JIT_SEQUENTIAL
Set to prevent spawning multiple concurrent C++ compiler instances.
HILTI_OPTIMIZER_PASSES
Colon-separated list of optimizer passes to activate. If unset uses the default-enabled set.
HILTI_PATH
Replaces the built-in search path for *.hlt source files.
HILTI_PRINT_SETTINGS
Set to see summary of compilation options.
spicy-driver
is a standalone Spicy host application that compiles and executes Spicy parsers on the fly, and then feeds them data for parsing from standard input.
usage-spicy-driver
spicy-driver
supports the same environment variables as spicyc
.
If there's only single public
unit in the Spicy source code, spicy-driver
will automatically use that for parsing its input. If there's more than one public unit, you need to tell spicy-driver
which one to use through its --parser
(or -p
) option. To see the parsers that are available, use --list-parsers
(or -l
).
In addition to the names shown by --list-parsers
, you can also specify a parser through a port or MIME type if the corresponding unit defines them through properties <unit_meta_data>
. For example, if a unit defines %port = 80/tcp
, you can use spicy-driver -p 80/tcp
to select it. To specify a direction, add either %orig
or %resp
(e.g., -p 80/tcp%resp
); then only units with a port tagged with an &originator
or &responder
attribute, respectively, will be considered. If a unit defines %mime-type = application/test
, you can select it through spicy-driver -p application/test
. (Note that there must be exactly one unit with a matching property for this all to work, otherwise you'll get an error message.)
spicy-driver
provides a batch input mode for processing multiple interleaved input flows in parallel, mimicking how host applications like Zeek would be employing Spicy parsers for processing many sessions concurrently. The batch input must be prepared in a specific format (see below) that provides embedded meta information about the contained flows of input. The easiest way to generate such a batch is a Zeek script coming with Spicy
</_static/record-spicy-batch.zeek>
. If you run Zeek with this script on a PCAP trace, it will record the contained TCP and UDP sessions into a Spicy batch file:
# zeek -b -r http/methods.trace record-spicy-batch.zeek
tracking [orig_h=128.2.6.136, orig_p=46562/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46563/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46564/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46565/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46566/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46567/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
[...]
tracking [orig_h=128.2.6.136, orig_p=46608/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46609/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
tracking [orig_h=128.2.6.136, orig_p=46610/tcp, resp_h=173.194.75.103, resp_p=80/tcp]
recorded 49 sessions total
output in batch.dat
You will now have a file batch.dat
that you can use with spicy-driver -F batch.data ...
.
The batch created by the Zeek script will select parsers for the contained sessions through well-known ports. That means your units need to have a %port
property matching the responder port of the sessions you want them to parse. So for the HTTP trace above, our Spicy source code would need to provide a public unit with property %port = 80/tcp;
.
In case you want to create batches yourself, we document the batch format in the following. A batch needs to start with a line !spicy-batch v2<NL>
, followed by lines with commands of the form @<tag> <arguments><NL>
.
There are two types of input that the batch format can represent: (1) individual, uni-directional flows; and (2) bi-directional connections consisting in turn of one flow per side. The type is determined through an initial command: @begin-flow
starts a flow flow, and @begin-conn
starts a connection. Either form introduces a unique, free-form ID that subsequent commands will then refer to. The following commands are supported:
@begin-flow FID TYPE PARSER<NL>
Initializes a new input flow for parsing, associating the unique ID
FID
with it.TYPE
must be eitherstream
for stream-based parsing (think: TCP), orblock
for parsing each data block independent of others (think: UDP).PARSER
is the name of the Spicy parser to use for parsing this input flow, given in the same form as withspicy-driver
's--parser
option (i.e., either as a unit name, a%port
, or a%mime-type
).@begin-conn CID TYPE ORIG_FID ORIG_PARSER RESP_FID RESP_PARSER<NL>
Initializes a new input connection for parsing, associating the unique connection ID
CID
with it.TYPE
must be eitherstream
for stream-based parsing (think: TCP), orblock
for parsing each data block independent of others (think: UDP).ORIG_FID
is separate unique ID for the originator-side flow, andORIG_PARSER
is the name of the Spicy parser to use for parsing that flow.RESP_FID
andRESP_PARSER
work accordingly for the responder-side flow. The parsers can be given in the same form as withspicy-driver
's--parser
option (i.e., either as a unit name, a%port
, or a%mime-type
).@data FID SIZE<NL>
A block of data for the input flow
FID
. This command must be followed directly by binary data of lengthSIZE
, plus a final newline character. The data represents the next chunk of input for the corresponding flow.@data
can be used only inside corresponding@begin-*
and@end-*
commands bracketing the flow ID.@end-flow FID<NL>
Finalizes parsing of the input flow associated with
FID
, releasing all state. This must come only after a corresponding@begin-flow
command, and every@begin-flow
must eventually be followed by an@end-flow
.@end-conn CID<NL>
Finalizes parsing the input connection associated with
CID
, releasing all state (including for its two flows). This must come only after a corresponding@begin-conn
command, and every@begin-conn
must eventually be followed by an@end-end
.
spicy-dump
is a standalone Spicy host application that compiles and executes Spicy parsers on the fly, feeds them data for proessing, and then at the end prints out the parsed information in either a readable, custom ASCII format, or as JSON (--json
or -J
). By default, spicy-dump
disables showing the output of Spicy print
statements, --enable-print
or -P
reenables that.
usage-spicy-dump