New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[preview, wip] lua output v15.2 #902

Closed
wants to merge 31 commits into
base: master
from

Conversation

1 participant
@inliniac
Member

inliniac commented Mar 20, 2014

Continuation of #901. Now passes prscript and builds on OpenBSD and FreeBSD.

Prscript:

inliniac added some commits Mar 13, 2014

lua: support regular lua C library
Not all systems have luajit or a need for luajit. For low bandwidth
and offline support regular lua may be sufficient.
output: Lua HTTP log initial implementation
Initial version of a HTTP LUA logger. Execute lua scripts from the
Tx-log API.
output: add submodules list to OutputCtx
Add a list to the OutputCtx that can contain OutputModule structures.
This will be used by a 'parent' module to register submodules directly.
output-lua: display warning if no lua support
Display a warning that the lua module is not available if we're
not compiled against lua(jit).
output-lua: support submodules
Use the OutputCtx::submodules list to register additional log modules.
Currently this is hardcoded to the 'lua' module.
output-lua: new file for common functions
Add output-lua-common.[ch] to store functions common to various parts
of the lua output framework.
output-lua: packet logger support
Through 'needs' the script init function can indicate it wants to
see packets and select a condition function. Currently only alerts
is an option:

    function init (args)
        local needs = {}
        needs["type"] = "packet"
        needs["filter"] = "alerts"
        return needs
    end
output-lua: add example packet log script
Example packet log script that outputs to stdout in the alert-
fast log format.
output-lua: move LuaPrintStack to common
It's a utility function that will be used in several other places
as well.
lua/luajit: use HAVE_LUA mostly
Only use HAVE_LUAJIT if things are done differently from HAVE_LUA,
like in the states pool.
lua: deal with FreeBSD and OpenBSD
FreeBSD pkg-config lua-5.1.pc, lib liblua-5.1.so
OpenBSD pkg-config lua51.pc, lib liblua5.1.so

Default (linux) pkg-config: lua5.1.pc, lib liblua5.1.so
output-lua: initial packet support
Add key for storing packet pointer in the lua stack and a utility
function to retrieve it from lua callbacks.
output-lua: add stack utility functions
Add utility functions for placing things on the stack for use
by the scripts. Functions for numbers, strings and byte arrays.

Add callback for returing IP header info: ip version, src ip,
dst ip, proto, sp, dp (or type and code for icmp and icmpv6):
SCPacketTuple
output-lua: rename LuaPacketLogger to ..Alerts
As the script is called for each alert, not for each packet, name
the script LuaPacketLoggerAlerts.
output-lua: support File logging
Add file logger support. The script uses:

function init (args)
    local needs = {}
    needs['type'] = 'file'
    return needs
end

The type is set to file to make it a file logger.
output-lua: register common callbacks
Clean up callback registration in the setup-stage and register
common callbacks.
output-lua: improve error handling and documentation
Better document the various functions and improve error handling.
output-lua: TxLogger use proper stack function
Use proper wrapper to setup the stack.
output-lua: add SCLogPath callback
Add a lua callback for getting Suricata's log path, so that lua scripts
can easily get the logging directory Suricata uses.

Update the Setup logic to register callbacks before the scripts 'setup'
is called.

Example:

    name = "fast_lua.log"
    function setup (args)
        filename = SCLogPath() .. "/" .. name
        file = assert(io.open(filename, "a"))
    end
output-lua: make packet ptr available to all scripts
TxLogger and Packet logger need it to be able to use the Tuple
callback.
output-log: expose SCLog functions to lua scripts
The lua scripts can use SCLogDebug, SCLogInfo, SCLogNotice, SCLogWarning,
SCLogError. The latter 2 won't be able to add an error code though.
output-lua: improve error checking for init()
If init doesn't properly init the script, skip the script and error
out.
output-lua: add all packets logging support
If the script needing a packet doesn't specify a filter, it will
be run against all packets. This patch adds the support for this
mode. It is a packet logger with a condition function that always
returns true.
lua: introduce util-lua.[ch]
Shared functions for all lua parts of the engine.
lua: add flow store and retrieval wrappers
Add flow store and retrieval wrappers for accessing the flow through
Lua's lightuserdata method.

The flow functions store/retrieve a lock hint as well.
detect-lua: convert extensions to use flow wrappers
Use the new flow wrapper functions in the lua flowvar and flowint
extensions.
lua: add SCFlowTuple lua function
Like SCPacketTuple, only retrieves Tuple from the flow.

Minimal log function:

    function log(args)
        ipver, srcip, dstip, proto, sp, dp = SCFlowTuple()
        print ("Flow IPv" .. ipver .. " src " .. srcip .. " dst " .. dstip ..
               " proto " .. proto .. " sp " .. sp .. " dp " .. dp)
    end
output-lua: rule info callback
SCRuleIds(): returns sid, rev, gid:

    function log(args)
        sid, rev, gid = SCRuleIds()

SCRuleMsg(): returns msg

    function log(args)
        msg = SCRuleMsg()

SCRuleClass(): returns class msg and prio:

    function log(args)
        class, prio = SCRuleClass()
        if class == nil then
            class = "unknown"
        end
output-lua: add SCPacketTimeString
Add SCPacketTimeString to get the packets time string in the format:
    11/24/2009-18:57:25.179869

Example use:

    function log(args)
        ts = SCPacketTimeString()
output-lua: add file callbacks
SCFileInfo: returns fileid (number), txid (number), name (string),
            size (number), magic (string), md5 in hex (string)

Example:

    function log(args)
        fileid, txid, name, size, magic, md5 = SCFileInfo()

SCFileState: returns state (string), stored (bool)

Example:
    function log(args)
        state, stored = SCFileState()
output-lua: expose flow start time string
SCFlowTimeString: returns string form of start time of a flow

Example:

    function log(args)
        startts = SCFlowTimeString()
        ts = SCPacketTimeString()
        if ts == startts then
            print("new flow")
        end

@inliniac inliniac closed this Mar 27, 2014

@inliniac inliniac deleted the dev-lua-output-v15.2 branch Mar 27, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment