Log API v4.2 #797

merged 27 commits into from Jan 27, 2014


None yet

1 participant

added some commits Dec 5, 2013
@inliniac Introduce packet logging output API
This patch introduces a new API for outputs that log based on the
packet, such as alert outputs. In converts fast-log to the new API.

The API gets rid of the concept of each logger being a thread module,
but instead there is one thread module that runs all packet loggers.
Through the registration function OutputRegisterPacketModule a log
module can register itself to be considered for each packet.

Each logger registers itself to this new API with 2 functions and the
OutputCtx object that was already used in the old implementation.
The function pointers are:

LogFunc:       the log function

ConditionFunc: this function is called before the LogFunc and only
               if this returns TRUE the LogFunc is called.

For a simple alert logger like fast-log, the condition function will
simply return TRUE if p->alerts.cnt > 0.
@inliniac Packet logging API: convert unified2
Convert unified2 alert to new logging API.
@inliniac Introduce TX logging API
This patch introduces a new API for logging transactions from
tx-aware app layer protocols. It runs all the registered loggers
from a single thread module. This thread module takes care of the
transaction handling and flow locking. The logger just gets a
transaction to log out.

All loggers for a protocol will be run at the same time, so there
will not be any timing differences.

Loggers will no longer act as Thread Modules in the strictest sense.
The Func is NULL, and SetupOuputs no longer attaches them to the
thread module chain individually. Instead, after registering through
OutputRegisterTxModule, the setup data is used in the single logging

The logger (LogFunc) is called for each transaction once, at the end
of the transaction.
@inliniac TX logging API: convert HTTP log
Convert the HTTP log to the new TX logging API.
@inliniac Convert log-drop to packet logger api. 35aa6c1
@inliniac dns: convert dns logger to TX logger API
Make sure to use the new logger TX API. For this the transaction
handling was improved as well.
@inliniac log-tls: clean ups
Make all functions static. Remove separate ipv4 and ipv6 registration
functions. Move register function to the bottom so that we no longer
need function prototype declarations.
@inliniac log-tls: convert to packet logger API
This patch converts log-tls to use the packet logger API. The packet
logger API was choosen as the TLS parser is not transaction aware.

To make sure the state is only logged once, the flag
SSL_AL_FLAG_STATE_LOGGED was added to the parser. This flag is checked
by the condition function, and set at the end of the Logger function.
@inliniac alert-debug log cleanups
Make all funcs but registration static.
Remove stale registation prototypes.
Move registation func to the bottom.
@inliniac alert-debuglog: port to packet logger api
Convert AlertDebugLog to Packet logger API. Convert packet args to
@inliniac alert-debuglog: minor cleanups
Clean up log functions after packet logger conversion. No more
PacketQueue arguments.
@inliniac alert-pcapinfo: clean up
Make functions static.
Move registration to the bottom.
@inliniac alert-pcapinfo: convert to packet logger API
Convert pcap-info to use the packet logger API.
@inliniac alert-syslog: cleanup
Remove separate ipv4 and ipv6 registration functions.
Make all functions static.
Move registration function to the bottom.
Simplify OS_WIN32 wrappers usage.
@inliniac alert-syslog: convert to packet logger API
Convert Syslog alert logger to packet logger API.
@inliniac prelude: fix configure and cleanup
Fixes configure enabling of prelude. CFLAGS is reset, so the previous
adding of -DPRELUDE was nixed. Using AC_DEFINE now.

- make functions static
- simplify handling of no prelude support
- move registration to the bottom
@inliniac prelude: convert to packet logger API
Convert prelude logger to use the packet logger API.
@inliniac log-file: cleanups
Make all functions static.
Move registration to the bottom.
@inliniac Introduce 'file' logging API
This patch introduces a new logging API for logging extracted file info.
It allows for registration of a callback that is called once per file:
when it's considered 'closed'.

Users of this API register their Log Function through:

The API uses a magic settings globally. This might be changed later.
@inliniac log-file: convert to file-logger API
Use file logger API.

Also, check if the protocol is HTTP before getting the HTTP
@inliniac profiling: add logger api labels 4c024f9
@inliniac app-layer: add logger check to API
The new API call:
    int AppLayerParserProtocolHasLogger(uint8_t ipproto,
                                        AppProto alproto)

Returns TRUE if a logger is registered on the ip/alproto pair, and
FALSE otherwise.
@inliniac tx-logger: speed up
By bailing out early in case no logger is enabled for the protocol,
a significant speed up is reached.
@inliniac log-filestore: tag truncated files as such
Tag truncated files as truncated in the same way log-file does.
@inliniac log-filestore: cleanups
Remove unused code.
Make functions static.
Move registration to the bottom.
@inliniac Introduce Filedata Logger API
A new logger API for registering file storage handlers. Where the
FileLog handler is called once per file, this handler will be called
for each data chunk so that storing the entire file is possible.

The logger call in the API is as follows:
    typedef int (*FiledataLogger)(ThreadVars *, void *thread_data,
        const Packet *, const File *, const FileData *, uint8_t flags);

All data is const, thus should be read only. The final flags field
is used to indicate to the caller that the file is new, or if it's
being closed.

Files use an internal unique id 'file_id' which can be used by the
loggers to create unique file names. This id can use the 'waldo'
feature of the log-filestore module. This patch moves that waldo
loading and storing logic to this API's implementation. A new
configuration directive 'file-store-waldo: <filename>' is added,
but the existing waldo settings will also continue to work.
@inliniac log-filestore: convert to FiledataLog API
This patch converts the log-filestore module to use the new
FiledataLog API.
@inliniac inliniac merged commit b27d03e into master Jan 27, 2014
@inliniac inliniac deleted the dev-log-api-v4.2 branch Jan 31, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment