Log api v4.1 #791
Closed
Log api v4.1 #791
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch introduces a new API for outputs that log based on the
packet, such as alert outputs. In converts fast-log to the new API.
The API gets rid of the concept of each logger being a thread module,
but instead there is one thread module that runs all packet loggers.
Through the registration function OutputRegisterPacketModule a log
module can register itself to be considered for each packet.
Each logger registers itself to this new API with 2 functions and the
OutputCtx object that was already used in the old implementation.
The function pointers are:
LogFunc: the log function
ConditionFunc: this function is called before the LogFunc and only
if this returns TRUE the LogFunc is called.
For a simple alert logger like fast-log, the condition function will
simply return TRUE if p->alerts.cnt > 0.
Convert unified2 alert to new logging API.
This patch introduces a new API for logging transactions from tx-aware app layer protocols. It runs all the registered loggers from a single thread module. This thread module takes care of the transaction handling and flow locking. The logger just gets a transaction to log out. All loggers for a protocol will be run at the same time, so there will not be any timing differences. Loggers will no longer act as Thread Modules in the strictest sense. The Func is NULL, and SetupOuputs no longer attaches them to the thread module chain individually. Instead, after registering through OutputRegisterTxModule, the setup data is used in the single logging module. The logger (LogFunc) is called for each transaction once, at the end of the transaction.
Convert the HTTP log to the new TX logging API.
Make sure to use the new logger TX API. For this the transaction handling was improved as well.
Make all functions static. Remove separate ipv4 and ipv6 registration functions. Move register function to the bottom so that we no longer need function prototype declarations.
This patch converts log-tls to use the packet logger API. The packet logger API was choosen as the TLS parser is not transaction aware. To make sure the state is only logged once, the flag SSL_AL_FLAG_STATE_LOGGED was added to the parser. This flag is checked by the condition function, and set at the end of the Logger function.
Make all funcs but registration static. Remove stale registation prototypes. Move registation func to the bottom.
Convert AlertDebugLog to Packet logger API. Convert packet args to const.
Clean up log functions after packet logger conversion. No more PacketQueue arguments.
Make functions static. Move registration to the bottom.
Convert pcap-info to use the packet logger API.
Remove separate ipv4 and ipv6 registration functions. Make all functions static. Move registration function to the bottom. Simplify OS_WIN32 wrappers usage.
Convert Syslog alert logger to packet logger API.
Fixes configure enabling of prelude. CFLAGS is reset, so the previous adding of -DPRELUDE was nixed. Using AC_DEFINE now. Cleanups: - make functions static - simplify handling of no prelude support - move registration to the bottom
Convert prelude logger to use the packet logger API.
Make all functions static. Move registration to the bottom.
This patch introduces a new logging API for logging extracted file info.
It allows for registration of a callback that is called once per file:
when it's considered 'closed'.
Users of this API register their Log Function through:
OutputRegisterFileModule()
The API uses a magic settings globally. This might be changed later.
Use file logger API. Also, check if the protocol is HTTP before getting the HTTP fields.
The new API call:
int AppLayerParserProtocolHasLogger(uint8_t ipproto,
AppProto alproto)
Returns TRUE if a logger is registered on the ip/alproto pair, and
FALSE otherwise.
By bailing out early in case no logger is enabled for the protocol, a significant speed up is reached.
Tag truncated files as truncated in the same way log-file does.
Remove unused code. Make functions static. Move registration to the bottom.
A new logger API for registering file storage handlers. Where the
FileLog handler is called once per file, this handler will be called
for each data chunk so that storing the entire file is possible.
The logger call in the API is as follows:
typedef int (*FiledataLogger)(ThreadVars *, void *thread_data,
const Packet *, const File *, const FileData *, uint8_t flags);
All data is const, thus should be read only. The final flags field
is used to indicate to the caller that the file is new, or if it's
being closed.
Files use an internal unique id 'file_id' which can be used by the
loggers to create unique file names. This id can use the 'waldo'
feature of the log-filestore module. This patch moves that waldo
loading and storing logic to this API's implementation. A new
configuration directive 'file-store-waldo: <filename>' is added,
but the existing waldo settings will also continue to work.
This patch converts the log-filestore module to use the new FiledataLog API.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rebase of #777
Merging this tomorrow unless there are major concerns. Further updates will be incremental. cc @decanio @regit @poona @ken-tilera @jasonish
There are some open issues, wrt to the setup code as pointed out by @regit. Also, I see room for some optimization wrt ip string creation and such.
Prscript: