Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Callback on include #727

Closed
wants to merge 14 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/capi.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,40 @@ contains the file name and line number where the error or warning occurs.
you're using :c:func:`yr_compiler_add_string`. The ``user_data`` pointer is the
same you passed to :c:func:`yr_compiler_set_callback`.

By default, for rules containing references to other files
(``include "filename.yara"``), yara will try to find those files on disk.
However, if you want to fetch the imported rules from another source (eg: from a
database or remote service), a callback function can be set with
:c:func:`yr_compiler_set_include_callback`.
The callback receives the following parameters:
* ``include_name``: name of the requested file.
* ``calling_rule_filename``: the requesting file name (NULL if not a file).
* ``calling_rule_namespace``: namespace (NULL if undefined).
* ``user_data`` pointer is the same you passed to :c:func:`yr_compiler_set_include_callback`.
It should return the requested file's content as a string. The memory for this
string should be allocated by the callback function. Once it is safe to free the
memory used to return the callback's result, the include_free function passed to
:c:func:`yr_compiler_set_include_callback` will be called. If the memory does
not need to be freed, NULL can be passed as include_free instead.

The callback function has the following prototype:

.. code-block:: c

const char* include_callback(
const char* include_name,
const char* calling_rule_filename,
const char* calling_rule_namespace,
void* user_data);

The free function has the following prototype:

.. code-block:: c

void include_free(
const char* callback_result_ptr,
void* user_data);

After you successfully added some sources you can get the compiled rules
using the :c:func:`yr_compiler_get_rules()` function. You'll get a pointer to
a :c:type:`YR_RULES` structure which can be used to scan your data as
Expand Down Expand Up @@ -402,6 +436,15 @@ Functions
pointer is passed to the callback function.


.. c:function:: void yr_compiler_set_include_callback(YR_COMPILER* compiler, YR_COMPILER_INCLUDE_CALLBACK_FUNC callback, YR_COMPILER_INCLUDE_FREE_FUNC include_free, void* user_data)

Set a callback to provide rules from a custom source when ``include``
directive is invoked. The *user_data* pointer is untouched and passed back to
the callback function and to the free function. Once the callback's result
is no longer needed, the include_free function will be called. If the memory
does not need to be freed, include_free can be set to NULL.


.. c:function:: int yr_compiler_add_file(YR_COMPILER* compiler, FILE* file, const char* namespace, const char* file_name)

Compile rules from a *file*. Rules are put into the specified *namespace*,
Expand Down Expand Up @@ -668,6 +711,7 @@ Functions
Enables the specified rule. After being disabled with :c:func:`yr_rule_disable`
a rule can be enabled again by using this function.


Error codes
-----------

Expand Down
30 changes: 30 additions & 0 deletions docs/yarapython.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,36 @@ should be accepted in the source files, for example:
If the source file contains include directives the previous line would raise
an exception.

If includes are used, a python callback can be set to define a custom source for
the imported files (by default they are read from disk). This callback function
is set through the ``include_callback`` optional parameter.
It receives the following parameters:
*``requested_filename``: file requested with 'include'
*``filename``: file containing the 'include' directive if applicable, else None
*``namespace``: namespace
And returns the requested rules sources as a single string.

.. code-block:: python
import yara
import sys
if sys.version_info >= (3, 0):
import urllib.request as urllib
else:
import urllib as urllib

def mycallback(requested_filename, filename, namespace):
if requested_filename == 'req.yara':
uf = urllib.urlopen('https://pastebin.com/raw/siZ2sMTM')
sources = uf.read()
if sys.version_info >= (3, 0):
sources = str(sources, 'utf-8')
return sources
else:
raise Exception(filename+": Can't fetch "+requested_filename)

rules = yara.compile(source='include "req.yara" rule r{ condition: true }',
include_callback=mycallback)

If you are using external variables in your rules you must define those
external variables either while compiling the rules, or while applying the
rules to some file. To define your variables at the moment of compilation you
Expand Down
118 changes: 100 additions & 18 deletions libyara/compiler.c
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,12 @@ YR_API int yr_compiler_create(

new_compiler->errors = 0;
new_compiler->callback = NULL;
new_compiler->include_callback = _yr_compiler_default_include_callback;
new_compiler->include_free = _yr_compiler_default_include_free;
new_compiler->last_error = ERROR_SUCCESS;
new_compiler->last_error_line = 0;
new_compiler->current_line = 0;
new_compiler->last_result = ERROR_SUCCESS;
new_compiler->file_stack_ptr = 0;
new_compiler->file_name_stack_ptr = 0;
new_compiler->fixup_stack_head = NULL;
new_compiler->allow_includes = 1;
Expand Down Expand Up @@ -182,38 +183,119 @@ YR_API void yr_compiler_set_callback(
}


int _yr_compiler_push_file(
YR_COMPILER* compiler,
FILE* fh)
const char* _yr_compiler_default_include_callback(
const char* include_name,
const char* calling_rule_filename,
const char* calling_rule_namespace,
void* user_data)
{
if (compiler->file_stack_ptr < MAX_INCLUDE_DEPTH)
char* buffer;
char* s = NULL;
#ifdef _WIN32
char* b = NULL;
#endif
char* f;
FILE* fh;
char* file_buffer;
size_t file_length;

if(calling_rule_filename != NULL)
{
compiler->file_stack[compiler->file_stack_ptr] = fh;
compiler->file_stack_ptr++;
return ERROR_SUCCESS;
buffer = (char*) calling_rule_filename;
}
else
{
compiler->last_result = ERROR_INCLUDE_DEPTH_EXCEEDED;
return ERROR_INCLUDE_DEPTH_EXCEEDED;
buffer = "\0";
}

s = strrchr(buffer, '/');

#ifdef _WIN32
b = strrchr(buffer, '\\'); // in Windows both path delimiters are accepted
#endif
#ifdef _WIN32
if (s != NULL || b != NULL)
#else
if (s != NULL)
#endif
{
#ifdef _WIN32
f = (b > s)? (b + 1): (s + 1);
#else
f = s + 1;
#endif
strlcpy(f, include_name, sizeof(buffer) - (f - buffer));
f = buffer;
// SECURITY: Potential for directory traversal here.
fh = fopen(buffer, "rb");
// if include file was not found relative to current source file,
// try to open it with path as specified by user (maybe user wrote
// a full path)
if (fh == NULL)
{
f = (char*) include_name;
// SECURITY: Potential for directory traversal here.
fh = fopen(include_name, "rb");
}
}
else
{
f = (char*) include_name;
// SECURITY: Potential for directory traversal here.
fh = fopen(include_name, "rb");
}
if (fh != NULL)
{
file_buffer = NULL;
file_length = 0;

fseek(fh, 0, SEEK_END);
file_length = ftell(fh);
fseek(fh, 0, SEEK_SET);
file_buffer = (char*) yr_malloc(file_length+1);
if(file_buffer)
{
if(file_length != fread(file_buffer, 1, file_length, fh))
{
return NULL;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a memory leak here. If the file can't be fully read for some reason file_buffer is never freed. Add a yr_free(file_buffer) before the return statement. A call to fclose is also required to prevent leaking file handles.

}
else
{
file_buffer[file_length]='\0';
}
}
fclose(fh);
return file_buffer;
}
else{
return NULL;
}
}


FILE* _yr_compiler_pop_file(
YR_COMPILER* compiler)
void _yr_compiler_default_include_free(
const char* callback_result_ptr,
void* user_data)
{
FILE* result = NULL;

if (compiler->file_stack_ptr > 0)
if(callback_result_ptr != NULL)
{
compiler->file_stack_ptr--;
result = compiler->file_stack[compiler->file_stack_ptr];
yr_free((void*)callback_result_ptr);
}
}

return result;

YR_API void yr_compiler_set_include_callback(
YR_COMPILER* compiler,
YR_COMPILER_INCLUDE_CALLBACK_FUNC include_callback,
YR_COMPILER_INCLUDE_FREE_FUNC include_free,
void* user_data)
{
compiler->include_callback = include_callback;
compiler->include_free = include_free;
compiler->incl_clbk_user_data = user_data;
}


int _yr_compiler_push_file_name(
YR_COMPILER* compiler,
const char* file_name)
Expand Down
42 changes: 31 additions & 11 deletions libyara/include/yara/compiler.h
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,17 @@ typedef void (*YR_COMPILER_CALLBACK_FUNC)(
void* user_data);


typedef const char* (*YR_COMPILER_INCLUDE_CALLBACK_FUNC)(
const char* include_name,
const char* calling_rule_filename,
const char* calling_rule_namespace,
void* user_data);

typedef void (*YR_COMPILER_INCLUDE_FREE_FUNC)(
const char* callback_result_ptr,
void* user_data );


typedef struct _YR_FIXUP
{
void* address;
Expand Down Expand Up @@ -103,9 +114,6 @@ typedef struct _YR_COMPILER
char* file_name_stack[MAX_INCLUDE_DEPTH];
int file_name_stack_ptr;

FILE* file_stack[MAX_INCLUDE_DEPTH];
int file_stack_ptr;

char last_error_extra_info[MAX_COMPILER_ERROR_EXTRA_INFO];

char lex_buf[LEX_BUF_SIZE];
Expand All @@ -114,8 +122,12 @@ typedef struct _YR_COMPILER

char include_base_dir[MAX_PATH];
void* user_data;
void* incl_clbk_user_data;

YR_COMPILER_CALLBACK_FUNC callback;
YR_COMPILER_INCLUDE_CALLBACK_FUNC include_callback;
YR_COMPILER_INCLUDE_FREE_FUNC include_free;


} YR_COMPILER;

Expand All @@ -134,14 +146,6 @@ typedef struct _YR_COMPILER
fmt, __VA_ARGS__);


int _yr_compiler_push_file(
YR_COMPILER* compiler,
FILE* fh);


FILE* _yr_compiler_pop_file(
YR_COMPILER* compiler);


int _yr_compiler_push_file_name(
YR_COMPILER* compiler,
Expand All @@ -151,6 +155,15 @@ int _yr_compiler_push_file_name(
void _yr_compiler_pop_file_name(
YR_COMPILER* compiler);

const char* _yr_compiler_default_include_callback(
const char* include_name,
const char* calling_rule_filename,
const char* calling_rule_namespace,
void* user_data);

void _yr_compiler_default_include_free(
const char* callback_result_ptr,
void* user_data);

YR_API int yr_compiler_create(
YR_COMPILER** compiler);
Expand All @@ -166,6 +179,13 @@ YR_API void yr_compiler_set_callback(
void* user_data);


YR_API void yr_compiler_set_include_callback(
YR_COMPILER* compiler,
YR_COMPILER_INCLUDE_CALLBACK_FUNC include_callback,
YR_COMPILER_INCLUDE_FREE_FUNC include_free,
void* user_data);


YR_API int yr_compiler_add_file(
YR_COMPILER* compiler,
FILE* rules_file,
Expand Down