New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

F/python #387

Merged

lbudai merged 30 commits into master from f/python

Feb 23, 2015

Collaborator

bazsi commented Feb 17, 2015

Please consider my Python related work for merging. It ports the Python module in the incubator to the syslog-ng proper.

It's not yet complete, but I intended to submit the functional parts to make it more accessible for everyone else.

I still have upcoming work that I've left out of this branch (e.g. template function support and "root" level Python language support), however those still have critical issues.

This one has much less test coverage than I would prefer (e.g. exactly None), at least an end-to-end test would be great to have. It's really difficult to do unit-like testing as most of it is just glue code.

Anyway, I indend to add some testing to the func-test suite in the nearfuture.

Please merge!
Thanks.

bazsi added the in progress label

bazsi and others added 29 commits

February 19, 2015 10:54


          logthrdestdrv: fix time_reopen initialization

3d554ad

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          cfg.c: create separate method for registering builtin plugins

143b3db

Signed-off-by: Balazs Scheidler <bazsi@balabit.hu>


          python: initial port from the incubator

c7f7fbc

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: rename grammar prefix from python_driver_ to simply python_

b9cd73f

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: convert python_plugins to an array

d563d9c

I have some pending patches that add additional features to the Python
module that are not yet ready for prime time, but which add entries to the
python_plugins as an array.  (template functions and root-level Python code
blocks support)

Do the conversion here, users will come later.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: add LogMessage wrapper

459d3da

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: remove seq_num as LogThreadedDestDriver already supports it

e075744

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: remove msg_set_context() calls as LogThreadedDestDriver alrea…

7daa6dd

…dy does that

This is already factored into LogThreadedDestDriver, so no need to
do that ou rselves.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: introduce separate python-globals.h

aa3784c

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: extract value-pairs application into a separate module

1ebfb78

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: report errors reported by queue() properly

aacb846

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: rename python_worker prefix to python_dd

25eb289

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: don't handle exceptions in _py_function_return_value_as_bool

9ffd211

As it is explicitly handled by all call sites (the way it should be) and
which is being refactored into a single spot in upcoming patches.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: get the function name from the object instead of passing func…

2b0884b

…tion names around

Instead of having to pass function names around, use a new function
that extracts this information from a callable.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: extract functions from python_dd_init() & python_dd_deinit()

314c824

In order to simplify program flow and to break the code to smaller chunks,
use a separate function to initialize Python code.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: generalize py_invoke_queue() to py_invoke_function()

3bec624

This change makes it possible to rework all Python function invocation
into simpler forms, that is made possible by previous refactorings
(most notable the py_get_callable_name() function).

This makes the Python related functions a bit more plug & play, even
though only very limited functions can be called (e.g. one with only
a single argument and either a bool or None return value). Exceptions are
also handled properly.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: implement init() & deinit() calls with the new Python helpers

ce1e45b

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: rename python_dd_perform_imports

105b5cc

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: extract _py_do_import()

90429aa

Make a new function available that imports a Python module, which can then
be used for more than implementing the imports() option of the Python
destination.

This is the basis of the _py_init_bindings() refactor upcoming in the next patch.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: introduce _py_init_bindings and _py_free_bindings

8f84089

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: simplify _py_init_bindings()

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: move python_dd_insert() function

38681fc

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: make it possible to call functions without dropping the retur…

2227b20

…n value

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: use classes as destinations

5dc8ca8

Instead of using a function based interface without proper instantiation,
use a class with a definite interface in the Python part.  The method names
are intended to match that of the Java destination of a similar role.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: in case value-pairs() is not specified pass a LogMessage object

e5e9cf9

Instead of always converting the LogMessage to a dict, make it possible
to use the underlying C structure with a thin Python wrapper directly.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: handle NULL exception in _py_format_exception_text()

0cd15f2

It may happen that the Python C API returns NULL without an exception being
set. Fix potential segfault by checking if the exception is NULL.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: add exception information on module import failures

92c64ea

It was difficult to diagnose the "Error loading Python module" error
message, as syntax errors were not reported.

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: explicitly log the case when send() returns failure

a7738cf

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>


          python: add functional testcase

290553b

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>

bazsi force-pushed the f/python branch from aa1f051 to 290553b Compare

February 19, 2015 09:55

Collaborator Author

bazsi commented Feb 19, 2015

I have rebased this against the latest master and also added an automated testcase to functest, and a few fixes that I've found while adding the testcase :)

It should be a better form for merging.

Collaborator

lbudai commented Feb 19, 2015

Thanks! I'll try to review the patch in this week( +end :-) ).


          python: fix make dist

527ef57

Signed-off-by: Balazs Scheidler <balazs.scheidler@balabit.com>

Collaborator

lbudai commented Feb 21, 2015

Nice work :-)

I had a list of questions/notes, but size of my list reduces as refactors are coming
and after 5dc8ca8 notes are gone...
(maybe buf1, buf2 not the best names for variables, but... well... it is OK)

One question:
Is there any other way for importing a python class than setting PYTHONPATH before running syslog-ng?

Collaborator Author

bazsi commented Feb 21, 2015

I have some followup work and a few fixes in f/python-template-functions

That should conclude the python module for a short term until I submit my debugger cli in python work I worked on today.

You can avoid having to set PYTHONPATH by installing the extension module using a setup.py or something. Maybe we should also add /etc/syslog-ng or a subdirectory automatically. How do you do that with the Java plugin?

Collaborator

lbudai commented Feb 23, 2015

In Java dest. we have an option for setting the classpath.

lbudai added a commit that referenced this pull request


          Merge pull request #387 from balabit/f/python

4b2d218

F/python

lbudai merged commit 4b2d218 into master

lbudai removed the in progress label

lbudai mentioned this pull request

F/python template functions #392

Merged

bazsi deleted the f/python branch

November 17, 2015 07:16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet