Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbld/python: use python3 for every platform & remove xenial & add type support #4057

Merged
merged 27 commits into from Oct 13, 2022

Conversation

alltilla
Copy link
Collaborator

@alltilla alltilla commented Jun 24, 2022

For the reviewers

Kira fails, because it needs 3 things to work:

  1. There is an internal MR, which sets up kira, to use python3 instead of python2.
  2. There is an internal MR, which modifies the python-parser test config a bit, to work with the more up-to-date hy module.
  3. A new dbld-kira image build is needed, to have the latest hy package. It will only be generated when this PR is merged.

Kira tests with dbld-kira image generated from this PR, and with both of the 2 internal MRs:


Why remove xenial

It does not work with python3.

Its standard support ended in April 2021. Also the python3 version on it is python3.5, which's support ended in September 2020.

We have kept xenial until now, because we could use it with python2, but now, as we started to use python3 everywhere, and it does not work here, it is time to remove xenial from the supported platforms.


GitHub Actions devshell run

https://github.com/alltilla/syslog-ng/actions/runs/2746728090


Local package tests

Steps:

  1. Nightly packages generated in my fork: https://github.com/alltilla/syslog-ng/actions/runs/2555378620
  2. Indexer script ran locally.
  3. HTTP server ran locally on the indexed dir.
  4. Steps reproduced from test-apt-packages.yml.
# debian-stretch

root@1ce2b5554965:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,cryptofuncs,system-source,kvformat,dbparser,xml,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@1ce2b5554965:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007f33bb773000)


# debian-buster

root@c3e009d19ef1:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,cryptofuncs,system-source,kvformat,dbparser,xml,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@c3e009d19ef1:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007fef1ddd3000)


# debian-bullseye

root@6ad3c5ec24eb:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,mqtt,cryptofuncs,system-source,kvformat,dbparser,xml,kafka,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@6ad3c5ec24eb:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.9.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0 (0x00007f3f0b24a000)


# debian-testing

root@388badd95de9:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,mqtt,cryptofuncs,system-source,kvformat,dbparser,xml,kafka,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@388badd95de9:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.10.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0 (0x00007fdbe3012000)


# ubuntu-bionic

root@42a5b91ef236:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,cryptofuncs,system-source,kvformat,dbparser,xml,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@42a5b91ef236:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.6m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0 (0x00007f3cb201e000)


# ubuntu-focal

root@afd70a2bcbee:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,cryptofuncs,system-source,kvformat,dbparser,xml,kafka,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@afd70a2bcbee:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f235cffe000)


# ubuntu-impish

root@dde280b75341:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,mqtt,cryptofuncs,system-source,kvformat,dbparser,xml,kafka,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@dde280b75341:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.9.so.1.0 => /lib/x86_64-linux-gnu/libpython3.9.so.1.0 (0x00007f773936f000)


# ubuntu-jammy

root@e3fcfa1c879a:/# syslog-ng --version
syslog-ng 3 (3.37.1.51.gd265828)
Config version: 3.35
Installer-Version: 3.37.1.51.gd265828
Revision: 3.37.1.51.gd265828-snapshot+20220624T112802
Compile-Date: Jun 24 2022 11:28:02
Module-Directory: /usr/lib/syslog-ng/3.37
Module-Path: /usr/lib/syslog-ng/3.37
Include-Path: /usr/share/syslog-ng/include
Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'
Available-Modules: afuser,afsnmp,redis,afstomp,timestamp,stardate,mqtt,cryptofuncs,system-source,kvformat,dbparser,xml,kafka,regexp-parser,afsmtp,map-value-pairs,rate-limit-filter,http,basicfuncs,afsql,linux-kmsg-format,pseudofile,azure-auth-header,affile,json-plugin,tfgetent,disk-buffer,mod-python,appmodel,add-contextual-data,afsocket,afmongodb,syslogformat,confgen,afprog,secure-logging,csvparser,tags-parser,sdjournal,pacctformat,cef,geoip2-plugin,hook-commands,afamqp,graphite,riemann,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
root@e3fcfa1c879a:/# ldd /usr/lib/syslog-ng/3.37/libmod-python.so | grep python
        libpython3.10.so.1.0 => /lib/x86_64-linux-gnu/libpython3.10.so.1.0 (0x00007f1489a07000)

Signed-off-by: Attila Szakacs attila.szakacs@oneidentity.com

@alltilla alltilla added this to the syslog-ng-4.0 milestone Jun 24, 2022
@alltilla alltilla marked this pull request as draft July 1, 2022 08:48
@alltilla alltilla changed the title dbld: use python3 for every platform & remove xenial dbld/python: use python3 for every platform & remove xenial & add type support Jul 15, 2022
@alltilla alltilla marked this pull request as ready for review July 15, 2022 14:59
@kira-syslogng
Copy link
Contributor

Build FAILURE

@kira-syslogng
Copy link
Contributor

Build FAILURE

@alltilla
Copy link
Collaborator Author

@kira-syslogng test this please test branch=ose-with-python-3;

Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Oct 13, 2022
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Oct 13, 2022
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Oct 13, 2022
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
alltilla and others added 15 commits October 13, 2022 21:03
python-types provides the same functionality.

Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
python-types offers the same functionality.

Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
…ossible

`!py_bytes_or_string_to_string()` checks it for us.

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
`_dict_clone_value()` limits us, to only be able to get python values
as strings.

Also, it extracts the python value to C, which leaves us room for error
(which we happen to have, because we lose the knowledge about whether
it is shown as bytes or str in python)

The only 2 users of `_dict_clone_value()` uses it to check a python
variable's value, so it is easier to add an assert function for that,
which can handle any kinds of types (as you build up a "script", you
can inject any types of variables).

Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
In the previous 2 commits we have introduced 2 breaking changes:
  1. Previously if someone tried to set an NV value with not a
     string/bytes type in a python code, we threw an error.
     With the previous commit we changed the behavior and we store it
     with the correct type.
  2. Previously the python NV accessor function always returned
     string/bytes type. With the previous commit we return the same type
     as it is stored in the `LogMessage`.

The 1. point is not a big deal for backward compatibility reasons,
because we can be more forgiving, as long as we do not fail with
previously working code.

The 2. point is a bigger deal, because older codes expect strings/bytes,
but we can return other types, based on how it is stored in the
`LogMessage`.

To fix these, I have added a compatibility flag to the `PyLogMessage`
class. If the flag is set, we throw an error when setting a non
string/bytes value in python code, and we return string/bytes typed
value when accessing NVs in python code, no matter how it is stored in
the `LogMessage`.

Unfortunately, setting this flag can only be done where we create the
`PyLogMessage` instance in C code, as we only have a reference to the
`GlobalConfig` there. We cannot do that in the python code, because the
constructor there executes a callback (`py_log_message_new_empty()`),
which cannot have any user_data provided. This means we can only set it
in parser and destination, and not in source.
Not setting it in the source is fine, because there, we have always
created an empty `LogMessage`, so there is no risk of accessing NVs with
non bytes/string type.

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
It is implemented in a parametrized fashion, which might seems off, but
more input sets will be added in the following commits.

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
We want to add the config version 3.x compat to its state in prepare
time, so we can create the PyLogMessage accordingly.

This commit paves the way for that.

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>
IMHO we should not "fix" this problem with the blacklist,
but the proper fix would need a lot more work, so for now,
it is better to use the blacklist workaround correctly,
until we have a proper fix.

Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
Signed-off-by: Attila Szakacs <attila.szakacs@oneidentity.com>
@kira-syslogng
Copy link
Contributor

Build FAILURE

@kira-syslogng
Copy link
Contributor

Build FAILURE

@MrAnno MrAnno merged commit 8468190 into syslog-ng:master Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants