Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Under python3 the LogService._get_raw_data fails on undecodable data #5430

Closed
apardyl opened this issue Feb 19, 2019 · 0 comments · Fixed by #5431
Closed

Under python3 the LogService._get_raw_data fails on undecodable data #5430

apardyl opened this issue Feb 19, 2019 · 0 comments · Fixed by #5431
Labels
bug needs triage Issues which need to be manually labelled

Comments

@apardyl
Copy link
Contributor

apardyl commented Feb 19, 2019

Bug report summary

Under python3 the LogService._get_raw_data fails with an exception when undecodable bytes are read from a log file (e.g. the file encoding is different from what locale.getpreferredencoding() specifies or some garbage data is logged by the monitored application). This blocks any LogService-based collector until a log rotation happens or Netdata is restarted (the method will retry reading the same line every time it is invoked).

OS / Environment

Arch Linux, python3.7

Netdata version (ouput of netdata -V)

v1.12.0-46-gf1d69813 (built from master)

Component Name

collectors/python.d.plugin

Steps To Reproduce

As an example with fail2ban module:

head -c 200 < /dev/urandom >> /var/log/fail2ban.log

This will result with having error.log flooded with

python.d ERROR: fail2ban: fail2ban: update() unhandled exception: 'utf-8' codec can't decode byte 0xbd in position 0: invalid start byte
Expected behavior

_get_raw_data ignores the encoding errors by skipping invalid bytes or the entire line.
@apardyl apardyl mentioned this issue Feb 19, 2019
Merged
ilyam8 pushed a commit that referenced this issue Feb 20, 2019
@apardyl @ilyam8
fix #5430 - LogService._get_raw_data under python3 fails on undecodab…
c649de8 
…le data (#5431)

* Fix LogService._get_raw_data under python3

* Move python version check to init

* Mark open_args as private
@netdatabot netdatabot added bug needs triage Issues which need to be manually labelled labels Feb 20, 2019
@cakrit cakrit mentioned this issue Feb 21, 2019
Closed
kiku-jw pushed a commit to kiku-jw/netdata that referenced this issue Mar 4, 2019
@apardyl @kiku-jw
fix netdata#5430 - LogService._get_raw_data under python3 fails on un…
556f155 
…decodable data (netdata#5431)

* Fix LogService._get_raw_data under python3

* Move python version check to init

* Mark open_args as private
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug needs triage Issues which need to be manually labelled
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix #5430 - LogService._get_raw_data under python3 fails on undecodable data apardyl/netdata
2 participants
@apardyl @netdatabot