mmnormalize_{variable,tokenized,regex}.sh tests are failing #489

Whissi · 2015-08-18T18:06:49Z

The

mmnormalize_variable.sh
mmnormalize_tokenized.sh
mmnormalize_regex.sh

tests are failing:

FAIL: mmnormalize_variable.sh
=============================

===============================================================================
[mmnormalize_variable.sh]: basic test for mmnormalize module variable-support
rsyslogd started with pid  25366
starting run 1
Sending file './testsuites/date_time_msg' 1 times.
^M00000001 kb sent
runtime: 0.000
End of tcpflood Run
doing shutdown
imdiag[13500]: mainqueue empty
wait on shutdown
content-check failed
FAIL mmnormalize_variable.sh (exit status: 1)

FAIL: mmnormalize_tokenized.sh
==============================

===============================================================================
[mmnormalize_tokenized.sh]: test for mmnormalize tokenized field_type
rsyslogd started with pid  25409
starting run 1
Sending file './testsuites/tokenized_input' 1 times.
^M00000001 kb sent
runtime: 0.000
End of tcpflood Run
doing shutdown
imdiag[13500]: mainqueue empty
wait on shutdown
content-check failed
FAIL mmnormalize_tokenized.sh (exit status: 1)

FAIL: mmnormalize_regex.sh
==========================

===============================================================================
[mmnormalize_regex.sh]: test for mmnormalize regex field_type
rsyslogd started with pid  25566
starting run 1
Sending file './testsuites/regex_input' 1 times.
^M00000001 kb sent
runtime: 0.000
End of tcpflood Run
doing shutdown
imdiag[13500]: mainqueue empty
wait on shutdown
content-check failed
FAIL mmnormalize_regex.sh (exit status: 1)

Confirmed against rsyslog-8.12.0 release tarball and git at 1e26405.

The text was updated successfully, but these errors were encountered:

Whissi · 2015-08-18T18:36:23Z

OK, this is a problem which only happens with liblognorm-1.1.2. It is working fine with liblognorm-1.1.1.

Do we need to mask liblognorm-1.1.2 for rsyslog or does the test requires an update to work with liblognorm-1.1.2?

alorbach · 2015-08-19T13:38:46Z

I will look into this.

alorbach · 2015-08-20T14:21:50Z

Are you certain these tests worked with liblognorm-1.1.1?
I also posted on the mailinglist that at least one of the configs does not work.

Whissi · 2015-08-20T14:50:47Z

100% sure. I just verified again:

[...]
PASS: mmnormalize_variable.sh
PASS: mmnormalize_tokenized.sh
PASS: mmnormalize_regex_defaulted.sh
PASS: mmnormalize_regex_disabled.sh
PASS: mmnormalize_regex.sh
[...]
============================================================================
Testsuite summary for rsyslog 8.12.0
============================================================================
# TOTAL: 158
# PASS:  153
# SKIP:  5
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

If you could bring back 1.1.1 for precise in the ppa Travis would pass, too (when we would pin the package to 1.1.1).

rgerhards · 2015-08-21T09:01:03Z

I think this problem is related to this issue here: rsyslog/liblognorm#57

Maybe the tests in rsyslog also just need to be updated, maybe there indeed was a problem that neither @janmejay nor me noticed after the initial discussion.

Breaks rsyslog. See http://lists.adiscon.net/pipermail/rsyslog/2015-August/041013.html rsyslog/rsyslog#489 Signed-off-by: Lars Wendler <polynomial-c@gentoo.org>

rgerhards · 2015-08-23T12:49:41Z

I have probably traced the problem down to the root cause: the testbench generates rulebases with incomplete lines (\n missing). As it looks, the new liblognorm code does not accept such incomplete lines, but the old one seemed to process them. I'll dig into it some more, but thought I share the news (I need to continue tomorrow).

This problem occurs with the very last line of a rulebase (at EOF). If it is not properly terminated (LF missing), it is silently ignored. Previous versions did obviously process lines in this case. While technically this is invalid input, we can't outrule that such rulebases exist. For example, they do in the rsyslog testbench, which made us aware of the problem (see rsyslog/rsyslog#489 ). I think the proper way of addressing this is to process such lines without termination, as many other tools do as well.

This problem occurs with the very last line of a rulebase (at EOF). If it is not properly terminated (LF missing), it is silently ignored. Previous versions did obviously process lines in this case. While technically this is invalid input, we can't outrule that such rulebases exist. For example, they do in the rsyslog testbench, which made us aware of the problem (see rsyslog/rsyslog#489 ). I think the proper way of addressing this is to process such lines without termination, as many other tools do as well. closes rsyslog#135

rgerhards · 2015-08-24T09:28:20Z

These tests (plus two not mentioned) now work with the new version of liblognorm, with the exception of the _regex test. It looks like this test does something wrong, the root cause is that it tests regex when it is not supported. Will look into it, but may wait until @janmejay has time to dig further.

janmejay · 2015-08-24T17:34:35Z

Rainer, I'm buried in work add of now, I may take at least a few weeks
before I can look at it. I'm so sorry!

Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Aug 24, 2015 2:58 PM, "Rainer Gerhards" notifications@github.com wrote:

These tests (plus two not mentioned) now work with the new version of
liblognorm, with the exception of the _regex test. It looks like this test
does something wrong, the root cause is that it tests regex when it is not
supported. Will look into it, but may wait until @janmejay
https://github.com/janmejay has time to dig further.

—
Reply to this email directly or view it on GitHub
#489 (comment).

rgerhards · 2015-08-24T18:13:37Z

Sent from phone, thus brief.
Am 24.08.2015 19:34 schrieb "Janmejay Singh" notifications@github.com:

Rainer, I'm buried in work add of now, I may take at least a few weeks
before I can look at it. I'm so sorry!

No Problem. If i don't see it quickly enough, I'll comment out the test.
Regex should not be used by regular users anyways, so it's not a big deal
IMHO.

Rainer

Regards,
Janmejay

PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.

On Aug 24, 2015 2:58 PM, "Rainer Gerhards" notifications@github.com
wrote:

These tests (plus two not mentioned) now work with the new version of
liblognorm, with the exception of the _regex test. It looks like this
test
does something wrong, the root cause is that it tests regex when it is
not
supported. Will look into it, but may wait until @janmejay
https://github.com/janmejay has time to dig further.

—
Reply to this email directly or view it on GitHub
#489 (comment).

—
Reply to this email directly or view it on GitHub.

rgerhards · 2015-08-25T08:02:37Z

@janmejay as it looks, at least in travis the regex test works now. I still have problems in one environment, may be environment-induced. I would appreciate if you could look into the issue when you have time later. Note that I am not yet sure if we should support regex in v2 natively, so this may become a non-issue within the next few weeks.

janmejay · 2015-08-25T11:43:47Z

I think its valuable to support regexp for odd unforeseen cases. I'll
definitely look at it once I have a little breathing room.

On Tue, Aug 25, 2015 at 1:32 PM, Rainer Gerhards notifications@github.com
wrote:

@janmejay https://github.com/janmejay as it looks, at least in travis
the regex test works now. I still have problems in one environment, may be
environment-induced. I would appreciate if you could look into the issue
when you have time later. Note that I am not yet sure if we should support
regex in v2 natively, so this may become a non-issue within the next few
weeks.

—
Reply to this email directly or view it on GitHub
#489 (comment).

Regards,
Janmejay
http://codehunk.wordpress.com

rgerhards · 2015-08-27T06:55:46Z

@janmejay I opened an issue tracker where we can discuss the need to re-introduce regex: rsyslog/liblognorm#143

janmejay · 2015-08-27T11:30:52Z

Sounds good.

On Thu, Aug 27, 2015 at 12:25 PM, Rainer Gerhards notifications@github.com
wrote:

@janmejay https://github.com/janmejay I opened an issue tracker where
we can discuss the need to re-introduce regex: rsyslog/liblognorm#143
rsyslog/liblognorm#143

—
Reply to this email directly or view it on GitHub
#489 (comment).

Regards,
Janmejay
http://codehunk.wordpress.com

…issue #135 This fix is required to pass rsyslog's test suite, see rsyslog/rsyslog#489

Whissi · 2015-10-09T15:10:24Z

Short update: We disabled PCRE in liblognorm-1.1.2 like suggested in rsyslog/liblognorm#143 and cherry-picked rgerhards/liblognorm@4b35ca1 which fixed the issue for the remaining two tests mmnormalize_variable.sh and mmnormalize_tokenized.sh.

Now that rsyslog-8.13.0 finally passed the test suite on all our supported plattforms we added it to the tree.

Not sure if this issue should stay open until rsyslog requirements for liblognorm are bumped to require a version with the patch included or not so people experiencing the same problem will find it (and a solution).

rgerhards · 2018-01-19T09:39:17Z

As it looks, this is solved. I do also see no related failures in our CI runs. If the problem re-occurs, let me know and I can re-open.

lock · 2019-12-26T14:19:07Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

Whissi mentioned this issue Aug 19, 2015

Enable mmnormalize for test coverage within Travis #492

Merged

alorbach self-assigned this Aug 19, 2015

rgerhards mentioned this issue Aug 24, 2015

v1.1.2+ does not process last line if it misses the terminating LF rsyslog/liblognorm#135

Closed

rgerhards mentioned this issue Aug 24, 2015

"fix": process last line if it misses the terminating LF rsyslog/liblognorm#136

Merged

Whissi added a commit to Whissi/gentoo-overlay that referenced this issue Oct 9, 2015

dev-libs/liblognorm: Revbump to v1.1.2-r1 to include fix for uptream …

4b5ef22

…issue #135 This fix is required to pass rsyslog's test suite, see rsyslog/rsyslog#489

rgerhards added this to the v8.33 milestone Nov 26, 2017

rgerhards unassigned alorbach Nov 26, 2017

rgerhards closed this as completed Jan 19, 2018

lock bot locked as resolved and limited conversation to collaborators Dec 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mmnormalize_{variable,tokenized,regex}.sh tests are failing #489

mmnormalize_{variable,tokenized,regex}.sh tests are failing #489

Whissi commented Aug 18, 2015

Whissi commented Aug 18, 2015

alorbach commented Aug 19, 2015

alorbach commented Aug 20, 2015

Whissi commented Aug 20, 2015

rgerhards commented Aug 21, 2015

rgerhards commented Aug 23, 2015

rgerhards commented Aug 24, 2015

janmejay commented Aug 24, 2015

rgerhards commented Aug 24, 2015

rgerhards commented Aug 25, 2015

janmejay commented Aug 25, 2015

rgerhards commented Aug 27, 2015

janmejay commented Aug 27, 2015

Whissi commented Oct 9, 2015

rgerhards commented Jan 19, 2018

lock bot commented Dec 26, 2019

mmnormalize_{variable,tokenized,regex}.sh tests are failing #489

mmnormalize_{variable,tokenized,regex}.sh tests are failing #489

Comments

Whissi commented Aug 18, 2015

Whissi commented Aug 18, 2015

alorbach commented Aug 19, 2015

alorbach commented Aug 20, 2015

Whissi commented Aug 20, 2015

rgerhards commented Aug 21, 2015

rgerhards commented Aug 23, 2015

rgerhards commented Aug 24, 2015

janmejay commented Aug 24, 2015

rgerhards commented Aug 24, 2015

rgerhards commented Aug 25, 2015

janmejay commented Aug 25, 2015

rgerhards commented Aug 27, 2015

janmejay commented Aug 27, 2015

Whissi commented Oct 9, 2015

rgerhards commented Jan 19, 2018

lock bot commented Dec 26, 2019