testAddBanInvalidEncoded fails under OSX

[yarikoptic]

see
http://nipy.bic.berkeley.edu/builders/fail2ban-py2.6-osx-10.6_master/builds/151/steps/shell_1/logs/stdio:

======================================================================
FAIL: testAddBanInvalidEncoded (fail2ban.tests.databasetestcase.DatabaseTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./fail2ban/tests/databasetestcase.py", line 200, in testAddBanInvalidEncoded
    or readtickets[0] == tickets[0]
AssertionError

and confirming that while debugging while running it query returns without "data":

bash-3.2$ fg
python2.7 bin/fail2ban-testcases -l debug testAddBanInvalidEncoded      (wd: ~/fail2ban)

[(u'127.0.0.1', 0, {}), (u'127.0.0.2', 0, {}), (u'127.0.0.3', 0, {})]
(Pdb) print list(cur.execute("SELECT ip, timeofban, data FROM bans WHERE 1"))
[(u'127.0.0.1', 0, {}), (u'127.0.0.2', 0, {}), (u'127.0.0.3', 0, {})]
(Pdb) print cur
<sqlite3.Cursor object at 0x1012ab260>
(Pdb) bt
  /Users/yoh/fail2ban/bin/fail2ban-testcases(126)<module>()
-> tests_results = testRunner.run(tests)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py(151)run()
-> test(result)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py(70)__call__()
-> return self.run(*args, **kwds)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py(108)run()
-> test(result)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py(70)__call__()
-> return self.run(*args, **kwds)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/suite.py(108)run()
-> test(result)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py(391)__call__()
-> return self.run(*args, **kwds)
  /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py(327)run()
-> testMethod()
  /Users/yoh/fail2ban/fail2ban/tests/databasetestcase.py(195)testAddBanInvalidEncoded()
-> readtickets = self.db.getBans(jail=self.jail)
  /Users/yoh/fail2ban/fail2ban/server/database.py(464)getBans()
-> for ip, timeofban, data in self._getBans(**kwargs):
  /Users/yoh/fail2ban/fail2ban/server/database.py(89)wrapper()
-> return f(self, self._db.cursor(), *args, **kwargs)
> /Users/yoh/fail2ban/fail2ban/server/database.py(442)_getBans()
-> return cur.execute(query, queryArgs)

but if I just reopen that sqlite db from another python session -- load is there:

>>> import sqlite3
>>> _db = sqlite3.connect('fail2ban_3zziBK.db', check_same_thread=False, detect_types=sqlite3.PARSE_DECLTYPES)
>>> cur = _db.cursor()
>>> list(cur.execute("SELECT ip, timeofban, data FROM bans WHERE 1"))
[(u'127.0.0.1', 0, u'{"matches": ["user \\"\ufffd\ufffd\ufffd\ufffd\ufffd\\"", "user \\"\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\\""], "failures": 0}'), (u'127.0.0.2', 0, u'{}'), (u'127.0.0.3', 0, u'{}')]
>>> list(cur.execute("SELECT ip, timeofban, data FROM bans WHERE 1"))
[(u'127.0.0.1', 0, u'{"matches": ["user \\"\ufffd\ufffd\ufffd\ufffd\ufffd\\"", "user \\"\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\\""], "failures": 0}'), (u'127.0.0.2', 0, u'{}'), (u'127.0.0.3', 0, u'{}')]
>>> list(cur.execute("SELECT ip, timeofban, data FROM bans WHERE 1"))
[(u'127.0.0.1', 0, u'{"matches": ["user \\"\ufffd\ufffd\ufffd\ufffd\ufffd\\"", "user \\"\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\\""], "failures": 0}'), (u'127.0.0.2', 0, u'{}'), (u'127.0.0.3', 0, u'{}')]

[opoplawski]

I appear to be seeing this building the fail2ban rpm, but not when I run the test directly out of the build directory, so I think something in the environment is affecting it.

Success:
tickets = [FailTicket: ip=127.0.0.1 time=0 #attempts=0 matches=['user "\xd1\xe2\xe5\xf2\xe0"', 'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"'], FailTicket: ip=127.0.0.2 time=0 #attempts=0 matches=['user "\xd1\xe2\xe5\xf2\xe0"', u'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"'], FailTicket: ip=127.0.0.3 time=0 #attempts=0 matches=['user "\xd1\xe2\xe5\xf2\xe0"', u'user "\xe4\xf6\xfc\xdf"']]

readtickets = [FailTicket: ip=127.0.0.1 time=0 #attempts=0 matches=['user "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd"', 'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"'], FailTicket: ip=127.0.0.2 time=0 #attempts=0 matches=['user "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd"', 'user "\xc3\x83\xc2\xa4\xc3\x83\xc2\xb6\xc3\x83\xc2\xbc\xc3\x83\xc2\x9f"'], FailTicket: ip=127.0.0.3 time=0 #attempts=0 matches=['user "\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd\xef\xbf\xbd"', 'user "\xc3\xa4\xc3\xb6\xc3\xbc\xc3\x9f"']]

Fail:
readtickets = [FailTicket: ip=127.0.0.1 time=0 #attempts=1 matches=[], FailTicket: ip=127.0.0.2 time=0 #attempts=1 matches=[], FailTicket: ip=127.0.0.3 time=0 #attempts=1 matches=[]]

Works again if I set LANG to en_US.UTF-8 (from C, the default in rpm).

[sebres]

Should we possibly add to this test case some conditions, like os.environ["LANG"] or sys.getdefaultencoding()?

[opoplawski]

I'm now seeing a failure with en_US.UTF-8 and 0.9.4:

+ umask 022
+ cd /builddir/build/BUILD
+ cd fail2ban-0.9.4
+ export LANG=en_US.UTF-8
+ LANG=en_US.UTF-8
+ ./fail2ban-testcases-all-python3 --no-network
Testing using /usr/bin/python3.5
Fail2ban 0.9.4 test suite. Python 3.5.1 (default, Mar  4 2016, 13:43:50) [GCC 6.0.0 20160302 (Red Hat 6.0.0-0.14)]. Please wait...
........................s......................................................................F.................F......................................................E..........................................................................................s..
======================================================================
FAIL: testAddBanInvalidEncoded (fail2ban.tests.databasetestcase.DatabaseTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./fail2ban/tests/databasetestcase.py", line 201, in testAddBanInvalidEncoded
    or readtickets[0] == tickets[0]
AssertionError: False is not true

[sebres]

I can suspect that has changed again in python 3.5... will test it now

[sebres]

Seems to be also already fixed in f2b-perfom-prepare-716-cs:

$ echo $LANG
en_US.UTF-8
$ PYTHONPATH=. python3.5 bin/fail2ban-testcases -fn test
Fail2ban 0.9.3.dev test suite. Python 3.5.1 (default, Dec 18 2015, 00:00:00) [GCC 4.8.4]. Please wait...
........................s............................s.............................sssss..........................s.......s..................................s............sssssssssss.............s......sssss...........................................................................................
...............ssssss........................
----------------------------------------------------------------------
Ran 342 tests in 4.727s

OK (skipped=33)

$ PYTHONPATH=. python3.5 bin/fail2ban-testcases test
Fail2ban 0.9.3.dev test suite. Python 3.5.1 (default, Dec 18 2015, 00:00:00) [GCC 4.8.4]. Please wait...
........................s.................................................................................................s..............................................................................................................................................................................
.............................................
----------------------------------------------------------------------
Ran 342 tests in 25.360s

OK (skipped=2)

[opoplawski]

Hmm, mixed results for me - koji build did not have this test failure, local mock build did. Any chance for a simple backport to 0.9.4?

[sebres]

local mock build did

What did you mean exactly?

Any chance for a simple backport to 0.9.4?

Well if I knew why (need more info). This can be even not so "simple"...
According to @yarikoptic we will try to make from this a version 0.10.0 (why not really 1.0?)...

[opoplawski]

mock is tool for doing local package builds. Well, currently this test failure is blocking releasing 0.9.4 for Fedora. Unless we decide to ignore the test failure, which I'd rather not do unless we know it's just a test issue.

[sebres]

I know what is mock, the question was rather "concerns this also my branch or not", because you wrote "mixed results"

[opoplawski]

What I meant was using your branch, the test still failed in my local mock build, but not with my koji build. Strange.

[sebres]

Then I think the mock would use something "wrong" (environment, etc.)
Can you try to build it with mock, translating parameters -l heavydebug testAddBanInvalidEncoded to the test-cases to see more