Add real-time error messages during testing

[jaimedelreyg]

Sometimes it is useful to have some information about your test asserts during execution, particularly in long period tests. I propose to add real-time error messages using python's logging module.

[okken]

This is a nice feature. I'd like to look into it.
It needs to add real-time logging without duplicating the overall error report.

[okken]

What do you think about this option?
We could add the flag --log-check-level and maybe the setting log_check_level, defaulting to NOTSET.
And if set, the failures are logged.
This would add real time logging support, but not change the default behavior.

[jaimedelreyg]

yeah, of course it could be a great idea make this feature optional :)

[kapis90]

I would add something else on my side. It would be wonderful to have logged the passes as well (optionally ofc)
Here is a sample output:

test_sandbox.py::TestSandbox::test_case1 
------------------------------- live log setup --------------------------------
13:17:04.319 [INFO] INIT
-------------------------------- live log call --------------------------------
13:17:04.320 [FAIL] check 2 == 1 FAIL1
13:17:04.853 [PASS] check 1 == 1 PASS
13:17:04.854 [FAIL] check 0 == 1 FAIL2
FAILED                                                                   [100%]
testsuites\sandbox_test\test_sandbox.py:42 (TestSandbox.test_case1)
FAILURE: check 2 == 1: FAIL1
test_sandbox.py:44 in test_case1() -> self.steps.step_check()
scripts\steps.py:25 in step_check() -> check.equal(2, 1, "FAIL1")

FAILURE: check 0 == 1: FAIL2
test_sandbox.py:44 in test_case1() -> self.steps.step_check()
scripts\steps.py:27 in step_check() -> check.equal(0, 1, "FAIL2")

------------------------------------------------------------
Failed Checks: 2

What I did, I added two logging levels (PASS and FAIL) and just log them with message.
I think adding different log levels should not be a problem. I can create them on root logger in my project, and just specify them from command line as you suggest

Why I even need a pass in test case?
Well, I am using pytest for functional tests in my project, after execution I need to create a report what was tested. Having this in log will simplify the process

@okken, does it make sense ?

[okken]

@kapis90 I understand the request.
I also understand the want. It's really a separate request, I think, from the original want.
PR #115 solves the original issue, hopefully, but doesn't address the pass logging.

I've added #116 for your request.

[okken]

See changelog for set_failure_callback() on version 2.1.0

[kapis90]

@kapis90 I understand the request. I also understand the want. It's really a separate request, I think, from the original want. PR #115 solves the original issue, hopefully, but doesn't address the pass logging.

I've added #116 for your request.

I agree with you, it can be treated as separate issue