use pytest

[sils]

http://pytest.org/latest/goodpractises.html#integrating-with-distutils-python-setup-py-test

should be easy and cool :)

[sils]

maybe we want tox too, sounds like it could even simplify our CI scripting.

[sils]

So I'm not sure if we really want that.

1. I like our output better
2. Our test framework seems just faster
3. Our test framework just executes our test file and the other framework executes the Unittest objects - because of that it doesn't execute the module code and that leads to failing tests.

I started migrating out our test framework to cotest, not sure though if we shouldn't just make our tests independent on module code and use pytest. Any opinions?

[AbdealiLoKo]

Everything you said has plugins in pytest.

There must be something for formatting
Three a plugin to run tests in parallel
There must be a way to skip tests if needed

I think it may be wiser to make our own pytest plugins if there are any issues with it in general

-----Original Message-----
From: "Lasse Schuirmann" notifications@github.com
Sent: ‎02-‎08-‎2015 23:57
To: "coala-analyzer/coala" coala@noreply.github.com
Subject: Re: [coala] use pytest (#875)

So I'm not sure if we really want that.
I like our output better
Our test framework seems just faster
Our test framework just executes our test file and the other framework executes the Unittest objects - because of that it doesn't execute the module code and that leads to failing tests.
I started migrating out our test framework to cotest, not sure though if we shouldn't just make our tests independent on module code and use pytest. Any opinions?
—
Reply to this email directly or view it on GitHub.

[sils]

@AbdealiJK following tests fail when executed with pytest:

http://pastebin.com/F1NWHCV8

[sils]

Marking wontfix, for now our framework works and we need to focus.

[sils]

I think we'd greatly profit from using doctest which pytest provides so reopening this.

[Makman2]

Did you miss to reopen?

[sils]

@AbdealiJK DBus test is a huge problem because you rely on a custom main routine. Can you help? Currently looking into the rest...

[AbdealiLoKo]

https://pypi.python.org/pypi/pytest-xprocess seems to be the answer to your issue. It spawns a process at the beginning and keeps that process going throughout the testing time.

Although we want it only for 1 test, this seems the way to go.

-----Original Message-----
From: "Lasse Schuirmann" notifications@github.com
Sent: ‎01-‎01-‎2016 22:51
To: "coala-analyzer/coala" coala@noreply.github.com
Cc: "AbdealiJK" abdealikothari@gmail.com
Subject: Re: [coala] use pytest (#875)

@AbdealiJK DBus test is a huge problem because you rely on a custom main routine. Can you help? Currently looking into the rest...
—
Reply to this email directly or view it on GitHub.

[sils]

@AbdealiJK sure that's the way to go? PyTest also supports test functions, so instead of putting the process spawning into the main routine we could put it into a test_something method which invokes another python file (cleaner anyway, one file for the actual test, another file for the coala dbus server)?

[AbdealiLoKo]

I tried that when I first mentioned pytest (as far as I remember). It
didn't work out properly if I remember correctly.
Basically the test_* don't run parallelly on a single core and some other
complications that I don't remember made me not go through that route.

I think it's better to spawn a test dbus server - that way even
DbusDocumentTest.py and other Dbus*Test.py can also test do unit tests
against the server later (right now they don't)

On Sat, Jan 2, 2016 at 6:41 PM, Lasse Schuirmann notifications@github.com
wrote:

@AbdealiJK https://github.com/AbdealiJK sure that's the way to go?
PyTest also supports test functions, so instead of putting the process
spawning into the main routine we could put it into a test_something
method which invokes another python file (cleaner anyway, one file for the
actual test, another file for the coala dbus server)?

—
Reply to this email directly or view it on GitHub
#875 (comment)
.

[sils]

@AbdealiJK do you have any hint about the other failures in pytest? Many are because of the test skipping but some seem to be very strange, like some tests of is_applicable of some actions which are really simple.

[AbdealiLoKo]

What exactly do you mean by the failures of pytest ? The major issues I
noticed with pytest was :

• the thing with dbus where we needed an external app running
• the skip_test functionality needed to be revamped
• the fact that it was slower than our tests quite a bit
• And in general lack of flexibility as it's not our code and we had to
  follow someone else's docs
  Is that what you were looking for ?
  The first two took time and we didn't want to spend time on it earlier.
  The second two were a trade off between maintaining our own barebones code
  and using a full-fledged 3rd part library.

And what did you mean by the is_applicable thing ? What's strange about it ?

On Sat, Jan 2, 2016 at 7:31 PM, Lasse Schuirmann notifications@github.com
wrote:

@AbdealiJK https://github.com/AbdealiJK do you have any hint about the
other failures in pytest? Many are because of the test skipping but some
seem to be very strange, like some tests of is_applicable of some actions
which are really simple.

—
Reply to this email directly or view it on GitHub
#875 (comment)
.

[sils]

FWIW I'm almost done with replacing our skip functionality, @AbdealiJK can you look into DBus?

[sils]

Ok, that's what I get after #1183:
http://pastebin.com/ZrMy5D3k

[sils]

Alright, the DBusTest issue is resolved, think we can make the switch soon

[AbdealiLoKo]

Now a new issue came up. If I do py.test -k "coalaTest" the test fails. This happens in coalaTest ConfigurationGatheringTest and CliParserTest - probably because they all use sys.argv