Addressed flaky Travis parallel test execution by adding a retry #287
Conversation
ssbarnea
force-pushed
the
feature/travis
branch
2 times, most recently
from
8835dbe
to
0ee1974
Compare
| @@ -1692,9 +1694,9 @@ def test_create_version(self): | |||
|
|
|||
| def test_create_version_with_project_obj(self): | |||
| project = self.jira.project(self.project_b) | |||
| version = self.jira.create_version('new version 1', project, | |||
| version = self.jira.create_version('new version 2', project, | |||
There was a problem hiding this comment.
Are we running tests in parallel?! (If so, where is this configured?)
Specifically, I know we run multiple Travis builds concurrently, but within a single Travis build (or local test run, for that matter) do multiple tests run in parallel?
This change, and the setup of the JiraTestManager imply that we do, but I can't see any evidence of this in the configuration or Travis test run logs, so I'm a bit confused.
There was a problem hiding this comment.
@asqui Yes, Travis is running 3 test executions in parallel, one for each python version 27,34,37 but tests are run sequentially for each of them. That's a good thing, when it works and instead of going the dirty way of running them sequentially we should do the right thing and fix them.
Also running tests in parallel will allow us to discover some real world scenarios that may not happen on a empty jira instance that is used exclusively for the purpose of one test. Is not quite a load test but if JIRA does not have atomic operations we should know it and adapt the library to make it more resilient to the server misbehaviours.
Based on the execution history I am inclined to believe that Jira is misbehaving and that is caching some results: example you may get a 5xx or 4xx error trying to create a resource (like a group or user) that was recently removed. By putting a sleep() or 1-2 seconds we may workaround this but I am not 100% sure yet as is very hard to get confirmation due to the flaky nature of the tests.
Still, I observed that these errors do not happen randomly over all the tests, when they happen they usually happen inside specific ones.
@asqui It would be great if you can have a look as this is the most important issue with the library, one that is preventing us from starting to release often and to provide bug fixes. Feel free to use the feature/travis branch to test other changes.
If you want to simulate what happens with Travis but on your own machine you could try to run detox which is mainly running tox in parallel.
The entire idea is that we should support running tests in parallel and that's why we generate the TestManager.jid unique id, which is supposed to be unique per execution: we should never have two test executions that run in parallel that are using the same jid.
Again, I am talking about parallelism regarding the whole execution of the test suites, not tests themselves, these are sequential and even the order is important.
|
I would suggest using random keys/names for test projects and issues. That should ensure that the next or parallel test will not fail. WDYT? |
ssbarnea
force-pushed
the
feature/travis
branch
from
da5777b
to
8ca9987
Compare
Change-Id: I016f823097c4edc89d8f0711fce9e4ae3fde6654 Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I5cfec378205eda2f30697588f6833689724b6acf Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: I63267bdc817319e8da02f6f4a1952db0cd9587b6 Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
Change-Id: Ia376514bc7d3e1478ae277568b60b0185464b58d
ssbarnea
force-pushed
the
feature/travis
branch
15 times, most recently
from
864a03e
to
741f6ac
Compare
ssbarnea
force-pushed
the
feature/travis
branch
8 times, most recently
from
509d1e1
to
96184fb
Compare
…anism. Change-Id: I8affe5ab3f9cfa8946cfdc9f969e338fbb43302b
ssbarnea
force-pushed
the
feature/travis
branch
from
96184fb
to
9c022e8
Compare
ssbarnea
changed the title
Fixes #143