Integration tests fail randomly

[pecko]

While we merge #932 these tests failed randomly on rultor.

RtReleaseAssetsITCase, RtReleasesITCase, RtContentsITCase have failed sometimes.

I have reproduced it once on Eclipse: com.jcabi.github.RtReleaseAssetsITCase.uploadsSameAssetInTwoReleases() failed.

[pecko]

Since tests print

• HTTP response status is not equal to 201:

or

• HTTP response status is not equal to 204:

the problem can be caused with some temporary connection issue on github.
I propose that we try merge rultor more times.

[pecko]

@darkled @dmarkov @yegor256

I have also tried rultor build locally. The build pass all tests.

However, there are a difference.
When running a build after 'at rultor merge', the build is called with:

mvn clean install -Pjcabi-github -Pqulice --errors --settings ../settings.xml

This run all integration tests and some of them fails.
For instance,

...
Running com.jcabi.github.RtReleaseAssetsITCase 
...
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.605 sec <<< FAILURE! - in com.jcabi.github.RtReleaseAssetsITCase uploadsSameAssetInTwoReleases(com.jcabi.github.RtReleaseAssetsITCase)

I have tried:

mvn clean install -Pjcabi-github -Pqulice --errors
mvn clean install -Pqulice

It skips a lot integration tests. Other tests pass. The same issue happens on Travis and AppVeyor
For instance,

Running com.jcabi.github.RtReleaseAssetsITCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 sec - in com.jcabi.github.RtReleaseAssetsITCase

Actually, all 6 tests are skipped because test's setUp method fails.
I suppose that we need add properties as follows:

• failsafe.github.key
• failsafe.github.key.second
• failsafe.github.repo
• failsafe.github.organization

I have tried to add the failsafe.github.key property and all 6 tests in this test run (and pass, but, IMO, it's randomly). Also, a lot integration tests run (not all)

Questions:

1. Are there some other required properties and what I need to set here?
2. I suppose that they are defined in yegor256/home#assets/jcabi/settings.xml. Where is this file? Can I get it?
3. How I start rultor to run/use .rultor.xml from the jcabi-github project, for instance?

[pecko]

@yegor256 @dmarkov @darkled

I have tried the build more times (with the failsafe.github.key property). One build fails with the following failure:

Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 14.375 sec 
<<< FAILURE! - in com.jcabi.github.RtReleasesITCase
canRemoveRelease(com.jcabi.github.RtReleasesITCase)  Time elapsed: 0.564 sec  
<<< FAILURE!
java.lang.AssertionError: HTTP response status is not equal to 201:
422 Unprocessable Entity [https://api.github.com/repos/pecko/PgflwUceR09aJ0OAkVZo/releases]

This is definitely random issue and not related to rultor.
Maybe we should separate integration tests until solving this issue,

Possible reasons:

• temporary github issue
• the jcabi-github project acquires ports too aggressive (see Travis builds fail unexcpected #1165)

One idea: try these tests three times. If a test fails all three times we would decide that it is failed.
I know that it isn't solution, but currently we can't merge anything to the jcabi-github project.

[yegor256]

@pecko keep in mind that we discourage abstract discussions in tickets. If you think that the project has a bug, report a new ticket and explain what is the problem. It will be fixed. Than you can return to this ticket and continue. Check these articles: http://www.yegor256.com/2014/11/24/principles-of-bug-tracking.html http://www.yegor256.com/2015/01/15/how-to-cut-corners.html

[pecko]

@yegor256

:( not addressed again.

All discussion is related to this issue: "RtReleaseAssetsITCase and RtReleasesITCase tests fail randomly". I have changed the title to "Integration tests fail randomly", because it fails for different integration tests.

I have opened two new issues related to my question(s) above.

I am interested in this problem because my task #932 blocked with this issue :(.

[pecko]

@yegor256

This PR is relevant: #1162. The com.jcabi.github.RtReposITCase test also fails sometimes.

[dmarkov]

@pecko I am aware of the task, give me some time to find a developer...

[dmarkov]

@pecko many thanks, 15 mins added to your acc for reporting this bug, pmt ID 66776531