[regression] Inconsistent results on travis-ci

[nemesifier]

After the last release I started to get inconsistent results on travis again (python3).

I cannot replicate and fix the issues.

I opened this issue even though I don't have much details to offer, hoping that if more people are experiencing the same problem they will help to debug, for now I will try to pin the test requirements to the previous stable version.

[samuelcolvin]

I'm getting this too, it's not the same as #457. Which I think was caused by installing some packages locally.

Example: https://travis-ci.org/samuelcolvin/arq/jobs/238303765

[timothycrosley]

@nemesisdesign: Do you mean inconsistent from local running of isort? I couldn't reproduce this issue, but I did notice you only run isort for Python3 - so it couldn't be inconsistencies between Python versions. It looks like on travis it's incorrectly identifying a project level import, is this correct?

Thanks!

~Timothy

[krzysztof]

I don't think it was a case for @nemesisdesign build, but in general you might get inconsistencies if travis is cacheing pip and using cached isort in version other than your local one (clearing travis repo cache might help).

[samuelcolvin]

another example: https://travis-ci.org/em-2/em2/jobs/239309361

pytest-isort on travis wants blank lines but pytest-isort locally doesn't care. This is fixed by running isort itself locally which adds the new lines

[nemesifier]

@krzysztof thank you for the tip. I've ran into the same issue before and it was a pain to fix. But this time I tried the same fix I applied last time and it didn't work.

I worked around it by pinning isort to 4.2.5 for the moment.

[nemesifier]

@timothycrosley in short, the problem is that I cannot get the same output of the travis build in my local development environment, I'm having this issues in several python packages.

I usually create my virtualenvs with the following command:

mkvirtualenv --python=/usr/bin/python3 <name>

May it be a problem with virtualenvwrapper? I really don't know :-/

[nemesifier]

Or maybe is a misunderstanding in how the library should be used? Not sure!

[krzysztof]

@nemesisdesign It looks like a problem with the isort distinction between first/third party packages, it sometimes happens when they are installed with dev dependencies (not your case). Nonetheless, you can try creating an .editorconfig file and setting "known_third_party" value, which specifies the dependencies for some of the requirements you have here: https://github.com/openwisp/django-netjsonconfig/blob/master/requirements.txt

For example, our editorconfig:
https://github.com/inveniosoftware/invenio-records/blob/master/.editorconfig#L40
Libraries we install from as dev dependencies:
https://github.com/inveniosoftware/invenio-records/blob/master/requirements-devel.txt

[sobolevn]

@krzysztof You are my hero of the day. Solved my inconsistent test results with:

[*.py]
known_first_party = split_settings
default_section = THIRDPARTY

[nemesifier]

@krzysztof @timothycrosley:
I tried updating to the latest version (4.2.15) and the build on travis now only complains about missing blank lines:
https://travis-ci.org/netjson/django-netjsongraph/jobs/245409125

[nemesifier]

I solved that build issue by specifying known_first_party and default_section as suggested:
openwisp/django-netjsongraph@3f29fdb

It's a shame I will have to do this for all my projects though.. there are quite a few openwisp2 modules out there I will need to upgrade..

[jleclanche]

I had the same issue as described above, despite already having known_third_party = mylib. Had to add default_section = THIRDPARTY as well to get consistent results, I don't really understand why that's needed.

[sidewayss]

I am having a similar issue, but not with Python. It is happening on my clone of the MuseScore repo, and is illustrated by these two log files:
https://api.travis-ci.org/v3/job/484839047/log.txt
https://api.travis-ci.org/v3/job/484839034/log.txt

These are two Travis jobs that ran concurrently on the same git commit and push, because I have Travis setup on my repo and this branch is also attached to a pull request that runs Travis. These two log files are far from identical. Travis is producing different results on the same branch, the same commit.

Travis does have two separate commit ids for the two jobs. I don't know why that is. The issues are in the final stage of each build, running ctest scripts. Travis, using is the same code, generates different files that fail diff tests for different reasons. I don't expect you to know about MuseScore, but you can see near the bottom of the file, where it summarizes the failures, that in one log there are 27 failures, and in the other there are 25. Sounds impossible, but it's true, and there are specific diff differences that I can point to if you care to see them. Of course non of these failures happen when I run ctest locally, which is the worst part of this.

If this comment is deemed unrelated to this issue, I'll create a separate issue. I'm simply exasperated and looking for any way to resolve this, hoping that someone here might be able to clarify and point me in the right direction. Thanks!

[timothycrosley]

I've done a significant amount of work to improve consistency across environments and platforms in the last few releases, i hope I have resolved these issues and am therefore closing this report. If I closed incorrectly please comment to let me know!

Thanks!

~Timothy

[samuelcolvin]

Thanks Timothy, I use isort on almost all my projects and it's now extremely consistent across different platforms.