Docker test environment updates

[wiredfool]

Docker test environment updates

• Enable centos-7, fedora-25,
• Femove fedora-24

[wiredfool]

I really don't understand the coverage reports here. Looking at one of the docker runs, ubuntu-xenial:

• According to travis, we're covering python scripts: https://travis-ci.org/python-pillow/Pillow/jobs/299056249#L2287
• According to coveralls, we're not. And we're not covering much of the C files https://coveralls.io/jobs/30973729
• According to CodeCov, we're either not affecting the coverage, or it's up 0.92%, or they don't have any record of the base version.
• On my machine, I'm only seeing evidence of python coverage, not c.

[hugovk]

• According to travis, we're covering python scripts: https://travis-ci.org/python-pillow/Pillow/jobs/299056249#L2287

That coverage report is Python-only and comes before after_success.sh is run, where the C coverage is combined. Combined coverage is meant to be uploaded to Coveralls/Codecov.

[hugovk]

Here's the Codecov report for this commit, which looks more-or-less sensible Python + C coverage: https://codecov.io/gh/python-pillow/Pillow/tree/387ec79cebad1be4fe8670200032afea3ae20402

[hugovk]

I did notice a drop in coverage today:

https://codecov.io/gh/python-pillow/Pillow/commits

PR #2841 did nothing to change coverage (just some lines a test changed), and the PR build itself reported no change in coverage.

So what changed?

The "Make and test with coverage" PR was merged in the interim: python-pillow/docker-images#12.

• Changed test to use tempfile #2841 PR build: https://codecov.io/gh/python-pillow/Pillow/tree/28119dd68dc7915b7fd2d007f9e0742f1b6744fd
• Changed test to use tempfile #2841 merge build: https://codecov.io/gh/python-pillow/Pillow/tree/79f2ae0c7560bd2ac63befeb3553e369acdc75c5

Build	Files	Tracked lines	Lines hit	Lines missed	Coverage
PR	Project Totals (165 files)	22,027	18,312	3,715	83.13%
merge	Project Totals (165 files)	22,508	18,392	4,116	81.71%

So tracked lines went up, as did lines hit and lines missed, causing a dip in the coverage percentage.

These changes are only in C files. Python numbers are identical.

Checking a few C files, the main difference is return NULL lines which were white before (not counted) are now green (covered) or more likely red (not covered). I also noticed some case XYZ: statements changing from white to red/green.

https://codecov.io/gh/python-pillow/Pillow/src/28119dd68dc7915b7fd2d007f9e0742f1b6744fd/_imaging.c#L2944
https://codecov.io/gh/python-pillow/Pillow/src/79f2ae0c7560bd2ac63befeb3553e369acdc75c5/_imaging.c#L2944

I'm not quite sure what accounts for this. I'd guess it might be different versions of software used to instrument or measure or merge the code coverage in the new Dockerfiles, and that's changing the mixup.

Whatever it is must account for the +0.92% in this commit, which removes one old dockerfile, and adds two newer ones.

[hugovk]

Just some notes about the differences between "native" and Docker builds:

native:

coverage erase
python setup.py clean
CFLAGS="-coverage" python setup.py build_ext --inplace

coverage run --append --include=PIL/* selftest.py
coverage run --append --include=PIL/* -m nose -vx Tests/test_*.py

Docker:

make clean && \
make install-coverage && \
/usr/bin/xvfb-run -a python ./test-installed.py -v --with-coverage

Docker, extracting commands:

# make clean
python setup.py clean
rm PIL/*.so || true
rm -r build || true
find . -name __pycache__ | xargs rm -r || true

# make install-coverage
CFLAGS="-coverage" python setup.py build_ext install
python selftest.py --installed

# test-installed.py
-> calls `nose.main()` with Tests/test*.py

Main thing is native is in-place, Docker is installed.

[hugovk]

Comparing Travis build logs between native 3.6 and Docker ubuntu-xenial-amd64:

Docker can't find the coverage command in after_success.sh to do coverage report. This is odd, but doesn't seem to be the real problem. It clearly ran the nose tests with coverage, because it output a summary. Was that in a different context to Dockerfile's pip install coveralls? Anyway, it gets installed in the current context in the very next line during pip install codecov. Let's fix this.

The main problem looks like installed builds don't collect C coverage properly. Just a few lines earlier is the C coverage results from lcov --remove coverage.info '/usr/*' -o coverage.filtered.info, showing in-place:

Summary coverage rate:
  lines......: 79.4% (9593 of 12089 lines)
  functions..: 87.2% (860 of 986 functions

Installed:

Summary coverage rate:
  lines......: 14.3% (1712 of 12001 lines)
  functions..: 16.3% (158 of 972 functions)

I can reproduce this on a Mac. genhtml coverage.filtered.info --output-directory out can be used to create HTML reports, confirming low numbers. In-place v. installed:

In-place v. installed:

[wiredfool]

I was looking in the wrong place earlier for the C coverage files, so that's explained a bit.

Ok, so in place is required for C coverage. I find that unfortunate, mainly in terms of testing and other uses for having the whole pile of docker images locally. Essentially, as the Pillow directory is mounted into the docker file, it restricts the docker image tree from ever having more than one concurrent run, as the incompatible .so files will stomp all over eachother. So, no parallel runs, debugging is harder, and all that entails. It's less of an issue on travis, but that's only because they're all running on independent machines anyway.

Was that in a different context to Dockerfile's pip install coveralls

Everything in after-success.sh is in the context of the test machine, not the docker image. So, possibly different python, different OS, different architecture, definitely different virtualenvs. The only thing that's actually in the docker context is the /test script.

[hugovk]

It may just be a matter of paths and configuration. In any case, having in-place on Travis doesn't prevent the possibility of installed for local debugging.

[hugovk]

And still not quite sure how this accounts for the coverage percentage change but let's see after switching Docker to in-place.