1.1.1: Testing astropy.wcs fails with Python-3.5 and wcslib-5.12

[olebole]

On Debian, the astropy.wcs tests fails reproducible with some corrupted memory:

astropy/wcs/tests/test_wcs.py::test_calc_footprint_3 PASSED
astropy/wcs/tests/test_wcs.py::test_sip PASSED
astropy/wcs/tests/test_wcs.py::test_printwcs PASSED
astropy/wcs/tests/test_wcs.py::test_invalid_spherical PASSED
astropy/wcs/tests/test_wcs.py::test_no_iteration PASSED
astropy/wcs/tests/test_wcs.py::test_sip_tpv_agreement PASSED
astropy/wcs/tests/test_wcs.py::test_tpv_copy PASSED
astropy/wcs/tests/test_wcs.py::test_hst_wcs *** Error in `/usr/bin/python3.5':
    free(): corrupted unsorted chunks: 0x000000000bc2d530 ***

I wonder whether this is connected with #4441. wcslib is compiled here with flex-2.5.39 (the current version in Debian).

Version 1.1 compiles without this problem, so this is a regression.

[embray]

Probably related to #4441. I doubt this is particular to v1.1.1, which did not contain any significant changes to WCS code at the C level. That is, it probably is not a regression compared to v1.1, just to be clear.

[lupinix]

I can confirm the issue with Fedora. We are working on wcslib update there. Builds with wcslib 4.25.1 don't show this issue.

astropy/wcs/tests/test_pickle.py ......
astropy/wcs/tests/test_profiling.py ..................................
astropy/wcs/tests/test_utils.py .........................................
astropy/wcs/tests/test_wcs.py ......................................................................./var/tmp/rpm-tmp.7zgjNg: line 36: 28927 Aborted                 (core dumped) py.test-3.5 -k "not test_web_profile" astropy
error: Bad exit status from /var/tmp/rpm-tmp.7zgjNg (%check)

Full build log at https://copr-be.cloud.fedoraproject.org/results/lupinix/wcslib5-test/fedora-rawhide-x86_64/00153269-python-astropy/build.log.gz

[olebole]

@embray I tested both versions 1.1 and 1.1.1 in an identical environment (including wcslib 5.12) astropy-1.1 does not have this issue, but astropy-1.1.1 has.

[olebole]

I could track this down to commit 6d4552e: After reverting this patch, all tests succeed (on x86_64). I agree that this is strange; however I doublechecked it.

[embray]

How strange. Have you managed to get a stack trace for the error? I almost wonder if something is awry with the re module in Python 3.5.

[embray]

Though I'll also need to go through that commit with a fine-toothed comb and make sure it's not doing anything materially different from the previous code. It looked right to me before but maybe there's a subtle mistake in it somewhere.

[astrofrog]

@olebole - could you also try and simplify the regular expression to see if the error goes away?

[embray]

Yeah looking at it again I see several areas where it could be simplified (not that it's that complex in the first place).

[embray]

Here's a simpler equivalent regexp:

'''^[AB]P?_1?[0-9]_1?[0-9][A-Z]?$'''

[embray]

See also this note: 6d4552e#commitcomment-15453711

That might also be related--I'm not sure if it's properly deleting all the SIP keywords because of this.

[embray]

@olebole @lupinix Could one or both of you confirm whether the patch in #4492 fixes the issue? Thanks.

[olebole]

I just made a test build with both the simplified regexp and the list() wrapping, which was successful; they fix the issue. Do you need to have them tested separately as well?

[embray]

@olebole I'd be curious, yeah. At this point I think the fix in #4492 is more likely the culprit than anything in the regexp (though I'll probably commit the simplified regexp separately too).

[lupinix]

I can confirm: Builds fine now: https://copr.fedoraproject.org/coprs/lupinix/wcslib5-test/build/153475/

[olebole]

I can confirm that the list() wrapping in #4492 solved the problem.

[astrofrog]

@olebole - so just to be clear, we can close this?

[olebole]

Yep.