astropy.log: The process cannot access the file because it is being used by another process

[embray]

c:\windows\temp\tmpyzcvmqastropy_config\astropy\astropy.log: The process cannot access the file because it is being used by another process

This is an error I get sometimes when the tests are run in Windows on Jenkins. Right now this is only happening in my staging branch where I'm testing #495, though I'd like to merge that work.

It always happens at the exact same point in the tests. To give a little further context:

astropy\io\fits\tests\test_image.py:883: TestImageFunctions.test_scale_back PASSED
astropy\io\fits\tests\test_image.py:907: TestImageFunctions.test_scale_back_compressed PASSED
astropy\io\fits\tests\test_image.py:948: TestImageFunctions.test_insufficient_compression_allocation error: c:\windows\temp\tmpyzcvmqastropy_config\astropy\astropy.log: The process cannot access the file because it is being used by another process

E:\Builds\astropy-winxp32-staging-iguananaut\workspaces\COMPILER\mingw32\NUMPY_VER\numpy-1.5.1\PYTHON_VER\python-2.7>exit 1 

It seems like the test runner itself is crashing, because the tests just stop there. The test it's stopping at is a slightly bizarre one, but it doesn't do anything I can think of that would affect the log, so I'm not convinced the test itself is relevant (though I might try disabling it). This occurs on some test configurations but not others, though I haven't discerned any pattern so I think it's more or less non-deterministic. When I run the tests manually in the same configuration it does not occur, so it seems like it might be due to an interaction with Jenkins.

Any hypotheses?

[embray]

Hrm...disabling that one test did seem to go better, though the full build still didn't succeed due to github timing out on some of the later configurations. But until that happened all the previous configurations succeeded. Giving it one more try.

[eteq]

What I don't understand is why it should care at all that another file object is using it. There isn't supposed to be any locking or anything. So why doesn't it just append to the file with no concern that another object is using it?

[embray]

Well I found that the test it's occurring on maybe does have something to do with it--when I disable that test I don't see this error. It's still very odd though.

[eteq]

Definitely very odd. I guess the workaround is to skip that one test if you're on windows? If it appears somewhere else later perhaps it requires more investigation...

[embray]

I'll go ahead and assign myself to this so that it doesn't get lost, and I'll just disable that test on Windows so I can move forward on merging #495. I still want to figure out what's going on here at some point though.

[embray]

I'm getting this error now in a different test that was included in 3e0fe42.

I'm not sure the error is directly related to the logging, because if I disable all logging to files all that happens is the process exits without reporting anything. It's more like it's segfaulting.

[embray]

The attached PR fixes the issue as originally reported, and fixes a new issue that cropped up in 3e0fe42 which had the same symptoms and turns out to be caused by the same thing.

In short, my hacked around fake realloc was causing an access violation on Windows, though not usually on Linux. This refactors the compress_hdu() function so that such a hack is not necessary.

[embray]

Assuming the tests for this pass on Travis, I need to merge this soon as it's causing the tests to fail on Windows.

[eteq]

Seems fine to me, but why did you delete the test for it? Or is that using some of the code you deleted (I didn't see where, but I admit I'm not fully understanding this)

[embray]

@eteq As explained in the commit message, the test that was deleted no longer makes sense with the way the code was reorganized, but there's a different test that tests the same corner case implicitly (it's a difficult case to force explicitly which is what the old test did).