Skip to content

Datetime fix for MinGW. #214

Closed
wants to merge 6 commits into from

4 participants

@87
87 commented Feb 20, 2012

Hi,

This should fix the 32/64 bits issues with the time type when using MinGW. (See also here)

@rgommers
NumPy member

It fixes all the failures with MinGW + python 2.5, with python 3.1 a few issues are left. This is with PR-213 and this one together.

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_datetime.py", line 1706, in test_datetime_y2038
    a = np.datetime64('2038-01-20T13:21:14')
OSError: Failed to use mktime to convert local time to UTC

======================================================================
ERROR: test_maskna.test_array_maskna_astype
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest
    self.test(*self.arg)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_maskna.py", line 212, in test_array_maskna_astype
    b = a.astype(dt2)
TypeError: complex() argument must be a string or a number

======================================================================
ERROR: Failure: OSError (Failed to use mktime to convert local time to UTC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\failure.py", line 37, in runTest
    reraise(self.exc_class, self.exc_val, self.tb)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\_3.py", line 7, in reraise
    raise exc_class(exc_val).with_traceback(tb)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\loader.py", line 389, in loadTestsFromName
    addr.filename, addr.module)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 915, in <module>
    class TestArgmax(TestCase):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 941, in TestArgmax
    np.datetime64('2041-12-03T14:05:03')], 5),
OSError: Failed to use mktime to convert local time to UTC
@mwiebe
NumPy member
mwiebe commented Feb 20, 2012

My guess would be that it's somehow not triggering the 64-bit time_t.

@87
87 commented Feb 20, 2012

Hmm, maybe mktime needs to be _mktime64 for the same reasons. I don't get why it works for Python 2.5 though..

@rgommers
NumPy member

The errors on 2.5 were never the same ones as on 2.7/3.1. Perhaps related to the Python 2.5 installers being built with VS 2005 (IIRC) and 2.7/3.1 with VS 2008.

@87
87 commented Feb 20, 2012

Ralf, if you change mktime on this line to _mktime64, will it fix the issue on Python 3.1?

@87
87 commented Feb 20, 2012

Ah, I think Python 2.5 might even have been compiled with VS 2003, if I remember correctly.

@87
87 commented Feb 20, 2012

I think I understand why Python 2.5 succeeds. It doesn't use the custom functionality, and therefore falls back on a 32-bits time_t and -time functionality. Because the datetime code makes sure that 32-bits time_t size is handled with a kind of overflow mechanism for dates beyond 2038, it does not fail in those cases..

@rgommers
NumPy member

Han, after changing mktime to _mktime64 as you suggest, the same tests are still failing. Just different errors than before:

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_datetime.py", line 1707, in test_datetime_y2038
    assert_equal(str(a)[:-5], '2038-01-20T13:21:14')
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_maskna.test_array_maskna_astype
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest
    self.test(*self.arg)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_maskna.py", line 212, in test_array_maskna_astype
    b = a.astype(dt2)
TypeError: complex() argument must be a string or a number

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmax)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 979, in test_combinations
    assert_equal(np.argmax(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1047, in test_combinations
    assert_equal(np.argmin(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time
@87
87 commented Feb 20, 2012

That is strange, have you applied all patches here? This seems to be the same bug we had before (with a 32-bits time_t), but that should have been fixed.

@rgommers
NumPy member

Yes, PR-213, then this PR, then the one-line change rawtime = _mktime64(&tm_); in `datetime_strings.c.

The difference with the 2.5 errors from before is that those were using localtime and 3.1 is now failing on _localtime64_s.

@rgommers
NumPy member

That's it for tonight for me.

@87
87 commented Feb 20, 2012

Okay, I'll take a look at it tomorrow.

@87
87 commented Feb 21, 2012

I tried to reproduce the issue with 64-bits localtime, but MinGW and win64 don't go together. I had to fix some issues with the installer, but after getting things to work on a 32-bit Python32 environment, it crashes after a few tests.

@87
87 commented Feb 21, 2012

Hmm, it also crashed on XP, using Python26 and using MinGW 4.6.1.

The crash happens for me in test_arrayprint, in the TestComplexArray class and can be reproduced by doing print np.array([0j], np.complex64)

@87
87 commented Feb 21, 2012

I pushed a few changes to add a function get_mktime to wrap mktime and _mktime64 and also added an improvement to the dll finder to look for the correct architecture.

@87
87 commented Feb 21, 2012

The test_datetime passes for me on WinXP with Python26 using MinGW. The test_multiarray crashes, however, just like the general test. (Is that the reason for you to stick to the older version of MinGW? I cannot find a download for it, though...)

@rgommers
NumPy member

This gives the same errors as before, on both Python 2.7 and 3.1:

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python27\lib\site-packages\numpy\core\tests\test_datetime.py", line 1707, in test_datetime_y2038
    assert_equal(str(a)[:-5], '2038-01-20T13:21:14')
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmax)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py", line 979, in test_combinations
    assert_equal(np.argmax(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1047, in test_combinations
    assert_equal(np.argmin(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time
@87
87 commented Feb 21, 2012

Yeah, I figured. I'm a bit at a loss, though..

@mwiebe
NumPy member
mwiebe commented Feb 22, 2012

What I would suggest is instrumenting that part of the code so that before the call to the failing function call, it prints out the value of the parameters being passed in. This way, we could validate that nothing has munged the value before getting there, or create a separate tiny test program to experiment with what's going on.

@87
87 commented Feb 24, 2012

I cannot find the installer for MinGW 3.4.5, it is too old. I tried to download it from the Wayback Machine, but they don't have it either. The newest version of MinGW passes the datetime tests, but crashes on other tests.

Do you have a version I can download from somewhere, Ralf? Otherwise I'm afraid there is not much more I can do to solve these issues..

@rgommers
NumPy member

The installer I used was MinGW-5.1.6.exe, which installs 3.4.5. All the part are still linked at http://wiki.codeblocks.org/index.php?title=MinGW_installation#GCC_3.4.5_manual_install for example. But I have to say that this is getting very questionable to use. Would be much better indeed to make gcc-4.x work. I'll have a look now if David's branch is public, that may solve your crashes.

@rgommers
NumPy member

His branch doesn't seem to be public, I asked on the list for it.

@87
87 commented Mar 1, 2012

Thanks! I've installed the separate files for version 3.4.5 from the given site, but I am running into some problems with it. At first MinGW could not find the .a files that were generated by the setup script (which I had to put into a MinGW directory manually), but now I get a few errors I cannot get around:

Running spawnve(0,'C:\\MinGW\\bin\\g++.exe',['C:\\MinGW\\bin\\g++.exe', '-mno-cygwin', '_configtest.o', '-lmsvcr90', '-o', '_configtest.exe'],os.environ)
_update_environment(...)
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../mingw32/lib/libmsvcr90.a(dkxds01142.o):(.text+0x0): multiple definition of `atexit'
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../crt2.o:crt1.c:(.text+0x2b0): first defined here
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../mingw32/lib/libmsvcr90.a(dkxds00725.o):(.text+0x0): multiple definition of `_onexit'
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../crt2.o:crt1.c:(.text+0x2c0): first defined here
collect2: ld returned 1 exit status
failure.

Which gives a "broken toolchain" error. Thing is, according to the symbol export, there should not be multiple definitions.
Did you run into these issues? (If so, how did you solve it?)

@rgommers
NumPy member
rgommers commented Mar 2, 2012

That is a very common issue, and looks familiar. Seems to be a MinGW bug, http://comments.gmane.org/gmane.comp.emulators.wine.devel/17437.

I documented how I set up Wine, see the "To install Wine on OS X..." paragraph at https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt. I think my mention of an issue with Wine + MinGW is exactly this, and I fixed it by installing Wine with http://code.google.com/p/osxwinebuilder/.

@charris
NumPy member
charris commented Mar 24, 2012

Where are we with this?

@rgommers
NumPy member

Down to 3 failures, but those are very hard to fix. MinGW is a real problem; 3.4.5 is very hard to even find, while 4.x is not supported. David was close to finishing the move to 4.x, but ran into some problems with DLLs and out of time. I'm not sure what the best way forward is at this point.

@charris
NumPy member
charris commented Apr 7, 2012

Perhaps this could go in now and additional fixes could be made later. Does that seem reasonable?

@rgommers
NumPy member
rgommers commented Apr 8, 2012

That's fine with me.

@charris
NumPy member
charris commented Apr 15, 2012

Pushed in 913e55d..c17738a.

@charris charris closed this Apr 15, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.