Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Datetime fix for MinGW. #214

Closed
wants to merge 6 commits into from

4 participants

@87
87 commented

Hi,

This should fix the 32/64 bits issues with the time type when using MinGW. (See also here)

@rgommers
Owner

It fixes all the failures with MinGW + python 2.5, with python 3.1 a few issues are left. This is with PR-213 and this one together.

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_datetime.py", line 1706, in test_datetime_y2038
    a = np.datetime64('2038-01-20T13:21:14')
OSError: Failed to use mktime to convert local time to UTC

======================================================================
ERROR: test_maskna.test_array_maskna_astype
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest
    self.test(*self.arg)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_maskna.py", line 212, in test_array_maskna_astype
    b = a.astype(dt2)
TypeError: complex() argument must be a string or a number

======================================================================
ERROR: Failure: OSError (Failed to use mktime to convert local time to UTC)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\failure.py", line 37, in runTest
    reraise(self.exc_class, self.exc_val, self.tb)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\_3.py", line 7, in reraise
    raise exc_class(exc_val).with_traceback(tb)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\loader.py", line 389, in loadTestsFromName
    addr.filename, addr.module)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 915, in <module>
    class TestArgmax(TestCase):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 941, in TestArgmax
    np.datetime64('2041-12-03T14:05:03')], 5),
OSError: Failed to use mktime to convert local time to UTC
@mwiebe
Owner

My guess would be that it's somehow not triggering the 64-bit time_t.

@87
87 commented

Hmm, maybe mktime needs to be _mktime64 for the same reasons. I don't get why it works for Python 2.5 though..

@rgommers
Owner

The errors on 2.5 were never the same ones as on 2.7/3.1. Perhaps related to the Python 2.5 installers being built with VS 2005 (IIRC) and 2.7/3.1 with VS 2008.

@87
87 commented

Ralf, if you change mktime on this line to _mktime64, will it fix the issue on Python 3.1?

@87
87 commented

Ah, I think Python 2.5 might even have been compiled with VS 2003, if I remember correctly.

@87
87 commented

I think I understand why Python 2.5 succeeds. It doesn't use the custom functionality, and therefore falls back on a 32-bits time_t and -time functionality. Because the datetime code makes sure that 32-bits time_t size is handled with a kind of overflow mechanism for dates beyond 2038, it does not fail in those cases..

@rgommers
Owner

Han, after changing mktime to _mktime64 as you suggest, the same tests are still failing. Just different errors than before:

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_datetime.py", line 1707, in test_datetime_y2038
    assert_equal(str(a)[:-5], '2038-01-20T13:21:14')
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_maskna.test_array_maskna_astype
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\nose-3.0.0.dev-py3.1.egg\nose\case.py", line 178, in runTest
    self.test(*self.arg)
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_maskna.py", line 212, in test_array_maskna_astype
    b = a.astype(dt2)
TypeError: complex() argument must be a string or a number

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmax)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 979, in test_combinations
    assert_equal(np.argmax(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python31\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1047, in test_combinations
    assert_equal(np.argmin(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time
@87
87 commented

That is strange, have you applied all patches here? This seems to be the same bug we had before (with a 32-bits time_t), but that should have been fixed.

@rgommers
Owner

Yes, PR-213, then this PR, then the one-line change rawtime = _mktime64(&tm_); in `datetime_strings.c.

The difference with the 2.5 errors from before is that those were using localtime and 3.1 is now failing on _localtime64_s.

@rgommers
Owner

That's it for tonight for me.

@87
87 commented

Okay, I'll take a look at it tomorrow.

@87
87 commented

I tried to reproduce the issue with 64-bits localtime, but MinGW and win64 don't go together. I had to fix some issues with the installer, but after getting things to work on a 32-bit Python32 environment, it crashes after a few tests.

@87
87 commented

Hmm, it also crashed on XP, using Python26 and using MinGW 4.6.1.

The crash happens for me in test_arrayprint, in the TestComplexArray class and can be reproduced by doing print np.array([0j], np.complex64)

@87
87 commented

I pushed a few changes to add a function get_mktime to wrap mktime and _mktime64 and also added an improvement to the dll finder to look for the correct architecture.

@87
87 commented

The test_datetime passes for me on WinXP with Python26 using MinGW. The test_multiarray crashes, however, just like the general test. (Is that the reason for you to stick to the older version of MinGW? I cannot find a download for it, though...)

@rgommers
Owner

This gives the same errors as before, on both Python 2.7 and 3.1:

======================================================================
ERROR: test_datetime_y2038 (test_datetime.TestDateTime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python27\lib\site-packages\numpy\core\tests\test_datetime.py", line 1707, in test_datetime_y2038
    assert_equal(str(a)[:-5], '2038-01-20T13:21:14')
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmax)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py", line 979, in test_combinations
    assert_equal(np.argmax(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time

======================================================================
ERROR: test_combinations (test_multiarray.TestArgmin)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Z:\Users\rgommers\.wine\drive_c\Python27\lib\site-packages\numpy\core\tests\test_multiarray.py", line 1047, in test_combinations
    assert_equal(np.argmin(arr), pos, err_msg="%r"%arr)
OSError: Failed to use '_localtime64_s' to convert to a local time
@87
87 commented

Yeah, I figured. I'm a bit at a loss, though..

@mwiebe
Owner

What I would suggest is instrumenting that part of the code so that before the call to the failing function call, it prints out the value of the parameters being passed in. This way, we could validate that nothing has munged the value before getting there, or create a separate tiny test program to experiment with what's going on.

@87
87 commented

I cannot find the installer for MinGW 3.4.5, it is too old. I tried to download it from the Wayback Machine, but they don't have it either. The newest version of MinGW passes the datetime tests, but crashes on other tests.

Do you have a version I can download from somewhere, Ralf? Otherwise I'm afraid there is not much more I can do to solve these issues..

@rgommers
Owner

The installer I used was MinGW-5.1.6.exe, which installs 3.4.5. All the part are still linked at http://wiki.codeblocks.org/index.php?title=MinGW_installation#GCC_3.4.5_manual_install for example. But I have to say that this is getting very questionable to use. Would be much better indeed to make gcc-4.x work. I'll have a look now if David's branch is public, that may solve your crashes.

@rgommers
Owner

His branch doesn't seem to be public, I asked on the list for it.

@87
87 commented

Thanks! I've installed the separate files for version 3.4.5 from the given site, but I am running into some problems with it. At first MinGW could not find the .a files that were generated by the setup script (which I had to put into a MinGW directory manually), but now I get a few errors I cannot get around:

Running spawnve(0,'C:\\MinGW\\bin\\g++.exe',['C:\\MinGW\\bin\\g++.exe', '-mno-cygwin', '_configtest.o', '-lmsvcr90', '-o', '_configtest.exe'],os.environ)
_update_environment(...)
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../mingw32/lib/libmsvcr90.a(dkxds01142.o):(.text+0x0): multiple definition of `atexit'
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../crt2.o:crt1.c:(.text+0x2b0): first defined here
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../../mingw32/lib/libmsvcr90.a(dkxds00725.o):(.text+0x0): multiple definition of `_onexit'
C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/../../../crt2.o:crt1.c:(.text+0x2c0): first defined here
collect2: ld returned 1 exit status
failure.

Which gives a "broken toolchain" error. Thing is, according to the symbol export, there should not be multiple definitions.
Did you run into these issues? (If so, how did you solve it?)

@rgommers
Owner

That is a very common issue, and looks familiar. Seems to be a MinGW bug, http://comments.gmane.org/gmane.comp.emulators.wine.devel/17437.

I documented how I set up Wine, see the "To install Wine on OS X..." paragraph at https://github.com/numpy/numpy/blob/master/doc/HOWTO_RELEASE.rst.txt. I think my mention of an issue with Wine + MinGW is exactly this, and I fixed it by installing Wine with http://code.google.com/p/osxwinebuilder/.

@charris
Owner

Where are we with this?

@rgommers
Owner

Down to 3 failures, but those are very hard to fix. MinGW is a real problem; 3.4.5 is very hard to even find, while 4.x is not supported. David was close to finishing the move to 4.x, but ran into some problems with DLLs and out of time. I'm not sure what the best way forward is at this point.

@charris
Owner

Perhaps this could go in now and additional fixes could be made later. Does that seem reasonable?

@rgommers
Owner

That's fine with me.

@charris
Owner

Pushed in 913e55d..c17738a.

@charris charris closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
45 numpy/core/src/multiarray/datetime_strings.c
@@ -25,7 +25,11 @@
#include "datetime_strings.h"
/* Platform-specific time_t typedef */
-typedef time_t NPY_TIME_T;
+#if defined(NPY_MINGW_USE_CUSTOM_MSVCR)
+ typedef __time64_t NPY_TIME_T;
+#else
+ typedef time_t NPY_TIME_T;
+#endif
/*
* Wraps `localtime` functionality for multiple platforms. This
@@ -43,7 +47,7 @@ get_localtime(NPY_TIME_T *ts, struct tm *tms)
func_name = "localtime_s";
goto fail;
}
- #elif defined(__GNUC__) && defined(NPY_MINGW_USE_CUSTOM_MSVCR)
+ #elif defined(NPY_MINGW_USE_CUSTOM_MSVCR)
if (_localtime64_s(tms, ts) != 0) {
func_name = "_localtime64_s";
goto fail;
@@ -88,7 +92,7 @@ get_gmtime(NPY_TIME_T *ts, struct tm *tms)
func_name = "gmtime_s";
goto fail;
}
- #elif defined(__GNUC__) && defined(NPY_MINGW_USE_CUSTOM_MSVCR)
+ #elif defined(NPY_MINGW_USE_CUSTOM_MSVCR)
if (_gmtime64_s(tms, ts) != 0) {
func_name = "_gmtime64_s";
goto fail;
@@ -118,6 +122,37 @@ get_gmtime(NPY_TIME_T *ts, struct tm *tms)
}
/*
+ * Wraps `mktime` functionality for multiple platforms. This
+ * converts a local time struct to an UTC value.
+ *
+ * Returns timestamp on success, -1 on failure.
+ */
+static NPY_TIME_T
+get_mktime(struct tm *tms)
+{
+ char *func_name = "<unknown>";
+ NPY_TIME_T ts;
+#if defined(NPY_MINGW_USE_CUSTOM_MSVCR)
+ ts = _mktime64(tms);
+ if (ts == -1) {
+ func_name = "_mktime64";
+ goto fail;
+ }
+#else
+ ts = mktime(tms);
+ if (ts == -1) {
+ func_name = "mktime";
+ goto fail;
+ }
+#endif
+ return ts;
+fail:
+ PyErr_Format(PyExc_OSError, "Failed to use '%s' to convert "
+ "local time to UTC", func_name);
+ return -1;
+}
+
+/*
* Converts a datetimestruct in UTC to a datetimestruct in local time,
* also returning the timezone offset applied.
*
@@ -234,10 +269,8 @@ convert_datetimestruct_local_to_utc(npy_datetimestruct *out_dts_utc,
tm_.tm_isdst = -1;
/* mktime converts a local 'struct tm' into a time_t */
- rawtime = mktime(&tm_);
+ rawtime = get_mktime(&tm_);
if (rawtime == -1) {
- PyErr_SetString(PyExc_OSError, "Failed to use mktime to "
- "convert local time to UTC");
return -1;
}
View
6 numpy/distutils/mingw32ccompiler.py
@@ -250,6 +250,7 @@ def find_python_dll():
# - in system32,
# - ortherwise (Sxs), I don't know how to get it.
lib_dirs = []
+ lib_dirs.append(sys.prefix)
lib_dirs.append(os.path.join(sys.prefix, 'lib'))
try:
lib_dirs.append(os.path.join(os.environ['SYSTEMROOT'], 'system32'))
@@ -302,13 +303,16 @@ def generate_def(dll, dfile):
def find_dll(dll_name):
+ arch = {'AMD64' : 'amd64',
+ 'Intel' : 'x86'}[get_build_architecture()]
+
def _find_dll_in_winsxs(dll_name):
# Walk through the WinSxS directory to find the dll.
winsxs_path = os.path.join(os.environ['WINDIR'], 'winsxs')
if not os.path.exists(winsxs_path):
return None
for root, dirs, files in os.walk(winsxs_path):
- if dll_name in files:
+ if dll_name in files and arch in root:
return os.path.join(root, dll_name)
return None
Something went wrong with that request. Please try again.