64-bit python on Windows 64-bit fails

[newville]

there were many issues with 64-bit python on Windows, basically preventing it from working at all.

[klauer]

I gave this a try after you mentioned it previously, and I can't seem to replicate the issue. Perhaps your PATH was set incorrectly, leading to the infamous dll hell?

Working output with x64 Python, 2.7.9:

C:\dev\pyepics>set PATH=c:\mc\envs\nsls2\;C:\dev\pyepics\dlls\win64
C:\dev\pyepics>set PYEPICS_LIBCA=C:\dev\pyepics\dlls\win64\ca.dll

C:\dev\pyepics>python -c "import epics; pv = epics.PV('foobar'); print pv; print pv.get(); import platform; print platform.architecture()[0]"
<PV 'foobar': not connected>
1.2345
64bit

[newville]

Oh, no, I beat hard on this, and tried many versions of 64-bit ca.dll/Com.dll. I believe this is now working, though only on 2.7.9 (not 2.7.8), and I broke something on other systems. I'll write a more complete of the solution shortly....

[newville]

I believe this is now resolved, and pyepics can work reliably on 64-bit Windows. There are 3 (separate) issues and caveats:

1. The fix requires Python 2.7.9. Notably, the current Anaconda 64-bit uses Python 2.7.8, and even with the other fixes, gives sporadic seg-faults, though many simple tests do work. I am not certain, but this may be due to Python Issue #13096: "Fixed segfault in CTypes POINTER handling of large values".
2. The layout of the function arguments, held in structure/classes event_handler_args and connection_args``need to add an 8-bit alignment padding after thechidfield or the data is garbled and cannot be extracted reliably. This was found by trial and error, and a test for 64-bit Python on Windows indbr.py(dbr.PY64_WINDOWS`) is used to add these paddiings.
3. An apparently "known issue" (but scantily documented except in Python issue tracker), requries callbacks that use large, complex data structure for function args to be declared to use

  callback = ctypes.CFUNCTYPE(None, ctypes.POINTER(args))(func)

on 64-bit Windows instead of (what is *required on all other systems):

  callback = ctypes.CFUNCTYPE(None, args)(func)

Again, the 64-bit-Windows test is used to make this distinction. This also has the consequence that each of the callback functions need to have

if dbr.PY64_WINDOWS: args = args.contents

to use the args the same way on all platforms.

[klauer]

I see why my test worked now - I saw the issue, unknowingly pulled in your fixes that came before the issue posted, and successfully ran the test. At least you have confirmation that it works elsewhere, I suppose...

That's very strange about the POINTER argument, and the struct padding is surprising too.

The byte sizes of the event_handler_args elements in order are: 8, 4, 1(=padding), 4, 4, 8, 4 -- that's ... odd. type (after the padding) is then at 0-based byte 13. And sizeof(c_long) == sizeof(chid_t) == 4 doesn't change from 32-bit to 64-bit, yet padding is inserted after chid_t in connection_args...

[newville]

@klauer

The byte sizes of the event_handler_args elements in order are: 8, 4, 1(=padding), 4, 4, 8, 4 --
that's ... odd. type (after the padding) is then at 0-based byte 13. And sizeof(c_long) ==
sizeof(chid_t) == 4 doesn't change from 32-bit to 64-bit, yet padding is inserted after chid_t
in connection_args...

Yes, it is all very weird, and I don't fully understand it. It's like there is some implied added bits (maybe related to passing a pointer to the structure). I wouldn't be surprised if there was a better solution, but this seems to pass many tests, so it can't be very wrong.

FWIW, in a simple C test on 64-bit windows (just adding a printf to caget.c), sizeof(chid) appears to 8!. I saw that, and thought "oh, that must be the problem". But if I make the chid_t declaration in the event and connection args a ctypes.c_long64, I never get the right value other items in the structure ('op', 'count', 'status', which can be easily verified as "very wrong").

The errors I saw, even with the CPOINTER change and the "padding" change with 2.7.8 (and I was really hoping to use Anaconda!) were even weirder:

Callbacks and events worked and gave sensible data values. What failed was doing a ca_element_count(chid), ca_field_type(chid), or ca_name(chid) on an existing, connected channel, which gave a seg-fault, and so no data, but didn't stop the Python interpreter. These functions only takes chid as the argument. And it always worked the first time, but like after an event had been seen, it failed. This makes no sense to be, unless it's something like chid is really not a C long. But, then what the heck is it, and why does it work everywhere else?

I'm willing to say these are all bugs in ctypes that got fixed between 2.7.8 and 2.7.9, but it makes very scary to say this actually works on 64-bit Windows!

[newville]

OK -- it gets slightly weirder (but perhaps in a better way):

For 64-bit Python 3.4, the Epics CA chid does need to be declared as 8 bytes (ctypes.c_int64), This makes sense, and corresponds to the fact that chid is 8 bytes for all 64-bit systems and that ctypes.c_long is 8 bytes on 64-bit Unices, but 4 bytes 64-bit Windows. With that, the extra byte of padding is not needed with Python 3.4. Using a 4-byte chid and a 1-byte padding is still needed for 64-bit Python 2.7.9.

So, I would say the situation on 64-bit Python 3 is understandable: chid has to be explicitly declared as 8-bytes, and large data structures for function arguments have to be wrapped/unwrapped with ctypes.POINTER. The behavior for 64-bit Python 2.7.9 is very weird, and it doesn't work at all for 64-bit Python 2.7.8.

OK, with all that, 3.2.4 is released!