Incompatibility with Python 3.7.6

[ZLLentz]

Python 3.7.6 was released on Dec 11th and is incompatible with pyepics, failing on import. Steps to reproduce:

$ git clone git@github.com:pyepics/pyepics.git
$ cd pyepics
$ conda create --name py375 python=3.7.5
$ conda create --name py376 python=3.7.6
$ conda activate py375
$ python
>>> import epics
$ conda activate py376
$ python
>>> import epics
Traceback (most recent call last):
  File "/u1/zlentz/conda/envs/py376/lib/python3.7/ctypes/__init__.py", line 97, in CFUNCTYPE
    return _c_functype_cache[(restype, argtypes, flags)]
KeyError: (None, (<class 'epics.dbr.access_rights_handler_args'>,), 1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/afs/slac.stanford.edu/u/lu/zlentz/pyepics/epics/__init__.py", line 29, in <module>
    from . import ca
  File "/afs/slac.stanford.edu/u/lu/zlentz/pyepics/epics/ca.py", line 761, in <module>
    dbr.access_rights_handler_args)
  File "/afs/slac.stanford.edu/u/lu/zlentz/pyepics/epics/dbr.py", line 339, in make_callback
    return ctypes.CFUNCTYPE(None, args)(func)
  File "/u1/zlentz/conda/envs/py376/lib/python3.7/ctypes/__init__.py", line 99, in CFUNCTYPE
    class CFunctionType(_CFuncPtr):
TypeError: item 1 in _argtypes_ passes a struct/union with a bitfield by value, which is unsupported.

Using

python                    3.7.6                h357f687_2    conda-forge

[newville]

@ZLLentz Thanks, I have not yet tried 3.7.6 myself. I'm not sure I fully understand that error message, but it looks like tolerance for not-so-well-defined wrapping of a C structure has been reduced.

And it does look like that access_rights_handler_args should be defined (according to the CA reference docs) as:

typedef struct ca_access_rights {
    unsigned    read_access:1;
    unsigned    write_access:1;
} caar;

/* arguments passed to user access rights handlers */
struct  access_rights_handler_args {
    chanId  chid;   /* channel id */
    caar    ar;     /* new access rights state */
};

which we translated to Python as

class access_rights_handler_args(ctypes.Structure):
    "access rights arguments"
    _fields_ = [('chid', chid_t),
                ('read_access', uint_t, 1),
                ('write_access', uint_t, 1)]

So, maybe the exception message is saying that we need to make a mapping of a ca_access_rights structure and then wrap that -- and maybe have to then change the unwrapping of that structure for ca._onAccessRightsEvent. That's kind of a guess, but if you're up for a bit of hacking around, maybe that's something to try.

Does anyone else have an idea about why this might be failing?

[klauer]

Here's where that exception comes from:
https://github.com/python/cpython/blob/00ac28ac4d06a311fc2386ef921b2603735ffae6/Modules/_ctypes/_ctypes.c#L2298-L2302

I wonder if this is just due to how bitfield layout isn't so well defined? Reference I'd be curious to hear from an expert about it.

An easy workaround would be to remove the bitfield, changing it into uint8 access and picking off the bits ourselves in 2 properties - read_access/write_access.

[newville]

@klauer Thanks -- I thought you were the expert ;). I thought maybe the message was enforcing that the second element of the access_rights_handler_args had to be a ctypes.Structure reflecting the ca_access_rights struct.

@ZLLentz I was able to reproduce this, and it looks like a simple conda update --all from Anaconda Python will install Py 3.7.6, breaking pyepics.

This does seem slightly urgent, but I don't really know what the solution will be. I'll try to look at this soon.

[newville]

@klauer Would you be willing to check if this looks right to you:

dbr.py:

class access_rights_handler_args(ctypes.Structure):
    "access rights arguments"
    _fields_ = [('chid', chid_t), ('access', uint_t)]

ca.py:

def _onAccessRightsEvent(args):
    'Access rights callback'
    try:
        entry = _chid_cache[_chid_to_int(args.chid)]
    except KeyError:
        return
    read = bool(args.access % 2)
    write = bool(args.access>=2)
    entry.run_access_event_callbacks(read, write)

This seems like it works for me with like 2 minutes of testing with 2 PVs (one RW, one RO).

[klauer]

@newville Wish I could say I'm an expert on bitfield portability and ctypes magic ;)

I think I'd do something like this, which keeps backward-compatibility and is a bit friendlier:

class access_rights_handler_args(ctypes.Structure):
    "access rights arguments"
    _fields_ = [('chid', chid_t), ('access', uint_t)]

    @property
    def read_access(self):
        return bool(self.access & 1)

    @property
    def write_access(self):
        return bool((self.access >> 1) & 1)

What do you think?