Binary | and & operators

[ARF1]

I am trying to evaluate a expression of the form: a | (b << shift_amount)

I.e. I want to combine some bits from a with some bits from b.

While Numexpr seems to be be ok with binary shift operators, binary or seems not to work. Am I right to assume that it does not exist? That only a logical or exists?

Would it be possible to add binary and and or?

This would be very useful for binary data format processing in vectorized form: I am unpacking an array packed 11-bit integers which would benefit from multi-core execution.

Sample traceback:

Traceback (most recent call last):
  File "C:\Anaconda\envs\wxcomp\lib\site-packages\numexpr\necompiler.py", line 794, in evaluate
    compiled_ex = _numexpr_cache[numexpr_key]
KeyError: ('slc | tmp1', (('optimization', 'aggressive'), ('truediv', False)), (('slc', <class 'numpy.int64'>), ('tmp1', <class 'numpy.int64'>)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\My User\Documents\My Project\grib\grib\transform_gribfiles.py", line 57, in <module>
    main()
  File "C:\Users\My User\Documents\My Project\grib\grib\transform_gribfiles.py", line 21, in main
    a = rec.data
  File "C:\Users\My User\Documents\My Project\grib\grib\grib_reader.py", line 227, in data
    return self._unpacker_gpd_simple()
  File "C:\Users\My User\Documents\My Project\grib\grib\grib_reader.py", line 323, in _unpacker_gpd_simple
    x3 = unpackIntegersNumexpr(self.sections[7][5:], num_points, bit_len)
  File "C:\Users\My User\Documents\My Project\grib\grib\grib_reader.py", line 568, in unpackIntegersNumexpr
    slc = ne.evaluate('slc | tmp1')
  File "C:\Anaconda\envs\wxcomp\lib\site-packages\numexpr\necompiler.py", line 797, in evaluate
    NumExpr(ex, signature, **context)
  File "C:\Anaconda\envs\wxcomp\lib\site-packages\numexpr\necompiler.py", line 609, in NumExpr
    precompile(ex, signature, context)
  File "C:\Anaconda\envs\wxcomp\lib\site-packages\numexpr\necompiler.py", line 553, in precompile
    ast = typeCompileAst(ast)
  File "C:\Anaconda\envs\wxcomp\lib\site-packages\numexpr\necompiler.py", line 199, in typeCompileAst
    % (ast.value + '_' + retsig + basesig))
NotImplementedError: couldn't find matching opcode for 'or_bll'
Press any key to continue . . .

Versions:

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Numexpr version:   2.6.1
NumPy version:     1.11.2
Python version:    3.5.2 |Continuum Analytics, Inc.| (default, Jul  5 2016, 11:45:57) [MSC v.1900 32 bit (Intel)]
AMD/Intel CPU?     True
VML available?     True
VML/MKL version:   Intel(R) Math Kernel Library Version 11.3.3 Product Build 20160413 for 32-bit applications
Number of threads used by default: 2 (out of 2 detected cores)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

[robbmcleod]

You are correct that a Python & maps to a C && and | -> ||. These are the operations in interp_body.cpp:

    case OP_AND_BBB: VEC_ARG2(b_dest = (b1 && b2));
    case OP_OR_BBB: VEC_ARG2(b_dest = (b1 || b2));

There's no more space in the ascii wordspace for more operations to be added. I'm actively working on NumExpr3 which will have a 4-byte wordspace in which case I have bitwise & and |whereas and maps to && and or to ||. It's a big project though...

[ARF1]

@robbmcleod Thanks for the explanation.

I leave it up to you whether you want to keep this issue open as a reminder or close it because nothing immediate can be done about it.

[robbmcleod]

If it's of interest the alpha for 3.0 is up.:

NumExpr-3.0 branch

| is now bitwise or whereas or is logical or now. You also have ^ added and such, plus support for uint16 which would be better in your case.

[benjimin]

Would it be difficult to add bitwise_or and bitwise_and as functions?

I wasn't sure whether functions are counted the same as operators toward the wordspace limit? (It looked like trigonometric functions are not among the 128 opcodes..) Or is the next release expected to be version 3 soon anyway?

[robbmcleod]

Functions works sort of like UTF-8, in that you have an op-code to indicate that there's a second function code following in the extended space. However, there's only support for floating-point and complex number functions in 2.6.

I think version 3.0 will move into a beta stage by the end of the month. There's a couple of seg faults to deal with but I know where the problems lie.