ImportError (google.protobuf) with rdm_test_server.py

[ssilverman]

When I try to run rdm_test_server.py, I get this:

Traceback (most recent call last):
File "/usr/local/bin/rdm_test_server.py", line 38, in
from ola.ClientWrapper import ClientWrapper, SelectServer
File "/usr/local/lib/python2.7/site-packages/ola/ClientWrapper.py", line 28, in
File "/usr/local/lib/python2.7/site-packages/ola/OlaClient.py", line 21, in
File "/usr/local/lib/python2.7/site-packages/ola/rpc/StreamRpcChannel.py", line 20, in
ImportError: No module named google.protobuf

I'm using homebrew to install ola. I've tried reinstalling protobuf and ola, but I get the same error.

System: OSX 10.14.6

[peternewman]

Hi @ssilverman ,

We are having the same issue too, you can see our progress in #1584 .

We're currently doing:
pip install --user protobuf==3.1.0

I think Homebrew have switched to Python 3 only for Protobuf, see #1538 for our progress on that front.

Although did ./configure succeed for you? It may just be a path issue in that case. Could you post your config.log (probably before trying to re-run it).

[ssilverman]

Before I try anything I'll gladly send you the config.log file, but do you know the path of the one you're looking for?

[peternewman]

I don't know specifically, but this looks relevant:
https://apple.stackexchange.com/questions/164525/what-directory-does-homebrew-use-to-build-programs

Other than that, it will likely be in an ola or olad folder if you can find any config.log files on your machine.

[ssilverman]

There's no config.log files that I could find, related to the ola install. After installing ola, however, there's also no configure file. Are you referring to one that comes with a git clone or one that comes with the homebrew install?

[kripton]

@ssilverman : Could you please retry, now that #1605 has been merged? It might have fixed your issue :)

[peternewman]

The RDM tests still don't run under Python 3 due to a few outstanding issues @kripton (xrange being one of the easier ones), so while it may have progressed things and #1655 will have done too, the tests themselves still aren't quite there unfortunately.

[brucelowekamp]

It was never complete, but if you're interested I had some of the work done in the py3-build branch of https://github.com/brucelowekamp/ola.git. There are some imports and other things that I think were necessary beyond the conversion of print and xrange (which is of course the bulk of the changes). I *think* this was the result of running futurize and a couple other tools over the codebase. But I remember there being some unicode stuff in a web module that I couldn't sort out, and there were definitely a few fun things like: ./tools/rdm/TestDefinitions.py: ALL_MODES = ALLOWED_MODES + [3] + range(0x80, 0xe0) that didn't get converted automatically to list(range), and I realized I needed to go back and find them. That was when I narrowed the scope to just the API... Bruce

…

On Wed, Nov 18, 2020 at 7:32 PM Peter Newman ***@***.***> wrote: The RDM tests still don't run under Python 3 due to a few outstanding issues @kripton <https://github.com/kripton> (xrange being one of the easier ones), so while it may have progressed things and #1655 <#1655> will have done too, the tests themselves still aren't quite there unfortunately. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNHN5KRMPXNZF7RGFFGRG3SQRRSDANCNFSM4JRA4RUQ> .

[peternewman]

Thanks @brucelowekamp . I've also done the print changes in #1685 which I'll merge in as it didn't seem to hurt and seemed to be sufficient to get the compileall stuff to work, although flake8 needs more work, and actually running seems to need more still in some cases.

I went for a slightly different xrange/range solution as in theory it should be more efficient on python2 I think:
https://github.com/OpenLightingProject/ola/pull/1685/files#diff-5b3f8e5df8e5a430e32762b12ed99f846058b721ae6049489f7a6058fbd8f45eR25-R31

[brucelowekamp]

IMHO I would rather change the xranges to range and from builtins import range (or just from future import * so no one needs to think about it) but if you want to leave the xranges in place, then from past.builtins import xrange https://python-future.org/compatible_idioms.html#xrange I think is preferable. IMHO it’s better to use range everywhere b/c you can wind up thinking about xrange and range as different things, which they are in python 2 but not python 3 (unless you from past.builtins import range, but let’s not go there), and it’s better to think about them as the same function and write python 3 code moving forward. Bruce

On Sat, Nov 21, 2020 at 5:29 PM Peter Newman ***@***.***> wrote: Thanks @brucelowekamp <https://github.com/brucelowekamp> . I've also done the print changes in #1685 <#1685> which I'll merge in as it didn't seem to hurt and seemed to be sufficient to get the compileall stuff to work, although flake8 needs more work, and actually running seems to need more still in some cases. I went for a slightly different xrange/range solution as in theory it should be more efficient on python2 I think: https://github.com/OpenLightingProject/ola/pull/1685/files#diff-5b3f8e5df8e5a430e32762b12ed99f846058b721ae6049489f7a6058fbd8f45eR25-R31 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNHN5NNSX3GB7DJTFWGOYDSRA5NDANCNFSM4JRA4RUQ> .

-- Sent from my mobile

[peternewman]

IMHO I would rather change the xranges to range and
from builtins import range
(or just from future import * so no one needs to think about it)

I thought theoretically xrange should be more efficient on Python 2 from my understanding? Although how much difference it makes in say the licence check when it's just four values I don't really know.

From what I can tell builtins isn't core either, compared to future, i.e. it's another dependency?

but if you want to leave the xranges in place, then
from past.builtins import xrange
https://python-future.org/compatible_idioms.html#xrange
I think is preferable.

From my, admittedly very quick look, past.builtins appeared to be a separate module which we're then requiring people to have for compatibility.

IMHO it’s better to use range everywhere b/c you can wind up thinking about
xrange and range as different things, which they are in python 2 but not
python 3 (unless you from past.builtins import range, but let’s not go
there), and it’s better to think about them as the same function and write
python 3 code moving forward.

Yeah I can see the logic behind that, I guess I was initially driven by doing it as a quick win on a few bits of code. I certainly haven't made the whole codebase pass yet. If the performance issue is fairly minor then doing a find/replace of xrange to range seems pretty trivial too.

[brucelowekamp]

I guess the question with performance is whether it matters there. For the tests, code generators, etc, I would assume not. If some of the rdm code is used where it matters, might be worth looking at. My impression was that most of the code was for testing and didn't appear to be performance sensitive, but you're a better judge of that. I'm amazed at how fast python is even with stupid array manipulations (I've never switched my code to numpy) so I would be surprised if it matters. As future can be installed with pip, IMHO it's not a concern. It's not like needing to find an ancient protobuf binary ;) I might have some slight concern with the fact that while it is still being maintained, it's obvious no one is aggressively trying to maintain full forward compatibility in 3.x, but I would assume that's definitely not a concern with using future (since it really only does something in py2, which isn't changing) and I doubt it would be a problem with the small set of features ola might import from past.builtins. Bruce

…

On Sat, Nov 21, 2020 at 11:10 PM Peter Newman ***@***.***> wrote: IMHO I would rather change the xranges to range and from builtins import range (or just from future import * so no one needs to think about it) I thought theoretically xrange should be more efficient on Python 2 from my understanding? Although how much difference it makes in say the licence check when it's just four values I don't really know. From what I can tell builtins isn't core either, compared to *future*, i.e. it's another dependency? but if you want to leave the xranges in place, then from past.builtins import xrange https://python-future.org/compatible_idioms.html#xrange I think is preferable. From my, admittedly very quick look, past.builtins appeared to be a separate module which we're then requiring people to have for compatibility. IMHO it’s better to use range everywhere b/c you can wind up thinking about xrange and range as different things, which they are in python 2 but not python 3 (unless you from past.builtins import range, but let’s not go there), and it’s better to think about them as the same function and write python 3 code moving forward. Yeah I can see the logic behind that, I guess I was initially driven by doing it as a quick win on a few bits of code. I certainly haven't made the whole codebase pass yet. If the performance issue is fairly minor then doing a find/replace of xrange to range seems pretty trivial too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADNHN5PHLE4SVBURUSBLDRLSRCFJVANCNFSM4JRA4RUQ> .

[ssilverman]

Update (including all the messages in case others search for anything similar): After installing protobuf with pip, I saw this error when trying to run rdm_test_server.py:

Traceback (most recent call last):
  File "/usr/local/bin/rdm_test_server.py", line 43, in <module>
    from ola.testing.rdm import TestDefinitions
  File "/usr/local/lib/python2.7/site-packages/ola/testing/rdm/TestDefinitions.py", line 24, in <module>
  File "/usr/local/lib/python2.7/site-packages/ola/testing/rdm/ResponderTest.py", line 33, in <module>
  File "/usr/local/lib/python2.7/site-packages/ola/testing/rdm/TimingStats.py", line 19, in <module>
  File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py", line 142, in <module>
  File "/usr/local/lib/python2.7/site-packages/numpy/core/__init__.py", line 47, in <module>
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /usr/local/opt/python@2/bin/python2.7),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: No module named _multiarray_umath

I installed numpy again using: pip install --user numpy, and then I saw this error:

Traceback (most recent call last):
  File "/usr/local/bin/rdm_test_server.py", line 1058, in <module>
    main()
  File "/usr/local/bin/rdm_test_server.py", line 1023, in main
    ('pids.proto', 'draft_pids.proto'))
  File "/usr/local/lib/python2.7/site-packages/ola/PidStore.py", line 1214, in GetStore
OSError: [Errno 2] No such file or directory: '/usr/local/Cellar/ola/0.10.7/share/ola/pids'

From brew info ola:

ola: stable 0.10.7 (bottled), HEAD
Open Lighting Architecture for lighting control information
https://www.openlighting.org/ola/
/usr/local/Cellar/ola/0.10.7_5 (462 files, 20.6MB) *
  Poured from bottle on 2020-12-13 at 23:06:34
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/ola.rb
License: GPL-2.0
==> Dependencies
Build: autoconf ✘, automake ✘, libtool ✔, pkg-config ✔
Required: liblo ✔, libmicrohttpd ✔, libusb ✔, numpy ✔, protobuf@3.6 ✔, python@3.9 ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 43 (30 days), 123 (90 days), 371 (365 days)
install-on-request: 42 (30 days), 120 (90 days), 359 (365 days)
build-error: 0 (30 days)

It looks like the search directory needs to append that "_5"? (A simple ln -s 0.10.7_5 0.10.7 fixed the problem.)

[peternewman]

I guess the question with performance is whether it matters there. For the
tests, code generators, etc, I would assume not. If some of the rdm code
is used where it matters, might be worth looking at. My impression was
that most of the code was for testing and didn't appear to be performance
sensitive, but you're a better judge of that. I'm amazed at how fast
python is even with stupid array manipulations (I've never switched my code
to numpy) so I would be surprised if it matters.

Yeah we're only using numpy for some stats, not for performance reasons. You're generally right, I'm sure people would like to run tests quickly, especially if they're iterating, but it's only the stuff in the python folder that might be used in a realtime way (e.g. the DMX and RDM manipulation during a show, and generally only the DMX side of that).

As future can be installed with pip, IMHO it's not a concern. It's not
like needing to find an ancient protobuf binary ;) I might have some
slight concern with the fact that while it is still being maintained, it's
obvious no one is aggressively trying to maintain full forward
compatibility in 3.x, but I would assume that's definitely not a concern
with using future (since it really only does something in py2, which isn't
changing) and I doubt it would be a problem with the small set of features
ola might import from past.builtins.

It's not so much pip, it's more that means they all need to be in the various packaging systems people package OLA for.

I think I was using 3.1.0 as it was consistent across the Ubuntu and Mac packages (probably mostly limited by the old Ubuntu version, but we've moved on since then so on the Travis Bionic we're using the built in deb packaged version with Python 2 (and only not on Python 3 as I think the way I'm switching Python misses the links to the relevant dist packages for some reason, probably mixing pip/pyenv and deb).

[peternewman]

Traceback (most recent call last):
  File "/usr/local/bin/rdm_test_server.py", line 1058, in <module>
    main()
  File "/usr/local/bin/rdm_test_server.py", line 1023, in main
    ('pids.proto', 'draft_pids.proto'))
  File "/usr/local/lib/python2.7/site-packages/ola/PidStore.py", line 1214, in GetStore
OSError: [Errno 2] No such file or directory: '/usr/local/Cellar/ola/0.10.7/share/ola/pids'

From brew info ola:

ola: stable 0.10.7 (bottled), HEAD
Open Lighting Architecture for lighting control information
https://www.openlighting.org/ola/
/usr/local/Cellar/ola/0.10.7_5 (462 files, 20.6MB) *
  Poured from bottle on 2020-12-13 at 23:06:34

It looks like the search directory needs to append that "_5"? (A simple ln -s 0.10.7_5 0.10.7 fixed the problem.)

Good spot @ssilverman . This feels like a bug with Homebrew given that file just contains the contents of piddatadir which itself is just datadir which is just datarootdir which is just prefix/share:
https://github.com/OpenLightingProject/ola/blob/0.10.7/python/ola/Makefile.mk#L52-L53

Possibly it didn't get updated when the bottle got built.

This will go away when this goes in, but just because the variant gets reset:
Homebrew/homebrew-core#65415

Do you want to report this to them given you've got a Mac to test it on. Feel free to @mention me/this in it too.

[peternewman]

Does the olad web interface work in this setup too? I guess you only tested after adding the symlink.

[peternewman]

@ssilverman FYI the RDM tests should finally work on Python 3 in #1761 if you want to take a look?

[peternewman]

Closing this as it should have been resolved by #1761 . Let us know if you find any outstanding RDM Python bugs.