Join GitHub today
"Illegal Instruction" by googlesamples-assistant-pushtotalk in Pi Zero(grpc issue) #235
I installed Googlesamples-assistant-pushtotalk(Google Assistant Service) in Pi Zero recently , but it displays "Illegal instruction" error message and I can't execute it.
This messages didn't appear when I installed before(3/25/2018).
Raspberry Pi Zero W
I executed official installation procedure.
I think that it is caused by grpcio pip package .
I found grpcio package installation with "grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl" from log of installation Google Assistant. This file name looks like to compliant armv6 architecture, but "Illegal Instruction" error was occurred.
So, I tried to install grpcio from source before Google Assistant installation.
Then, "Illegal instruction" error was not occurred and I can use pushtotalk well.
In Pi Zero, latest version of grpcio(1.11.0) is installed from binary package. but previous version(1.10.0) is installed from source.
I think this is why "Illegal Instruction" is occurred recently.
Would you check this problem please?
Thank you for taking your time.
Not sure if this happens to be the root cause or not but just by looking at PyPI index, I can confirm that none of gRPC ARM packages are “arm6”. All of them are “arm7l” designated. https://pypi.org/project/grpcio/1.11.0/#files
I’m curious how you got the arm6 installed from a binary wheel?
according a installation log of google assistant...
I have confirmed this file can be downloaded now.
It seems that this file is a binary file to complient armv7 architecture, because I heard other user could run Google Assistant with this package in Raspberry Pi 3.
I don't grasp what happened because I don't know well about python package system.
Ah, I can see what's happened here. It hasn't come up before.
Before piwheels, PyPI didn't accept Arm wheel uploads, so we automated building and hosting Arm wheels built from sdist of all packages on PyPI.
Since PyPI uploads moved to warehouse's back-end (long before the very recent front-end switch), Arm wheel uploads have been permitted.
Since Raspbian userland is 32-bit only, regardless on Pi model/CPU, all wheels built on a Pi 2/3 are identical to those built on Pi 1/Zero, despite the platform tag being armv6l/armv7l. But they're actually armv6 because of the agnostic userland.
We build "armv7l" wheels and provide duplicate files tagged as "armv6l" because they are compatible with all Pis in Raspbian. See RTIMULib for example.
When a compatible wheel is available on PyPI, all we do is download it, not build it from source. If this happens it is usually a pure python wheel which is not problem. However, in this case Armv7 wheels have been provided but not Armv6 (note: PyPI/warehouse does not allow uploading multiple identical files even with different tags like we do on piwheels).
It's configured as an additional index in
Precedence is determined by the provider of the best matching file. Usually (e.g numpy) piwheels is preferred due to providing a wheel over PyPI's sdist, but in the case of a pure python package providing a wheel on PyPI (e.g. gpiozero), they are equal and PyPI is preferred as the primary index.
If the user is on an "armv7" platform (Pi 2 or 3), pip will only match armv7 wheels. armv6 is simply not a match, the same way x86 isn't. So in this case, the user would get the armv7 wheel from pypi.
If the user is on an armv6 platform (Pi 1/Zero) and no armv7 wheel is provided on PyPI, pip's choice is between a seemingly valid platform wheel from piwheels vs. sdist from PyPI. piwheels wins.
However, if the original wheel is true armv7 (as opposed to armv6 userland posing as armv7) we have a problem, because piwheels copied it to "armv6l" and is providing that to Pi 1/Zero users.
How was the armv7 wheel built? QEMU maybe? If it contains Armv7 instructions it will not work on Pi 1/Zero (and I'm not even sure about Pi 2/3).
I will test and report
I have successfully built an Arm wheel on a Pi 3, and tested installing it on another Pi 3. It seems to work but I'm not sure how to test it.
I have also duplicated it as armv6l.
You can test:
pip install http://bennuttall.com/files/grpcio-1.11.0-cp35-cp35m-linux_armv7l.whl # pi 2/3 pip install http://bennuttall.com/files/grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl # pi 1/zero
If those work ok, I'd suggest the maintainers don't bother uploading emulated platform wheels to PyPI, and let piwheels build them natively. Any reason not to?
@bennuttall I am not 100% sure who asked us (@kpayson64 should have more context), but I think we have had people pinging us a couple times we forgot to upload the ARM wheels simultaneously with the rest of our wheels. I suspect the reason might be not all ARM users are on Raspbian or Raspberry Pi? Frankly, we are not in love with building on ARM either, and we would love to defer to other people if we could, but I strongly suspect that's not an option. @proppy, do you think that is a viable solution for everyone?
Is this the right place for this issue? Should it be raised with grpc/grpc perhaps? Is gprc a google project or not?
Yeah it's a complex beast. piwheels solves one problem for one set of users really well, i.e. raspbian / raspberry pi users. We don't claim or pretend to be trying to solve a wider Arm issue. But maintainers providing Arm wheels upstream (i.e. PyPI) must ensure they are compatible with other Arm boards. We (piwheels) do not recommend users of other Arm boards use piwheels as their repo! (they can replicate the effort if they want!)
If piwheels is breaking something for Pi 1/Zero users then I'll remove it, but moving forward if the authors can agree on a coherent solution, that would be great
@bennuttall gRPC was started by Google and donated to and currently owned by CNCF. I don't think there's much to be discussed from the gRPC side. I just brought it up because you suggested completely outsourcing the ARM wheel production to pywheels, and I wanted to get a quick assessment on its fundamental feasibility from the community's perspective, as people on this thread are probably more familiar with the community needs around the ARM distribution than most, but beyond that there's not much we can do from gRPC's side. What I think makes sense regardless of the upstream is for pywheels to either serve a correct arm6 build or not serve one at all so users end up fetching and building from source.
Sorry, I can't test it because I'm so busy.
I think Google Assistant team has a responsibility to test a solution and fix this problem.
There are many people that failed to install Google Assistant in own Pi Zero even if they followed official install procedure. It isn't user's problem and it is a problem of official guide, isn't it?
As a result, it seems that these people have had misunderstanding about "Google Assistant is not supported Pi Zero." I would appreciate if you write clearly support status of Pi Zero in official document to avoid misunderstanding of these users.
And as outside opinion from user-side third person, I think it is a good way that build from source if you can't cope with ARM complex environment properly. It may requires a lot of time, but it is better than halt caused by illegal binary.
I have already reported all information I have. Please handle this issue as you wish.
Thank you for your support.
I've replaced our wheels with the ones I built last night, so Pi 2/3 users will still get the armv7 wheel from PyPI, but Pi 1/Zero users will get the armv6 one from piwheels (whereas without piwheels they'd get sdist and would have to build - it took about 20 mins to build on a Pi 3, which could be more like 2 hours on a Pi 1).
I just discussed the matter with @kpayson64 who was trying to build the ARMv6 builds with QEMU before and he told me the generated artifacts for ARM6 from QEMU were broken and that's why we never published ARM6 ones on PyPI.
@bennuttall Can you modify the pywheels process such that it builds from source instead of renaming the ARMv7 version? I can imagine this might cause problems not just for gRPC but also other packages.
Today, I had the opportunity to build Google Assistant with Pi Zero.
Thank you for prompt response and improvement. I will tell it to Pi Zero users around.
There seems to remain discussion about the provision of the armv6 package, so I leave this issue open.
Thank you very much.
This was referenced
Apr 21, 2018
referenced this issue
Apr 21, 2018
referenced this issue
Apr 22, 2018
There's no need to rebuild all wheels, because "armv6" and "armv7" wheels are identical when built on a Pi. It's only when there's an armv7 wheel on PyPI (this is the first one I've seen, as it's a relatively new addition that causes an issue).
It may be that we can check the wheel downloaded and force a rebuild when we see it's armv7. See bennuttall/piwheels#66
I've been following the install guide at http://wiki.seeedstudio.com/ReSpeaker_2_Mics_Pi_HAT/ and hit the same problem on my Pi Zero (v1.3).
I wasn't sure quite what the conclusion of the discussion above meant, but I tried doing "python -m pip install grpcio" on my Pi Zero to manually install the package, and that appeared to install an "armv6" version, however I still received an "Illegal Instruction" on running the demo. Last few lines of terminal below...
(env) root@speaky:/home/pi/projects/speaky# python -m pip install grpcio
I have blacklisted the
At a later date I will allow this package to be built for Armv6 as well which should resolve the issue without making people build from source.
I deleted the venv I had before and cleaned out my .cache/pip directory and started again with the Google Assistant section of http://wiki.seeedstudio.com/ReSpeaker_2_Mics_Pi_HAT/.
When it gave the "Illegal instruction" this time, I did a manual install of grpcio, and got the results below...
This looks like it's followed the same grpcio install process as last time, the .whl is the same and the "six>=1.5.2" seems the same warning.
I'll do some poking around to see if I can find anything that looks like an odd conflicting package, but my Python-fu is quite weak, so don't keep your fingers crossed!
tl;dr - YAY it works!
After some tedious shenanigans with reinstalling and compile troubles last night, I finally did a fairly thorough clean out and reinstall following Dimeiza's path of installing grpcio before any of the assistant stuff.
For some reason this also required me to "apt install libffi-dev libssl-dev" before the google-assistant-sdk would install. I also decided to bin the venv approach and be root as I'll be rebuilding the Zero when this works.
So the steps I ended up with were:
After some pondering I see "Press Enter to send a new request..." and yup, it works!
Thanks for the help Ben. I'll do some more poking at the google-assistant-demo script to see if there's a clue about which package is causing the problem there. I guess it may be another armv6/7 packaging issue?