[Bug] qmk command failure

[elfmimi]

Describe the Bug

edited after seeing rgoulter's comment
qmk script does not properly detect absence of required modules, namely hid and pyusb.

the error message looks like this.

File "${HOME}/qmk_firmware/lib/python/qmk/cli/console.py", line 9, in
import hid
ModuleNotFoundError: No module named 'hid'

this occurs after renewing a repository across the last breaking change.

How to reproduce

condition 1: make sure user.developer is set to None or False with qmk config. ex. qmk config user.developer=None
condition 2: make sure hid module is not installed. ex. pip3 uninstall hid

then qmk hello to test its behaviour.

Quick fix

manually install required modules for now.
pip3 install hid pyusb

or

comment out this line > # from . import console

System Information

• QMK Firmware version: latest master 1646c0f

Additional Context

#12455 (comment)

[rgoulter]

qmk script does not properly detect absence of required module, hid.

As well as hid, pyusb is also needed.

From e.g. QMK MSYS, when I checked out the develop branch I needed to pip3 install both of those.

[elfmimi]

when pyusb is not installed, at this line, find_spec('usb.core') will throw an exception.

ModuleNotFoundError: No module named 'usb'

edit: above is the case when user.developer == True. (and pyusb is not installed)
condition must be something like: not (find_spec('usb') and find_spec('usb.core'))

[djfiander]

These two modules are listed in the "requirements-dev.txt" file, but not in the "requirements.txt" file, which is what util/qmk_install.sh uses, among other things.

[rgoulter]

More specifically, #12828 added hid and pyusb to requirements-dev.txt but not to requirements.txt.

The above PR also changed the cli GitHub Action: it was updated to install from requirements-dev.txt, although as noted above, the various scripts in util/install used by util/qmk_install.sh install per requirements.txt.

This is less of an issue for those using the global qmk installed by pip, which added hid, pyusb to its setup.py, as of version 0.0.46. (@elfmimi another workaround would be to install a recent version of the qmk cli with something like pip3 install qmk).

@skullydazed could you clarify: the comment here indicates that hid and pyusb were added to requirements.txt. https://github.com/qmk/qmk_firmware/pull/12828/files#r637416230 -- they don't appear to have been added to requirements.txt in this repository. Am I misunderstanding something?

[rgoulter]

IIUC, #13079 fixes this too. Please check, @elfmimi

[elfmimi]

IIUC, #13079 fixes this too. Please check, @elfmimi

umm? this is a recurrent pointer. @rgoulter

[rgoulter]

😅 So it is. Sorry, I meant to refer to #13134. @elfmimi

[elfmimi]

moving pyusb and hid from requirements-dev.txt to requirements.txt is not a solution.
and possibility of this error remains. see #12455 (comment)

ImportError: Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll

quote from #12978 (comment)
@skullydazed > I’ll take a closer look in a few days. We may need to rework how those imports happen.

so I guess we are still at a situation where only first aid is applied.

[skullydazed]

After writing that comment I created (and we merged) #13199. If there is still a problem being encountered we can continue to work through it. I know it's frustrating when you are subject to a corner case but we are trying to solve the general problem here (allowing us to add new requirements without breaking the ability to use essential commands like qmk setup) rather than narrowly solving only the pyhid problem.

[rgoulter]

you are subject to a corner case

Adding a Python dependency which has a native dependency could be described as a corner case.

I'd quibble, though: I think "running qmk setup by following QMK docs" (which was broken by the change to the deps) couldn't reasonably be called a corner case. (Somewhat exasperated by that the people who are new to the project won't be familiar with how to work around the problem).

I tried re-running the commands in a Docker container per #12978 (comment), and still observed the same error. Whereas I'd expect the commands given in the docs to work in a fresh Linux environment.

[skullydazed]

I'd quibble, though: I think "running qmk setup by following QMK docs" (which was broken by the change to the deps) couldn't reasonably be called a corner case. (Somewhat exasperated by that the people who are new to the project won't be familiar with how to work around the problem).

It's not happening for most people, which is why I call it a corner case. I'm dedicated to making this work.

I tried re-running the commands in a Docker container per #12978 (comment), and still observed the same error. Whereas I'd expect the commands given in the docs to work in a fresh Linux environment.

Apparently I haven't broken my machine enough so that it "doesn't work on my machine." 🙂 I'll get the metaphorical sledgehammer out to properly recreate your situation, but I may not be able to do that for a few days.

[rgoulter]

The effort is appreciated. Thanks for looking into it. :)

[Aeners]

I'm also getting this error:

> qmk setup
> Error: %s: %s ('ImportError', ImportError('Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll')) ...(and more about hdapi)

Although running pip3 install hid pyusb gives me:

> pip3 install hid pyusb
> Requirement already satisfied: hid in /usr/local/lib/python3.9/site-packages (1.0.4)
> Requirement already satisfied: pyusb in /usr/local/lib/python3.9/site-packages (1.1.1)

I tried following issues instruction but I had no luck solving my problem.
If you find something still interested :)

[skullydazed]

@rgoulter I've opened #13324 to solve this.

@Aeners You can run util/qmk_install.sh to install the underlying OS requirements. Normally qmk setup would do this but there's a problem that won't be fixed until #13324 is merged.

[NiklasBr]

You can run util/qmk_install.sh to install the underlying OS requirements. Normally qmk setup would do this but there's a problem that won't be fixed until #13324 is merged.

This helped me move forward through the setup guide. Thanks!

[shelaf]

#13324 has been merged, but I still get an error on MSYS2.

$ qmk setup
ERROR Could not find qmk_firmware!
Would you like to clone qmk/qmk_firmware to C:/msys64/home/shela/qmk_firmware? [y/n] y
Cloning into '/home/shela/qmk_firmware'...
...
Updating files: 100% (24854/24854), done.
Submodule 'lib/chibios' (https://github.com/qmk/ChibiOS) registered for path 'lib/chibios'
Submodule 'lib/chibios-contrib' (https://github.com/qmk/ChibiOS-Contrib) registered for path 'lib/chibios-contrib'
Submodule 'lib/googletest' (https://github.com/qmk/googletest) registered for path 'lib/googletest'
Submodule 'lib/lufa' (https://github.com/qmk/lufa) registered for path 'lib/lufa'
Submodule 'lib/printf' (https://github.com/qmk/printf) registered for path 'lib/printf'
Submodule 'lib/ugfx' (https://github.com/qmk/uGFX) registered for path 'lib/ugfx'
Submodule 'lib/vusb' (https://github.com/qmk/v-usb) registered for path 'lib/vusb'
Cloning into '/home/shela/qmk_firmware/lib/chibios'...
Cloning into '/home/shela/qmk_firmware/lib/chibios-contrib'...
Cloning into '/home/shela/qmk_firmware/lib/googletest'...
Cloning into '/home/shela/qmk_firmware/lib/lufa'...
Cloning into '/home/shela/qmk_firmware/lib/printf'...
Cloning into '/home/shela/qmk_firmware/lib/ugfx'...
Cloning into '/home/shela/qmk_firmware/lib/vusb'...
Submodule path 'lib/chibios': checked out 'ffe54d63cb10a355add318f8e922e39f1c3d4bfd'
Submodule path 'lib/chibios-contrib': checked out '61baa6b036138c155f7cfc5646d833d9423f3243'
Submodule path 'lib/googletest': checked out 'ec44c6c1675c25b9827aacd08c02433cccde7780'
Submodule path 'lib/lufa': checked out 'ce10f7642b0459e409839b23cc91498945119b4d'
Submodule path 'lib/printf': checked out 'd3b984684bb8a8bdc48cc7a1abecb93ce59bbe3e'
Submodule path 'lib/ugfx': checked out '40b48f470addad6a4fb1177de1a69a181158739b'
Submodule path 'lib/vusb': checked out 'bdb53e4c043d089279d9891b68bea77614cb97ee'
INFO Successfully cloned https://github.com/qmk/qmk_firmware to C:/msys64/home/shela/qmk_firmware!
INFO Added https://github.com/qmk/qmk_firmware as remote upstream.
Warning: Could not import qmk.cli.console: ImportError, Unable to load any of the following libraries:libhidapi-hidraw.so libhidapi-hidraw.so.0 libhidapi-libusb.so libhidapi-libusb.so.0 libhidapi-iohidmanager.so libhidapi-iohidmanager.so.0 libhidapi.dylib hidapi.dll libhidapi-0.dll
INFO QMK Doctor is checking your environment.
INFO QMK home: C:/msys64/home/shela/qmk_firmware
INFO Detected Windows 10 (10.0.19041).
ERROR Can't find arm-none-eabi-gcc in your path.
ERROR Can't find avr-gcc in your path.
ERROR Can't find avrdude in your path.
ERROR Can't find dfu-programmer in your path.
ERROR Can't find dfu-util in your path.
Would you like to install dependencies? [Y/n] <class 'EOFError'>
ERROR
Traceback (most recent call last):
  File "C:/msys64/mingw64/lib/python3.8/site-packages/milc/milc.py", line 476, in __call__
    return self.__call__()
  File "C:/msys64/mingw64/lib/python3.8/site-packages/milc/milc.py", line 479, in __call__
    return self._subcommand(self)
  File "C:/msys64/home/shela/qmk_firmware/lib/python/qmk/cli/doctor.py", line 95, in doctor
    if yesno('Would you like to install dependencies?', default=True):
  File "C:/msys64/mingw64/lib/python3.8/site-packages/milc/questions.py", line 46, in yesno
    answer = input(format_ansi(prompt % args))
EOFError

The workaround for this issue is to run qmk setup -y.

[Aeners]

@rgoulter I've opened #13324 to solve this.

@Aeners You can run util/qmk_install.sh to install the underlying OS requirements. Normally qmk setup would do this but there's a problem that won't be fixed until #13324 is merged.

The workaround worked well for me 🚀
Thanks guys

[CreamyCookie]

@skullydazed

pip3 install hid pyusb
Requirement already satisfied: hid in /home/pirat/.local/lib/python3.8/site-packages (1.0.4)
Requirement already satisfied: pyusb in /home/pirat/.local/lib/python3.8/site-packages (1.1.1)

and the utils script did not help either. The only sub commands I get are

Sub-commands:
  {config,clone,setup}

I've did the setup in the past, and updated that local repo, so maybe it has something to do with that.

Any ideas what I can do?

Update: What solved the problem for me was python3 -m pip install -U qmk (or remove then reinstall, which is what I actually did)

[skullydazed]

@shelaf

#13324 has been merged, but I still get an error on MSYS2.

This is a new problem which I have addressed in qmk/qmk_cli#63.

@CreamyCookie Glad you got that figured out.

[elfmimi]

@skullydazed almost, almost there.
Now that the title of this issue "qmk command failure" does not apply, except for the corner case.
But there still is a tiny catch which is better ironed out.

Our scenario follows this description.

this occurs after renewing a repository across the last breaking change.

Let's assume one had qmk command version 0.0.45 installed and the repository was at somewhere around tag/0.12.34 .
After updating one's working repository to the latest master as of today with git pull, next thing one would do is ...
> make <keyboard>:<keymap>
And then this command do nothing. ???
Nothing. because the message from qmk hello asking for whether to install hid and pyusb is being thrown to the void of /dev/null .

Yes, I know updating qmk command to the latest will solve the situation.
Seeing people stumbling on issues like this and complaining and whining is frustrating.

[CreamyCookie]

I think it would make sense to mention the update command (python3 -m pip install -U qmk) somewhere in the output when qmk notices that only a few subcommands can be used (or maybe even for other problems).