Python 3.9 breaks the package?

[Boulder08]

I've installed Python v3.9 and after that, launching a scheduled Python script to collect data from the Vallox API fails with this error:

File "C:\Program Files\Python39\lib\site-packages\vallox_websocket_api\__init__.py", line 1, in <module>
    from .client import Client
  File "C:\Program Files\Python39\lib\site-packages\vallox_websocket_api\client.py", line 7, in <module>
    from .messages import (LogReadRequest, LogReadResponse1, LogReadResponse2,
  File "C:\Program Files\Python39\lib\site-packages\vallox_websocket_api\messages.py", line 91, in <module>
    LogReadResponse2 = GreedyRange(
  File "C:\Program Files\Python39\lib\site-packages\construct\core.py", line 444, in compile
    module = importlib.util.module_from_spec(module_spec)

AttributeError: module 'importlib' has no attribute 'util'

This is the code I've tested it with:

# -*- coding: utf-8 -*-
import asyncio
import datetime
from vallox_websocket_api import Client

client = Client('192.168.0.60')

async def run():
    metrics = await client.fetch_metrics()
    poimi_muuttujat(metrics)

def poimi_muuttujat(metrics):
    global poistoilma
    global jateilma
    global ulkoilma
    global tuloilmakennosta
    global tuloilmakennosta_heater
    global kennontila
    global tulokierrokset
    global poistokierrokset
    global puhallin
    poistoilma = metrics['A_CYC_TEMP_EXTRACT_AIR']
    jateilma = metrics['A_CYC_TEMP_EXHAUST_AIR']
    ulkoilma = metrics['A_CYC_TEMP_OUTDOOR_AIR']
    tuloilmakennosta = metrics['A_CYC_TEMP_SUPPLY_CELL_AIR']
    tuloilmakennosta_heater = metrics['A_CYC_TEMP_SUPPLY_AIR']
    kennontila = metrics['A_CYC_CELL_STATE']
    tulokierrokset = metrics['A_CYC_SUPP_FAN_SPEED']
    poistokierrokset = metrics['A_CYC_EXTR_FAN_SPEED']
    puhallin = metrics['A_CYC_FAN_SPEED']

asyncio.get_event_loop().run_until_complete(run())

It's strange that if I open the script in IDLE and launch it from there, no problems.
All the old Python versions are uninstalled.

[tirkarthi]

See construct/construct#930 PR to construct. This seems to be a behavior change in Python's importlib. See also https://issuetracker.google.com/issues/170125513

[Boulder08]

This issue is fixed by adding
import importlib.util
at the beginning of the messages.py file.

[yozik04]

Which version of ‘construct’ you have installed? There would be a lot of attention to this problem from homeassistant community as my lib and a lot of others use ‘construct’ as dependency.

[yozik04]

This issue is fixed by adding
import importlib.util
at the beginning of the messages.py file.

If this is the fix then this import can be anywhere? Not only in vallox/construct libs?

[Boulder08]

It's v2.10.67.

[Boulder08]

This issue is fixed by adding
import importlib.util
at the beginning of the messages.py file.

If this is the fix then this import can be anywhere? Not only in vallox/construct libs?

I'm not savvy at all, but I'd figure that it's not global so if you use it inside the module, you have to import it there.

[slovdahl]

I'm also having the same problem, and it makes no sense at all. I'm on macOS 11.6.3 at the moment and trying to run a simple test client with vallox_websocket_api in iTerm gives me the same module 'importlib' has no attribute 'util' error. I have tested it with in Python 3.10.2, 3.9.10 and 3.8.12, and I get the same error in all three cases.

import asyncio
from vallox_websocket_api import Vallox

client = Vallox('1.2.3.4')

async def run():
    profile = await client.get_profile()
    print("Profile: %s" % profile)

asyncio.get_event_loop().run_until_complete(run())

If a manually add the changes from construct/construct#930 to construct in site-packages, it works fine.

[yozik04]

I can not reproduce. I test the lib in 5 different python versions: ["3.6", "3.7", "3.8", "3.9", "3.10"].
I do not get any errors. Maybe construct version must be stricter.

install_requires=[
        "websockets >= 9.1, < 11.0",
        "construct >= 2.9.0, < 3.0.0"
],

[slovdahl]

My guess is that it has somthing to do with either the OS, the surrounding environment or how the Python binary/package is built/packaged. Did you try it with the minimal example I provided too? And which OS did you test it on?

I added a bit more info here too: construct/construct#930 (comment)

[yozik04]

Can you run tests?
python3 setup.py test

I run the lib on Ubuntu server in venv Python 3.8.10 to hook the unit up with MQTT.
I run tests on Mac Big Sur with Python 3.9 and all is green.

[slovdahl]

Interesting, the tests do work.

Can't remember where I read it, but someone speculated that it might be enough that one file loads importlib.util in "the right way" during the lifetime of the Python process, and that could explain why it works in some cases but not in others.

[Boulder08]

In my case, I also got it working if I ran the script from IDLE. If I ran it using the command line, it didn't work. That's what puzzled me a lot.

[yozik04]

On my Mac tests do work in IDE (PyCharm) and from console.

[slovdahl]

@yozik04 what about this simple test script, does it also work on your Mac? And how is Python installed in that case?

import asyncio
from vallox_websocket_api import Vallox

client = Vallox('1.2.3.4')

async def run():
    profile = await client.get_profile()
    print("Profile: %s" % profile)

asyncio.get_event_loop().run_until_complete(run())

[yozik04]

Works fine:

python3 slovdahl.py
Profile: PROFILE.HOME

python3 --version
Python 3.9.7

[yozik04]

websockets==9.1
construct==2.10.67

[yozik04]

@slovdahl, @Boulder08 I have asked construct devs to merge PR to fix this problem. Please verify. Should be fixed in construct 2.10.68.

[slovdahl]

@slovdahl, @Boulder08 I have asked construct devs to merge PR to fix this problem. Please verify. Should be fixed in construct 2.10.68.

Great, I can confirm that it starts working after running pip3 install construct==2.10.68. 👍

[yozik04]

Great! Happy to help.