0.5.0 testing #29

Magalex2x14 · 2020-01-22T10:29:13Z

To discuss and test a new data collection method.
0.5.0 trying to close #9 #14 #23 #25

Magalex2x14 · 2020-01-22T12:12:53Z

Raspberry PI 3b+, HassOS 3.8, HASSio 0.104.3 - works out of the box

HASSio (Arch Linux x86_64 + Docker) - works out of the box

Mac mini, Debian Buster, Python virtualenv, HA 0.104.3 - works after granting permissions with:

sudo setcap 'cap_net_raw,cap_net_admin+eip' `readlink -f \`which python3\``

Raspberry PI 3b+, Debian Buster, python virtualenv, HA 0.104.3 - works after granting permissions

Ernst79 · 2020-01-22T12:30:58Z

@Magalex2x14. Testing on Raspberry PI 3b+, Debian Buster, python virtualenv, HA 0.104.3.

Small remark, you are probably aware of, but the documentations needs to be updated. It still shows the old config option hcitool_active and the new options are not explained yet and are not in the example configuration.

I added both new options in my config and removed hcitool_active. After a restart of home assistant, I get the following message. Could be related to hci_interface: 0? How do I check which number I need?

2020-01-22 13:21:50 ERROR (Thread-8) [asyncio] Fatal write error on socket transport
protocol: <aioblescan.aioblescan.BLEScanRequester object at 0x5d6655b0>
transport: <_SelectorSocketTransport fd=27 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 857, in write
    n = self._sock.send(data)
PermissionError: [Errno 1] Operation not permitted
2020-01-22 13:21:53 ERROR (Thread-18) [asyncio] Fatal write error on socket transport
protocol: <aioblescan.aioblescan.BLEScanRequester object at 0x5fb8a530>
transport: <_SelectorSocketTransport fd=37 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 857, in write
    n = self._sock.send(data)
PermissionError: [Errno 1] Operation not permitted
2020-01-22 13:22:54 ERROR (Thread-20) [asyncio] Fatal write error on socket transport
protocol: <aioblescan.aioblescan.BLEScanRequester object at 0x5b4b2490>
transport: <_SelectorSocketTransport fd=38 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 857, in write
    n = self._sock.send(data)
PermissionError: [Errno 1] Operation not permitted

Magalex2x14 · 2020-01-22T13:30:57Z

It may be Python have insufficient rights to access hci interface.
Try:

sudo setcap 'cap_net_raw,cap_net_admin+eip' `readlink -f \`which python3\``

Ernst79 · 2020-01-22T13:57:20Z

Sorry, it didn't help, still have the same error.

Magalex2x14 · 2020-01-22T14:17:53Z

I 'll get to the computer - I 'll take a closer look. Yes, zero for hci0 interface, one for hci1, and so on. Try to remove this option from the configuration (default is 0).

Magalex2x14 · 2020-01-22T14:20:23Z

To check interface you can try this command:
hcitool dev

Ernst79 · 2020-01-22T14:23:11Z

hci0 is correct, but also without this option in config, still the same error. I'm running home assistant in a venv,

aqualx · 2020-01-22T16:21:47Z

Seems that this approach works nice on hassio.
But sometimes get very strange values from CGG1 : 23.485714285714284

Have such settings:

  - platform: mitemp_bt
    rounding: False
    decimals: 2
    period: 60
    log_spikes: True
    use_median: False

Magalex2x14 · 2020-01-22T16:25:20Z

@Ernst79 I checked the rights on my server:

~$ sudo getcap `readlink -f \`which python3\``
/usr/bin/python3.7 = cap_net_admin,cap_net_raw+eip

After resetting rights (it seems that I gave them out sometime during my experiments):

~$ sudo setcap '' `readlink -f \`which python3\``
~$ sudo getcap `readlink -f \`which python3\``
/usr/bin/python3.7 =

I reproduced this problem:

2020-01-22 19:07:17 ERROR (Thread-3) [asyncio] Fatal write error on socket transport
protocol: <aioblescan.aioblescan.BLEScanRequester object at 0x7f9a527810f0>
transport: <_SelectorSocketTransport fd=27 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 857, in write
    n = self._sock.send(data)
PermissionError: [Errno 1] Operation not permitted
2020-01-22 19:07:20 ERROR (Thread-4) [asyncio] Fatal write error on socket transport
protocol: <aioblescan.aioblescan.BLEScanRequester object at 0x7f9a415df208>
transport: <_SelectorSocketTransport fd=27 read=idle write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 857, in write
    n = self._sock.send(data)
PermissionError: [Errno 1] Operation not permitted

The problem is solved by this:

~$ sudo setcap 'cap_net_raw,cap_net_admin+eip' `readlink -f \`which python3\``
~$ sudo getcap `readlink -f \`which python3\``
/usr/bin/python3.7 = cap_net_admin,cap_net_raw+eip

It is important to note that after changing the rights, it is not necessary to restart, but to stop and start HA (since the python process does not exits upon restart).

Magalex2x14 · 2020-01-22T16:30:09Z

@aqualx This is a normal result with rounding: False option. The component averages all the values taken over the measurement period. Therefore, if you do not need all the resolution obtained in this case, then enable rounding with rounding: True.

aqualx · 2020-01-22T16:42:54Z

But there is option
decimals: 2...

My goal is to receive all raw values from sensor and feed PID controller with them. CGG1 is very stable in values and has no drift in values. Unlike LYWSDCGQ that always drifts within +/-0.1°C

Magalex2x14 · 2020-01-22T16:47:39Z

Hmmm! You're right, it makes no sense to have two parameteters in this case. Now decimals only works when rounding is enabled. It is necessary to remove the rounding option, and enable it only if the number of decimal places is indicated in config.

Magalex2x14 · 2020-01-22T17:13:48Z

My goal is to receive all raw values from sensor and feed PID controller with them.

Averaging always take place (with use_median: True it is median, otherwise it is mean). According to the information I have, your sensor gives about 20 readings per minute (look at "last mean of" sensor attribute) - our component calculate average of them, and then sensor entity is updated with this average. In your case, enabling rounding is safe - you will still get the most accurate value within the resolution specified by decimals option. If this is important for you, then open a separate issue - we will discuss it there.

Ernst79 · 2020-01-22T18:14:13Z

@Magalex2x14 You pointed me in the right direction. When running the getcap command, I got the following.

sudo getcap `readlink -f \`which python3\``
/home/pi/.local/bin/python3.8 = cap_net_admin,cap_net_raw+eip

So, it refers to python3.8, which I installed manually somewhere in the past. But my home assistant is running in a python3.7 virtual environment. Running the following command fixed it.

sudo setcap 'cap_net_raw,cap_net_admin+eip' /usr/bin/python3.7

The sensor is running fine now. Thanks.

gadgetchnnel · 2020-01-24T20:01:05Z

I've installed 0.5.1-alfa (I was previously successfully using 0.4.2) and I didn't get any errors in the log (I ran the setcap command to give access) but the sensors were unavailable.

I enabled debug logging for the component and got the following:

2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Starting
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Spawning HCIdump thread.
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] HCIdump thread: Init
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] HCIdump thread: Init finished
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Starting HCIdump thread.
2020-01-24 19:34:11 DEBUG (Thread-3) [custom_components.mitemp_bt.sensor] HCIdump thread: Run
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] update_ble called
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Time to analyze...
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Getting data from HCIdump thread
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] HCIdump thread: joining
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] 'NoneType' object has no attribute 'call_soon_threadsafe'
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] HCIdump thread: joined
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Spawning HCIdump thread.
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] HCIdump thread: Init
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] HCIdump thread: Init finished
2020-01-24 19:34:11 DEBUG (SyncWorker_7) [custom_components.mitemp_bt.sensor] Starting HCIdump thread.
2020-01-24 19:34:11 DEBUG (Thread-4) [custom_components.mitemp_bt.sensor] HCIdump thread: Run

The 'NoneType' object has no attribute 'call_soon_threadsafe' message (seems like that should be ERROR rather than DEBUG) looks like the reason.
Looking at the code, could it be that the join() method is being called before the event loop has been created?

Magalex2x14 · 2020-01-24T20:12:51Z

@gadgetchnnel How much time has passed since the HA was launched? At the start of HA, sometimes a “strange” things happens (my vision :)
If the time specified in the period option (60s by default) has already passed, then yes, we need to dig ...

gadgetchnnel · 2020-01-24T20:31:06Z

@Magalex2x14 I think that the issue may be that Python 3.7 on my Raspberry Pi 3 B+ wasn't built with Bluetooth socket support:

>>> import aioblescan as aiobs
>>> mysocket = aiobs.create_bt_socket(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/srv/homeassistant/lib/python3.7/site-packages/aioblescan/aioblescan.py", line 1208, in create_bt_socket
    sock = socket.socket(family=socket.AF_BLUETOOTH,
AttributeError: module 'socket' has no attribute 'AF_BLUETOOTH'

I've reverted to 0.4.3 for now.

Magalex2x14 · 2020-01-24T20:48:33Z

Hmmm... Do you use your own build? I have not seen such a problem... What is your environment? (I ask, because I wonder if this could be a mass problem)

And about the fact that thread.join() is called before creating the loop, you're right. But I do not classify this as an error, since, for example, this happens rarely for me, and at the first start my system even manages to get some data from the sensors before calling the first join().
I tried to illustrate this:

Sorry for the scribble )
With the subsequent runs of update_ble, everything is fine, since it is postponed for the period of time.

gadgetchnnel · 2020-01-24T21:00:40Z

@Magalex2x14 Yes, I had to build Python 3.7 (back whe HA dropped support for Python 3.5) since my Pi 3 B+ is still on Raspbian Stretch and didn't have anything newer than Python 3.5 in its repository. I must have missed this in the build options.

It's probably time to bite the bullet and finally try to update it to Buster and use the official Python version from the repository.

Ernst79 · 2020-01-25T18:55:34Z

@Magalex2x14 I've been testing the 0.5.2 release, but I noticed that the memory consumption seems to increase over time pretty quick. Memory usage has increased with 14% in about 12 h when running 0.5.2. Not sure if it is related to 0.5.2 (or even to mitemp_bt), but after going back to 0.4.3, it only increased 2% in 10 hours time. The sudden drops in memory use are restarts of home assistant.

I will keep an eye on it the coming days. I'm not sure is can do any harm. I'm not even sure that it is related to mitemp_bt. But maybe you can also check it on your end.

Magalex2x14 · 2020-01-25T19:28:05Z

The memory usage of version 0.5 should be more, since it stores hcidump completely in memory (in 0.4 it was a file).
But the uptrend is alarming, yes. I don't know much about memory management in Python. Apparently it's time to learn it )

Magalex2x14 · 2020-01-25T23:36:33Z

Successfully discovered a problem with mem_top.
The source of the problem is the list storing the hcidump (references count grows infinitely).
The plan is clear - I need to find a way to correctly receive the list (mutable object) from the another thread, and it is advisable to make sure that the start of a new thread is possible before hcidump parsing (since this is a "long" process).
There is still the thought of not stopping the dumpthread at all, that is, starting it once and parsing a copy of the hcidump list...

Ernst79 · 2020-01-26T10:16:43Z

Unfortunately, I can’t help you with that. It’s way out of my programming skills.

Just a confirmation, I did install 0.5.2 again, same uptrend in memory came back, about 10% in 10 hours on a Raspberry 3B.

Magalex2x14 · 2020-01-26T15:15:19Z

Solution found. After about 15 hours of testing, mem_top shows no signs of leakage. I pushed 0.5.3-beta with fix.

Ernst79 · 2020-01-26T19:04:23Z

Tested for 3 hours now, no more increase in memory, so it seems to be fixed. Great work!

Magalex2x14 · 2020-01-26T19:24:17Z

I added homeassistant service metrics to my influx_db using telegraf (procstat), since the total memory usage on my server does not give an objective picture - now I’ll observe... I noticed that even with a disabled component there is a trend to increase the used memory. But it seems that this trend is decreasing over time. Time will tell...

Ernst79 · 2020-01-26T19:54:40Z

I also still have a very small upward trend in memory usage, but I don’t think it’s related to mitemp_bt. As far as I can remember, this has always been the case.

Magalex2x14 · 2020-02-03T18:21:05Z

OK, guys. More than a week has passed, and I see no reason to leave 0.5 in beta status. I suggest releasing it to the master, if you have no arguments against.

Ernst79 · 2020-02-03T19:43:51Z

Yes, go ahead. It’s running fine here for a week or so.

Magalex2x14 · 2020-02-04T22:11:01Z

I close, because v0.5 was released

Magalex2x14 pinned this issue Jan 22, 2020

Magalex2x14 closed this as completed Feb 4, 2020

Magalex2x14 unpinned this issue Feb 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.5.0 testing #29

0.5.0 testing #29

Magalex2x14 commented Jan 22, 2020 •

edited

Magalex2x14 commented Jan 22, 2020 •

edited

Ernst79 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Ernst79 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Ernst79 commented Jan 22, 2020 •

edited

aqualx commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

aqualx commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020 •

edited

Ernst79 commented Jan 22, 2020

gadgetchnnel commented Jan 24, 2020 •

edited

Magalex2x14 commented Jan 24, 2020

gadgetchnnel commented Jan 24, 2020 •

edited

Magalex2x14 commented Jan 24, 2020

gadgetchnnel commented Jan 24, 2020

Ernst79 commented Jan 25, 2020 •

edited

Magalex2x14 commented Jan 25, 2020

Magalex2x14 commented Jan 25, 2020

Ernst79 commented Jan 26, 2020

Magalex2x14 commented Jan 26, 2020 •

edited

Ernst79 commented Jan 26, 2020

Magalex2x14 commented Jan 26, 2020

Ernst79 commented Jan 26, 2020

Magalex2x14 commented Feb 3, 2020 •

edited

Ernst79 commented Feb 3, 2020

Magalex2x14 commented Feb 4, 2020

0.5.0 testing #29

0.5.0 testing #29

Comments

Magalex2x14 commented Jan 22, 2020 • edited

Magalex2x14 commented Jan 22, 2020 • edited

Ernst79 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Ernst79 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Ernst79 commented Jan 22, 2020 • edited

aqualx commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

aqualx commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020

Magalex2x14 commented Jan 22, 2020 • edited

Ernst79 commented Jan 22, 2020

gadgetchnnel commented Jan 24, 2020 • edited

Magalex2x14 commented Jan 24, 2020

gadgetchnnel commented Jan 24, 2020 • edited

Magalex2x14 commented Jan 24, 2020

gadgetchnnel commented Jan 24, 2020

Ernst79 commented Jan 25, 2020 • edited

Magalex2x14 commented Jan 25, 2020

Magalex2x14 commented Jan 25, 2020

Ernst79 commented Jan 26, 2020

Magalex2x14 commented Jan 26, 2020 • edited

Ernst79 commented Jan 26, 2020

Magalex2x14 commented Jan 26, 2020

Ernst79 commented Jan 26, 2020

Magalex2x14 commented Feb 3, 2020 • edited

Ernst79 commented Feb 3, 2020

Magalex2x14 commented Feb 4, 2020

Magalex2x14 commented Jan 22, 2020 •

edited

Magalex2x14 commented Jan 22, 2020 •

edited

Ernst79 commented Jan 22, 2020 •

edited

Magalex2x14 commented Jan 22, 2020 •

edited

gadgetchnnel commented Jan 24, 2020 •

edited

gadgetchnnel commented Jan 24, 2020 •

edited

Ernst79 commented Jan 25, 2020 •

edited

Magalex2x14 commented Jan 26, 2020 •

edited

Magalex2x14 commented Feb 3, 2020 •

edited