Exposure as Transport #15
Comments
|
Hi @smar000 It is fully exposed as a transport, using the standard asyncio Protocol/Transport model. Have a look at client.py, or - more simply: from evohome_rf import Gateway
def process_message(msg) -> None:
# e.g. msg.payload is the the message dict, msg.src.id is the source device address
dtm = f"{msg.dtm:%H:%M:%S.%f}"[:-3]
print(f"{dtm} {msg}")
async main(serial_port, **kwargs):
gwy = Gateway(serial_port, **kwargs)
protocol, _ = gwy.create_client(process_message)
await asyncio.create_task(gwy.start())
if __name__ == "__main__":
asyncio.run(main("/dev/ttyUSB0"))The idea is - as I'm sure you'll now - is to add your evohome_rf --> RESTful gateway in Receiving messages (an inbound msg) is pretty good, but sending commands (an outbound msg) isn't fully fleshed out yet - you can use There are a shedload of config parameters available (see Workflow is a bit broken, presently: I'll tidy up the master branch today/tomorrow & move dev to another branch. Any questions, just ask! |
|
Thanks for the quick response back, and that's great. I haven't actually had a chance to look at your latest code but will hopefully do so over the weekend. Thanks again and I'm sure I will have questions... :) |
|
No worries: I'd be very happy to see someone else using it! |
|
You have clearly done a tremendous amount of work - really impressive!! One small issue - I am getting the following error/stack trace: Commenting out the respective lines in |
|
Thanks - I will fix this - it seems your nanoCUL is not FTDI? I had a look at your code - this library will do a lot of the heavy lifting for you - designed exactly for your use-case. Good luck! Hmmm. Should be: if os.name == "posix":
try:
ser_instance.set_low_latency_mode(True) # only for FTDI?
except AttributeError: # TODO: also: ValueError?
passAnyway, removing that code block will do no harm. |
|
Yes absolutely - I should be able shrink my code down signficantly. WRT the latency issue, I tried with 2 nanoCULs. One with an FTDI and one with CH340. Both gave the same issue, so I'm not entirely sure that it is FTDI related. Anyway, its running without the lines, and so not a big issue! |
|
Hi @zxdavb I have got a skeleton version of my system working with your library - looking great so far. Would you mind giving me a full sample scheme showing controller, dhw, ufh controller, trvs, zones etc (I had a look at Also are you open to exposing some further properties/attributes, e.g. in Message object, in the Thanks! |
@smar000 apologies for barging in on the thread, but, I was literally just looking at seeing if evohome_rf could be used for a project to integrate directly with MQTT so your question is great timing. Currently using an HGI80 on a really old and heavily customised Domoticz install just for evohome. I've been considering migrating to Home Assistant but my whole system is modular built using node-red & MQTT, so while moving to HA would get me an up-to-date evohome integration I'm trying to move away from using a myriad of different smart home software just to provide specific integrations. I'd much rather have hardware <-> MQTT directly! Is your evohome2mqtt project public anywhere? Happy to help with testing in return for access, I've got a spare HGI80 that I used for development in the past so would run with that so as not to impact my live system. |
|
Hi @martynwendon I have an existing evohome -> MQTT system in my repo (evolistener/gateway), where I rewrote a lot of the domoticz packet decoding/encoding to python. That system is working reasonably well. Having looked at @zxdavb work here though, he has a built a much more robust - and comprehensive - packet decoding/encoding library, and so it makes sense for me to migrate over to his library. My current 'skeleton' system is no where near ready for public use, and will be many weeks/month or so away before I upload it publicly. |
|
@zxdavb I came across an interesting error a short while ago: I certainly haven't changed controllers, and don't recognise 01:191706 as one of my devices... I don't believe any of my neighbours have an evohome system either. Any ideas? Thanks. |
I think I did try this a few years ago but didn't have much success with the HGI80. I don't recall what the problem was, but I didn't have a spare HGI80 back then so didn't spend too much time on it. I stayed with Domoticz as that was already working. I'll take another look though since I have the spare HGI80 now so can afford to play around a bit more.
Agreed, the progress has been amazing and I've been tracking the thread on the Home Assistant forum for quite some time. I had a bit of spare time so thought I'd dig into possibly writing my own MQTT direct gateway using the evohome_rf library, hence I spotted your question.
No problem at all, I'll check back at a later date. In the meantime if you do want anybody to test please let me know. |
This is a cut and past job - will give the idea - note the mapping between UFH circuit and a evohome zone |
Everything you might need should be exposed, or example: Is there anything specific you think you might need, but you can't find? |
This is simply saying that an
This confirms the diagnosis - the source address is corrupt (t was likely the HGI, which evohome_rf is getting to send This unfortunately common, and I have to work out a way of handling this. The controller doesn't have this issue, because it knows teh 'truth', whilst evohome_rf relies upon eavesdropping. |
|
I am not sure if you need to have evohome_rf maintain state? Possibly you can do the equivalent of this: ... and just consume (i.e. forward via MQTT) the messages? Have you tried to do anything like: ...and I'm always keen on receiving packet logs to test against! |
|
Many thanks for the above - very helpful, and noted on the likely cause of the error message.
I haven't yet, but I should have the packet logs saved. Let's see how I get on, and if the problem persists, I will get back to you.
Thanks and yes, for the most part I found they are. One specific one that I didn't fine is the Not a big deal TBH, but I thought it might be helpful to have this within the Message class itself. Thanks again. |
|
You can use a packet log as a source, rather than a serial port - the will be useful fro test/dev: I think Re: Opentherm - I am driven/limited by other projects, from which I am reluctant to deviate far... I see that JSON as distinct from my schema. E.g. https://github.com/mvn23/pyotgw |
Thanks and that is fine. After thinking about it, and seeing the different results under With regards to sending commands, is there any callback that confirms the status of the sending, e.g. command sent/success/fail notifications? I see various packet messages showing retry/expired etc, but haven't (yet) looked to see how it all works in your code. EDIT: Also is there any way to 'label' or tag devices? For example, if say there are multiple TRVs in a room, can we identify specific TRVs by name? |
|
Another question - in your schema, what is the correct way to define
Hope the above makes sense! Thanks. |
|
OMG, the answer to this will be worth making into a wiki entry... 1st (useful to know): the output schema can be used as an input schema, so just use: and hit Ctrl-C after the last 2nd: I have not had a UFH system to test against, so expect some issues. To answer this question:
Please start by providing me the schema, so we can discuss it without confusion. 3rd (the biggie): There are three means used by evohome_rf to construct the schema (in order of development):
There are advantages to all 3 options - and each has issues, too. One is that when an received packet - usually an eavesdropped packet) does not match the schema: The schema is important, it is used to construct some meta-data (e.g. zone heat demand from the child devices of that zone), and the objects (so you can, e.g. set the mode of a zones). 4th...
I just want to say, just want to avoid confusion: the schema should reflect reality: (evohome_rf doesn't change the behaviour of evohome), so 'making' the controller the sensor for a zone is only appropriate if it is the sensor for that zone. There is no simple means to learn the sensor for a zone, when that sensor is the controller - I do have some code for this, but it is only 'nearly-deterministic' and is currently disabled. That is, the only 'hard/fast' way to know which zone the controller is a sensor for (if any) is via an input schema (which we hope is 'true' to reality for the above reasons). |
What version of pyserial / pyserial-asyncio are you using? |
|
Thanks for the above - very helpful. I did try the cli, but didn't realise that it built the full schema for me - definitely one for the wiki! Schema output from discovery below: I have an HCC80R installed, and so can give you any data that you may need from this. Also in answer to your 4th point, yes, the scenario is "actual", in that my Given the importance of getting the schema correct, it might be helpful for end users if there was a pre-defined command for the cli that just went and got the schema and saved it to a file (or stdout) once the schema was complete. WDYT? |
pyserial is 3.4, and pyserial-asyncio is version 0.5 |
|
Just a follow on WRT the schema file. I added the above schema and tried to run the client with the schema as input. It is giving me the following error for the ufh_controller section: This is the ufh_controller section in the auto-discovered schema: Any suggestions as to what the issue may be? Thanks. |
So simply change the above to have: But! Don't be surprised it it doesn't work - I think I made some breaking changes - I will check it now. Just checked: this seems to work OK (a minimal schema, the rest done by discovery): Later, I'll test a full schema & let you know. Of course, you can test it too. |
As I said, above: there is no deterministic way to work out what zone the controller is a sensor for (if any) - you have to analyse the meta-data, and even then it's not fully guaranteed to be 'truth'. The best scenario is that you have N+1 zones (presumably all with sensors), and only N sensors (other than the controller). |
Reasonable idea - will add it as a feature, but until then: TBH, you don't really need a schema, except for:
Everything else can be discovered for evohome (that doesn't apply for other RAMSES-based systems, like Sundial, etc.). In fact, it is probably better to keep it minimal, rather than risk it being wrong: Note: evohome_rf will track all controllers (i.e. systems) it sees, but by default the first controller seen is the controller, (unless the allow/block lists come into play) - The schema allows you to ensure that this is your controller. |
|
I need a packet log from you
Let me test it (as I said, I haven't done a lot of UFH testing). Use this for now: |
Thanks and yes, I will be testing too. Packet log with above schema attached: |
Even starting with just the above, autodiscovery seems to overwrite the zone 7 sensor value to null. |
I haven't been seeing this. The callback doesn't seem to have been called on any of my expired commands that I could see.
It isn't any specific Command, though more often with the more "complex" commands such as EDIT: Hope this helps. On a separate note, do you know if it is possible to get an actuator to report its state (3EF0) by sending it an appropriate RQ command? |
|
OK, will have a look. |
|
A simple way to test is to send a command to a device that does not respond back, e.g, if I create a Command with the following arguments |
|
OK, I found the bugs (x2) - it works now, but is not ideal - tests only when a packet arrives - if that packet is 15 seconds after the expiry, you have to wait 15 seconds. Below, the packet expired at I will fix it presently. |
|
Try 0.9.7 - only the above issue remains. |
Working! Thanks again for the quick turn-around. |
|
I have pushed an update to master that will make callbacks for failed Command much faster. Something like: |
|
Hi again @zxdavb One thing to clarify - is it still the recommendation that the schema should only contain the controller ID and |
|
Yes, Controller ID only, and an allow_list - there are a few exceptions:
Eavesdropping is not generally recommended, but is needed to confirm a schema. |
Ok thanks and noted. |
|
Yes - may need allow_list off & eavesdropping on to get full schema - then make a schema, use an allow_list & turn off eavesdropping for ongoing, until any changes are made at some future time. TBH, you can manage without a full schema in most circumstances, but an allow_list is virtually required. I use the allow list in ramses_rf, but also in evohome_cc too - that it, you may want to consider using it in evohome_listener. |
|
Thanks and that makes sense. I will look to do it the same way.
For my own understanding, the allow_list is required to prevent messages appearing from (a) neighbouring evohome setups and (b) messages getting garbled in reception thus appearing to be from incorrect device ids? |
|
Yes (to both). The big issue is that valid packets can - as you say - have corrupted (but still valid) devices addresses. There are a lot of controls in place for this, but ultimately an allow list is required. |
|
Understood! Thanks. |
|
@zxdavb Looking at Also when should we use the Thanks. |
|
(not sure if I understand your question 100%, so here goes)
These are used for probing new device types or for development - not for production - discovery is a separate thing, and is built into the library. The client.py tool's main reason for being is to: a) perform ad-hoc stuff / learn about the protocol (e.g. SCAN_FULL), or to demonstrate how the library can be used (e.g. evohome-listener, or evohome_cc). I see you've pulled some ideas from client.py (which the idea), but you may also want to look at evohome_cc\__init__.py (although it is a little more obtuse). All the CLI switches are a 'cheat' - all that matters is passing the config/schema/device_list Generally, eavesdropping is to be avoided, especially without an I use voluptuous roughly the same way you use configparser, and so see my schema.py: CONFIG_SCHEMA = vol.Schema(
{
vol.Optional(DISABLE_DISCOVERY, default=False): bool,
vol.Optional(DISABLE_SENDING, default=False): bool,
vol.Optional(ENABLE_EAVESDROP, default=False): bool,
vol.Optional(REDUCE_PROCESSING, default=0): vol.All(
int, vol.Range(min=0, max=DONT_CREATE_MESSAGES)
),
vol.Optional(MAX_ZONES, default=DEFAULT_MAX_ZONES): vol.All(
int, vol.Range(min=1, max=16)
),
vol.Optional(USE_SCHEMA, default=True): vol.Any(None, bool),
vol.Optional(ENFORCE_ALLOWLIST, default=None): vol.Any(None, bool),
vol.Optional(ENFORCE_BLOCKLIST, default=None): vol.Any(None, bool),
vol.Optional(USE_NAMES, default=None): vol.Any(None, bool),
vol.Optional(EVOFW_FLAG, default=None): vol.Any(None, str),
},
extra=vol.ALLOW_EXTRA, # TODO: remove for production
)There is one thing that ramses_rf needs to do, and does not yet do well - that is periodically re-discover... This would be most useful or OTBs (R8810As) - I will ad this soon enough. HTH, ask again if not. |
|
Great, that clears up quite a bit. Will also have a look at your evohome_cc_init.py_. Thanks again. |
|
Hi David I noticed that zone names are available in |
|
I am happy with the split between I am happy to push in (impose) a (hand-crafted) ... however, the latter two can be managed with There are two bugs presently affecting
HTH |
|
BTW, you make a good point - eavesdropping of |
Thanks, will check these out. Noted on the bugs. |
Ok! |
|
Hi David Thanks for all your help (and patience!) throughout the migration. I am almost there I think. One final (hopefully!) question for now WRT the allow_list. Is this supposed to only permit exactly those devices listed or will it automatically allow any "18:" HGI device types through? I ask as I have a number of spurious messages purporting to be from HGI devices (e.g. I am seeing messages from the following HGI IDs now as I type: 18:000030, 18:000070, 18:007030, 18:056026, 18:136712; these are all in addition to my actual ID of 18:000730). My
Anyway, thanks again! |
|
Sorry, yet another query/clarification. I noticed that zone/dhw mode "until" datetimes are formatted (in the helper.) without the Should the |
|
Hi again @zxdavb One further query - I noticed that you have the default HGI device address in the e.g. some messages from my log file: |
|
Currently. Because you don't know the HGI80s address before you send a packet, you use
This is planned.
I am willing to take a view on this (the issue is whether it will be a breaking change), but is is kind of 'optional', see: https://docs.python.org/3/library/datetime.html#datetime.datetime.fromisoformat |
Understood!
Ah ok, I wasn't aware that it was optional. It's not that much of a problem, but openHAB's mqtt binding requires the "T" out of the box. Without it, additional formatting steps need to be carried out which has to be repeated for each datetime item etc. For now, I've added a small conversion before publishing to the mqtt broker. Maybe make it optional via config If you do re-consider it in the future? |
Making it a config option is not useful - I'll change the format if I can. |
Thanks. As I said, not a significant issue from my side. Anyway, my migration to your framework seems to be complete for now, and so I will close this thread. Sincere thanks once again for all your support throughout. |
Hi @zxdavb
Not an issue, but I'm the author of evogateway/listener - you may recall we had a brief chat a few months back in the automatedhome forum. I was wondering if you've had any further thoughts on exposing your library as a transport. No urgency!
Many thanks.
The text was updated successfully, but these errors were encountered: