Unable to connect to a test device with keys in SDK, but not in DCL

[priyankub]

The problem

I tried connecting to a test device with keys in the SDK (Github PAA) via different methods, but it keeps failing with error 101, because the matter server seems to not update certs from SDK, if it is already in DCL.

The device works with Matter integration in Alexa, Homekit and SmartThings, but not with HA, because HA likely prioritizes DCL over SDK for certs.

Logs:
Scan the QR using an iPad running the HA companion app -

2024-01-17 15:53:50 core-matter-server matter_server.server.device_controller[126] INFO Starting Matter commissioning with code using Node ID 6.
2024-01-17 15:53:59 core-matter-server chip.EM[126] ERROR Failed to Send CHIP MessageCounter:264160583 on exchange 37472i sendCount: 4 max retries: 4
2024-01-17 15:53:59 core-matter-server chip.SC[126] ERROR PASESession timed out while waiting for a response from the peer. Expected message type was 33
2024-01-17 15:54:02 core-matter-server chip.CTL[126] ERROR Failed in verifying 'Attestation Information' command received from the device: err 101. Look at AttestationVerificationResult enum to understand the errors
2024-01-17 15:54:02 core-matter-server chip.CTL[126] ERROR Failed to perform commissioning step 13
2024-01-17 15:54:02 core-matter-server chip.EM[126] ERROR Failed to send Solitary ack for MessageCounter:13165336 on exchange 37480i:src/messaging/ExchangeContext.cpp:103: CHIP Error 0x00000002: Connection aborted
2024-01-17 15:54:02 core-matter-server matter_server.server.client_handler[126] ERROR [281472898205456] Error handling message: CommandMessage(message_id='<REDACTED>', command='commission_with_code', args={'code': 'MT:<REDACTED>', 'network_only': True})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/matter_server/server/client_handler.py", line 188, in _run_handler
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/matter_server/server/device_controller.py", line 196, in commission_with_code
    raise NodeCommissionFailed(
matter_server.common.errors.NodeCommissionFailed: Commission with code failed for node 6

2024-01-17 15:48:32 core-matter-server matter_server.server.device_controller[126] INFO Starting Matter commissioning with code using Node ID 5.
2024-01-17 15:48:42 core-matter-server chip.EM[126] ERROR Failed to Send CHIP MessageCounter:264160578 on exchange 37463i sendCount: 4 max retries: 4
2024-01-17 15:48:42 core-matter-server chip.SC[126] ERROR PASESession timed out while waiting for a response from the peer. Expected message type was 33
2024-01-17 15:48:44 core-matter-server chip.CTL[126] ERROR Failed in verifying 'Attestation Information' command received from the device: err 101. Look at AttestationVerificationResult enum to understand the errors
2024-01-17 15:48:44 core-matter-server chip.CTL[126] ERROR Failed to perform commissioning step 13
2024-01-17 15:48:44 core-matter-server chip.EM[126] ERROR Failed to send Solitary ack for MessageCounter:248381685 on exchange 37471i:src/messaging/ExchangeContext.cpp:103: CHIP Error 0x00000002: Connection aborted
2024-01-17 15:48:44 core-matter-server matter_server.server.client_handler[126] ERROR [281472898205456] Error handling message: CommandMessage(message_id='<REDACTED>', command='commission_with_code', args={'code': 'MT:<REDACTED>', 'network_only': True})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/matter_server/server/client_handler.py", line 188, in _run_handler
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/matter_server/server/device_controller.py", line 196, in commission_with_code
    raise NodeCommissionFailed(
matter_server.common.errors.NodeCommissionFailed: Commission with code failed for node 5
2024-01-17 15:48:44 core-matter-server chip.IN[126] ERROR Data received on an unknown session (LSID=7217). Dropping it!

HA companion app in an Android device:

2024-01-17 15:55:02 core-matter-server matter_server.server.device_controller[126] INFO Starting Matter commissioning with IP using Node ID 7.
2024-01-17 15:55:04 core-matter-server chip.CTL[126] ERROR Failed in verifying 'Attestation Information' command received from the device: err 101. Look at AttestationVerificationResult enum to understand the errors
2024-01-17 15:55:04 core-matter-server chip.CTL[126] ERROR Failed to perform commissioning step 13
2024-01-17 15:55:04 core-matter-server matter_server.server.client_handler[126] ERROR [281472898205456] Error handling message: CommandMessage(message_id='<REDACTED>', command='commission_on_network', args={'setup_pin_code': <REDACTED>, 'ip_addr': 'REDACTED'})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/matter_server/server/client_handler.py", line 188, in _run_handler
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/matter_server/server/device_controller.py", line 275, in commission_on_network
    raise NodeCommissionFailed(
matter_server.common.errors.NodeCommissionFailed: Commission using IP failed for node 7
2024-01-17 15:55:04 core-matter-server chip.EM[126] ERROR Failed to send Solitary ack for MessageCounter:156302796 on exchange 37488i:src/messaging/ExchangeContext.cpp:103: CHIP Error 0x00000002: Connection aborted

What version of Home Assistant Core has the issue?

core-2024.1.3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Matter

Link to integration documentation on our website

https://www.home-assistant.io/integrations/matter

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @home-assistant/matter, mind taking a look at this issue as it has been labeled with an integration (matter) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of matter can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign matter Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

matter documentation
matter source
_{^{(message by IssueLinks)}}

[marcelveldt]

Did you restart the server before commisisoning ? The certs are only refreshed at startup.

[priyankub]

The keys were added to the SDK about 8 months back and I have been trying to add this device for more than a month now. So I am not sure if restarting would have done anything differently.
But yes, I restarted the addon, HA, factory reset the device, but nothing worked

[priyankub]

The device still does not work

[drempelg]

The device in question is a device that isn't currently in the DCL, I can't go further than that on other details.

I will say however, that I have successfully commissioned the device using iOS and Android as the app platforms, and Homekit, Smartthings, Alexa, and Google Home as the eco systems with no problems.

I have a fresh ODroid HA setup that I brought online today, that is perfectly capable of matter commissioning devices that are in the DCL and certified.

The differences appear to be, Home Assistant as the Eco System while the device is not in the DCL.

Unfortunately I cannot give any more information than that.

[marcelveldt]

It would help me if you can give me some more details. As you know we are based on the official Matter SDK, just like every platform you mention. But the other platforms seem to still accept devices with invalid attestation certificates where we do not. You need a valid certificate either on DCL (prod) or github (dev/test).

So far we have tested with many, many devices, also ones still in development and this has worked every single time.
So there must be some edge case going on here...

Where are the certificates currently placed ?
We fetch the latest certificates from both Github and DCL at startup.

[priyankub]

The certificates are in Github/SDK, not in DCL. It was already mentioned in the first comment. Sorry if it was unclear.
Does HA likely prioritize DCL over SDK for certs?

[drempelg]

The PAA's are in the DCL and the SDK, but there is no Cert Decl recorded yet in the DCL (obviously since it hasn't been certified yet, we have a test one we generated for development) and the product id isn't recorded in the DCL.

Our DAC's and PAI's are production ones (same method for being generated as other products we've released and certified, and we've verified that HA can indeed matter commission those certified released products).

We aren't really sure what's happening other than the logs are showing that Home Assistant isn't finding the appropriate PAA when everyone else is, and that's the only other difference we can think of (something to do with the DCL and HA).

SDK version is 1.2.

[aleksrozman]

Is it perhaps the issue that according to the code you are only loading

GIT_CERTS = [
    "Chip-Test-PAA-FFF1-Cert",
    "Chip-Test-PAA-NoVID-Cert",
]

So anything else in the Github is not being loaded? I have both matter server and home assistant running in containers and will try tomorrow with the device in question to validate my hypothesis.

[priyankub]

I forked this repo and rewrote the certificate fetching helper to prefer git certs
When pairing using an iPad because that allows to pair uncertified accessory, and choosing pair anyway, I first get an attestation error, and then mDNS errors! I will see if I have amy luck fixing those now

INFO Starting Matter commissioning with code using Node ID 6 (attempt 0/3).
2024-01-30 23:12:30 78653e92-matter-server-test chip.CTL[126] ERROR Failed in verifying 'Attestation Information' command received from the device: err 101. Look at AttestationVerificationResult enum to understand the errors
2024-01-30 23:12:30 78653e92-matter-server-test chip.CTL[126] ERROR Failed to perform commissioning step 13
2024-01-30 23:12:30 78653e92-matter-server-test chip.EM[126] ERROR Failed to send Solitary ack for MessageCounter:42957945 on exchange 43557i:src/messaging/ExchangeContext.cpp:103: CHIP Error 0x00000002: Connection aborted
2024-01-30 23:12:35 78653e92-matter-server-test matter_server.server.device_controller[126] INFO Starting Matter commissioning with code using Node ID 6 (attempt 1/3).
2024-01-30 23:13:05 78653e92-matter-server-test chip.CTL[126] ERROR Discovery timed out
2024-01-30 23:13:05 78653e92-matter-server-test chip.ZCL[126] ERROR Secure Pairing Failed
2024-01-30 23:13:06 78653e92-matter-server-test chip.DIS[126] ERROR Timeout waiting for mDNS resolution.
2024-01-30 23:13:10 78653e92-matter-server-test matter_server.server.device_controller[126] INFO Starting Matter commissioning with code using Node ID 6 (attempt 2/3).

[priyankub]

FWIW, it is able to connect to other devices in the same network like an Echo Dot, so it is not likely an mDNS issue.

[priyankub]

@aleksrozman/ @drempelg - Could you try the modified matter-server addon on your side and see if you have more success than me. Once you install it, configure the integration to use port 5581
Looking at the logs, it is downloading all Git certificates, and since that is downloaded the last, it would rewrite other certs:

2024-01-31 11:17:36 78653e92-matter-server-test matter_server.server.helpers.paa_certificates[126] INFO Fetched 122 PAA root certificates from DCL.
2024-01-31 11:17:36 78653e92-matter-server-test matter_server.server.helpers.paa_certificates[126] INFO Fetching the latest PAA root certificates from Git.
2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.helpers.paa_certificates[126] INFO Fetched 90 PAA root certificates from Git.
2024-01-31 11:17:57 78653e92-matter-server-test FabricAdmin[126] WARNING Allocating new controller with CaIndex: 1, FabricId: 0x0000000000000002, NodeId: 0x000000000001B669, CatTags: []
2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.device_controller[126] INFO Loaded 0 nodes from stored configuration
2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Loading vendor info from storage.
2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Loaded 156 vendors from storage.
2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Fetching the latest vendor info from DCL.
2024-01-31 11:17:58 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Fetched 155 vendors from DCL.
2024-01-31 11:17:58 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Saving vendor info to storage.

[aleksrozman]

I've tried similar code and ran into the same mDNS looking issue. I haven't made much progress yet to narrow down as it looked like I needed to debug on the CHIP code.

…

On Wed, Jan 31, 2024, 10:22 AM priyankub ***@***.***> wrote: @aleksrozman/ @drempelg <https://github.com/drempelg> - Could you try the modified matter-server addon <https://github.com/priyankub/hassio-addons> on your side and see if you have more success than me. Once you install it, configure the integration to use port 5581 Looking at the logs, it is downloading all Git certificates, and since that is downloaded the last, it would rewrite other certs: 2024-01-31 11:17:36 78653e92-matter-server-test matter_server.server.helpers.paa_certificates[126] INFO Fetched 122 PAA root certificates from DCL. 2024-01-31 11:17:36 78653e92-matter-server-test matter_server.server.helpers.paa_certificates[126] INFO Fetching the latest PAA root certificates from Git. 2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.helpers.paa_certificates[126] INFO Fetched 90 PAA root certificates from Git. 2024-01-31 11:17:57 78653e92-matter-server-test FabricAdmin[126] WARNING Allocating new controller with CaIndex: 1, FabricId: 0x0000000000000002, NodeId: 0x000000000001B669, CatTags: [] 2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.device_controller[126] INFO Loaded 0 nodes from stored configuration 2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Loading vendor info from storage. 2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Loaded 156 vendors from storage. 2024-01-31 11:17:57 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Fetching the latest vendor info from DCL. 2024-01-31 11:17:58 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Fetched 155 vendors from DCL. 2024-01-31 11:17:58 78653e92-matter-server-test matter_server.server.vendor_info[126] INFO Saving vendor info to storage. — Reply to this email directly, view it on GitHub <#108324 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH2PJVYCNG4UZGFBBINPMLYRJVVDAVCNFSM6AAAAABCA7TMOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJZGQ2TAMRRHE> . You are receiving this because you commented.Message ID: ***@***.***>

[marcelveldt]

There are 2 issues in this report. mdns issues have NOTHING to do with certificates but networking.
If you see mdns errors being reported by the SDK, it means that the device couldn't be resolved on the network, either because the device is not responding or there isa network (configuration) issue.

Make sure you have IPv6 enabled on your local network and the Home Assistant host and make sure you have a FLAT network. So no vLANS and especially no mdns forwarders. Also some network gear have flaky implementations of IGMP/MLD snooping causing havoc with the IPv6 based multicast traffic from matter.

@priyankub sounds like you discovered a bug in the certificates retrieval ? Are you planning on doing a PR to the matter server ? BTW: was it really needed to push your own fork to pypi ? May I ask you friendly to remove that again to avoid confusion? Thanks! Let's work together and not against eachother, that would be nice

[priyankub]

@marcelveldt - I am a novice/part time/hobby developer, and mostly re-engineer code! Sorry I did not know of another way other than push to pypi. I had no intention of working against.
I submitted a PR for the git cert retrieval.

The mdns error is strange because I am able to pair this device to an Echo in the same network. I will continue to debug when I can

[marcelveldt]

This report can be closed, thanks to your own contribution. Thanks for identifying and fixing the issue @priyankub