0.80 introduces new upnp component but it is not working

[bboti86]

Home Assistant release with the issue:
0.80.0

Last working Home Assistant release (if known):
This is a new feature

Operating environment (Hass.io/Docker/Windows/etc.):
Running on Hass.io ersion 136 on HassOS 1.11

Component/platform:
https://www.home-assistant.io/components/upnp/

Description of problem:
I've tried to set up upnp ports for my Home Assistant and also for the add-ons that I want to be able to view from outside, but I'm getting the error from below. The router's configuration shows no sign of the upnp'd ports and of course the ports are still closed.
The router is a Netgear R8000.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

 upnp: 
   port_mapping: true
   ports: 
     1xx0: 11xx0
     3xx8: 13xx8
     7xx1: 17xx1
     8xx1: 18xx1
     9xx1: 19xx1
     hass: 443

Traceback (if applicable):

Error setting up entry R8000 (Gateway) for upnp
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/homeassistant/config_entries.py", line 236, in async_setup
result = await component.async_setup_entry(hass, self)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/components/upnp/__init__.py", line 113, in async_setup_entry
device = await Device.async_create_device(hass, ssdp_description)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/components/upnp/device.py", line 36, in async_create_device
upnp_device = await factory.async_create_device(ssdp_description)
  File "/usr/local/lib/python3.6/site-packages/async_upnp_client/__init__.py", line 848, in async_create_device
service = await self.async_create_service(service_desc_xml, description_url)
  File "/usr/local/lib/python3.6/site-packages/async_upnp_client/__init__.py", line 875, in async_create_service
return self.create_service(service_description_xml, scpd_xml)
  File "/usr/local/lib/python3.6/site-packages/async_upnp_client/__init__.py", line 882, in create_service
state_vars = self.create_state_variables(scpd_xml)
  File "/usr/local/lib/python3.6/site-packages/async_upnp_client/__init__.py", line 902, in create_state_variables
state_var = self.create_state_variable(state_var_xml)
  File "/usr/local/lib/python3.6/site-packages/async_upnp_client/__init__.py", line 908, in create_state_variable
state_variable_info = self._state_variable_parse_xml(state_variable_xml)
  File "/usr/local/lib/python3.6/site-packages/async_upnp_client/__init__.py", line 927, in _state_variable_parse_xml
state_variable_xml.find('service:sendEventsAttribute', NS).text == 'yes'
AttributeError: 'NoneType' object has no attribute 'text'

Additional information:

[dgomes]

CC @StevenLooman

[dgomes]

Any chance of you running a wireshark capture or changing the code to print the XML ?

[bboti86]

I can do both. I'm not at home now so I can't do the wireshark, but I can access the hass.io instance and I can change the code.
Let me know what I should do!

[dgomes]

We just need the xml that is breaking the current implementation. So if you can provide us that it would be super!

[StevenLooman]

You can also use the traffic-logger from async_upnp_client. Add this to your home assistant configuration.yaml:

logger:
  default: warning
  logs:
    async_upnp_client: debug
    async_upnp_client.igd: debug
    async_upnp_client.traffic: debug

Note that this silences all home assistant messages, so you probably want to remove this later on.

[StevenLooman]

While I have closed the referred issue in the async_upnp_client project/handled this error (hopefully), I'm still curious about the XML!

[bboti86]

I've set the logging as requested and this is what I see: https://pastebin.com/ZFp9rzSt (it is impossible to paste and XML here without being processed, so I had to use pastebin)
Is this the one you were looking for?
B

[StevenLooman]

Actually, I'm expecting a few files containing XML. Can you try using Wireshark?

[bboti86]

Hey! I've fired up Wireshark but the problem is that I am connecting the raspberry pi running HassOS with an Ethernet cable but the laptop with WiFi so I only see the broadcast packages. But there is no XML at all there.
Anything else I could try?
B

[muspelheim]

same on tplink router

[StevenLooman]

@bboti86 Ok, lets try in a different way. On the RPi you're running HassOS on, try to run this:

$ python -m netdisco

I get this part:

igd:
[{'host': '192.168.178.1',
  'manufacturer': 'Ambit',
...
  'ssdp_description': 'http://192.168.1.1:80/RootDevice.xml',
...
  'upnp_device_type': 'urn:schemas-upnp-org:device:InternetGatewayDevice:1'}]

The ssdp_description is the part we're looking for. Now, using a tool to get files from http sources, such as wget, curl, or your browser, go to the URL from the ssdp_description.

For me, this gives:

$ curl http://192.168.1.1:80/RootDevice.xml
<?xml version="1.0" encoding="utf-8"?>
<root xmlns="urn:schemas-upnp-org:device-1-0">
...
<URLBase>http://192.168.178.1:80</URLBase>
...
<serviceList>
<service>
<serviceType>urn:schemas-upnp-org:service:Layer3Forwarding:1</serviceType>
<serviceId>urn:upnp-org:serviceId:L3Forwarding1</serviceId>
<SCPDURL>/Layer3Forwarding.xml</SCPDURL>
<controlURL>/Layer3Forwarding</controlURL>
<eventSubURL>/Layer3Forwarding</eventSubURL>
</service>
</serviceList>
<deviceList>
...
</root>

Now, for all <service>s you see in the result, get all the files pointed to in the <SCDPURL> part. You might need to prepend the URLBase value to get a valid URL. The service in the listing above has the URL: http://192.168.178.1:80/Layer3Forwarding

For example, for the part above:

$ curl http://192.168.178.1:80/Layer3Forwarding
<scpd xmlns="urn:schemas-upnp-org:service-1-0">
<specVersion>
<major>1</major>
<minor>0</minor>
</specVersion>
<actionList>
<action>
<name>SetDefaultConnectionService</name>
<argumentList>
<argument>
<name>NewDefaultConnectionService</name>
<direction>in</direction>
<relatedStateVariable>DefaultConnectionService</relatedStateVariable>
</argument>
</argumentList>
</action>
<action>
<name>GetDefaultConnectionService</name>
<argumentList>
<argument>
<name>NewDefaultConnectionService</name>
<direction>out</direction>
<relatedStateVariable>DefaultConnectionService</relatedStateVariable>
</argument>
</argumentList>
</action>
</actionList>
<serviceStateTable>
<stateVariable sendEvents="yes">
<name>DefaultConnectionService</name>
<dataType>string</dataType>
</stateVariable>
</serviceStateTable>
</scpd>

Please do this for the first URL and for all services and either attach the files here, or upload them somewhere.

@muspelheim Can you try to do the same?

[bboti86]

Hi!

Thanks for the detailed instructions!
In the HassOS - Hass.IO architecture you don't have access to the other docker containers so first I had to run
pip3 install netdisco
than the actual command was
python3 -m netdisco
(just for future reference if somebody else needs this as well)

So the URL was:
http://192.168.1.1:5000/Public_UPNP_gatedesc.xml
The xml file behind that is: https://pastebin.com/Ch5Mga07

There are multiple services:
#1: http://192.168.1.1:5000/Public_UPNP_Layer3F.xml : https://pastebin.com/NbisTNcY
#2: http://192.168.1.1:5000/Public_UPNP_WAND.xml : https://pastebin.com/4H2pD9Yb
#3: http://192.168.1.1:5000/Public_UPNP_WANEtherLinkCfg.xml : https://pastebin.com/qqRHATkF
#4: http://192.168.1.1:5000/Public_UPNP_WANIPConn.xml : https://pastebin.com/h0Xh7CUL
#5: http://192.168.1.1:5000/Public_UPNP_WANPPPConn.xml : https://pastebin.com/tsUh7FDt

I hope this helps!
B

[bboti86]

BTW, when running netdisco I've found something weird:
netdisco discovers the router twice, first as an igd device:
igd:
[{'host': '192.168.1.1',
'manufacturer': 'NETGEAR, Inc.',
'model_name': 'R8000',
'model_number': 'R8000',
'name': 'R8000 (Gateway)',
'port': 5000,
'serial': '4MXXXXXXXX61',
'ssdp_description': 'http://192.168.1.1:5000/Public_UPNP_gatedesc.xml',
'udn': 'uuid:1d3c4d73-ccad-a89e-3b09-dfXXXXXX8c',
'upnp_device_type': 'urn:schemas-upnp-org:device:InternetGatewayDevice:1'}]

than as a router:
netgear_router:
[{'host': '192.168.1.1',
'manufacturer': 'NETGEAR, Inc.',
'model_name': 'R8000',
'model_number': 'R8000',
'name': 'R8000 (Gateway)',
'port': 5000,
'serial': '4XXXXXXXXX561',
'ssdp_description': 'http://192.168.1.1:5000/Public_UPNP_gatedesc.xml',
'udn': 'uuid:1d3c4d73-ccad-a89e-3b09-dfXXXXXX8c',
'upnp_device_type': 'urn:schemas-upnp-org:device:InternetGatewayDevice:1'}]

Can this be a problem as well?

B

[dgomes]

Different components will make use of the discovery information

[StevenLooman]

@bboti86 Thanks! It is indeed missing something in the XML. In case you're interested: the used library, async_upnp_client, expects all stateVariables in the given XML, to state whether changes send events or not.

I've updated async_upnp_client to handle this case and assume it does not. When #17560 is merged, please do test.

[Anonym-tsk]

Now i have other warnings (0.80.3)

2018-10-18 18:07:30 WARNING (MainThread) [async_upnp_client] Invalid XML for state variable/send events:
<ns0:stateVariable xmlns:ns0="urn:schemas-upnp-org:service-1-0">
      <ns0:name>OSMajorVersion</ns0:name>
      <ns0:dataType>i4</ns0:dataType>
    </ns0:stateVariable>

2018-10-18 18:07:30 WARNING (MainThread) [async_upnp_client] Invalid XML for state variable/send events:
<ns0:stateVariable xmlns:ns0="urn:schemas-upnp-org:service-1-0">
      <ns0:name>OSMinorVersion</ns0:name>
      <ns0:dataType>i4</ns0:dataType>
    </ns0:stateVariable>

2018-10-18 18:07:30 WARNING (MainThread) [async_upnp_client] Invalid XML for state variable/send events:
<ns0:stateVariable xmlns:ns0="urn:schemas-upnp-org:service-1-0">
      <ns0:name>OSBuildNumber</ns0:name>
      <ns0:dataType>i4</ns0:dataType>
    </ns0:stateVariable>

2018-10-18 18:07:30 WARNING (MainThread) [async_upnp_client] Invalid XML for state variable/send events:
<ns0:stateVariable xmlns:ns0="urn:schemas-upnp-org:service-1-0">
      <ns0:name>OSMachineName</ns0:name>
      <ns0:dataType>string</ns0:dataType>
    </ns0:stateVariable>

[dgomes]

@Anonym-tsk this is a new issue, please open a new GitHub issue ;)