REST sensors show as "Unavailable" state after upgrading to Home Assistant 0.117.0

[mkanet]

The problem

Hi Guys, after upgrading to Home Assistant 0.117.0 four of my REST sensors started producing an "Unavailable" state along with respective "failed with [Errno 111] Connect call failed" errors in the log. My configuration has not changed at all before or after the upgrade.. Also, I can reach ALL the respective API endpoints with a regular web browser without any issues.

Environment

• Home Assistant Core release with the issue: Home Assistant 0.117.0
• Last working Home Assistant Core release (if known): 0.116
• Operating environment (OS/Container/Supervised/Core): VirtualBox VM running on Windows 10
• Integration causing this issue: Template
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/template

Problem-relevant configuration.yaml

sensor:
  - platform: rest
    resource: http://192.168.1.2:61208/api/3/fs
    name: Horizon C Drive Used
    value_template: '{{ value_json[0].percent | round }}'
    unit_of_measurement: "%"
    verify_ssl: false
    timeout: 30
    scan_interval: 16
  - platform: rest
    resource: http://192.168.1.2:61208/api/3/fs
    name: Horizon E Drive Used
    value_template: '{{ value_json[1].percent | round }}'
    unit_of_measurement: "%"
    verify_ssl: false
    timeout: 30
    scan_interval: 16
  - platform: rest
    name: diskio_0
    resource: http://192.168.1.2:61208/api/3/diskio
    value_template: '{{ ( value_json[0] ) }}'
    json_attributes_path: "$.0"
    json_attributes:
      - time_since_update
      - read_bytes
      - write_bytes
    verify_ssl: false
    timeout: 30
    scan_interval: 16
  - platform: rest
    name: diskio_1
    resource: http://192.168.1.2:61208/api/3/diskio
    value_template: '{{ ( value_json[1] ) }}'
    json_attributes_path: "$.1"
    json_attributes:
      - time_since_update
      - read_bytes
      - write_bytes
    verify_ssl: false
    timeout: 30
    scan_interval: 16

It is important to note that I have SEVERAL other REST sensors (few examples below) that connect to the same server that don't have any issues at all. The only difference I see is in the way value_template is defined when parsing a json response with more than 1 key.

  - platform: rest
    resource: http://192.168.1.2:61208/api/3/cpu
    name: Horizon CPU
    value_template: '{{ value_json.total | round }}'
    unit_of_measurement: "%"
    verify_ssl: false
    timeout: 30
    scan_interval: 16
  - platform: rest
    resource: http://192.168.1.2:61208/api/3/mem
    name: Horizon RAM
    value_template: '{{ value_json.percent | round }}'
    unit_of_measurement: "%"
    verify_ssl: false
    timeout: 30
    scan_interval: 16
  - platform: rest
    resource: http://192.168.1.2:61208/api/3/memswap
    name: Horizon Pagefile
    value_template: '{{ value_json.percent | round }}'
    unit_of_measurement: "%"
    verify_ssl: false
    timeout: 30
    scan_interval: 16

Traceback/Error logs

I get the below errors constantly in the log file:

2020-10-29 12:04:26 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.1.2:61208/api/3/fs failed with [Errno 111] Connect call failed ('192.168.1.2', 61208)
2020-10-29 12:04:26 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.1.2:61208/api/3/fs failed with [Errno 111] Connect call failed ('192.168.1.2', 61208)
2020-10-29 12:04:26 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.1.2:61208/api/3/diskio failed with [Errno 111] Connect call failed ('192.168.1.2', 61208)
2020-10-29 12:04:26 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.1.2:61208/api/3/diskio failed with [Errno 111] Connect call failed ('192.168.1.2', 61208)

Additional information

I looked at Home Assistant 0.117.0 breaking changes for templates.. however, I can't tell what change(s) broke my 4 REST sensors. It seems like something changed in Home Assistant adversely affecting REST sensors that have lines like below:

value_template: '{{ ( value_json[0] ) }}'
value_template: '{{ ( value_json[1] ) }}'

[probot-home-assistant]

rest documentation
rest source
_{^{(message by IssueLinks)}}

[cogneato]

possibly related to #42552

[mkanet]

@cogneato I would like to revert back to version 0.116.4 in order to use my REST sensors. Do you guys have the information you need from me; or, should I wait?

[cogneato]

@mkanet I think you've provided what you can as far as I can tell. You should go ahead and downgrade if you need to.

[mkanet]

Thank you.

[elupus]

My guess is it is related to: #41973

You don't happen to be running proxies do you?

[elupus]

Scratch that. Seem you say it works with simpler templates.

[fsghzh]

same problem here. downgrade to 0.116.4 solved the problem

[mkanet]

I haven't heard any news about a fix for this yet. I'm still using version 0.116.4

[bdraco]

If this is still a problem, please turn on debug logging so we can get more data about why it cannot connect

[mkanet]

@bdraco or @cogneato Could one of you please tell me what specific logger settings I should set? I want to be sure I capture all the debug information for the appropriate component(s) since it's a lot of trouble for me to upgrade to 0.117.0 again, capture the log, then restore version 0.116.4 again. Will the below settings capture everything needed related to this issue?

logger:
  default: info
  logs:
    homeassistant.core: debug

I want to be certain I capture all the information you guys need before restoring to version 0.116.4 again.

@fsghzh Maybe, you wouldn't mind also doing the same in order to confirm you're having the same issue?

[fsghzh]

Hi, I upgraded to 0.117.4. Still the same Problem.
This is one of the sensors:

• platform: rest
  name: WP_Temperatur_Aussen
  resource: "http://192.168.168.24/usr-cgi/xml.cgi?A|27|27"
  value_template: "{{ value_json['PCOWEB']['PCO']['ANALOG']['VARIABLE']['VALUE'] }}"
  unit_of_measurement: °C
  device_class: temperature
  scan_interval: 300

Here is the log entry:

2020-11-05 11:42:03 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:03 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:04 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|27|27 failed with peer unexpectedly closed connection
2020-11-05 11:42:04 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:04 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|29|29 failed with peer unexpectedly closed connection
2020-11-05 11:42:04 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:05 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.rest
2020-11-05 11:42:06 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|28|28 failed with peer unexpectedly closed connection
2020-11-05 11:42:06 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:06 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|31|31 failed with peer unexpectedly closed connection
2020-11-05 11:42:06 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:11 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|30|30 failed with peer unexpectedly closed connection
2020-11-05 11:42:11 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:11 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|40|40 failed with peer unexpectedly closed connection
2020-11-05 11:42:11 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:11 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?A|41|41 failed with peer unexpectedly closed connection
2020-11-05 11:42:11 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|44|44 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|85|85 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|80|80 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|82|82 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|84|84 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|136|136 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.
2020-11-05 11:42:12 ERROR (MainThread) [homeassistant.components.rest.data] Error fetching data: http://192.168.168.24/usr-cgi/xml.cgi?D|140|140 failed with peer unexpectedly closed connection
2020-11-05 11:42:12 WARNING (MainThread) [homeassistant.components.sensor] Platform rest not ready yet. Retrying in 60 seconds.

[bdraco]

Can you check to see if the webserver at 192.168.168.24 is using proper line endings? It might be this case python-hyper/h11#7