opensky with Update Error

[ingeniumdesign]

Hi all,

The problem

all the time i have a Warning: with opensky Plugin.
Github opensky

Environment

i use the last HA Version. core-2021.1.5

Problem-relevant configuration.yaml

# OpenSky https://www.home-assistant.io/integrations/opensky/
- platform: opensky
  name: opensky
  latitude: !secret homeassistant_latitude
  longitude: !secret homeassistant_longitude
  radius: 20

SCAN_INTERVAL = timedelta(seconds=12) # opensky public limit is 10 seconds

Traceback/Error logs

WARNING (MainThread) [homeassistant.components.sensor] Updating opensky sensor took longer than the scheduled update interval 0:00:12

my Log is full with opensky Warnings 👎

Additional information

Thanks all for help!

[frenck]

Hi there,

Thanks for opening up an issue. When creating an issue, an issue template is presented. It looks like you've (partly) removed it. This causes your issue report to be incomplete.

Please edit your issue to have the full issue template back again and provide as much of the requested data as possible. Without it, we won't be able to help you.

Thanks already! 👍

[ingeniumdesign]

okay, i updated the issue 👍

[brinzlee]

I have this issue too and have also noted the sensor appears on the main map of HomeAssistant as a tracker...this has only recently appeared.

[akeslo]

experiencing the same issue

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:605
Integration: Sensor (documentation, issues)
First occurred: 7:37:57 PM (8 occurrences)
Last logged: 7:41:57 PM

Updating opensky sensor took longer than the scheduled update interval 0:00:12

[brayStorm]

#43571

[frederickjh]

This is a duplicate of #43571. So this issue could be closed.
A pull request not related to this issue, was put in that could fix this but the pull request was rejected by the HA developers. The code that fixes this is available in the pull request coders fork and can be installed as a custom component.

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[brinzlee]

I tried using this again the other day and still had lots of errors in my logs. What exactly has been updated.

[frederickjh]

My guess is that nothing has been updated. They have a bot running around closing issues that are "stale". If no one cares enough to work on an issue or keep commenting on it, then they just close it. Not sure it is the best policy to encourage the community to report bugs. Now before taking the time to report a bug you have to think if a developer will take enough interest to do something or not. Otherwise you are just wasting your own time too. I have been bit by this in the HA issue queue before so for me to report something now it has to be big otherwise let some other person roll the dice. Unsubscribing as the notice that github-actions just posted in the "kiss of death" to an issue.

[ingeniumdesign]

the Error log is full from:
[homeassistant.components.sensor] Updating opensky sensor took longer than the scheduled update interval 0:00:12

[h1ghrise]

same here.
Installed opensky today, and already my logs are full with

2021-09-26 21:32:33 WARNING (MainThread) [homeassistant.components.sensor] Updating opensky sensor took longer than the scheduled update interval 0:00:12
2021-09-26 21:32:33 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.opensky fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 446, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 649, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/opensky/sensor.py", line 140, in update
    states = self._session.get(OPENSKY_API_URL).json().get(ATTR_STATES)
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm using openWRT and stubby for DNS resolution. Stubby is the local Port 53 DNS, which forwards all DNS requests encrypted (DoT) to a DNSSEC and DoT enabled DNS server. Might be related?

[mbuzina]

Same for me, just found this after filing an issue myself: #61808

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[niemyjski]

Same issue

[chris-github1975]

I have the same issue, fresh install today on..
Home Assistant Core 2022.5.5
Home Assistant Supervisor 2022.05.3
Home Assistant OS 7.6

[rohrsh]

Yep, me too

[BettySwallocks]

Is this connected to the opensky public api having 100 calls per day limit? Based on this any HA system using the standard config, with 12 second intervals hardcoded in the sensor.py, would have used their limit by 00:20 (twenty minutes) into a new day.

[BettySwallocks]

Registered users can get 1000 calls per day. The script would have to be altered to include code to authenticate at appropriate intervals. If using 95 second (or so) intervals then you could get a full 24 hours of coverage, but many flights would be completely missed. So seems that this integration may be dead?

[six-shooter]

Registered users can get 1000 calls per day. The script would have to be altered to include code to authenticate at appropriate intervals. If using 95 second (or so) intervals then you could get a full 24 hours of coverage, but many flights would be completely missed. So seems that this integration may be dead?

how can I put the interval at 95/100 seconds?

[BettySwallocks]

So this depends on your build & HA type.
On my Raspi docker Supervisor image I open my own Putty session to my Pi - so not using the built in file editor or Visual Studo editor. From a command prompt as sudo I locate my most recent docker image path and then find the sub-path /usr/src/homeassistant/homeassistant/components/opensky/ and edit (nano) the sensors.py script. I see the hardcoded interval on or around line 30:

SCAN_INTERVAL = timedelta(seconds=12) # opensky public limit is 10 seconds

Modify that 12 value to what you want and save the script under the same name.

But, as I previously said, the issue is likely to be Opensky organisation putting harsh limits on anonymous API calls, so you will probably miss lots of flights with a 95 second interval,

P.S. I've found that any patch updates to your HA will undo your interval edit...

[JelleKoster]

The necromancy on this issue (which was for a different error) seems to be the same as separate fresh issue #74935.
Might be best to continue there?

I'm also encountering this, by the way.

[BettySwallocks]

This is an open issue with several contributors, including one of the main devs (frenk) - why would we move our comments over to another issue?

[vingerha]

OK done, hopefully this will work with the rest of your library! As I said seems to work fine in my case

Hi, I was almost on the way to add the user/pwd myself as a PR but then noticed this one. Above you mention "new standards" As I can hack a bit but not embedded into the HA dev cycle, what does this mean to the existing sensor which (as I may say) looks pretty straightforward...so why new standards ?

[OzGav]

The are a bunch of design decisions that are rules for devs to follow. @joostlek is improving the current integration. If you want authentication now then you can do your own or use my fork (link further up) as a custom component

[vingerha]

Thanks....then similarly as you ..will steer away as this is not my skillset. I already forked yours and will test later this week when back home. Another thing I want to add is that the discovered airplanes end up in a separate sensor as attributes, so they can be further used in HA, e.g. use their position to show on a map card if they fulfill conditions. I already created something similar with floghtaware but this is not a free service

[joostlek]

Next steps will be adding config flow, so it can be set up via UI. Then we can look into adding authentication. Then I would like to have someone who is a contributing user to OpenSky (like you have these sensors hooked up at your home) and check if there's a way we can detect if someone is a contributing user (that grants free access to own states and gives you 8000 credits) and add sensors for that

[joostlek]

Next release you can authenticate yourself with Opensky :)

[joostlek]

Have you tried it yet 👀

[kbrint]

I couldn't see how to do it on 2023.9.0.

…

On Thu, Sep 7, 2023 at 2:56 PM Joost Lekkerkerker ***@***.***> wrote: Have you tried it yet 👀 — Reply to this email directly, view it on GitHub <#45453 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMSWDRK2ZMJRMQY27P33TXZI7HJANCNFSM4WPRLG4Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- kevin brintnall

[joostlek]

[vingerha]

It throws an error when tryin to login with my user/pwd, this user/pwd is working fine with a adapted fork from @OzGav which is running in parallel on my other instance

EDIT: correction, the other instance also has an issue... 'too many requests'

2023-09-08 07:35:22.308 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/python_opensky/opensky.py", line 121, in _request
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 429, message='', url=URL('https://opensky-network.org:443/api/states/all?time=0&extended=true')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 80, in ban_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 31, in headers_middleware
response = await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle
result = await handler(request, **request.match_info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 63, in with_admin
return await func(self, request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 234, in post
return await super().post(request, flow_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper
result = await method(view, request, data, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 110, in post
result = await self._flow_mgr.async_configure(flow_id, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 298, in async_configure
result = await self._async_handle_step(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 394, in _async_handle_step
result: FlowResult = await getattr(flow, method)(user_input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/opensky/config_flow.py", line 120, in async_step_init
await opensky.authenticate(
File "/usr/local/lib/python3.11/site-packages/python_opensky/opensky.py", line 49, in authenticate
await self.get_states()
File "/usr/local/lib/python3.11/site-packages/python_opensky/opensky.py", line 166, in get_states
data = await self._request("states/all", data=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/python_opensky/opensky.py", line 133, in _request
raise OpenSkyConnectionError(msg) from exception
python_opensky.exceptions.OpenSkyConnectionError: Error occurred while communicating with OpenSky API

[joostlek]

What is your Poling rate? Could it be that you're completely maxing out on the API tokens?

[vingerha]

I donot know how often the integration polls?
For my other sensor which tracks the flight I use 60s, based on max 4000/day
....I think I need to review this now :)

[joostlek]

By default the polling rate is set to 90 seconds, which is updatable using the update entity service

[vingerha]

So that sort-of explains why I am 'over' 4000/d around 7/8 in the morning. I will modify my tracker sensor

[joostlek]

Haha, I am really amazed on how much people want to max out their API limits for OpenSky

[vingerha]

Not maxing out ...just never thought it fully through
I use the airplane iata (input_text) with opensky to get the icao24 in a another input_text
With the second inp.txt I run a command_line to get the lat/lon and present that in a map.
When I am tracking a flight I want it to ne every minute.
Current setup however runs always every 60s, even if I have no icao24 or not interested in tracking, so I need to spin this down :)

[joostlek]

What do you run in that command line? A web request to Opensky?

[joostlek]

Might be interesting to put that in a service call. Hmmm

[vingerha]

command_line: 
  - sensor:
        name: Flight Tracker Opensky
        unique_id: flight_tracker_opensky
        #device_class: timestamp
        scan_interval: 6000
        command: >
            curl -u "user:pwd" -s "https://opensky-network.org/api/states/all?time=0&icao24={{ states('input_text.aircraft_icao') }}" | jq '[.[] ] | .[1] | .[0] | {icao: .[0], iata: .[1], origin_country: .[2], timestamp: .[3], latitude: .[6], longitude: .[5], on_gound: .[8], velocity: .[9] }'   
        value_template: > 
            {{ value_json.timestamp }}        
        json_attributes:
          - icao
          - iata
          - origin_country
          - timestamp
          - latitude
          - longitude
          - on_ground
          - velocity

[vingerha]

The main issue is to link the flight number to icao24, there is no free service that can get that, at least, not that I know of

[joostlek]

But would a service to get a plane by icao24 be useful?

[vingerha]

If it replaces the one above, I guess so. The only thing is that it should not trigger all day long as mine does

[joostlek]

I mean, that's how you configure it, you could make whatever automation with it you want

[joostlek]

Anyway, back to the topic, does the update work?

[vingerha]

Not with me as still stuck with the limit...I will re later ..probably next weekz
EDIT: because travelling and no access to my dev instance via remote

[vingerha]

Got it working but
{{trigger.event.data.icao24 }} does not work
but this does
{{trigger.event.data.callsign }}
I am using that to populate a input_text helper

2023-09-08 13:36:45.306 ERROR (MainThread) [homeassistant.components.automation.message_flight_entry_notification] Error while executing automation automation.message_flight_entry_notification: string value is None for dictionary value @ data['value']

[joostlek]

Open a new issue for this, maybe easier to discuss there :)

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.