Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Met Office API Migration to DataHub #109301

Open
djpugh opened this issue Feb 1, 2024 · 35 comments
Open

Met Office API Migration to DataHub #109301

djpugh opened this issue Feb 1, 2024 · 35 comments
Labels
integration: metoffice

Comments

@djpugh
Copy link

djpugh commented Feb 1, 2024

The problem

The Integration does not support datahub api and datapoint api is being deprecated by April 20th 2024.

There is a MetOffice API migration that creates new credentials and new baseurl - see the docs - this is likely an issue with the upstream datapoint dependency, but this would likely need at least an update to the URL constants set in that library to use the new datahub at a minimum.

At least as of today, you are unable to subscribe to the legacy datapoint API on the metoffice website, so the integration will not work for any new users (as a bonus, the error message in the config flow is not very clear if the integration already has services enabled, and when trying to add the integration without existing services, there is no clear error message/behaviour)

What version of Home Assistant Core has the issue?

core-2024.1.6

What was the last working version of Home Assistant Core?

core-2024.1.6

What type of installation are you running?

Home Assistant OS

Integration causing the issue

metoffice

Link to integration documentation on our website

https://www.home-assistant.io/integrations/metoffice

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response
@home-assistant
Copy link

home-assistant bot commented Feb 1, 2024

Hey there @MrHarcombe, @avee87, mind taking a look at this issue as it has been labeled with an integration (metoffice) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of metoffice can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign metoffice Removes the current integration label and assignees on the issue, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

metoffice documentation
metoffice source
(message by IssueLinks)

@ResteNarquois
Copy link

You're probably aware of this, but just in case:

https://github.com/home-assistant/core/pull/108163

With the integration already being disabled from 2024.2 due to Python incompatibillty, seems like it will need updating to the new DataHub API anyway if it's ever going to get back into HA.

@MrHarcombe
Copy link
Contributor

MrHarcombe commented Feb 2, 2024 via email

@danuals
Copy link

danuals commented Feb 3, 2024

https://pypi.org/project/metoffice-datahub/

@MrHarcombe
Copy link
Contributor

MrHarcombe commented Feb 3, 2024 via email

@danuals
Copy link

danuals commented Feb 3, 2024 •
edited

https://github.com/MetOffice/weather_datahub_utilities

https://pypi.org/project/metoffice-weather-datahub/

any good?

@Squelch
Copy link

Squelch commented Feb 8, 2024 •
edited

After upgrading to core-24.2.0 the integration fails to load with a Python version error.

Logger: homeassistant.setup
Source: setup.py:221
First occurred: 9:29:25 AM (1 occurrences)
Last logged: 9:29:25 AM

Setup failed for 'metoffice': Dependency is disabled - Integration library not compatible with Python 3.12

https://github.com/MetOffice/weather_datahub_utilities

https://pypi.org/project/metoffice-weather-datahub/

any good?

That is the exemplar code, and it looks like a few adaptations need to be made to the existing integration to use the service fully. It's only been live since Nov 2023, and site specific forecasting is still regarded as beta.

I haven't looked too deeply at datapoint-python to see if it can be modified to allow use of DataHub, but it does look like it's gone stale given the low activity and the error above. An open issue concerning versioneer looks to be the problem. [Edit]@EJEP seems to be aware as of 3 hours ago (sorry forgot to tab refresh)]

@MrHarcombe Would a rewrite of the integration using the MetOffice examples be appropriate here, or should datapoint-python be modified?

@Levanterman
Copy link

Levanterman commented Feb 8, 2024 •
edited

image image

Met Office - No Module Named 'datapoint'

image

Stopped working here too, after update HA

@DustyArmstrong
Copy link
Contributor

I've created a PR for that should, if my understanding is correct, hopefully resolve the issue with versioneer that has been acknowledged by the code owner for possible merging or at least to prompt resolution.

I think for longevity's sake however it would be worth pursuing a more modern API wrapper particularly as the old format is due to be deprecated. Happy to help where I can, I'd like to keep the integration in HA.

@EJEP
Copy link

EJEP commented Feb 9, 2024

Hey, I'd sort of forgotten that the datapoint-python API wrapper was being used here. The latest version (0.9.9) should work in python 3.12 now.

I'm going to have a poke at the new Met Office API this weekend to see how much work it would be to use it with the existing code in datapoint-python. Depending on how different the new API is it might end up being easier to write a new integration from scratch though.

@DustyArmstrong
Copy link
Contributor

DustyArmstrong commented Feb 10, 2024 •
edited

Thanks @EJEP that's great news, I was going to do the same to see what it would involve but you're much more familiar with your code than I am. Still happy to assist but doubt there's much more I can offer!

It would be nice if the integration could bundle in its own internal requests to the API and less on 3rd party libraries, not that they aren't effective but just so we aren't leaning on someone else if they have other things going on and can't dedicate the time. Appreciate that would involve a bit of work though.

How do we go about a request for re-enabling the Met Office integration now in HA? Or is it being "disabled" more that it just wouldn't run (but now will). Just whether I'm safe to install 2024.2 yet! EDIT: For anyone else, it stays "not-loaded" with no option to re-enable so I have skipped 2024.2.1 for now.

@MrHarcombe
Copy link
Contributor

MrHarcombe commented Feb 10, 2024 via email

@DustyArmstrong
Copy link
Contributor

Fair enough, I wasn't aware that was a requirement but good to know.

Also, is the integration open for contribution with regards to its other aspects? There are some more features that I'd like to have (pulling out templow/night low) but I'm unsure on the status of development/contributing.

@DustyArmstrong DustyArmstrong mentioned this issue Feb 10, 2024
Merged
20 tasks
@EJEP
Copy link

EJEP commented Feb 10, 2024

I've got an integration with the new DataHub API working on a feature branch (migrate_to_datahub) of the existing datapoint-python repository. There are user-facing changes in the package API that will require changes in HomeAssistant. I had a quick look at the integration code but it wasn't obvious to me how the data from the API is used.

It might be up to a few weeks before I release an update with these changes in as I still need to document the changes and update the rest of the documentation, tests and build infrastructure.

@DustyArmstrong
Copy link
Contributor

Thanks EJEP, massive appreciation for that. I have a local dev copy of the integration now so looking forward to tinkering with the changes for DataHub when it's ready!

First I'm working on this but looks like the integration basically links Home Assistant components (e.g. ATTR_FORECAST_TEMPERATURE) with timestep elements (e.g. timestep.temperature.value) to create relevant entities through weather.py. I guess it will just depend on the changes you've made to the timesteps primarily.

@MrHarcombe
Copy link
Contributor

MrHarcombe commented Feb 10, 2024 via email

@DustyArmstrong
Copy link
Contributor

No problem and understandable, as I say I use the Met Office integration daily, it's the only weather platform I've found in HA that's actually accurate for my area. As accurate as any weather forecast can truly be, at least. I'm happy to contribute if it helps and keeps things going, it's been fun (and a learning experience).

@Hacs2023
Copy link

Hi,
I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.

Source: https://www.metoffice.gov.uk/services/data/datapoint

@MrHarcombe
Copy link
Contributor

MrHarcombe commented Feb 13, 2024 via email

@DustyArmstrong
Copy link
Contributor

Hi, I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.

Good for a little wiggle room! I suppose the main concern will be if new users can no longer sign up for the legacy Datapoint API (though the registration still appears to be open(?), haven't tried though). That in itself is a good driver to migrate to Datahub when able.

I want to have a mess around with Datapoint/hub over the weekend with the alt branch to see how that works and what might change or even need to change for the integration to migrate to it (I also really want to see about adding Night Minimum "Nm"/temp low reading). If EJEP is able and happy to finalise the Datahub branch over the next few weeks we can start working on migrating proper then!

@EJEP
Copy link

EJEP commented Feb 15, 2024

I had definitely got muddled up between a migration of endpoints in DataHub and the retirement of Datapoint! Good that there is more breathing room.

That said, I'm still planning to press on with the migration. I plan to write some docs for what is changing in the migration, but the summary is that the custom objects are mostly replaced with dicts, and the data is no longer split up into 'days'. The main change I foresee to the current HomeAssistant integration is that information is accessed by dict keys rather than object elements.

@Squelch
Copy link

Squelch commented Feb 16, 2024

Hi, I'm sure you are already aware, but have seen the retirement of datapoint is now March 2025, not April 2024 unless I'm not looking at the correct place.

Source: https://www.metoffice.gov.uk/services/data/datapoint

The confusion seems to have come from the banner on this page which then links to
the cutoff date of April 2024 which concerns DataHub API key migration only.

I have also confirmed that registration for DataPoint is still working by creating a new account, which appears to indicate that March 2025 does indeed give more wriggle room.

@pssc pssc mentioned this issue Feb 17, 2024
Open
@djpugh
Copy link
Author

djpugh commented Feb 25, 2024

Thanks for flagging this @Squelch - Like @EJEP I got muddled between them too - I think I've somehow managed to endup in the datahub emails after registering for a datapoint account, and got the associated migration emails (and then the DataHub API migration too), so got confused by the deprecation dates. I have got my API token working again from the datapoint account.

@EJEP, happy to have a look at anything/help out as I have some time on my hands at the moment!

@Squelch
Copy link

Squelch commented Feb 29, 2024

@djpugh Easily done apparently. I'm sure we're not the only ones to have been confused by the similar naming and dates...

@9812ga98sd
Copy link

Are the 3 hourly entities no longer functioning? Mine are all unavailable. Only the daily weather (with a menu to see the hourly) are available.

@jackwilsdon
Copy link
Contributor

Assuming you've just updated to 2024.4, the daily and hourly forecast entities have been merged: https://www.home-assistant.io/blog/2024/04/03/release-20244/#backward-incompatible-changes

@9812ga98sd
Copy link

Assuming you've just updated to 2024.4, the daily and hourly forecast entities have been merged: https://www.home-assistant.io/blog/2024/04/03/release-20244/#backward-incompatible-changes

I used met office 3 hourly forecasts and it was working untilt he lastest upgrade.

met office 3 hourly was the entity.

@9812ga98sd
Copy link

Is there any way to show the hourly forecast in the main card? That would be really useful.

@systanford
Copy link

systanford commented Apr 6, 2024 •
edited

Is there any way to show the hourly forecast in the main card? That would be really useful.

Second this.

Also:
I was referencing the attributes of various 3 hourly forecasts, including temperature, humidity, precipitation probability in a number of templates to drive a number of functions and actions in my instance, which are now unusable.
The data is nolonger shown /shown correctly under attributes.
The 3 hourly entities now just show one value, rather than multiple 3 hour time periods listed under attributes.

@MrHarcombe
Copy link
Contributor

MrHarcombe commented Apr 6, 2024 via email

@gjohansson-ST
Copy link
Member

It seems the original issue has been solved? Is that the right understanding?

@9812ga98sd
Copy link

9812ga98sd commented Apr 6, 2024 •
edited

No the issue has not been solved.

AFAIK the various entities are all still available - I was using the 3-hourly weather in an AppDaemon script and had to change the naming, but it still seems to be working. I think I need to put through a docs change to reflect the changes that were made in 2024.4.1 to bring the daily and 3-hourly under one entity... If you look in the settings, search in entities for "met office" you should still find the same default set of entities listed although some of the names may have slightly changed; for me, the 3-hourly weather is called "sensor.market_harborough_weather" because of my location. I also have similar named enabled entities for _probability_of_precipitation, _uv_index and _wind_speed. If you then check your filters and include the disabled ones you can see where the others are. The daily sensors, however, are all banned slightly differently, as my default is showing as "weather.met_office_market_harborough_daily". But, for me, everything still seems to be there. Hope that helps?
…
--- Ian Harcombe
On Sat, 6 Apr 2024, 01:50 systanford, @.> wrote: Is there any way to show the hourly forecast in the main card? That would be really useful. Second this. I was referencing the attributes of various 3 hourly forecasts in a number of templates to drive a number of functions and actions in my instance, which are now unusable. — Reply to this email directly, view it on GitHub <#109301 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQU6A5SPWWTPMIYGS7TQK3Y35BGTAVCNFSM6AAAAABCUV65DKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBQHAZTGNZQGU . You are receiving this because you were mentioned.Message ID: @.>

For myself I have

image

If i change the "select forecast type" to hourly, it fails with a code error "e is not defined"

All the weather to show options work.

Edit: Its working, this is what I did.

  1. Do not change anything in code editor.
  2. Keep it as weather.met_office_YOURLOCATION_daily
  3. Switch to code editor
  4. Switch back to Visual Editor
  5. Select Hourly. If it comes up with an error, change it back to weather.met_office_YOURLOCATION_daily. You may have edit the entity in the code editor.
  6. Eventually the weather will show up showing every 3 hours.
    Eventually it will work and show the hourly in

@jackwilsdon
Copy link
Contributor

jackwilsdon commented Apr 6, 2024 •
edited

The issue being discussed (the hourly/daily weather being merged) isn't related to the original issue raised here (migrating to a different API). If you're having problems with it, could you please raise a new issue?

@gjohansson-ST
Copy link
Member

I believe my question remains open, is the original issue solved and thereby we should close this issue?
It appears so given you're clearly using it?

Regarding entities or any other, please raise a new issue as already mentioned by @jackwilsdon 👍

@jackwilsdon
Copy link
Contributor

It looks like it's still using the datapoint API, which is deprecated:

core/homeassistant/components/metoffice/manifest.json

Line 9 in 57cc349

"requirements": ["datapoint==0.9.9"]

So I think this issue needs to remain open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
integration: metoffice
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests