Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

fetch weather once every hour max #2177

Closed
wants to merge 1 commit into from

8 participants

@ronie
Collaborator

@theuni mentioned on irc weather underground wasn't too thrilled with all the load we send their way.
and if i'm not mistaken, we're in the same boat with world weather online as well.

in an effort to keep them happy i suggest we switch to hourly weather retrieval
instead of every 30 minutes.

this could (theoretically) reduce the load by half.

@MartijnKaijser

Personally I would go even further and retrieve it even less (once in 2 hours or more) . How often is data updated at providers side? If that's even less we should use their update interval (if it's bigger of course).

@ronie
Collaborator

might depend on your location, but for me it's updated twice every hour.
regardless, since we provide hourly forecasts, we should at least update once every hour.

the 'current conditions' won't be accurate either anymore if switch to 2/4 hour updates.

the fact confluence doesn't show a weather icon anymore by default on the homescreen in frodo
should already have a big influence on reducing the number of api calls we make.
if we add this pr on top, it should get us back in the save zone i hope.

@jmarshallnz
Owner

AFAIAA the first call goes out at first view of the weather data which means the first call to the infomanager (any of the weather labels or bools). It then updates at most every 30 minutes, and only if another call goes via the infomanager.

i.e. you'll get updates every 30 minutes only if there's persistent calls into the infomanager from a control. Note that you may get that call even if the control is invisible - this needs checking. It's not clear to me whether the default skins (confluence/touched) still induces calls into the weather add-on or not outside of the weather window. Depending on how it's done it may be that there is some control awaiting visibility on Weather.IsFetched, or a hidden label which still updates it's content. Someone (hint, hint) needs to test whether this is actually the case or not.

The next thing to check is whether there's a cheap API call to get whether an update is available, rather than just pulling down all the data every 30mins. Is there also a way to pull down some minimal data (e.g. just current info without forecast?) that might be used for weather widgets in most skins?

IMO increasing the time between updates should be the last resort. Most regions receive updates twice an hour, so that should really be our update time.

@ronie
Collaborator

this someone already tested the shit out of it ;-)
albeit using reverse engineering, as i suck at reading c++ code.

in frodo, confluence will not trigger weather retrieval outside of the actual weather window.
invisible controls don't trigger weather retrieval unless you'd use Weather.IsFetched as a visible condition.

the problem with world weather online is the number of api calls we make,
so using 'cheap' calls won't help us there.

as for weather underground, i have no idea if it's the number of calls or the amount of data we fetch.
^ @theuni ?

a few other suggestion to reduce the load on our weather providers:

  • call out to our skinners to follow confluence and make all weather labels/icons outside of the weather window optional (and disabled by default)
  • i haven't tested this one, but i think we still fetch weather when you're on the weather window and the black or dim screensavers are active
@bossanova808

For my addon (OzWeather) - this pulls live radar images - having it 1 or 2 hours out of date will suck. Any chance some thought could be put in to making this variable depending on where the data is coming from? I.e. maybe the addon.xml could specify a frequency or something (that's a daft idea but you get my drift...).

@MartijnKaijser

If you want live weather look outside. Sorry but having it once every 30 minutes is plain silly and definitely hogging on our kind weather providers.

@bossanova808

It's not plain silly - and not all weather providers are those ones....and radar data is not about 'look outside' - it's about watching what will come in an hour or two, how big an event it is etc. And people often like to check the weather outside at night when it's not so easy to see....I for one love to know just how big a storm is coming, and looking out my pitch black window doesn't tell me that.

You might not have noticed this but weather apps & devices are hugely popular in general. I don't disagree with reducing bandwidth for those generous free providers, of course, but if you go down this path I think that there needs to be a good way to sub in a whole weather add on (e.g. I could write a full script instead of just a weather add on - but then all the users will complain it doesn't integrate nicely).

If you choose to have weather at all, then it might as well be good weather...which to most means frequently updated IF this can be achieved in a way that doesn't cause issues for upstream providers...and if those upstream providers ARE willing, then why limit it in the general case?

I am sure with some thought a good balance can be found...

@MartijnKaijser

We are talking about a media center here not some personal full blown high update 24 hours a day weather station. Once an hour is acurate enough. Want better data check their website. We talking about reducing millions of API calls here. Sorry but not all can agree here. If you want to further discus lets do that on the forum. If other ways are to large of code change i say that we at least do this for 12.1 release.

@Voyager1
Collaborator

IMO it's weird that the fetch interval is hard coded. Wouldn't it make more sense to have the default value set and even configurable in the specific weather plugin itself? Different services (e.g. OzWeather vs. WU) justify having different update intervals...

@bossanova808

"We are talking about a media center here " ... so what? Why should this limit what it can do. The major sections are Weather, Pictures, Videos (etc) and Music. For some of us, Weather it an important feature. 'Check their website' - sure - and next time you want to watch a video, go to YouTube in a browser, by the same argument.

Happy to go to forum - is there a thread yet?

What Voyager said sounds the most sensible...

@da-anda
Collaborator

IMO full weather data should only be fetched when in one of the weather windows or explicitly requested. It doesn't make sense to grab the full forecast when skins only display a little icon. So how about doing light requests by default and trigger full requests when entering the weather window and the info is older then X minutes? This could be combined with a "updateInterval" variable provided by the addons for both, light and full requests.

@ronie
Collaborator

@bossanova808 yeah, i agree. whatever problem we have with wu and the likes should not affect addons for other weather providers.

@Voyager1 moving the update interval to the addon makes it kind of user-configurable. is this something we should worry about? or should we just take the few that may change the update interval 'once a minute' for granted?

@da-anda several skins indeed only show a current conditions icon on home, but there's also quite a few that display a 'mini' 7 day forecast on the homescreen as well.

i think we need to get a clear answer from wu first, if it's the amount of data or the number of api calls we make that's bothering them.

@amet any thoughts on what would work for world weather online?

@HitcherUK

Out of curiosity, is the update on every hour and half hour or just 30 minutes from when it was started?

@Voyager1
Collaborator

@ronie - @da-anda - very good points. I like the idea of splitting the updates between current weather conditions (temp / rain-sun-etc.) and full forecast. There should be really two intervals. In terms of making the intervals configurable, we should have a default value that makes both our providers happy as well as work for 80%+ of the users. For current conditions that could be 30 mins, and for full forecast only every 4 hours or so would be sufficient. And make these intervals to be configurable via the addon-configuration screen.
To avoid aberrations you could have a minimum (no less than 15 minutes e.g.).

@amet

@ronie I am not sure, we are over 5 times what they allow us to use, so 30 min to 60 min is not going to change much.I am ok to remove it and make WU its replacement. I see no reason to have both

@HitcherUK

@amet Surely removing one altogether would increase the load on the remaining one?

Splitting current and full forecasts sounds like a good compromise to me.

@amet

@HitcherUK true that

@MartijnKaijser

Also keep in mind that if user base starts growing more the API calls are increased even more. We need to make a good way too make a lot of headroom available to allow us to grow without possibly loosing our weather provider like we did on Dharma.

@HitcherUK

Bear in mind I have no idea how this all works but couldn't the data be fetched and then stored on mirrors; then couldn't the script could fetch the info from these mirrors?

@amet

caching a response might be an option so someone else could reuse it in the next 30 min, that would however only work on major cities and regions that get a lot of hits, not sure they would allow us to do that

@da-anda
Collaborator

I guess some sort of push updates are technically not possible?

@jmarshallnz
Owner

The best thing is to contact the service providers and find out how we can best work with them to reduce load. It seems to me that there's very little data that we actually pull down every 30mins (a few kilobytes, right?) but there may well be something they have available to minimize load anyway which we can take advantage of.

In the meantime, moving the update time to the addon xml seems like a reasonable thing to do, as this will likely be per-service anyway.

@MartijnKaijser

Did someone contact our weather providers on this what they prefer?

@MartijnKaijser

@ronie is this one still needed since we won't be shipping WU by default anymore (probably)?

@ronie
Collaborator

nah, let's close it.

@ronie ronie closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 3, 2013
  1. @ronie
This page is out of date. Refresh to see the latest.
Showing with 1 addition and 1 deletion.
  1. +1 −1  xbmc/utils/Weather.cpp
View
2  xbmc/utils/Weather.cpp
@@ -384,7 +384,7 @@ void CWeatherJob::SetFromProperties()
}
}
-CWeather::CWeather(void) : CInfoLoader(30 * 60 * 1000) // 30 minutes
+CWeather::CWeather(void) : CInfoLoader(60 * 60 * 1000) // 60 minutes
{
Reset();
}
Something went wrong with that request. Please try again.