Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Functions don't work right now! The error message is: There seems to be no data in the URL. there must be something wrong with the website of wunderground. #39

Open
ThaliaW opened this issue Sep 8, 2017 · 24 comments
Open

Functions don't work right now! The error message is: There seems to be no data in the URL. there must be something wrong with the website of wunderground. #39

ThaliaW opened this issue Sep 8, 2017 · 24 comments

Comments

@ThaliaW
Copy link

ThaliaW commented Sep 8, 2017

No description provided.

@ThaliaW
Copy link
Author

ThaliaW commented Sep 8, 2017

there must be something wrong with the website of wunderground. I used to use the URL: https://www.wunderground.com/history/airport/SEA/2014/4/1/CustomHistory.html?format=1 to get historical data. It just doesn't work for me right now and it is also the case with this package.

@CraigMohn
Copy link

CraigMohn commented Sep 8, 2017 •
edited
Loading

This also happened a few months ago, and then wunderground started returning csv data again after a little less than a day. This recent outage started yesterday.

I think your URL, apparently taken from the vignette, needs to have the airport as KSEA rather than SEA. I get results in html format if I drop the ?format=1 part, but that isn't helpful without altering the weatherData code to extract the results from the html.

https://www.wunderground.com/history/airport/KSEA/2017/9/8/DailyHistory.html works.
https://www.wunderground.com/history/airport/KSEA/2017/9/8/DailyHistory.html?format=1 doesn't, at least not right now.

@venkatsriram
Copy link

Same here. It was working two days but it stopped working since yesterday. I get the below error message while using both checkDataAvailability and getDetailedWeather functions
Unable to get data from URL
Check if URL is reachable

@CraigMohn
Copy link

It appears that wunderground is back up. I got a free key to use the wunderground api interface last night, but did not get a chance to check if that interface was returning data during the csv outage or not.

@shumakerro
Copy link

Seems to have been fixed -- at least for the getWeatherForDate function.

@quasistatic89
Copy link

It still doesn't work for me. Any steps to try and get this to work ?

@ThaliaW
Copy link
Author

ThaliaW commented Sep 11, 2017

library(httr)

url = ("https://www.wunderground.com/history/airport/KRNT/2017/1/1/MonthlyHistory.html?")
#url = ("https://www.wunderground.com/history/airport/KRNT/2017/2/1/MonthlyHistory.html?")
#url = ("https://www.wunderground.com/history/airport/KKLS/2017/3/1/MonthlyHistory.html?")

req = GET(url)
dat = content(req, "text")

myStr = strsplit(dat,"Daily Weather History & Observations")[[1]][2]
myStr = strsplit(myStr,"background-color: #FFF")[[1]][2]
myStr = strsplit(myStr,"script type=")[[1]][1]
myStr = gsub("\n|\r|\t|||||","",myStr)
myStr = gsub("Rain,|Snow,|Fog,|Thunderstorm,","",myStr)
myStr = gsub("Rain|Snow|Fog|Thunderstorm| ","",myStr)
myStr = gsub('|',"",myStr)
myStr = gsub("<.+?>", ",", myStr)
myStr = substring(myStr,6,nchar(myStr)-6)
myStr = gsub(",,,","\n",myStr)

t <- read.csv(textConnection(myStr), header = F)
colnames(t) = c("ForecastDay","Empty","TempHigh","TempAvg","TempLow","DewHigh","DewAvg","DewLow","HumidHigh","HumidAvg","HumidLow","SeaLevelHigh","SeaLevelAvg","SeaLevelLow","VisibilityHigh","VisibilityAvg","VisibilityLow","WindHigh","WindAvg","WindLow","Precip")

@ThaliaW ThaliaW closed this as completed Sep 11, 2017
@ThaliaW ThaliaW reopened this Sep 11, 2017
@ThaliaW
Copy link
Author

ThaliaW commented Sep 11, 2017

Try codes above. It works for me.

@quasistatic89
Copy link

The script still doesn't work for me. myStr returns NA after the first strsplit. Any ideas to get the getWeatherForDate function working. WIll be much smoother. Thanks for your inputs.

@jienagu
Copy link

jienagu commented Sep 12, 2017

The code is not working, myStr returns NA too! I am wondering if Wunderground page has some updates but we are still waiting for "weatherData" to update?

Any ideas? Thank you!

@mattyjkelly
Copy link

Hi,
I have the same problem if i try and pull data from R using either the examples or httr as described above by ThaliaW. I can copy the link into a web browser and see the data so I think the link is correct.

Do I need to register for an API key to use this package?

Regards,
Matt

@CraigMohn
Copy link

I do not believe that weatherData uses the API interface to wunderground. I have been able to get JSON formatted historical data from wunderground all day using the API key, but the site still does not return data in the csv format that weatherData requires (the examples ending in ?format=1 test this). The API interface has a daily limit of 500 single-date queries at a rate not exceeding 10 per minute for the free key. Hopefully this is temporary, perhaps related to recent US weather events driving up query volumes for wunderground.

@SSabas
Copy link

SSabas commented Sep 13, 2017 •
edited
Loading

The website (wunderground.com) hasn't been down, they have just changed the URL directory structure that the R package weatherData relied on. I am yet to fully understand what has been done and thus the problem still has to be resolved (and neither I could get the httr based solution working).
If you would like to retrieve data directly from wunderground.com, then you can use purpose built R package rwunderground, which is the R interface to Weather Underground API. The key for the API can be accessed by creating an account on their website (and is free for limited number of downloads). NB! To note the packages do not have the same functionality.

@CraigMohn
Copy link

If anyone is using rwunderground or directly using the API key, it is useful to know that wunderground resets the quota at midnight US Eastern Time. There is a limit on the number of times you can go over the daily limit, you start with 3 "raindrop" credits to excuse exceeding the limits and avoid denial of service. The credits allow some amount in excess of the limits, my single data point says that 606 requests paced at 10 per minute on a 500 request limit resulted in the loss of one of these "raindrop" credits.

@Ram-N
Copy link
Owner

Ram-N commented Sep 13, 2017

All, yes looks like WU is no longer making it easy to get CSV files without API's. If anyone figures out a URL for directly fetching CSV's, I will modify the package. But until then, we are stuck.

@brunolucian
Copy link

This other package works for me!
https://github.com/ALShum/rwunderground

@aliceicecream
Copy link

Hi everyone,
So weatherdata package is really down?
I've been unable to extract weather data history for 2 weeks now, and I'm working in ice cream business, so it matters a lot to explain sales gaps etc...
Just to know if I have to get into this API's thing and figure out how to use a new package (weatherdata was sooo nice and easy to use!)

Thanks

@CraigMohn
Copy link

CraigMohn commented Sep 26, 2017 •
edited
Loading

The wunderground.com interface used by weatherdata appears to be defunct. I have written a few routines to retrieve and store (in .rda files) historical data results. These routines are a wrapper for rwunderground functions. Unfortunately, rwunderground historical data fetches fail when all observations have missing visibility data (true for many personal weather stations) and for date/station combinations where there is no data at all. I discovered this yesterday, and wrote a quick patch for the code.
I also calculate a bit of daily summary data you probably don't need.
You need to get an API key (free up to 500 inquiries per day, less than 10 per minute).
The wrapper code is in my project wunderWrapper.
My updated rwunderground is in a forked version of ALShum's cran package, but it doesn't appear to be appropriately visible on github, I will fix that this evening, when I get home.

@andypicke
Copy link

andypicke commented Sep 26, 2017 via email

@tybyers
Copy link

tybyers commented Sep 26, 2017

@aliceicecream : hey considering your business depends on weather data, you might want to check out Dark Sky and the R darksky package: https://github.com/hrbrmstr/darksky. Dark Sky can have its drawbacks, especially since the weather observations aren't "official," rather they are aggregated across many different weather stations. But the API is pretty easy to use, and you can do 1,000 free calls a day.

@andypicke funny to run into you here :). Cool project, thanks for posting!

@drnmy
Copy link

drnmy commented Oct 1, 2017

Since it was not working for me I wrote a small web page scraping function in R:

getTempData <- function(code, year) {

url <- sprintf("https://www.wunderground.com/history/airport/%s/%d/1/1/CustomHistory.html?dayend=31&monthend=12&yearend=%d&req_city=NA&req_state=NA&req_statename=NA", code, year, year)

print(url)

webpage <- getURL(url)

webpage <- readLines(tc <- textConnection(webpage)); close(tc)

pagetree <- htmlTreeParse(webpage, error=function(...){}, useInternalNodes = TRUE)

parse the tree by tables

tablehead <- xpathApply(pagetree, "//table[contains(@id,'obsTable')]/thead/tr/th", xmlValue)

results <- xpathApply(pagetree, "//table[contains(@id,'obsTable')]/tbody/tr/td", xmlValue)

results = gsub("\n", "", results)

Convert character vector to dataframe

content <- as.data.frame(matrix(results, ncol = 21, byrow = TRUE))

colnames(content) <- c("Month", "high", "avg", "low", "high", "avg", "low", "high", "avg", "low", "high", "avg", "low", "high", "avg", "low", "high", "avg", "high", "sum", "evt")

return(content)

}

Far from being perfect but its a good 5 minute start :)

@Ram-N
Copy link
Owner

Ram-N commented Oct 18, 2017

I recently learned about the following sites:

https://mesonet.agron.iastate.edu/request/download.phtml
https://mesonet.agron.iastate.edu/request/daily.phtml

I will work on redirecting WeatherData to this site, when I can.

@pssguy pssguy mentioned this issue Oct 31, 2017
Open
@alixunderplatz
Copy link

alixunderplatz commented Dec 11, 2017 •
edited
Loading

@Ram-N Hello! I might have a solution to this issue.

I used to write a little R-script myself to get weather data from WU stations. That was about a year ago. This is the URL-structure I used to obtain the desired data:

https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IFREESTA28&day=11&month=12&year=2017&graphspan=day&format=1

Using this URL-structure, you will get the raw underlying data of the measured weather components as comma seperated values:

Time,TemperatureC,DewpointC,PressurehPa,WindDirection,WindDirectionDegrees,WindSpeedKMH,WindSpeedGustKMH,Humidity,HourlyPrecipMM,Conditions,Clouds,dailyrainMM,SolarRadiationWatts/m^2,SoftwareType,DateUTC
2017-12-11 00:00:00,10.4,9.8,1012.8,Nord,356,2.9,4.0,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:00:00,
2017-12-11 00:05:00,10.5,9.9,1013.1,Ost,97,14.3,19.8,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:05:00,
2017-12-11 00:10:00,10.5,9.9,1013.1,Ost-Nordost,62,0.6,4.0,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:10:00,
2017-12-11 00:15:00,10.5,9.9,1013.1,Ost-Südost,119,6.4,11.9,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:15:00,
2017-12-11 00:20:00,10.5,9.9,1013.1,Nordost,54,3.2,7.9,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:20:00,
2017-12-11 00:25:00,10.6,10.0,1013.1,Ost-Nordost,62,11.9,15.8,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:25:00,
2017-12-11 00:30:00,10.5,9.9,1013.1,Ost,100,4.3,7.9,96,0.0,,,0.0,0.00,HP1000 V1.0.9,2017-12-10 22:30:00,

As far as I can remember I found this link using excel to acess the data table on the desired station-site. Not sure if this works for airports, but definitely for personal weather stations.

Maybe this may help you to solve the issue!
Greets, Alex

@DanRuderman
Copy link

@Ram-N Thanks for your continued work on this package. Might be good to put some kind of note on the CRAN site for the package saying that it's currently not working (I didn't see one there).

BTW another option might be the NOAA ftp site: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests