Error: Unable to download indicators - worked fine the day before #35

dennis-hammerschmidt · 2020-06-19T08:03:42Z

Hi Vincent,

I have written the following small function to get a few indicators from the WDI dataset:

WB_input <- function (year, countries) {
  dataSeries = c("DT.DOD.DECT.GN.ZS",
                 "DT.DOD.DECT.EX.ZS",
                 "DT.TDS.DECT.EX.ZS")
  data = WDI(
    indicator = dataSeries,
    country = countries,
    start = year,
    end = year
  )
  return(data)
}

and I executed it for a list of countries for 2018 yesterday, all worked fine. Today, I now get the following error message:

Error in 1:nrow(lab) : argument of length 0
In addition: Warning messages:
1: In WDI(indicator = dataSeries, country = countries, start = year,  :
  Unable to download data on countries:  ATA, ESH, GUF, RKS
2: In WDI(indicator = dataSeries, country = countries, start = year,  :
  Unable to download indicators  DT.DOD.DECT.GN.ZS ; DT.DOD.DECT.EX.ZS ; DT.TDS.DECT.EX.ZS

Literally yesterday, all worked fine (except for the missing countries ATA, ESH, GUF, RKS, of course).

My question is if anything has changed in the API settings that you're aware of or that this is a problem that occurs when no connection to the API could be established. I see from your repo that there was no change in your code so I suspect that it might be something on the WDI side.

The text was updated successfully, but these errors were encountered:

FabianFox · 2020-06-19T10:45:47Z

This problem also affects other packages that rely on the API (see: gshs-ornl/wbstats#23). Hence, I also assume that it is an issue with the World Bank API. Hopefully, it can be fixed easily.

dennis-hammerschmidt · 2020-06-19T11:08:42Z

Let's hope for the best 🤞😊 Thanks for your quick reply!

vincentarelbundock · 2020-06-19T11:15:40Z

The world bank website appears to be down or not serving the data anymore. For example, this link is taken straight from their documentation but it doesn't work:

http://api.worldbank.org/v2/country/br?format=json

This happens from time to time. Usually gets fixed pretty quickly.

dscrimager · 2020-06-21T20:38:36Z

The issue is that ALL the api calls now need the /v2/ embedded in them. Many of the api calls in the code still do not have that so they fail. I just did a code pull and verified that when I changed the ones without /v2/ to include it they did work. If those are updated I think all should be fine.

dscrimager · 2020-06-21T21:18:45Z

Ohh except for this one apparently :
http://databank.worldbank.org/data/download/WDI_csv.zip
which does NOT work with the new /v2/ you know just for more non compatibility - grrr...

vincentarelbundock · 2020-06-21T23:57:52Z

Thanks a lot for digging in @dscrimager ! Turns out the api calls you identified without the "v2" weren't culprits. Notice that they are only in the WDIcache function, which still worked. So that couldn't explain the breakage.

I found what I think is a solution and I pushed it to Github, along with a new test suite. The tests pass.

Could @dennis-hammerschmidt, @FabianFox and @dscrimager install the Github master and give it a try? If it works, I'll push to CRAN very very soon.

Thanks!

library(remotes)
install_github('vincentarelbundock/WDI')

library(WDI)

dataSeries = c("DT.DOD.DECT.GN.ZS",
               "DT.DOD.DECT.EX.ZS",
               "DT.TDS.DECT.EX.ZS")
dat <- WDI(indicator = dataSeries, country = 'all', start = 1990, end = 2020)
dat <- na.omit(dat)
head(dat)
#>    iso2c                                     country year DT.DOD.DECT.GN.ZS
#> 61    4E East Asia & Pacific (excluding high income) 1990          35.65028
#> 62    4E East Asia & Pacific (excluding high income) 1991          37.16875
#> 63    4E East Asia & Pacific (excluding high income) 1992          37.17847
#> 64    4E East Asia & Pacific (excluding high income) 1993          37.44126
#> 65    4E East Asia & Pacific (excluding high income) 1994          36.03527
#> 66    4E East Asia & Pacific (excluding high income) 1995          35.03725
#>    DT.DOD.DECT.EX.ZS DT.TDS.DECT.EX.ZS
#> 61          153.4848          18.89601
#> 62          150.2238          17.69950
#> 63          141.9283          16.80542
#> 64          141.6803          17.43847
#> 65          131.9295          14.74167
#> 66          127.6825          14.19330

^{Created on 2020-06-21 by the reprex package (v0.3.0)}

dscrimager · 2020-06-22T01:25:48Z

Weird - I just hammered a /v2/ everywhere I saw a url and then sourced it into my code and it worked. Admittedly, I have a demo tomorrow in front of 50 students that suddenly didn't work so I was a bit frantic to figure out how to get it going any way possible... I didn't grok the code as I should before I shot from the hip. I'll try it now.

dscrimager · 2020-06-22T01:47:32Z

I cannot get it to work - I deleted and re-added the JSON library. When I change this file and source it it works fine. I'm not expert enough with R packaging to know whats going on.
WDI .txt
I did remove the old one and install the new one from the git link as shown above. Tried a couple different ways. But - it could be me...

dat <- WDI(indicator = dataSeries, country = 'all', start = 1990, end = 2020)
Error in 1:nrow(lab) : argument of length 0
In addition: Warning message:
In WDI(indicator = dataSeries, country = "all", start = 1990, end = 2020) :
Unable to download indicators DT.DOD.DECT.GN.ZS ; DT.DOD.DECT.EX.ZS ; DT.TDS.DECT.EX.ZS

vincentarelbundock · 2020-06-22T02:01:41Z

Wow, @dscrimager, that's really weird. I added Travis and Appveyor tests, and everything passes on Windows and Linux (and my own machine is a Mac).

Since you've sourced some files, could you try:

remove.packages('WDI')
Clear workspace: rm(list=ls())
Restart R, making sure that no workspace data is loaded when you come back (often stored in .RData).
Install from Github: remotes::install_github('vincentarelbundock/WDI')
Try to run a few of the WDI calls in this test file to see if they work: https://github.com/vincentarelbundock/WDI/blob/master/tests/testthat/tests-wdi.R

vincentarelbundock · 2020-06-22T02:09:57Z

BTW, I'm pretty sure you didn't install the new version of WDI from Github, because this warning message should have changed.

dscrimager · 2020-06-22T02:48:13Z

OK, confirmed. Not sure why I was not getting the new one - it reported it was installing it. Sorry to have steered you wrong. Anyhow : Ran all tests and they passed on my 3.6.1 !
Thanks very much! It's a nice package to have.

dennis-hammerschmidt · 2020-06-22T11:48:48Z

I've followed the steps above but still get an error when applying it to my own function I mentioned in the beginning. Now it reads:

Unable to download indicator DT.DOD.DECT.GN.ZS
Unable to download indicator DT.DOD.DECT.EX.ZS
Unable to download indicator DT.TDS.DECT.EX.ZS
No indicator could be downloaded. Please check that (a) the country codes are correct, (b) the indicator exists for country and years requested, (c) the indicator name is correctly entered, and (d) the World Bank API servers are online. If you still think your command should have succeeded, please file a support request on Github: https://github.com/vincentarelbundock/WDI

Since this error is different from before, I guess I'm on the most recent version (as it was also installed using remotes::install_github('vincentarelbundock/WDI'))

I've also tried to run your test suite and get the following error for the first testthat

Error: Test failed: 'WDIcache'
* invalid JSON input
Backtrace:
 1. WDI::WDIcache()
 3. RJSONIO::fromJSON(series_url, nullValue = NA)
 5. RJSONIO::fromJSON(...)

When I re-installed WDI from github and re-ran the testthat suite once again, I no longer receive this error and all tests pass. The error above for my own function, however, still persists.

Similar to @dscrimager I'm using R 3.6.1 and am on a Mac OS Catalina (10.15.4).

a) to c) in the error suggestions hold as I've not changed anything in my function to when it was working last week. I'm not sure where I find information on the status of the WDI servers, though.

vincentarelbundock · 2020-06-22T12:12:39Z

@dennis-hammerschmidt That's very weird. It works on my computer:

library(WDI)
dataSeries <- c("DT.DOD.DECT.GN.ZS",
                "DT.DOD.DECT.EX.ZS",
                "DT.TDS.DECT.EX.ZS")
dat <- WDI(indicator = dataSeries, country = 'all', start = 1960, end = 2020)
tail(dat)
#>       iso2c  country year DT.DOD.DECT.GN.ZS DT.DOD.DECT.EX.ZS DT.TDS.DECT.EX.ZS
#> 15970    ZW Zimbabwe 2015          52.16885          224.6833          15.76324
#> 15971    ZW Zimbabwe 2016          59.78537          262.2356          28.83375
#> 15972    ZW Zimbabwe 2017          58.09882          244.9491          14.28684
#> 15973    ZW Zimbabwe 2018          39.80752          238.6148          11.70750
#> 15974    ZW Zimbabwe 2019                NA                NA                NA
#> 15975    ZW Zimbabwe 2020                NA                NA                NA

^{Created on 2020-06-22 by the reprex package (v0.3.0)}

Is this exactly the code you are running?

Also, the WDIcache test still passes on my computer this morning. Could there be something broken about your local RJSONIO installation? I would try this in a completely fresh R session:

library(remotes)
remove.packages(c('RJSONIO', 'WDI'))
install_github('vincentarelbundock/WDI')
library(WDI)
WDI(indicator = 'DT.DOD.DECT.GN.ZS', country = 'all', start = 1960, end = 2020)

dennis-hammerschmidt · 2020-06-22T12:38:49Z

Running your code for country = "all" works fine for me. However, supplying my own list of countries - even after removing those that do not have matches in the WDI database (e.g., Antarctica) - results in the aforementioned error. I've also tried to explicitly convert the country codes to iso2c to make sure there is no problem with that but this didn't help either.

For now, I'm using the work-around of downloading the information for all countries and merging it with my own list of countries which effectively does the same thing as supplying my own list of countries. I'm finalizing a project at the moment where this solution is sufficient but I will take a look at this in more detail toward the end of the week to see if there are some countries beyond the ones that were flagged that might cause the problem.

Until then, thank you so much for your help, I really appreciate it! :)

vincentarelbundock · 2020-06-22T13:25:34Z

Very weird. If you ever find out what the problem is let me know.

For what it's worth, this code works on my machine:

library(WDI)
dataSeries <- c("DT.DOD.DECT.GN.ZS",
                "DT.DOD.DECT.EX.ZS",
                "DT.TDS.DECT.EX.ZS")
countries <- c('YEM', 'DZA', 'USA', 'FRA')
dat <- WDI(indicator = dataSeries, country = countries, 
           start = 1960, end = 2020)

tail(dat)
#>     iso2c     country year DT.DOD.DECT.GN.ZS DT.DOD.DECT.EX.ZS
#> 237    YE Yemen, Rep. 2015          17.68016          386.1902
#> 238    YE Yemen, Rep. 2016          22.85435          738.1637
#> 239    YE Yemen, Rep. 2017          26.85555                NA
#> 240    YE Yemen, Rep. 2018          26.14523                NA
#> 241    YE Yemen, Rep. 2019                NA                NA
#> 242    YE Yemen, Rep. 2020                NA                NA
#>     DT.TDS.DECT.EX.ZS
#> 237          18.94454
#> 238          14.56294
#> 239                NA
#> 240                NA
#> 241                NA
#> 242                NA

unique(dat$country)
#> [1] "Algeria"       "France"        "United States" "Yemen, Rep."

^{Created on 2020-06-22 by the reprex package (v0.3.0)}

stevecondylios · 2020-06-23T16:39:25Z

In case it's of use, I had similar problems over the past week, starting here with the World Bank API's SSL cert expiring. Then a week later, v1 of the API seems to have been decommissioned. Two major events in the space of a week!

To solve for the SSL cert, I gsub'd the whole repo replacing https with http to temporarily solve the certificate issue.

All of the API calls were able to be converted by inserting /v2, and for good measure, appended ?format=json throughout.

vincentarelbundock · 2020-06-23T16:46:57Z

Thanks @stevecondylios

In the end, that's basically what I did to fix the problem.

priceR looks like a cool package. Will definitely give it a try when the opportunity comes.

vincentarelbundock · 2020-06-23T16:48:40Z

A version with the emergency fix is now up on CRAN so I'll close this issue.

In a few days I'll push a new version with more tests, input checks, and informative error messages.

khas11 · 2022-12-23T16:35:52Z

@vincentarelbundock
Hello,

I have a similar issue with WDI. Last time, I was able to fix it by following your recommendation:

remove.packages("WDI")
rm(list=ls())
.rs.restartR()
library(remotes)
remotes::install_github('vincentarelbundock/WDI')
library(WDI)

Today, I was trying again to work on my project and it throws an error again:

In open.connection(con, "rb") :
cannot open URL 'https://api.worldbank.org/v2/en/country/all/indicator/SI.POV.GINI?format=json&date=2009:2018&per_page=32500&page=6': HTTP status was '400 Bad Request'

Is there anything else I can do to fix the code?

vincentarelbundock · 2022-12-23T16:39:44Z

My guess is that the World Bank server is down. Let me know if it still doesn’t work in a couple weeks.

vincentarelbundock closed this as completed Jun 23, 2020

etiennebacher mentioned this issue Jun 29, 2020

Region names became codes #36

Closed

clemdmcl mentioned this issue Mar 30, 2021

These indicators could not be downloaded #46

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: Unable to download indicators - worked fine the day before #35

Error: Unable to download indicators - worked fine the day before #35

dennis-hammerschmidt commented Jun 19, 2020

FabianFox commented Jun 19, 2020

dennis-hammerschmidt commented Jun 19, 2020

vincentarelbundock commented Jun 19, 2020 •

edited

dscrimager commented Jun 21, 2020

dscrimager commented Jun 21, 2020

vincentarelbundock commented Jun 21, 2020

dscrimager commented Jun 22, 2020

dscrimager commented Jun 22, 2020 •

edited

vincentarelbundock commented Jun 22, 2020

vincentarelbundock commented Jun 22, 2020

dscrimager commented Jun 22, 2020

dennis-hammerschmidt commented Jun 22, 2020

vincentarelbundock commented Jun 22, 2020

dennis-hammerschmidt commented Jun 22, 2020

vincentarelbundock commented Jun 22, 2020

stevecondylios commented Jun 23, 2020 •

edited

vincentarelbundock commented Jun 23, 2020

vincentarelbundock commented Jun 23, 2020

khas11 commented Dec 23, 2022

vincentarelbundock commented Dec 23, 2022 via email

Error: Unable to download indicators - worked fine the day before #35

Error: Unable to download indicators - worked fine the day before #35

Comments

dennis-hammerschmidt commented Jun 19, 2020

FabianFox commented Jun 19, 2020

dennis-hammerschmidt commented Jun 19, 2020

vincentarelbundock commented Jun 19, 2020 • edited

dscrimager commented Jun 21, 2020

dscrimager commented Jun 21, 2020

vincentarelbundock commented Jun 21, 2020

dscrimager commented Jun 22, 2020

dscrimager commented Jun 22, 2020 • edited

vincentarelbundock commented Jun 22, 2020

vincentarelbundock commented Jun 22, 2020

dscrimager commented Jun 22, 2020

dennis-hammerschmidt commented Jun 22, 2020

vincentarelbundock commented Jun 22, 2020

dennis-hammerschmidt commented Jun 22, 2020

vincentarelbundock commented Jun 22, 2020

stevecondylios commented Jun 23, 2020 • edited

vincentarelbundock commented Jun 23, 2020

vincentarelbundock commented Jun 23, 2020

khas11 commented Dec 23, 2022

vincentarelbundock commented Dec 23, 2022 via email

vincentarelbundock commented Jun 19, 2020 •

edited

dscrimager commented Jun 22, 2020 •

edited

stevecondylios commented Jun 23, 2020 •

edited