Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP error 502 when downloading Vietnam's json data #426

Closed
biocyberman opened this issue Sep 28, 2021 · 6 comments · Fixed by #427 or #428
Closed

HTTP error 502 when downloading Vietnam's json data #426

biocyberman opened this issue Sep 28, 2021 · 6 comments · Fixed by #427 or #428
Labels
bug Something isn't working data-issue An issue with one of the supported datasets.

Comments

@biocyberman
Copy link
Contributor

I am investigating the error. It seems that the return data is not fully validated JSON. Data can be viewed directly via browser but failed when downloaded with fromJSON. For example:

library(purrr)
library(tidyr)
library(lubridate)
library(stringr)
library(stringi)
library(jsonlite)
data_urls = list(
      "provinces" = "https://covid.ncsc.gov.vn/api/v3/covid/provinces",
      "case_by_time" = "https://covid.ncsc.gov.vn/api/v3/covid/provinces?filter_type=case_by_time"
    )
json_reader <- function(file, verbose = FALSE, ...) {
        if (verbose) {
          message("Downloading data from ", file)
          data <- fromJSON(file, ...)
        } else {
          data <- suppressWarnings(
            suppressMessages(
              fromJSON(file, simplifyDataFrame = TRUE, flatten = TRUE)
            )
          )
        }
        return(tibble(data))
      } 
data = map(data_urls, json_reader)

Gave: Error in open.connection(con, "rb") : HTTP error 502.

@github-actions
Copy link

Thanks for opening an issue! We'll try and get back to you shortly. If you've identified an issue and would like to fix it please see our contribution guidelines.

@RichardMN
Copy link
Collaborator

Looking at it and having tried a few variant URLs I think this may be a glitch upstream on the server end. I get 502 for just about any URL going to the API directory, though not to the top level one, which could be consistent with the API urls being redirected to a specific server or port and that port failing. (E.g., https://bobcares.com/blog/502-bad-gateway-nginx/) I think we may wait twelve hours and see. (Though when this happened with Germany I think I wrote my first implementation of a JSON downloader and had it running roughly in time for the problem to fix itself upstream.)

@RichardMN RichardMN added bug Something isn't working data-issue An issue with one of the supported datasets. labels Sep 28, 2021
@biocyberman
Copy link
Contributor Author

I reported to NCSC colleages.

@seabbs
Copy link
Contributor

seabbs commented Sep 29, 2021

Looks like this may have cleared up on the latest automated check so sounds like it was server side.

@biocyberman
Copy link
Contributor Author

The problem is gone even though I haven't got a reply from NCSC :)

@trilwu
Copy link
Contributor

trilwu commented Sep 30, 2021

Hello from NCSC. It is an honor for us to have this data contributed to your project.

The stable domain is covid19.ncsc.gov.vn, the old URL: covid.ncsc.gov.vn (beta) is now deprecated. We have reopened for a few days so you can update the new URL. I made a pull request here #427

This was linked to pull requests Sep 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working data-issue An issue with one of the supported datasets.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants