Bug: geocode error, invalid string in json text

[kaijagahm]

When I run geocode(), I get the following error:

Error: lexical error: invalid string in json text.
                                       new
                     (right here) ------^

Note that I'm using the development version of hereR, because my code depends on the fix implemented in #98. If there's a stable released version that I could reference, that would be great, but I think that fix is too new to have made it into a release.

The complete code to generate this error, with a minimal example, is:

remotes::install_github("munterfinger/hereR@develop")
library(hereR)

# Create minimal df to generate the error
errorDF <- structure(list(togeocode = c("edinburgh", "newtonville massachusetts", 
"latham new york", "ponte vedra beach florida", "shanadoah virginia"
), id = 1:5), row.names = c(NA, 5L), class = "data.frame")

hereR::geocode(errorDF$togeocode, sf = FALSE)

Error: lexical error: invalid string in json text.
                                       new
                     (right here) ------^

# There seems to be some problem with the place names. I thought the "new" might refer to "latham new york", so I tried a minimal example that omits the localities containing "new":
noErrorDF <- structure(list(togeocode = c("edinburgh", "ponte vedra beach florida", "shanadoah virginia"
), id = 1:3), row.names = c(NA, 3L), class = "data.frame")

geocode(noErrorDF$togeocode, sf = FALSE)

# this works fine, with no error

Note that I've geocoded these exact same localities before, about two weeks ago, with no error. Not sure if something has been changed in either hereR or the API itself that would cause problems with something related to a json file?

[munterfi]

Hi, thanks for testing and reporting.

The error is due to a bug in the request creation of hereR.

Background: As the Matrix Routing API uses a POST instead of a GET request, the handling of POST requests was added to the function .get_content() (see #99), which sends the requests asynchronously to the API and extracts the response content if the response code is successful ("Status 200"). As .get_content() takes a character array of URLs as input, the POST requests are represented as concatenated strings ("<url> <headers> <request body>") and split into these parts before creating the handles for the request pool. The choice of spaces as delimiter " " was very unfortunate. Therefore the delimiter is now changed to a pipe space combination " | ".

Geocoding of addresses with spaces should work again:

library(hereR)
set_verbose(TRUE)

geocode(errorDF$togeocode, sf = FALSE)
#> Sending 5 request(s) to: 'https://geocode.search.hereapi.com/v1/geocode?...'
#> Received 5 response(s) with total size: 4.8 Kb
#>   id rank                                           address     type
#> 1  1    1                               Edinburgh, Scotland locality
#> 2  2    1                    Newtonville, MA, United States locality
#> 3  3    1                         Latham, NY, United States locality
#> 4  4    1              Ponte Vedra Beach, FL, United States locality
#> 5  5    1 Virginia Ave, Shenandoah, PA 17976, United States   street
#>         street house_number postal_code state_code country_code district
#> 1         <NA>         <NA>       EH2 3       <NA>          GBR     <NA>
#> 2         <NA>         <NA>       02460         MA          USA     <NA>
#> 3         <NA>         <NA>       12110         NY          USA     <NA>
#> 4         <NA>         <NA>       32082         FL          USA     <NA>
#> 5 Virginia Ave         <NA>       17976         PA          USA     <NA>
#>                city     county         state       country lng_access
#> 1         Edinburgh Midlothian          <NA>      Scotland         NA
#> 2       Newtonville  Middlesex Massachusetts United States         NA
#> 3            Latham     Albany      New York United States         NA
#> 4 Ponte Vedra Beach   St Johns       Florida United States         NA
#> 5        Shenandoah Schuylkill  Pennsylvania United States         NA
#>   lat_access lng_position lat_position
#> 1         NA     -3.20144     55.95438
#> 2         NA    -71.20698     42.35058
#> 3         NA    -73.76182     42.74412
#> 4         NA    -81.38653     30.22608
#> 5         NA    -76.20550     40.82896

… and the same for weather requests:

weather(errorDF$togeocode)
#> Sending 5 request(s) to: 'https://weather.ls.hereapi.com/weather/1.0/report.json?...'
#> Received 5 response(s) with total size: 39.5 Kb
#> Simple feature collection with 5 features and 30 fields
#> geometry type:  POINT
#> dimension:      XY
#> bbox:           xmin: -81.38564 ymin: 30.23969 xmax: -3.19648 ymax: 55.95206
#> geographic CRS: WGS 84
#>   id                                  station distance           timestamp
#> 1  1                                Edinburgh        0 2020-12-31 15:20:00
#> 2  2                              Newtonville        0 2020-12-31 09:51:00
#> 3  3                                   Latham        0 2020-12-31 07:51:00
#> 4  4                        Ponte Vedra Beach        0 2020-12-31 09:53:00
#> 5  5 Arrowhead [Virginia Beach (city) County]        0 2020-12-31 10:15:00
#>           state        country daylight                   description skyInfo
#> 1      Scotland United Kingdom        D Light rain. Overcast. Chilly.      18
#> 2 Massachusetts  United States        D    Broken clouds. Quite cool.      15
#> 3      New York  United States        N Light rain. Overcast. Chilly.      18
#> 4       Florida  United States        D                  Sunny. Mild.       1
#> 5      Virginia  United States        D        Light rain. Fog. Cool.      21
#>   temperature comfort highTemperature lowTemperature humidity dewPoint
#> 1        2.00   -0.21            4.30          -1.50       93     1.00
#> 2        6.11    3.76            7.70           0.80       86     3.89
#> 3        2.22   -2.23            1.72           1.11       89     0.61
#> 4       22.78   23.88           22.30          17.60       71    17.22
#> 5       10.00    9.13           14.00           8.70       93     8.89
#>   precipitation1H precipitation3H precipitation6H precipitation12H
#> 1               *               *               *                *
#> 2               *               *               *                *
#> 3               *               *            0.03                *
#> 4               *               *               *                *
#> 5            0.05               *               *                *
#>   precipitation24H airInfo windSpeed windDirection windDescShort
#> 1                *       *      7.41           240            SW
#> 2                *       *     11.12           240            SW
#> 3             0.13       *     18.53           280             W
#> 4                *       *     22.24           170             S
#> 5                *       *      7.41           250             W
#>   barometerPressure barometerTrend visibility snowCover ageMinutes activeAlerts
#> 1           1005.08                         *         *         10            0
#> 2           1016.70                     16.09         *         39           12
#> 3           1015.60         Rising      16.09         *        159            0
#> 4           1020.20                     16.09         *         37            4
#> 5           1022.02                      4.83         *         15            5
#>                     geometry
#> 1  POINT (-3.19648 55.95206)
#> 2  POINT (-71.20367 42.3501)
#> 3 POINT (-73.75901 42.74702)
#> 4 POINT (-81.38564 30.23969)
#> 5   POINT (-76.1872 36.8342)

This solution is still not very ideal, and I am thinking of a better solution: Use an object oriented approach for handling requests and request pools. However, R is not my favorite for object oriented programming (if then I would use the S3 object system) and time is somewhat limited at the moment.

Let me know if this solves the problem for now.

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.3 (2020-10-10)
#>  os       macOS Big Sur 10.16         
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       Europe/Zurich               
#>  date     2020-12-31                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date       lib source        
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.0.0)
#>  callr         3.5.1      2020-10-13 [1] CRAN (R 4.0.2)
#>  class         7.3-17     2020-04-26 [1] CRAN (R 4.0.3)
#>  classInt      0.4-3      2020-04-07 [1] CRAN (R 4.0.0)
#>  cli           2.2.0      2020-11-20 [1] CRAN (R 4.0.2)
#>  crayon        1.3.4      2017-09-16 [1] CRAN (R 4.0.0)
#>  curl          4.3        2019-12-02 [1] CRAN (R 4.0.1)
#>  data.table    1.13.2     2020-10-19 [1] CRAN (R 4.0.3)
#>  DBI           1.1.0      2019-12-15 [1] CRAN (R 4.0.2)
#>  desc          1.2.0      2018-05-01 [1] CRAN (R 4.0.0)
#>  devtools      2.3.2      2020-09-18 [1] CRAN (R 4.0.2)
#>  digest        0.6.27     2020-10-24 [1] CRAN (R 4.0.2)
#>  dplyr         1.0.2      2020-08-18 [1] CRAN (R 4.0.2)
#>  e1071         1.7-4      2020-10-14 [1] CRAN (R 4.0.2)
#>  ellipsis      0.3.1      2020-05-15 [1] CRAN (R 4.0.0)
#>  evaluate      0.14       2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.4.1      2020-01-08 [1] CRAN (R 4.0.0)
#>  fs            1.5.0      2020-07-31 [1] CRAN (R 4.0.2)
#>  generics      0.1.0      2020-10-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.2      2020-08-27 [1] CRAN (R 4.0.2)
#>  hereR       * 0.5.2.9000 2020-12-31 [1] local         
#>  htmltools     0.5.0      2020-06-16 [1] CRAN (R 4.0.2)
#>  jsonlite      1.7.1      2020-09-07 [1] CRAN (R 4.0.2)
#>  KernSmooth    2.23-17    2020-04-26 [1] CRAN (R 4.0.3)
#>  knitr         1.30       2020-09-22 [1] CRAN (R 4.0.2)
#>  lifecycle     0.2.0      2020-03-06 [1] CRAN (R 4.0.0)
#>  magrittr      2.0.1      2020-11-17 [1] CRAN (R 4.0.2)
#>  memoise       1.1.0      2017-04-21 [1] CRAN (R 4.0.0)
#>  pillar        1.4.7      2020-11-20 [1] CRAN (R 4.0.2)
#>  pkgbuild      1.1.0      2020-07-13 [1] CRAN (R 4.0.2)
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.0.0)
#>  pkgload       1.1.0      2020-05-29 [1] CRAN (R 4.0.2)
#>  prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.0.0)
#>  processx      3.4.4      2020-09-03 [1] CRAN (R 4.0.2)
#>  ps            1.4.0      2020-10-07 [1] CRAN (R 4.0.2)
#>  purrr         0.3.4      2020-04-17 [1] CRAN (R 4.0.0)
#>  R6            2.5.0      2020-10-28 [1] CRAN (R 4.0.2)
#>  Rcpp          1.0.5      2020-07-06 [1] CRAN (R 4.0.2)
#>  remotes       2.2.0      2020-07-21 [1] CRAN (R 4.0.2)
#>  rlang         0.4.9      2020-11-26 [1] CRAN (R 4.0.3)
#>  rmarkdown     2.5        2020-10-21 [1] CRAN (R 4.0.2)
#>  rprojroot     2.0.2      2020-11-15 [1] CRAN (R 4.0.2)
#>  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 4.0.0)
#>  sf            0.9-6      2020-09-13 [1] CRAN (R 4.0.2)
#>  stringi       1.5.3      2020-09-09 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 4.0.2)
#>  testthat      3.0.0      2020-10-31 [1] CRAN (R 4.0.2)
#>  tibble        3.0.4      2020-10-12 [1] CRAN (R 4.0.2)
#>  tidyselect    1.1.0      2020-05-11 [1] CRAN (R 4.0.0)
#>  units         0.6-7      2020-06-13 [1] CRAN (R 4.0.2)
#>  usethis       1.6.3      2020-09-17 [1] CRAN (R 4.0.2)
#>  vctrs         0.3.5      2020-11-17 [1] CRAN (R 4.0.2)
#>  withr         2.3.0      2020-09-22 [1] CRAN (R 4.0.2)
#>  xfun          0.19       2020-10-30 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1      2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

[kaijagahm]

This does fix the problem for now! Thanks once again for the quick response and solution. I really appreciate the work you're doing on this package :)