Extra brackets for nested array

[patperu]

'sofa' seems to write extra brackets for nested arrays. I'm note sure, but I think they are not necessary.

library('sofa')
library('jsonlite')

options(stringsAsFactors = FALSE)

# Example from jsonlite / json-mapping.pdf
x <- data.frame(author = c("Homer", "Virgil", "Jeroen"))
x$poems <- list(c("Iliad", "Odyssey"), c("Eclogues", "Georgics", "Aeneid"), vector());
names(x)

toJSON(x[1, ], pretty = TRUE)

# [
#    {
#        "author": "Homer",
#        "poems": [
#            "Iliad",
#            "Odyssey"
#        ]
#    }
# ]

# local CouchDB running?
# yes, WIN64
ping()
# $version
# [1] "1.6.1"

db_create(dbname = 'test')

for (i in 1:nrow(x)) {
  doc_create(dbname = 'test', doc = as.list(x[i,]))
}

# Copy via Futon

# {
#   "_id": "f0b286aebef7887771711ad921004cbc",
#   "_rev": "1-200f4b3b4640a536047b7147bb409206",
#   "author": "Homer",
#   "poems": [
#       [              <---------- extra bracket
#           "Iliad",
#           "Odyssey"
#       ]              <---------- extra bracket
#   ]
# }

# POST via 'curl'
curl -X GET http://127.0.0.1:5984/test/f0b286aebef7887771711ad92100adc1
{"_id":"f0b286aebef7887771711ad92100adc1",
    "_rev":"1-38e34a2b327b4c26458efa6b81fca866",
        "author":"Virgil",
           "poems":["Eclogues","Georgics","Aeneid"]}

# POST via 'sofa'
curl -X GET http://127.0.0.1:5984/test/f0b286aebef7887771711ad921008d1a
{"_id":"f0b286aebef7887771711ad921008d1a",
     "_rev":"1-6ced453d63a2535b44e124bdb3bb01d7",
         "author":"Virgil",
            "poems":[["Eclogues","Georgics","Aeneid"]]}

[sckott]

Thanks @patperu

But I think there's a difference here between the two things, that is:

toJSON(df[1, ], pretty = TRUE) is taking a row of a data.frame, whereas as.list(df[i,]) is converting to

$author
[1] "Homer"

$poems
$poems[[1]]
[1] "Iliad"   "Odyssey"

note that $poems is a list so that

toJSON(as.list(df[1,])$poems)
#> [["Iliad","Odyssey"]] 

which i think is that double bracketed array you're seeing

So you can just pass in JSON if you want for now

I plan on making it easy to simply pass in a data.frame and then internally we'd either make each row a document, each column, or the whole thing as one document, see #27

[patperu]

Thanks @sckott
Now I see the difference. I need to understand this JSON / list thing better.
For now I use 'sapply' to remove the brackets. My working example
https://github.com/patperu/fp7-funding-network-analysis/blob/7cd3cf5b38/R/write2CouchDB.R
Thanks again!

[sckott]

@patperu Reinstall the latest from github, and try directly from a data.frame to create documents, see examples https://github.com/sckott/sofa/blob/master/R/doc_create.r#L58-L66

Sorry, but I rearranged parameters in that fxn a bit so that the first is the things passed into the fxn to create documents, so the function can easily behave differently for a json character string, an R list, or a data.frame

See also the new bulk function bulk_create(), examples: https://github.com/sckott/sofa/blob/master/R/bulk_create.R#L16-L21

Or would you rather not create a separate document for each row or column? Is there a different use case you have?

[patperu]

Wow, that's great!!
So, with
library('sofa') e9808c4
"rows" and "columns" work perfect!

   doc_create(x, dbname = 'fp7_rows', how = "rows")
   doc_create(x, dbname = 'fp7_cols', how = "columns") 

but with "bulk_create" I get an error

    # bulk
    db_create(dbname = 'fp7_bulk')
    bulk_create(x, dbname="fp7_bulk")
    # Error: (500) - function_clause 

Gist: https://gist.github.com/patperu/4f92213f39877b900b90

And my use case is actually only the "row" variante.

[sckott]

bummer. what does that data.frame look like, can you paste in what is in str(x)

[patperu]

Yup

str(x)
'data.frame':   1 obs. of  20 variables:
 $ rcn                 : num 108737
 $ reference           : num 609698
 $ acronym             : chr "RENATECH"
 $ programmePga        : chr "FP7-PEOPLE"
 $ programmeCode       : chr "FP7-PEOPLE"
 $ subprogramme        : chr "FP7-PEOPLE-2013-NIGHT"
 $ title               : chr "REsearchers for NAture preservation and TECHnology use for the benefit of the society"
 $ startDate           : chr "2013-05-01 00:00:00"
 $ endDate             : chr "2013-10-31 00:00:00"
 $ projectUrl          : chr NA
 $ totalCost           : num 50616
 $ ecMaxContribution   : num 49997
 $ call                : chr "FP7-PEOPLE-2013-NIGHT"
 $ fundingScheme       : chr "CSA-SA"
 $ coordinator         : chr "INSTITUT JOZEF STEFAN"
 $ organisationId      : num 9113619
 $ coordinatorCountry  : chr "SI"
 $ participants        :List of 1
  ..$ : chr  "FAKULTETA ZA INFORMACIJSKE 'TUDIJE V NOVEM MESTU" "NACIONALNI INSTITUT ZA BIOLOGIJO"
 $ participantCountries:List of 1
  ..$ : chr "SI"
 $ subjects            :List of 1
  ..$ : chr "LIF"

[patperu]

@sckott ,
I get the error also when running the 'bulk_create' example with 'mtcars'.
'iris' is fine.

[sckott]

@patperu i think it may be lists within the data.frame (e.g., column participants) that's causing the problem

p.s. i just pushed up changes to add list and character methods for bulk_create() see examples

[sckott]

@patperu can paste in the output of dput(x) here? I tried to recreate that data.frame locally, and I think I did so, and it worked for me, so I'm not sure if I have a diff. data.frame than you or do you have the latest changes form this repo?

[patperu]

Hi @sckott No, it's still not working. Here is the df structure:

structure(list(rcn = 108737, reference = 609698, acronym = "RENATECH", 
    programmePga = "FP7-PEOPLE", programmeCode = "FP7-PEOPLE", 
    subprogramme = "FP7-PEOPLE-2013-NIGHT", title = "REsearchers for NAture preservation and TECHnology use for the benefit of the society", 
    startDate = "2013-05-01 00:00:00", endDate = "2013-10-31 00:00:00", 
    projectUrl = NA_character_, totalCost = 50616, ecMaxContribution = 49997, 
    call = "FP7-PEOPLE-2013-NIGHT", fundingScheme = "CSA-SA", 
    coordinator = "INSTITUT JOZEF STEFAN", organisationId = 9113619, 
    coordinatorCountry = "SI", participants = list(c("FAKULTETA ZA INFORMACIJSKE 'TUDIJE V NOVEM MESTU", 
    "NACIONALNI INSTITUT ZA BIOLOGIJO")), participantCountries = list(
        "SI"), subjects = list("LIF")), .Names = c("rcn", "reference", 
"acronym", "programmePga", "programmeCode", "subprogramme", "title", 
"startDate", "endDate", "projectUrl", "totalCost", "ecMaxContribution", 
"call", "fundingScheme", "coordinator", "organisationId", "coordinatorCountry", 
"participants", "participantCountries", "subjects"), row.names = 5L, class = "data.frame")

And I still get an error with

bulk_create(mtcars, dbname="bulktest_f") 

I think I'm using the latest version

 package * version  date       source                      
 sofa      0.1.7.99 2015-03-12 Github (sckott/sofa@aaaf2fc)

[sckott]

Thanks! Can you paste in your devtools::session_info() or sessionInfo() - Errors may be coming from different versions of dependencies, likely jsonlite could be the problem

[patperu]

Yes sure!

> devtools::session_info()$packages
 package    * version  date       source                      
 bitops     * 1.0-6    2013-08-17 CRAN (R 3.1.2)              
 devtools   * 1.7.0    2015-01-17 CRAN (R 3.1.2)              
 httr       * 0.6.1    2015-01-01 CRAN (R 3.1.2)              
 jsonlite   * 0.9.14   2014-12-01 CRAN (R 3.1.2)              
 plyr       * 1.8.1    2014-02-26 CRAN (R 3.1.2)              
 Rcpp       * 0.11.5   2015-03-06 CRAN (R 3.1.2)              
 RCurl      * 1.95-4.5 2014-12-06 CRAN (R 3.1.2)              
 rstudioapi * 0.2      2014-12-31 CRAN (R 3.1.2)              
 sofa         0.1.7.99 2015-03-12 Github (sckott/sofa@aaaf2fc)
 stringr    * 0.6.2    2012-12-06 CRAN (R 3.1.2)   

[patperu]

Hello @sckott
here comes the rest

> devtools::session_info()
Session info --------------
 setting  value                                      
 version  R version 3.1.2 Patched (2015-01-10 r67415)
 system   x86_64, mingw32                            
 ui       Rgui                                       
 language (EN)                                       
 collate  German_Germany.1252                        
 tz       Europe/Berlin                              

[sckott]

@patperu Okay, install again and tell me what you get