Creating dataset using create_dataset() yields 500 error

[thomascli19]

Please specify whether your issue is about:

Hi There,

I was trying out some of the functions for the dataverse package, and came across it creating a 500 error every time I tried to execute the function. Unfortunately, I can't put my server and API key, but I can show what I did attempt:

## load package
library("dataverse")

## code
meta2<-list(title="test",author="Li,Thomas",datasetContact="Fish, Fishy",dsDescription="FISH",subject="Quantitative Sciences",depositor="Fish, Fishy",dateOfDeposit="Fish Time",datasetContactEmail="fishyfishery@tunahalibut.com")
create_dataset("MLPOCs2018",body=meta2) 

When running the debugger, I noticed the 500 error occurs when the POST() function is used from the 'httr' package (which makes sense), so I am trying to see the cause of this (be it permission or something else).

Just in case, the sessionInfo() yields the following:

R version 3.6.0 (2019-04-26)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] tools     stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] zip_2.0.2         flowCore_1.50.0   magick_2.0        ggplot2_3.2.0     usethis_1.5.0     devtools_2.0.2   
 [7] magrittr_1.5      data.table_1.12.2 dplyr_0.8.1       dataverse_0.2.0  

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5    remotes_2.1.0       purrr_0.3.2         lattice_0.20-38     pcaPP_1.9-73       
 [6] colorspace_1.4-1    testthat_2.1.1      stats4_3.6.0        yaml_2.2.0          rlang_0.4.0        
[11] pkgbuild_1.0.3      pillar_1.4.1        glue_1.3.1          withr_2.1.2         BiocGenerics_0.30.0
[16] sessioninfo_1.1.1   matrixStats_0.54.0  robustbase_0.93-5   munsell_0.5.0       gtable_0.3.0       
[21] mvtnorm_1.0-11      memoise_1.1.0       Biobase_2.44.0      callr_3.2.0         ps_1.3.0           
[26] curl_3.3            parallel_3.6.0      DEoptimR_1.0-8      Rcpp_1.0.1          corpcor_1.6.9      
[31] backports_1.1.4     scales_1.0.0        desc_1.2.0          pkgload_1.0.2       jsonlite_1.6       
[36] graph_1.62.0        fs_1.3.1            digest_0.6.19       processx_3.3.1      grid_3.6.0         
[41] rprojroot_1.3-2     cli_1.1.0           lazyeval_0.2.2      tibble_2.1.3        cluster_2.1.0      
[46] crayon_1.3.4        rrcov_1.4-7         pkgconfig_2.0.2     MASS_7.3-51.4       xml2_1.2.0         
[51] prettyunits_1.0.2   assertthat_0.2.1    httr_1.4.0          rstudioapi_0.10     R6_2.4.0           
[56] compiler_3.6.0  

If needed, I can also provide the log from the server that shows the "API internal error"

Thanks!

[pdurbin]

If needed, I can also provide the log from the server that shows the "API internal error"

The server.log file would be quite valuable to us. Can you please either attach it here (you'll have to add ".txt") or email it to support@dataverse.org?

[thomascli19]

Yeah sure thing! I have attached the .txt file with this message.
dataverse_API_internal error.txt

[pdurbin]

@thomascli19 thanks, are you running Dataverse 4.15? If so, here's where the NullPointerException is being thrown:

https://github.com/IQSS/dataverse/blob/v4.15/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java#L273

[thomascli19]

I'm actually running version 4.9.4. Would this NullPointerException that is being thrown also appear in this version?

[pdurbin]

Line 273 looks the same:

https://github.com/IQSS/dataverse/blob/v4.9.4/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java#L273

Can you please try it on https://demo.dataverse.org to see if you get a 500 error there too?

[thomascli19]

Just tried and it yields the same error, even when I input the "body" argument as an empty list.

[pdurbin]

@thomascli19 that's not so great if "create dataset" doesn't work.

I assume you're aware that you could also use curl or the Python or Java libraries listed at http://guides.dataverse.org/en/4.15/api/client-libraries.html to create datasets. (I'm using pyDataverse myself.)

In #21 a new maintainer is being sought. Let me at least mention @monogan in case he has time to take a look.

[maia-sh]

Any news on this issue? I'm coming across the same problem with create_dataset(). I've tried different arguments including:

1. https://cran.r-project.org/web/packages/dataverse/vignettes/A-introduction.html

# create a list of metadata
metadat <- list(title = "My Study",
                creator = "Doe, John",
                description = "An example study")

# create the dataset
dat <- initiate_dataset("mydataverse", body = metadat)

2. https://cran.r-project.org/web/packages/dataverse/vignettes/D-archiving.html

# create the dataset
ds <- create_dataset("mydataverse")

Note: I also tried the SWORD-based workflow described on this same webpage to no avail. initiate_sword_dataset() runs but add_file() throws an error.

3. body parameters to match the online html tags for the required fields, similar to @thomascli19's example

# create a list of metadata
metadat <- list(title = "My Study",
                author = "Doe, John",
                datasetContact = "Doe, John",
                dsDescription = "An example study",
                subject = "Other")

# create the dataset
dat <- initiate_dataset("mydataverse", body = metadat)

[sindribaldur]

I'm got stuck here too. It would have saved me time if this was mentioned in the README. Will try pyDataverse for the time being.

[wibeasley]

I'm sorry this isn't working. I'll have time this weekend to look.

[kuriwaki]

This workaround (#82 (comment)) using initiate_sword_dataset() instead of create_dataset() and using. add_dataset_file() instead of add_file() appears to be a good workaround.

[Danny-dK]

I previously got the same errors, but have an httr code working in R now. If others can help me with my question in #82(comment) we may be able to solve the question here as well (as the dataverse package uses httr as well in a similar fashion)?

[kuriwaki]

With current version on dev the error seems a bit more narrowed down.

library("dataverse")
packageVersion("dataverse") # dev
#> [1] '0.3.11'
metadat <- list(
  title = "test upload",
  creator = "Shiro Kuriwaki",
  datasetContact = "Shiro Kuriwaki",
  description = "Test Create",
  Subject = "Other"
)

create_dataset(
  dataverse = "kuriwaki",
  body = jsonlite::toJSON(metadat, auto_unbox = TRUE),
  server = "demo.dataverse.org",
  key = rstudioapi::askForPassword()
)

^{Created on 2022-04-09 by the reprex package (v2.0.1)}

This gives

Error in create_dataset(dataverse = "kuriwaki", body = jsonlite::toJSON(metadat, : 
Forbidden (HTTP 403). Failed to Validation Failed: Description Text is required. (Invalid 
value:edu.harvard.iq.dataverse.DatasetField[ id=null ]), Subject is required. (Invalid 
value:edu.harvard.iq.dataverse.DatasetField[ id=null ]), Title is required. (Invalid 
value:edu.harvard.iq.dataverse.DatasetField[ id=null ]), Contact E-mail is required. (Invalid 
value:edu.harvard.iq.dataverse.DatasetField[ id=null ]), Author Name is required. (Invalid 
value:edu.harvard.iq.dataverse.DatasetField[ id=null ])..

So perhaps getting the metadata slots right could do it

[Danny-dK]

Yes. You either have to create a json file with the required metadata fields #82 (comment) of which you can get the correct json format at https://guides.dataverse.org/en/5.3/_downloads/dataset-create-new-all-default-fields.json, or you can recreate the json with a very ugly paste code (doesn't require the json file) #82 (comment), or you can do it with an ugly for function requiring some json files #82 (comment)). Even within the pydataverse either a json file or a csv template is required #82 (comment)), there does not seem to be a helper function to recreate the require metadata like in initiate_sword_dataset() #82 (comment)