Skip to content
This repository has been archived by the owner on May 10, 2022. It is now read-only.

access public data in figshare without access code #58

Closed
grunwald opened this issue Dec 13, 2013 · 5 comments
Closed

access public data in figshare without access code #58

grunwald opened this issue Dec 13, 2013 · 5 comments

Comments

@grunwald
Copy link

I am trying to download data into R from figshare using rfigshare with the commands:

url <- fs_download(877104)
data <- read.csv(url)

Without the access codes I cannot read the data into R. Is there a way of accessing figshare data that is public without access codes?
@cboettig
Copy link
Collaborator

The figshare API does not permit a client to access the file without authenticating, even if the file is public.

I can't say exactly why they've made that choice, but note that it does permit them to better track and manage usage of the API, which can help avoid bringing the system down by a single machine flooding the API with requests, etc.

Once you set up some API keys your command will work as advertised. Getting a set of keys really isn't so hard; hopefully our directions in the README help walk you through this.

@grunwald
Copy link
Author

Hi Carl,

Thanks for getting back to me. I got it to work on my machine but wanted to use figshare to house demo datasets available from within R for general use. I guess this will not work given the requirement for authentication. Do you know if other options for housing data in a repository and calling from within R?

Thanks,
Nik

On Dec 13, 2013, at 1:20 PM, Carl Boettiger notifications@github.com wrote:

The figshare API does not permit a client to access the file without authenticating, even if the file is public.

I can't say exactly why they've made that choice, but note that it does permit them to better track and manage usage of the API, which can help avoid bringing the system down by a single machine flooding the API with requests, etc.

Once you set up some API keys your command will work as advertised. Getting a set of keys really isn't so hard; hopefully our directions in the README help walk you through this.

—
Reply to this email directly or view it on GitHub.

Niklaus J. Grünwald
Research Plant Pathologist | Horticultural Crops Research Laboratory | USDA ARS
Professor (courtesy) | Department of Botany and Plant Pathology | Oregon State University
3420 NW Orchard Ave. | Corvallis, OR 97330 | USA | Tel 541.738-4049 | Fax 541.738-4025
grunwaldlab.cgrb.oregonstate.edu | phytophthora-id.org | oregonstate.edu/instruct/dce/phytophthora | phytophthora-smallrna-db.cgrb.oregonstate.edu

@cboettig
Copy link
Collaborator

Thanks for explaining the use-case. One option is to just use a dummy account. For instance, I just grabbed these credentials from their example docs:

client_key = '123456'
client_secret = '65xyAzi1'
token_key = 'n4oTQ22l4FxsuQlYZlhCFwYrrSgrlPn1lhIx32uzwzAwn4oTQ22l4FxsuQlYZlhC1F'
token_secret = '0MNOqkQncNHuKTi6fQ8MuA'
fs_auth(client_key, client_secret, token_key, token_secret)

Just run that code block on any machine with rfigshare installed and then the public commands should work.

As for your more general question, yeah, there's a large variety of such packages, though I suspect all will need some kind of authentication to upload the data, they may not require it to read public data. You might look at RDropbox or RGoogleDocs, though I haven't played with them myself.

Your simplest workflow might not require an R package anyhow. Assuming you're happy uploading the files through some web interface, you can usually access the public files directly with read.csv etc and the url. For instance, one can always stick a bunch of .csv files in a github repository and then read them directly into R (just use the url given by the 'raw' button).

HTH,
Carl

@grunwald
Copy link
Author

Thanks Carl. That helps tremendously!

Best,
Nik

On Dec 13, 2013, at 1:49 PM, Carl Boettiger notifications@github.com wrote:

Thanks for explaining the use-case. One option is to just use a dummy account. For instance, I just grabbed these credentials from their example docs:

client_key = '123456'
client_secret = '65xyAzi1'
token_key = 'n4oTQ22l4FxsuQlYZlhCFwYrrSgrlPn1lhIx32uzwzAwn4oTQ22l4FxsuQlYZlhC1F'
token_secret = '0MNOqkQncNHuKTi6fQ8MuA'
fs_auth(client_key, client_secret, token_key, token_secret)
Just run that code block on any machine with rfigshare installed and then the public commands should work.

As for your more general question, yeah, there's a large variety of such packages, though I suspect all will need some kind of authentication to upload the data, they may not require it to read public data. You might look at RDropbox or RGoogleDocs, though I haven't played with them myself.

Your simplest workflow might not require an R package anyhow. Assuming you're happy uploading the files through some web interface, you can usually access the public files directly with read.csv etc and the url. For instance, one can always stick a bunch of .csv files in a github repository and then read them directly into R (just use the url given by the 'raw' button).

HTH,
Carl

—
Reply to this email directly or view it on GitHub.

Niklaus J. Grünwald
Research Plant Pathologist | Horticultural Crops Research Laboratory | USDA ARS
Professor (courtesy) | Department of Botany and Plant Pathology | Oregon State University
3420 NW Orchard Ave. | Corvallis, OR 97330 | USA | Tel 541.738-4049 | Fax 541.738-4025
grunwaldlab.cgrb.oregonstate.edu | phytophthora-id.org | oregonstate.edu/instruct/dce/phytophthora | phytophthora-smallrna-db.cgrb.oregonstate.edu

@cboettig cboettig closed this as completed Jan 7, 2014
@muschellij2
Copy link

muschellij2 commented Aug 28, 2020 •
edited
Loading

I know this is a bit old, but I wanted to bump this a bit for anyone looking. I think it's possible to do this for some publicly-available data. Both is_mine and session in fs_details require authorization. Setting mine = FALSE and session = NULL I think can download publicly available data straight away:

library(rfigshare)
x = rfigshare::fs_details("11916087", mine = FALSE, session = NULL)
#> No encoding supplied: defaulting to UTF-8.
df = jsonlite::fromJSON(jsonlite::toJSON(x$files), flatten = TRUE)
head(df)
#>                                      download_url
#> 1 https://ndownloader.figshare.com/files/21855555
#> 2 https://ndownloader.figshare.com/files/21855558
#> 3 https://ndownloader.figshare.com/files/21855561
#> 4 https://ndownloader.figshare.com/files/21855564
#> 5 https://ndownloader.figshare.com/files/21855567
#> 6 https://ndownloader.figshare.com/files/21855573
#>                                   name       id     size
#> 1 AI1_NEO1B41100255_2016-10-17.gt3x.gz 21855555 33.59 MB
#> 2 AI1_NEO1F09120035_2016-10-17.gt3x.gz 21855558 36.04 MB
#> 3 AI2_NEO1B41100262_2016-10-17.gt3x.gz 21855561 39.84 MB
#> 4 AI2_NEO1F16120038_2016-10-17.gt3x.gz 21855564 41.67 MB
#> 5 AI3_CLE2B21130054_2017-06-02.gt3x.gz 21855567 46.45 MB
#> 6 AI3_CLE2B21130055_2017-06-02.gt3x.gz 21855573 44.68 MB
dl = rfigshare::fs_download("11916087", mine = FALSE, session = NULL)
#> No encoding supplied: defaulting to UTF-8.
dl
#>       [,1]                                             
#>  [1,] "https://ndownloader.figshare.com/files/21855555"
#>  [2,] "https://ndownloader.figshare.com/files/21855558"
#>  [3,] "https://ndownloader.figshare.com/files/21855561"
#>  [4,] "https://ndownloader.figshare.com/files/21855564"
#>  [5,] "https://ndownloader.figshare.com/files/21855567"
#>  [6,] "https://ndownloader.figshare.com/files/21855573"
#>  [7,] "https://ndownloader.figshare.com/files/21855576"
#>  [8,] "https://ndownloader.figshare.com/files/21855582"
#>  [9,] "https://ndownloader.figshare.com/files/21855588"
#> [10,] "https://ndownloader.figshare.com/files/21855591"
#> [11,] "https://ndownloader.figshare.com/files/21855639"
#> [12,] "https://ndownloader.figshare.com/files/21855642"
#> [13,] "https://ndownloader.figshare.com/files/21855645"
#> [14,] "https://ndownloader.figshare.com/files/21855648"
#> [15,] "https://ndownloader.figshare.com/files/21855651"
#> [16,] "https://ndownloader.figshare.com/files/21855654"
#> [17,] "https://ndownloader.figshare.com/files/21855657"
#> [18,] "https://ndownloader.figshare.com/files/21855660"
#> [19,] "https://ndownloader.figshare.com/files/21855663"
#> [20,] "https://ndownloader.figshare.com/files/21855666"
#> [21,] "https://ndownloader.figshare.com/files/21855669"
#> [22,] "https://ndownloader.figshare.com/files/21855672"
#> [23,] "https://ndownloader.figshare.com/files/21855675"
#> [24,] "https://ndownloader.figshare.com/files/21855678"
#> [25,] "https://ndownloader.figshare.com/files/21855681"
#> [26,] "https://ndownloader.figshare.com/files/21855684"
#> [27,] "https://ndownloader.figshare.com/files/21855687"
#> [28,] "https://ndownloader.figshare.com/files/21855690"
#> [29,] "https://ndownloader.figshare.com/files/21855693"
#> [30,] "https://ndownloader.figshare.com/files/21855696"
#> [31,] "https://ndownloader.figshare.com/files/21855753"
#> [32,] "https://ndownloader.figshare.com/files/21855762"
#> [33,] "https://ndownloader.figshare.com/files/21855765"
#> [34,] "https://ndownloader.figshare.com/files/21855768"
#> [35,] "https://ndownloader.figshare.com/files/21855771"
#> [36,] "https://ndownloader.figshare.com/files/21855774"
#> [37,] "https://ndownloader.figshare.com/files/21855777"
#> [38,] "https://ndownloader.figshare.com/files/21855780"
#> [39,] "https://ndownloader.figshare.com/files/21855783"
#> [40,] "https://ndownloader.figshare.com/files/21855786"
#> [41,] "https://ndownloader.figshare.com/files/21855795"
#> [42,] "https://ndownloader.figshare.com/files/21855798"
#> [43,] "https://ndownloader.figshare.com/files/21855801"
#> [44,] "https://ndownloader.figshare.com/files/21855804"
#> [45,] "https://ndownloader.figshare.com/files/21855807"
#> [46,] "https://ndownloader.figshare.com/files/21855810"
#> [47,] "https://ndownloader.figshare.com/files/21855813"
#> [48,] "https://ndownloader.figshare.com/files/21855816"
#> [49,] "https://ndownloader.figshare.com/files/21855819"
#> [50,] "https://ndownloader.figshare.com/files/21855822"
#> [51,] "https://ndownloader.figshare.com/files/21855825"
#> [52,] "https://ndownloader.figshare.com/files/21855828"
#> [53,] "https://ndownloader.figshare.com/files/21855831"
#> [54,] "https://ndownloader.figshare.com/files/21855834"
#> [55,] "https://ndownloader.figshare.com/files/21855837"
#> [56,] "https://ndownloader.figshare.com/files/21855840"
#> [57,] "https://ndownloader.figshare.com/files/21855843"
#> [58,] "https://ndownloader.figshare.com/files/21855846"
#> [59,] "https://ndownloader.figshare.com/files/21855849"
#> [60,] "https://ndownloader.figshare.com/files/21855852"
#> [61,] "https://ndownloader.figshare.com/files/21855855"
#> [62,] "https://ndownloader.figshare.com/files/21855858"
#> [63,] "https://ndownloader.figshare.com/files/21855861"
#> [64,] "https://ndownloader.figshare.com/files/21855864"
#> [65,] "https://ndownloader.figshare.com/files/21855867"
#> [66,] "https://ndownloader.figshare.com/files/21855870"
#> [67,] "https://ndownloader.figshare.com/files/21855873"
#> [68,] "https://ndownloader.figshare.com/files/21855876"
#> [69,] "https://ndownloader.figshare.com/files/21855879"
#> [70,] "https://ndownloader.figshare.com/files/21855882"
#> [71,] "https://ndownloader.figshare.com/files/21855885"
#> [72,] "https://ndownloader.figshare.com/files/21855888"
#> [73,] "https://ndownloader.figshare.com/files/21855891"
#> [74,] "https://ndownloader.figshare.com/files/21855894"
#> [75,] "https://ndownloader.figshare.com/files/21855897"
#> [76,] "https://ndownloader.figshare.com/files/21855900"
#> [77,] "https://ndownloader.figshare.com/files/21855903"
#> [78,] "https://ndownloader.figshare.com/files/21855906"
#> [79,] "https://ndownloader.figshare.com/files/21855909"
#> [80,] "https://ndownloader.figshare.com/files/21855912"
#> [81,] "https://ndownloader.figshare.com/files/21881742"

# example from fs_download
url <- fs_download(90818, mine = FALSE, session = NULL)
#> No encoding supplied: defaulting to UTF-8.
data <- read.csv(url)
head(data)
#>                                         Title   Goal  Raised X..Funded
#> 1                   Ancient Roman DNA Project $6,000 $10,171       170
#> 2                    Support zombie research! $3,500  $5,000       143
#> 3                         Turtles in the Deep $2,500  $3,243       130
#> 4                   Cancer? Yeast has answers $2,500  $2,835       113
#> 5          Doctor Zen and the Amazon Crayfish $1,000  $1,120       112
#> 6 Learner Control in Online Training Programs $1,200  $1,240       103
#>   Fully.funded. Contributors Contributions Mean.donation Description.length
#> 1           Yes          170           172           $60               5568
#> 2           Yes           79            81           $63               3738
#> 3           Yes           76            77           $43               4374
#> 4           Yes           61            63           $46               4147
#> 5           Yes           36            38           $31               1509
#> 6           Yes           33            33           $38               2585
#>   Tweets Facebook.likes Video.views
#> 1    142            196        1932
#> 2    117            594       25325
#> 3     15            105         461
#> 4    296            170        1238
#> 5     67             18         950
#> 6     14             69         240
#>                                                                                 Link
#> 1                   http://www.rockethub.com/projects/3709-ancient-roman-dna-project
#> 2                     http://www.rockethub.com/projects/3737-support-zombie-research
#> 3                         http://www.rockethub.com/projects/3810-turtles-in-the-deep
#> 4                    http://www.rockethub.com/projects/3753-cancer-yeast-has-answers
#> 5          http://www.rockethub.com/projects/3695-doctor-zen-and-the-amazon-crayfish
#> 6 http://www.rockethub.com/projects/3800-learner-control-in-online-training-programs
#>   Notes
#> 1      
#> 2      
#> 3      
#> 4      
#> 5      
#> 6


url <- fs_download(877104, mine = FALSE, session = NULL)
#> No encoding supplied: defaulting to UTF-8.
data <- read.csv(url)
head(data)
#>       X56       X187 X18       X9      X12      X10      X13    X10.1       X5
#> 1                     NA Athena_1 Athena_2 Athena_3 Athena_4 Athena_5 Athena_6
#> 2 Isolate Population   1        2        3        4        5        6        7
#> 3       1   Athena_1   1        0        1        0        1        0        0
#> 4       2   Athena_1   1        0        1        0        0        0        0
#> 5       3   Athena_1   1        0        1        0        0        0        0
#> 6       4   Athena_1   1        0        1        0        0        0        0
#>        X11       X8    X10.2      X9.1        X10.3           X6         X8.1
#> 1 Athena_7 Athena_8 Athena_9 Athena_10 Mt. Vernon_1 Mt. Vernon_2 Mt. Vernon_3
#> 2        8        9       10        11           12           13           14
#> 3        1        1        0         1            1            1            0
#> 4        1        1        1         1            1            1            0
#> 5        0        1        0         1            1            1            0
#> 6        0        1        0         1            1            1            0
#>          X12.1          X17        X12.2        X12.3        X13.1  X X.1 X.2
#> 1 Mt. Vernon_4 Mt. Vernon_5 Mt. Vernon_6 Mt. Vernon_7 Mt. Vernon_8 NA  NA  NA
#> 2           15           16           17           18           19 20  21  22
#> 3            0            1            0            0            0  1   0   0
#> 4            0            1            0            0            0  1   0   0
#> 5            0            1            0            0            0  1   0   0
#> 6            0            1            0            0            0  1   0   0
#>   X.3 X.4 X.5 X.6 X.7 X.8 X.9 X.10 X.11 X.12 X.13 X.14 X.15 X.16 X.17 X.18 X.19
#> 1  NA  NA  NA  NA  NA  NA  NA   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA
#> 2  23  24  25  26  27  28  29   30   31   32   33   34   35   36   37   38   39
#> 3   1   1   0   0   1   0   1    0    1    0    0    1    0    1    0    0    0
#> 4   1   1   0   0   1   0   1    0    1    0    0    1    0    1    0    0    0
#> 5   1   1   0   0   0   0   1    0    1    0    0    1    0    0    0    0    0
#> 6   1   1   0   0   0   0   1    0    1    0    0    1    0    0    0    0    0
#>   X.20 X.21 X.22 X.23 X.24 X.25 X.26 X.27 X.28 X.29 X.30 X.31 X.32 X.33 X.34
#> 1   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA   NA
#> 2   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54
#> 3    0    0    1    1    1    1    0    0    0    1    1    1    1    0    0
#> 4    0    0    1    1    1    1    0    0    0    1    1    1    1    1    0
#> 5    0    0    1    0    1    0    0    0    0    1    0    1    1    0    0
#> 6    0    0    1    0    1    0    0    0    0    1    0    0    1    0    0
#>   X.35 X.36
#> 1   NA   NA
#> 2   55   56
#> 3    0    1
#> 4    0    1
#> 5    0    0
#> 6    0    0

Created on 2020-08-28 by the reprex package (v0.3.0)

Session info 
devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/New_York            
#>  date     2020-08-28                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date       lib source                           
#>  assertthat    0.2.1       2019-03-21 [1] CRAN (R 4.0.0)                   
#>  backports     1.1.9       2020-08-24 [1] CRAN (R 4.0.0)                   
#>  callr         3.4.3       2020-03-28 [1] CRAN (R 4.0.0)                   
#>  cli           2.0.2       2020-02-28 [1] CRAN (R 4.0.0)                   
#>  colorspace    1.4-1       2019-03-18 [1] CRAN (R 4.0.0)                   
#>  crayon        1.3.4       2017-09-16 [1] CRAN (R 4.0.0)                   
#>  curl          4.3         2019-12-02 [1] CRAN (R 4.0.0)                   
#>  desc          1.2.0       2020-06-01 [1] Github (muschellij2/desc@b0c374f)
#>  devtools      2.3.1.9000  2020-08-25 [1] Github (r-lib/devtools@df619ce)  
#>  digest        0.6.25      2020-02-23 [1] CRAN (R 4.0.0)                   
#>  dplyr         1.0.2       2020-08-18 [1] CRAN (R 4.0.2)                   
#>  ellipsis      0.3.1       2020-05-15 [1] CRAN (R 4.0.0)                   
#>  evaluate      0.14        2019-05-28 [1] CRAN (R 4.0.0)                   
#>  fansi         0.4.1       2020-01-08 [1] CRAN (R 4.0.0)                   
#>  fs            1.5.0       2020-07-31 [1] CRAN (R 4.0.2)                   
#>  generics      0.0.2       2018-11-29 [1] CRAN (R 4.0.0)                   
#>  ggplot2       3.3.2       2020-06-19 [1] CRAN (R 4.0.0)                   
#>  glue          1.4.1       2020-05-13 [1] CRAN (R 4.0.0)                   
#>  gtable        0.3.0       2019-03-25 [1] CRAN (R 4.0.0)                   
#>  highr         0.8         2019-03-20 [1] CRAN (R 4.0.0)                   
#>  htmltools     0.5.0       2020-06-16 [1] CRAN (R 4.0.0)                   
#>  httpuv        1.5.4       2020-06-06 [1] CRAN (R 4.0.0)                   
#>  httr          1.4.2       2020-07-20 [1] CRAN (R 4.0.2)                   
#>  jsonlite      1.7.0       2020-06-25 [1] CRAN (R 4.0.0)                   
#>  knitr         1.29        2020-06-23 [1] CRAN (R 4.0.2)                   
#>  later         1.1.0.1     2020-06-05 [1] CRAN (R 4.0.0)                   
#>  lifecycle     0.2.0       2020-03-06 [1] CRAN (R 4.0.0)                   
#>  magrittr      1.5         2014-11-22 [1] CRAN (R 4.0.0)                   
#>  memoise       1.1.0       2017-04-21 [1] CRAN (R 4.0.0)                   
#>  munsell       0.5.0       2018-06-12 [1] CRAN (R 4.0.0)                   
#>  pillar        1.4.6       2020-07-10 [1] CRAN (R 4.0.2)                   
#>  pkgbuild      1.1.0       2020-07-13 [1] CRAN (R 4.0.2)                   
#>  pkgconfig     2.0.3       2019-09-22 [1] CRAN (R 4.0.0)                   
#>  pkgload       1.1.0       2020-05-29 [1] CRAN (R 4.0.0)                   
#>  plyr          1.8.6       2020-03-03 [1] CRAN (R 4.0.0)                   
#>  prettyunits   1.1.1       2020-01-24 [1] CRAN (R 4.0.0)                   
#>  processx      3.4.3       2020-07-05 [1] CRAN (R 4.0.0)                   
#>  promises      1.1.1       2020-06-09 [1] CRAN (R 4.0.0)                   
#>  ps            1.3.4       2020-08-11 [1] CRAN (R 4.0.2)                   
#>  purrr         0.3.4       2020-04-17 [1] CRAN (R 4.0.0)                   
#>  R6            2.4.1       2019-11-12 [1] CRAN (R 4.0.0)                   
#>  Rcpp          1.0.5       2020-07-06 [1] CRAN (R 4.0.0)                   
#>  remotes       2.2.0       2020-07-21 [1] CRAN (R 4.0.2)                   
#>  rfigshare   * 0.3.7       2015-06-15 [1] CRAN (R 4.0.0)                   
#>  RJSONIO       1.3-1.4     2020-01-15 [1] CRAN (R 4.0.0)                   
#>  rlang         0.4.7.9000  2020-08-25 [1] Github (r-lib/rlang@de0c176)     
#>  rmarkdown     2.3         2020-06-18 [1] CRAN (R 4.0.0)                   
#>  rprojroot     1.3-2       2018-01-03 [1] CRAN (R 4.0.0)                   
#>  scales        1.1.1       2020-05-11 [1] CRAN (R 4.0.0)                   
#>  sessioninfo   1.1.1       2018-11-05 [1] CRAN (R 4.0.0)                   
#>  stringi       1.4.6       2020-02-17 [1] CRAN (R 4.0.0)                   
#>  stringr       1.4.0       2019-02-10 [1] CRAN (R 4.0.0)                   
#>  testthat      2.99.0.9000 2020-08-25 [1] Github (r-lib/testthat@6a24275)  
#>  tibble        3.0.3       2020-07-10 [1] CRAN (R 4.0.2)                   
#>  tidyselect    1.1.0       2020-05-11 [1] CRAN (R 4.0.0)                   
#>  usethis       1.6.1.9001  2020-08-25 [1] Github (r-lib/usethis@860c1ea)   
#>  vctrs         0.3.2       2020-07-15 [1] CRAN (R 4.0.2)                   
#>  withr         2.2.0       2020-04-20 [1] CRAN (R 4.0.0)                   
#>  xfun          0.16        2020-07-24 [1] CRAN (R 4.0.2)                   
#>  XML           3.99-0.5    2020-07-23 [1] CRAN (R 4.0.2)                   
#>  yaml          2.2.1       2020-02-01 [1] CRAN (R 4.0.0)                   
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

@kaijagahm kaijagahm mentioned this issue Nov 11, 2020
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cboettig @muschellij2 @grunwald