rrricanes

[timtrice]

Summary

• What does this package do? (explain in 50 words or less):
  Scrapes and parses tropical cyclone advisories and GIS data from the National Hurricane Center. Can access real-time or archived data for storms since 1998.

• Paste the full DESCRIPTION file inside a code block below:

Package: rrricanes
Type: Package
Title: Web scraper for real-time and archived advisory products for Atlantic and 
    east Pacific hurricanes and tropical storms
Version: 0.1.0.1
Authors@R: person("Tim", "Trice", email = "tim.trice@gmail.com",
                  role = c("aut", "cre"))
Depends:
  R (>= 3.3.3)
Description: Get archived data of past and current hurricanes and tropical 
  storms for the Atlantic and eastern Pacific oceans. Data is available for 
  storms since 1998.
URL: http://www.timtrice.net
BugReports: https://github.com/timtrice/rrricanes/issues
License: MIT + file LICENSE
LazyData: TRUE
Imports:
  data.table (>= 1.9.6), 
  dplyr (>= 0.5.0), 
  httr (>= 1.2.1), 
  lubridate (>= 1.6.0), 
  magrittr (>= 1.5), 
  purrr (>= 0.2.2.2), 
  readr (>= 1.1.0), 
  rvest (>= 0.3.2), 
  stringr (>= 1.2.0), 
  tibble (>= 1.3.1), 
  tidyr (>= 0.6.2), 
  xml2 (>= 1.1.1)
Suggests:
  devtools, 
  knitr, 
  rmarkdown,
  testthat,
RoxygenNote: 6.0.1
VignetteBuilder: knitr

• URL for the package (the development repository, not a stylized html page):
  https://github.com/timtrice/rrricanes

• Please indicate which category or categories from our package fit policies this package falls under and why? (e.g., data retrieval, reproducibility. If you are unsure, we suggest you make a pre-submission inquiry.):
  data extraction, data visualization, data munging, data packages

• Who is the target audience?
  Those with an interest in studying tropical meteorology, tracking storm developments and comparing the forecast abilities of the National Hurricane Center storm-to-storm, year-to-year.

• Are there other R packages that accomplish the same thing? If so, what is different about yours?
  I have searched (I believe) extensively and have not found one at all. To my knowledge the only other hurricane package in CRAN or GitHub is HURDAT - and I wrote that one, too :)

Requirements

R 3.3.3
Most tidyverse packages.
rnaturalearthdata and rnaturalerathhires will be required for release 0.1.1.

Confirm each of the following by checking the box. This package:

• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• does not violate the Terms of Service of any service it interacts with.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage, using services such as Travis CI, Coeveralls and/or CodeCov.
• I agree to abide by ROpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:
  • The package contains a paper.md with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)

Detail

• Does R CMD check (or devtools::check()) succeed? Paste and describe any errors or warnings:
  In release 0.1.0.1, only 2 notes about global variables. This has largely been resolved in unreleased 0.1.1.

• Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:
  I used a CONTRIBUTORS.md file instead of CODE_OF_CONDUCT.md.
  I use @keywords internal for internal functions

• If this is a resubmission following rejection, please explain the change in circumstances:

• If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:

[maelle]

Thanks for your submission @timtrice ! 😸

I have two questions

• I see you have Appveyor CI set up but in the logs I see it happens with these R CHECK options --no-tests --no-manual --no-build-vignettes, could you try a build on Appveyor not skipping tests and building the vignettes? I get errors when doing it on my Windows PC (R 3.3.1 though, I'll try again on another PC later but maybe the R version is not the issue, hence the Appveyor suggestion 😉)

Tests results

> devtools::test()
Loading rrricanes
Loading required package: testthat

Attaching package: ‘testthat’

The following object is masked from ‘package:dplyr’:

    matches

Testing rrricanes
Test base functions.: .........................
Storm Discussions (discus): ...........................12
|====================================================================================|100% ~0 s remaining     ....
Forecast/Advisory Products (fstadv): 3
Test getting storm data.: .
|====================================================================================|100% ~0 s remaining     ..
Position Estimates (posest): .................
|====================================================================================|100% ~0 s remaining     4
|====================================================================================|100% ~0 s remaining     .....5
Scrapers: .................
Update (update): .................
|====================================================================================|100% ~0 s remaining     .

Failed -------------------------------------------------------------------------------------------------------
1. Error: Test discus() (@test_discus.R#54) ------------------------------------------------------------------
character string is not in a standard unambiguous format
1: discus(link = url) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/tests/testthat/test_discus.R:54
2: scrape_header(contents, ret = "date") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/discus.R:71
3: scrape_date(header) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:234
4: as.POSIXct(x, tz = "Etc/GMT+4") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:162
5: as.POSIXct.default(x, tz = "Etc/GMT+4")
6: as.POSIXct(as.POSIXlt(x, tz, ...), tz, ...)
7: as.POSIXlt(x, tz, ...)
8: as.POSIXlt.character(x, tz, ...)
9: stop("character string is not in a standard unambiguous format")

2. Error: Test get_discus() (@test_discus.R#71) --------------------------------------------------------------
character string is not in a standard unambiguous format
1: get_discus(link = url) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/tests/testthat/test_discus.R:71
2: purrr::map(filter_discus(products), discus) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/discus.R:41
3: .f(.x[[i]], ...)
4: scrape_header(contents, ret = "date") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/discus.R:71
5: scrape_date(header) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:234
6: as.POSIXct(x, tz = "Etc/GMT+4") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:162
7: as.POSIXct.default(x, tz = "Etc/GMT+4")
8: as.POSIXct(as.POSIXlt(x, tz, ...), tz, ...)
9: as.POSIXlt(x, tz, ...)
10: as.POSIXlt.character(x, tz, ...)
11: stop("character string is not in a standard unambiguous format")

3. Error: (unknown) (@test_fstadv.R#8) -----------------------------------------------------------------------
character string is not in a standard unambiguous format
1: get_fstadv(link = url.al011998) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/tests/testthat/test_fstadv.R:8
2: purrr::map(filter_fstadv(products), fstadv) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/fstadv.R:157
3: .f(.x[[i]], ...)
4: scrape_header(contents, ret = "date") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/fstadv.R:210
5: scrape_date(header) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:234
6: as.POSIXct(x, tz = "UTC") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:152
7: as.POSIXct.default(x, tz = "UTC")
8: as.POSIXct(as.POSIXlt(x, tz, ...), tz, ...)
9: as.POSIXlt(x, tz, ...)
10: as.POSIXlt.character(x, tz, ...)
11: stop("character string is not in a standard unambiguous format")

4. Error: (unknown) (@test_prblty.R#9) -----------------------------------------------------------------------
character string is not in a standard unambiguous format
1: get_prblty(al1998[1]) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/tests/testthat/test_prblty.R:9
2: purrr::map(filter_prblty(products), prblty) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/prblty.R:25
3: .f(.x[[i]], ...)
4: scrape_header(contents, ret = "date") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/prblty.R:52
5: scrape_date(header) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:234
6: as.POSIXct(x, tz = "Etc/GMT+4") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:156
7: as.POSIXct.default(x, tz = "Etc/GMT+4")
8: as.POSIXct(as.POSIXlt(x, tz, ...), tz, ...)
9: as.POSIXlt(x, tz, ...)
10: as.POSIXlt.character(x, tz, ...)
11: stop("character string is not in a standard unambiguous format")

5. Error: 1998, Tropical Storm Alex, Advisory 26 (@test_public.R#38) -----------------------------------------
character string is not in a standard unambiguous format
1: expect_identical(public(al.1998.alex.products.public[25]) %>% dplyr::select(Status) %>% dplyr::first(), "Tropical Disturbance") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/tests/testthat/test_public.R:38
2: identical(object, expected)
3: public(al.1998.alex.products.public[25]) %>% dplyr::select(Status) %>% dplyr::first()
4: eval(lhs, parent, parent)
5: eval(expr, envir, enclos)
6: public(al.1998.alex.products.public[25])
7: scrape_header(contents, ret = "date") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/public.R:70
8: scrape_date(header) at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:234
9: as.POSIXct(x, tz = "Etc/GMT+4") at C:\Users\msalmon.ISGLOBAL\Documents\rrricanes/R/scrapers.R:156
10: as.POSIXct.default(x, tz = "Etc/GMT+4")
11: as.POSIXct(as.POSIXlt(x, tz, ...), tz, ...)
12: as.POSIXlt(x, tz, ...)
13: as.POSIXlt.character(x, tz, ...)
14: stop("character string is not in a standard unambiguous format")

DONE =========================================================================================================
Don't worry, you'll get it.

• In the following (uncommented) code, why do you use assignments to the global environment?

    s <- sapply(x, function(z, n = names, l = link, m = msg){
        f <- paste("get", z, sep = "_")
        res <- do.call(f, args = list("link" = l, "msg" = m))
        if (!is.null(n[[z]])) {
            assign(n[z][[1]], res, envir = .GlobalEnv)
        } else {
            assign(z, res, envir = .GlobalEnv)
        }
    })

[timtrice]

@maelle , thank you for taking the time to review my project!

Regarding the code in question, it has been replaced with the following:

    ds <- purrr::map(products, .f = function(x) {
        sprintf("get_%s", x) %>%
            purrr::invoke_map(.x = list(link = link)) %>%
            purrr::flatten_df()})
    names(ds) <- products
    return(ds)

This is in branch develop-0.2.0. I believe I have modified all of the assign calls; they do not show up in R CMD checks (0 notes).

Regarding the tests; I apologize for them not being enabled. At one point earlier this week I had to disable them due to the NHC archives being down for some annual archive pages. This seems to have been resolved now. Tests have been added to both Travis and Appveyor.

A side note to that last point; I have noticed there are timeouts when accessing the NHC archives. This has been verified on numerous occasions with Travis and Appveyor during tests. I have a feature request pending to add handling for this. Additionally, in 0.2.0, load_storm_data has been added that accesses some already-processed datasets through the GitHub repo rrricanesdata. So I'm trying to develop some workaround to ensure users have multiple ways to get quick access to data they need.

[maelle]

@timtrice thanks, nice work! For review (and my editor checks :-) ) will master be updated?

Great on having already processed datasets, but I see that repo has no licensing information. I'm not experienced at all in licensing issues but I guess it'd be at least nice to add where this data comes from etc.

[maelle]

Now that I could check the package (on the right R version) here is something more formalized.

Editor checks:

• [ x] Fit: The package meets criteria for fit and overlap
• [ x] Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• [ x] License: The package has a CRAN or OSI accepted license
• [ x] Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

• Note that lazyeval should soon be replaced by rlang so it might be worth to have a look at https://github.com/tidyverse/rlang

• 2 long lines identified by goodpractice::gp()

── GP rrricanes ───────────────────────────────────────────────────────────────────────────────────────────────

It is good practice to

  ✖ write unit tests for all functions, and all package code in general. 93% of code lines are
    covered by test cases.

    R/base.R:74:NA
    R/base.R:75:NA
    R/base.R:76:NA
    R/base.R:77:NA
    R/base.R:78:NA
    ... and 64 more lines

  ✖ avoid long code lines, it is bad for readability. Also, many people prefer editor windows that
    are about 80 characters wide. Try make your lines shorter than 80 characters

    R\base.R:84:1
    R\scrapers.R:83:1
    tests\testthat\test_scrapers.R:46:1

---

• Typos identified via devtools::spell_check()

currentlby      get_fstadv.Rd:18
dataframej      build_archive_df.Rd:20

Reviewers: @jsta @robinsones
Due date: 2017-07-10

[timtrice]

@maelle, thank you for the compliment! Very much appreciated.

The master will be updated with develop-0.2.0 when I finish the GIS products. They probably could be released now but I'm trying to think over things and make sure there's not a better way to handle them (consolidating functions, for example). Plus one of the products is raster format so I'm looking over ways to view them with ggplot, if even possible, what the alternatives are, etc.

In hindsight I should have incorporated bug fixes into master. It was one of those things that I had already made new changes and started just hammering out bugs - before I knew it I lost track of where I was. This is the first project I'm trying to incorporate R package rules and github best practices along with what I know needs to be done. Some of it (as your example above) is just a sloppy "get it working" attempt before I get back to making the code cleaner.

Regarding rrricanesdata, I can certainly add a license and the script to add new datasets and update existing ones as need be. I'll add a README as well today.