nlrx

[nldoc]

Summary

• What does this package do? (explain in 50 words or less):

nlrx enables you to run agent-based models in NetLogo from R.
This makes it possible to design reproducible experiments (which can be
zipped with nlrx and shared between persons). Furthermore, it
coerces the spatial data from NetLogo into raster and sf objects.

• Paste the full DESCRIPTION file inside a code block below:

Package: nlrx
Type: Package
Title: Setup, run and analyze NetLogo model simulations from R via XML
Version: 0.1.0
Authors@R: c(person("Jan", "Salecker", email = "jsaleck@gwdg.de",
                     role = c("aut", "cre"), comment = c(ORCID = '0000-0002-9000-4229')),
             person("Marco", "Sciaini", email = "sciaini.marco@gmail.com", role = c("aut"), comment = c(ORCID = '0000-0002-3042-5435')))
Maintainer: Jan Salecker <jsaleck@gwdg.de>
Description: The nlrx package provides tools to setup, run and analyze NetLogo model simulations in R.
    nlrx experiments use a similar structure as NetLogos Behavior Space experiments.
    However, nlrx offers more flexibility and additional tools for running and anlyzing complex simulation designs and sensitivity analyses.
    The user defines all information that is needed in an intuitive framework, using S4 class objects.
    Experiments are submitted from R to NetLogo via XML files that are dynamically written, based on specifications defined by the user.
    By nesting model calls in future environments, large simulation design with many runs can be executed in parallel.
    This also enables simulating NetLogo experiments on remote HPC machines.
License: GPL-3
Encoding: UTF-8
LazyData: true
Roxygen: list(markdown = TRUE)
RoxygenNote: 6.1.0
URL: https://github.com/nldoc/nlrx/
BugReports: https://github.com/nldoc/nlrx/issues/
Imports:
    XML,
    lhs,
    sensitivity,
    dplyr,
    readr,
    purrr,
    furrr,
    magrittr,
    stats,
    utils,
    GenSA,
    genalg,
    raster,
    rstudioapi,
    sf,
    tidyr
Depends:
   methods,
    tibble,
    R (>= 2.10)
Suggests:
    testthat,
    knitr,
    rmarkdown
VignetteBuilder: knitr

• URL for the package (the development repository, not a stylized html page):

• https://github.com/nldoc/nlrx

• Please indicate which category or categories from our package fit policies this package falls under *and why(? (e.g., data retrieval, reproducibility. If you are unsure, we suggest you make a pre-submission inquiry.):

  • reproducibility, as it is the first package that lets you perform systematic packages with NetLogo from R
  • partly spatial, as the landscapes from NetLogo are retrieved as rasters and the agents as sf objects

•   Who is the target audience and what are scientific applications of this package?  

  • The audience is the scientific NetLogo community, that performs experiments within
    NetLogo. They now can use nlrx to parameterise NetLogo and scale their experiments up.
    With nlrx it is the first also possible to run NetLogo experiments on computing clusters.

• Are there other R packages that accomplish the same thing? If so, how does
  yours differ or meet our criteria for best-in-category?

• There is RNetLogo on CRAN, this package however is based on rJava (which is most often an obstacle) and has not as much auxiliary functions to run NetLogo and embed results in R objects.

Requirements

Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage, using services such as Travis CI, Coveralls and/or CodeCov.
• I agree to abide by ROpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Publication options

• Do you intend for this package to go on CRAN?
• Do you wish to automatically submit to the Journal of Open Source Software? If so:
  • The package has an obvious research application according to JOSS's definition.
  • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  • The package is deposited in a long-term repository with the DOI:
  • (Do not submit your package separately to JOSS)
• Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:
  • The package is novel and will be of interest to the broad readership of the journal.
  • The manuscript describing the package is no longer than 3000 words.
  • You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)

Detail

• Does R CMD check (or devtools::check()) succeed? Paste and describe any errors or warnings:

• Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:

• If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:

  • Robin Lovelace (@Robinlovelace i), as he has experience with spatial simulations and NetLogo

[sckott]

thanks for your submission @nldoc we're discussing and will get back to you soon

[sckott]

@nldoc I completely agree the lack of rJava dependency is a big plus. . But can you comment more on the benefit of your pkg over RNetLogo with respect to features/functionality.

[sckott]

There's also https://cran.rstudio.com/web/packages/NetLogoR/ - not sure if you were aware or not? Can you also tell us benefit of your pkg over NetLogoR as well please

[nldoc]

@nldoc I completely agree the lack of rJava dependency is a big plus. . But can you comment more on the benefit of your pkg over RNetLogo with respect to features/functionality.

The main purpose of nlrx is running a NetLogo model and collecting output data. Everything is designed to make this as simple and reproducible as possible.

Reproducability

• nlrx uses nested S4 classes to store all the information that is needed to run NetLogo model simulations.
• After running simulations, the model output is attached to the S4 class object, thus the complete experiment including all experiment definitions, the parameter input matrix, runtime, etc. is now stored in one single S4 class object that can be easily archived and shared with other users.
• For complete reproducability the NetLogo model files are needed. Thus, nlrx provides a function to generate a zip file from the model folder, including a rds file containing the S4 class object that holds the information on the experiment. This zip file could be given to anyone to reproduce nlrx NetLogo simulations.
• When using RNetLogo, everything has to be written from scratch and there is no easy way to share and reproduce complex simulation experiments.

Setting up experiments

• Because everything in nlrx is designed on running NetLogo simulations (and in particular Sensitivity Analysis), the setup is very intuitive and requires a minimum amount of code to be written.
• RNetLogo may be more general in its usage than nlrx because it allows controlling interactive NetLogo sessions from R, however to set up large sensitivity analysis experiments a lot of R code needs to be written.
• The nlrx design is in many parts adapted from the NetLogo built-in tool behavior space, which allows easy understanding of experiment setup for users that are already familiar with NetLogo

Auxiliary functions

• The nlrx package provides a collection of simulation design helper functions. These can be used to create parameter input matrices for different simulation methods, based on the defined model variables.
• For example, the simdesign_ff() creates a full factorial parametrer matrix. simdesign_morris() creates a morris screening parameter matrix (there are many more).
• After running a sensitivity analysis simdesign created with one of these helper functions (e.g. simdesign_morris()), nlrx provides functions to calculate sensitivity indices for each parameter and output
• RNetLogo does not provide any auxiliary functions to setup simulations, generate input parameter matrices or analyze model output
• Additionally, nlrx provides auxiliary functions that make it easy to convert spatial output from NetLogo models to spatial R formats (patches are stored as raster objects and turtles are stored as sf point objects). RNetLogo only provides functions to collect spatial data from NetLogo as data frames.

Summary
The main purpose of the nlrx package was not to replace RNetLogo. Our motivation was to develop a package that makes running sensitivity analysis with R and NetLogo easier and reproducible. While RNetLogo may be more general in its usage, nlrx has many benefits when it comes to its main task: running a NetLogo model many times, with different parameter combinations and collecting model output. For this task nlrx is better suited as RNetLogo because of easier setup (with less code), enhanced reproducability and easy exchange with other users, built-in auxiliary functions to create parameter matrices and calculate sensitivity indices, and increased stability by removing rJava dependency.

[nldoc]

There's also https://cran.rstudio.com/web/packages/NetLogoR/ - not sure if you were aware or not? Can you also tell us benefit of your pkg over NetLogoR as well please

The NetLogoR package follows a different approach because it re-implements the NetLogo syntax in R. While this package can be used to rebuild NetLogo models in R, it can not be used to control already existing NetLogo models, which is the main purpose of the nlrx package.

[marcosci]

@nldoc I completely agree the lack of rJava dependency is a big plus... But can you comment more on the benefit of your pkg over RNetLogo with respect to features/functionality?

What Jan (@nldoc ) did not mention is the way the Java virtual machine is handled - nlrx opens a new one for each NetLogo model run, whereas RNetLogo keeps a single one open until you restart the R session. This can be quite cumbersome, in terms of memory, and should be solved with the way nlrx handles this.

[sckott]

thanks for the additional information @nldoc and @marcosci - we're discussing and will get back asap

[annakrystalli]

Editor checks:

• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

Editor comments

Hello @nldoc 👋,

Thanks for your package submission! I look forward to learning more about it during the review. I've completed the editors checks and there's a few issues to clear before we proceed.

All tests and checks pass. However:

test coverage

17% test coverage is generally too low to be accepted currently and will likely need more tests. Particular attention is required on the scripts with 0% coverage. Is there are reason these scripts aren't currently being tested?

covr::package_coverage(pkg_dir)

nlrx Coverage: 17.55%
R/class_init.R: 0.00%
R/download_netlogo.R: 0.00%
R/export_nl.R: 0.00%
R/get_nl_spatial.R: 0.00%
R/import_nl.R: 0.00%
R/run_nl.R: 0.00%
R/simdesign_helper.R: 0.00%
R/util_eval.R: 0.00%
R/util_runnl_dyn.R: 0.00%
R/util_runnl.R: 0.00%
R/util_stuff.R: 0.00%
R/zzz.R: 0.00%
R/class_setget.R: 33.33%
R/analyze_nl.R: 94.02%
R/class_constr.R: 100.00%

Good Practice

goodpractice::gp(pkg_dir)

It is good practice to:

• not use "Depends" in DESCRIPTION, as it can
  cause name clashes, and poor interaction with other
  packages. Use "Imports" instead.

• avoid long code lines, it is bad for
  readability. Also, many people prefer editor windows that
  are about 80 characters wide. Try make your lines shorter
  than 80 characters. Have a look at styler pkg.

  R/class_init.R:37:1
  R/class_init.R:38:1
  R/class_init.R:39:1
  R/class_init.R:40:1
  R/class_init.R:41:1
  ... and 98 more lines
  

• avoid calling setwd(), it changes the global
  environment. If you need it, consider using on.exit() to
  restore the working directory.

  R/export_nl.R:39:3
  R/export_nl.R:40:11
  R/export_nl.R:48:3
  

• not import packages as a whole, as this can
  cause name clashes between the imported packages. Instead,
  import only the specific functions you need.

• fix this R CMD check WARNING: LaTeX errors when
  creating PDF version. This typically indicates Rd
  problems.

  LaTeX errors found: ! Paragraph ended before
  \Rd@code was complete. <to be read again> \par l.100 !
  Paragraph ended before \Rd@code was complete. <to be read
  again> \par l.125
  

• avoid 'T' and 'F', as they are just variables
  which are set to the logicals TRUE and FALSE by
  default, but are not reserved words and hence can be
  overwritten by the user. Hence, one should always use
  TRUE and FALSE for the logicals.

  R/analyze_nl.R:NA:NA
  R/analyze_nl.R:NA:NA
  

spell check

I've pulled out a few typos that looked real. Might be worth running spell check to satisfy yourself too.

devtools::spell_check(pkg_dir)

  WORD              FOUND IN
anlyzing          description:3
contatining       simdesign.Rd:20
reproducability   import_nl.Rd:25

The biggest issue currently is the test coverage. How long do you think it will take to develop new tests? I can begin looking for reviewers but we'll really need the tests to be complete before they begin and it's best to contact reviewers when the package is ready for review. Let me know if you'd like me to put the review on hold or contact reviewers if you think you can address the issues swiftly.

[annakrystalli]

@nldoc @marcosci Apologies! I only just realised this was a pre-submission enquiry. So the package has been approved for review! 🎉 If you would like to go ahead, you already have the editors checks. Just let me know how you would like to proceed and sorry for jumping the gun!