nlrx #262
Comments
|
thanks for your submission @nldoc we're discussing and will get back to you soon |
|
@nldoc I completely agree the lack of rJava dependency is a big plus. . But can you comment more on the benefit of your pkg over |
|
There's also https://cran.rstudio.com/web/packages/NetLogoR/ - not sure if you were aware or not? Can you also tell us benefit of your pkg over |
The main purpose of nlrx is running a NetLogo model and collecting output data. Everything is designed to make this as simple and reproducible as possible. Reproducability
Setting up experiments
Auxiliary functions
Summary |
The |
What Jan (@nldoc ) did not mention is the way the Java virtual machine is handled - nlrx opens a new one for each NetLogo model run, whereas RNetLogo keeps a single one open until you restart the R session. This can be quite cumbersome, in terms of memory, and should be solved with the way nlrx handles this. |
Editor checks:
Editor commentsHello @nldoc 👋, Thanks for your package submission! I look forward to learning more about it during the review. I've completed the editors checks and there's a few issues to clear before we proceed. All tests and checks pass. However: test coverage17% test coverage is generally too low to be accepted currently and will likely need more tests. Particular attention is required on the scripts with 0% coverage. Is there are reason these scripts aren't currently being tested? covr::package_coverage(pkg_dir)Good Practicegoodpractice::gp(pkg_dir)It is good practice to:
spell checkI've pulled out a few typos that looked real. Might be worth running spell check to satisfy yourself too. devtools::spell_check(pkg_dir)The biggest issue currently is the test coverage. How long do you think it will take to develop new tests? I can begin looking for reviewers but we'll really need the tests to be complete before they begin and it's best to contact reviewers when the package is ready for review. Let me know if you'd like me to put the review on hold or contact reviewers if you think you can address the issues swiftly. |
|
Dear @annakrystalli and @sckott, thank you very much for your encouraging response and first feedback.
We will develop tests for all functions (where possible) within the next two weeks. Unfortunately, some functions (especially those that call java via command line) can not be tested without providing an executable NetLogo version within the repository. However, including NetLogo in the package repository would need way too much space.
We solved most of these issues with our latest updates. The only issue that persists is the use of setwd() within the export_nl function. The function creates a zip file from a defined directory. If the zip file is created without setting the defined directory as working directory, the complete folder directory (back to the current working directory) is stored within the zip file. We did not find any other workaround beside setting the working directory via setwd(). If there are other solutions to create a proper zip file, we are happy to remove the setwd command. However, we followed the suggestion and added the on.exit() command to reset the working directory to the original directory.
As already mentioned, we will develop the remaining tests within the next two weeks. If there are no more issues than the ones stated above the package will be ready for review in two weeks at latest. Thus, we think it makes sense to already contact the reviewers. |
|
Thanks @annakrystalli for your efforts already 🎉 Regarding the tests: NetLogo is ~200 Mb big, so, unfortunately, we can not deliver it with the package. This also means that we can not properly test if running NetLogo itself works. Are you aware of a possibility to do that only on Travis (it could be skipped on CRAN then)? |
|
I would encourage you to look at how rtika handles this: https://github.com/ropensci/rtika . The package provides a function to install the large tika |
|
Thanks for your comments @noamross! @nldoc, re: Have you tried supplying the eg I think you could do: utils::zip(zipfile=outfile, files = list.files(folder, full.names = TRUE), extras = "-r") I also notice you are using eg you could do nltempfile <- file.path(nltempdir, "nlobject.rds") |
|
Dear @annakrystalli, |
|
Great! Thanks for your speedy efforts. I'll get on to the checks today and then contact reviewers. |
|
We have reviewers! 🎉 @nldoc could I ask that you add the review badge to your README? @marinapapa & @thomasp85 many thanks for agreeing to review! 🙏 Feel free to reach out with questions at any point. Reviewers: @marinapapa @thomasp85 |
|
I just added the review badge 👍 Very much looking forward to @marinapapa's & @thomasp85's comments, thank you for reviewing! |
|
Reviewers: @marinapapa @thomasp85 |
Package ReviewPlease check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
DocumentationThe package includes all the following forms of documentation:
Functionality
Final approval (post-review)
Estimated hours spent reviewing: Review CommentsDocumentationI feel the README could be a bit more detailed when it comes to describing what the package does. As it stands, you need to already be familiar with NetLogo to understand the need/use for the package. While a detailed description of NetLogo is not suited for a README, a small section before Installation would make the README more welcome to new users of actor based simulations in general. Further it would be good to link to the NetLogo online ressources to help users with researching further information. The README should also clearly state that you need to install NetLogo manually and how this is done. Omission of this indicates that NetLogo is installed as part of the package installation. The vignettes assumes the user is running on windows. While it may be impossible to create examples that run on all systems, as you have to point to the NetLogo installation it would be nice if the vignettes were structured in a way that made it easier to modify the code to your local system. This could e.g. be done by storing relevant paths in variables at the top of the vignette. I would like the Simdesing examples vignette to include more descriptive text to carry the reader through what is happening, and giving them an impediment to understand the steps. As it is now, it is mainly a collections of snippets to cope-paste (or at least it appears like that with the lack of motivation and discussion) Functions are in general well-documented. I think it would make sense to put FunctionalityAs discussed above, the assumption that NetLogo is already installed is not clearly communicated. The The test coverage is generally good, and it is nice to see that they have made a strategy for testing functionality that can't be tested on CRAN The package title needs to be in titlecase and the description shouldn't start with "The nlrx package..." (CRAN requirements, not mine). Apart from that the packaging guidelines seems to be followed (again, with the excdeption of the documentation discussed above) |
|
I hope it is as it should - otherwise I’m open to a review of my review🙂 |
Package ReviewHi all, here is my review. I really like your package so I hope my comments will help to improve it :)
DocumentationThe package includes all the following forms of documentation:
Functionality
Final approval (post-review)
Estimated hours spent reviewing: 15 Review CommentsIn general, I like the idea of the package and its functionalities. Some dependencies issues that were not stated made the review process a bit frustruating, but I think most of my points are very easy to tackle. Installation & Documentation
So it would be helpful if the information for the above points were stated in the installation guidelines/ README/ description.
Given that NetLogo is a program widely used for teaching, I think a bit more detailed help files chould be of great importance:
Functions:getstarted.Rmd
nl@experiment:
spatial-output.Rmd I couldn't get it to fully work. Two main problems:
And minor:
analyze_simple Renaming to no_analysis may be helpful (a 'simple' analysis is still an analysis). So far if the user runs analyze_nl with simple design they get NULL, which can be confusing. A message can be added to be printed in analuze_simple in order to inform the user that 'no analysis can take place if a simple design is chosen'. write_simoutput
util_runnl.R: load-model-parameters: General comments:
In total, I think it can be a very useful package. As I mentioned above, the package can gain a lot from improving its documentation. Also of course the variety of Netlogo models is very large so also by including another example of a quite different model may add value to it. |
|
And as @thomasp85 said, I m happy to review my review if needed 😊 |
|
Thank you very much for your detailed reviews @thomasp85 and @marinapapa |
|
Indeed thanks so much for the thorough review @marinapapa! @nldoc & @marcosci, over to you now. We normally ask to aim for responses to reviewers within two weeks but with the holidays coming, it is quite alright if you take some extra time over it. So wishing all in the thread Happy Holidays and enjoyable downtime! 🌟🎄🎅 |
annakrystalli
added
4/review(s)-in-awaiting-changes
and removed
3/reviewer(s)-assigned
labels
|
Hi Anna, Marina and Thomas - thanks a ton for the very nice reviews! Just trying to fix the issues @marinapapa had with the Edit: I figured it out - the library(nlrx)
nl <- nl(nlversion = "6.0.4",
nlpath = "/home/marco/Documents/rpackages/nlrx_usecase/1_Helper/NetLogo 6.0.4/",
modelpath = "/home/marco/Documents/rpackages/nlrx_usecase/1_Helper/NetLogo 6.0.4/app/models/Sample Models/Biology/Wolf Sheep Predation.nlogo",
jvmmem = 1024)
nl@experiment <- experiment(expname = "nlrx_spatial",
outpath="out/",
repetition = 1,
tickmetrics = "true",
idsetup = "setup",
idgo = "go",
idfinal = NA_character_,
idrunnum = NA_character_,
runtime = 100,
evalticks = seq(1,100),
metrics = c("count sheep","count wolves"),
metrics.turtles = c("who", "pxcor", "pycor", "breed"),
metrics.patches = c("pxcor", "pycor", "pcolor"),
constants = list("model-version" = "\"sheep-wolves-grass\"",
'initial-number-sheep' = 100,
'initial-number-wolves' = 50,
"grass-regrowth-time" = 30,
"sheep-gain-from-food" = 4,
"wolf-gain-from-food" = 20,
"sheep-reproduce" = 4,
"wolf-reproduce" = 5,
"show-energy?" = "false")
)
# Attach simdesign simple using only constants
nl@simdesign <- simdesign_simple(nl=nl,
nseeds=1)
#> Creating simple simulation design
# Run simulations and store output in results
results <- run_nl_all(nl = nl, cleanup = "all")
# Attach results to nl object:
setsim(nl, "simoutput") <- results
# Report spatial data:
results_spatial <- get_nl_spatial(nl,
turtles = TRUE,
patches = TRUE,
turtle_coords = "px",
format="spatial")
#> Joining, by = c("[run number]", "model-version", "initial-number-sheep", "initial-number-wolves", "grass-regrowth-time", "sheep-gain-from-food", "wolf-gain-from-food", "sheep-reproduce", "wolf-reproduce", "show-energy?", "random-seed", "[step]", "count sheep", "count wolves", "siminputrow")
devtools::session_info()
#> ─ Session info ──────────────────────────────────────────────────────────
#> setting value
#> version R version 3.5.1 (2018-07-02)
#> os Ubuntu 18.10
#> system x86_64, linux-gnu
#> ui X11
#> language en_US
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz America/North_Dakota/Beulah
#> date 2018-12-19
#>
#> ─ Packages ──────────────────────────────────────────────────────────────
#> package * version date lib source
#> assertthat 0.2.0 2017-04-11 [1] CRAN (R 3.5.0)
#> backports 1.1.2 2017-12-13 [1] CRAN (R 3.5.0)
#> bindr 0.1.1 2018-03-13 [1] CRAN (R 3.5.0)
#> bindrcpp * 0.2.2 2018-03-29 [1] CRAN (R 3.5.0)
#> callr 3.1.0 2018-12-10 [1] CRAN (R 3.5.1)
#> class 7.3-14 2015-08-30 [4] CRAN (R 3.5.0)
#> classInt 0.2-3 2018-04-16 [1] CRAN (R 3.5.0)
#> cli 1.0.1 2018-09-25 [1] CRAN (R 3.5.1)
#> codetools 0.2-15 2016-10-05 [4] CRAN (R 3.5.0)
#> crayon 1.3.4 2017-09-16 [1] CRAN (R 3.5.0)
#> DBI 1.0.0 2018-05-02 [1] CRAN (R 3.5.0)
#> desc 1.2.0 2018-05-01 [1] CRAN (R 3.5.0)
#> devtools 2.0.1 2018-10-26 [1] CRAN (R 3.5.1)
#> digest 0.6.18 2018-10-10 [1] CRAN (R 3.5.1)
#> dplyr 0.7.8 2018-11-10 [1] CRAN (R 3.5.1)
#> e1071 1.7-0 2018-07-28 [1] CRAN (R 3.5.1)
#> evaluate 0.12 2018-10-09 [1] CRAN (R 3.5.1)
#> fs 1.2.6 2018-08-23 [1] CRAN (R 3.5.1)
#> furrr 0.1.0 2018-05-16 [1] CRAN (R 3.5.0)
#> future 1.10.0 2018-10-17 [1] CRAN (R 3.5.1)
#> globals 0.12.4 2018-10-11 [1] CRAN (R 3.5.1)
#> glue 1.3.0 2018-10-29 [1] github (tidyverse/glue)
#> highr 0.7 2018-06-09 [1] CRAN (R 3.5.0)
#> hms 0.4.2 2018-03-10 [1] CRAN (R 3.5.0)
#> htmltools 0.3.6 2017-04-28 [1] CRAN (R 3.5.0)
#> knitr 1.21 2018-12-10 [1] CRAN (R 3.5.1)
#> lattice 0.20-38 2018-11-04 [4] CRAN (R 3.5.1)
#> listenv 0.7.0 2018-01-21 [1] CRAN (R 3.5.0)
#> magrittr 1.5 2014-11-22 [1] CRAN (R 3.5.0)
#> memoise 1.1.0 2017-04-21 [1] CRAN (R 3.5.0)
#> nlrx * 0.1.0 2018-12-19 [1] local
#> pillar 1.3.0 2018-07-14 [1] CRAN (R 3.5.1)
#> pkgbuild 1.0.2 2018-10-16 [1] CRAN (R 3.5.1)
#> pkgconfig 2.0.2 2018-08-16 [1] CRAN (R 3.5.1)
#> pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.5.1)
#> prettyunits 1.0.2 2015-07-13 [1] CRAN (R 3.5.0)
#> processx 3.2.1 2018-12-05 [1] CRAN (R 3.5.1)
#> ps 1.2.1 2018-11-06 [1] CRAN (R 3.5.1)
#> purrr * 0.2.5 2018-05-29 [1] CRAN (R 3.5.0)
#> R6 2.3.0 2018-10-04 [1] CRAN (R 3.5.1)
#> raster 2.8-4 2018-11-03 [1] CRAN (R 3.5.1)
#> Rcpp 1.0.0 2018-11-07 [1] CRAN (R 3.5.1)
#> readr 1.3.0 2018-12-11 [1] CRAN (R 3.5.1)
#> remotes 2.0.2 2018-10-30 [1] CRAN (R 3.5.1)
#> rlang 0.3.0.1 2018-10-25 [1] CRAN (R 3.5.1)
#> rmarkdown 1.11 2018-12-08 [1] CRAN (R 3.5.1)
#> rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.5.0)
#> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.5.1)
#> sf 0.7-1 2018-10-24 [1] CRAN (R 3.5.1)
#> sp 1.3-1 2018-06-05 [1] CRAN (R 3.5.0)
#> spData 0.2.9.6 2018-12-03 [1] CRAN (R 3.5.1)
#> spDataLarge 0.2.7.1 2018-09-16 [1] github (Nowosad/spDataLarge)
#> stringi 1.2.4 2018-07-20 [1] CRAN (R 3.5.1)
#> stringr 1.3.1 2018-05-10 [1] CRAN (R 3.5.0)
#> testthat 2.0.1 2018-10-13 [1] CRAN (R 3.5.1)
#> tibble 1.4.2 2018-01-22 [1] CRAN (R 3.5.0)
#> tidyr 0.8.2 2018-10-28 [1] CRAN (R 3.5.1)
#> tidyselect 0.2.5 2018-10-11 [1] CRAN (R 3.5.1)
#> units 0.6-2 2018-12-05 [1] CRAN (R 3.5.1)
#> usethis 1.4.0 2018-08-14 [1] CRAN (R 3.5.1)
#> withr 2.1.2 2018-03-15 [1] CRAN (R 3.5.0)
#> xfun 0.4 2018-10-23 [1] CRAN (R 3.5.1)
#> XML 3.98-1.16 2018-08-19 [1] CRAN (R 3.5.1)
#> yaml 2.2.0 2018-07-25 [1] CRAN (R 3.5.1)
#>
#> [1] /home/marco/R/x86_64-pc-linux-gnu-library/3.5
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/libraryCreated on 2018-12-19 by the reprex package (v0.2.1) |
|
Hello all and happy (belated) New Year! 💫 Just checking in to see how things are going. I also asked the editors about your query regarding the repeating warning message when running @noamross also provided some useful tips on how to troubleshoot and then set up a test to monitor the behaviour:
Let me know if you have any further questions! |
|
Happy New Year to you too! Let me give you a short update on the progress of our revisions. |
|
Thanks for the updates @nldoc! 👍 Have a nice weekend too. 😊 |
Detailed response to reviewersDear @marinapapa and @thomasp85, Reviewer comments by @thomasp85Documentation
Functionality
Reviewer comments by @marinapapa
Installation & Documentation
Functions:
General comments:
Cheers, |
|
Thank you @marcosci & @nldoc for the detailed response and improvements! @thomasp85 & @marinapapa , over to you. Please feedback on whether the responses have adequately addressed the issues you've raised. Feel free to reach out for any clarification. |
|
Dear Jan and Marco, I am sorry for not responding to your initial message about get_nl_spatial . It is indeed running smoothly now so no need for anything else :) The new documentation is great and all my initial issues are adequately addressed. I also like very very much the interfacing-different-models addition! Some comments:
I didn't try much to solve the errors, maybe it is something very straight forward. Everything else works nicely :) |
|
Hi Marina, thanks again for the comments! Cheers, PS: If you ever use gganimate for real flocking data or more complex models, please let me know :) would love to see that! |
|
Hi Jan and Marco I can only echo Marina in saying that all raised issues seems to have been solved. You have gone all-in on providing good documentation and guidance which warms my heart :-) I do not have any additional comments to the package as it appears now. best |
|
Approved! 🚀 Many many thanks @nldoc & @marcosci for submitting and being so responsive and @marinapapa & @thomasp85 for your most thorough and helpful reviews! 😺 To-dos:
Should you want to aknowledge your reviewers in your package DESCRIPTION, you can do so by making them Welcome aboard! We'd also love a blog post about your package, either a short-form intro to it (https://ropensci.org/tech-notes/) or long-form post with more narrative about its development. (https://ropensci.org/blog/). If you are interested, @stefaniebutland will be in touch about content and timing. We've started putting together a gitbook with our best practice and tips, this chapter starts the 3d section that's about guidance for after onboarding. Please tell us what could be improved, the corresponding repo is here. Feel free to submit to MEE next. 😁 |
|
Also, @marinapapa & @thomasp85, would you mind checking any remaining check boxes in your review comment to indicate formal acceptance? Many thanks again! |
|
Many thanks to you, @annakrystalli, @marinapapa, @thomasp85 and the whole rOpenSci community. We are very glad that nlrx is now a part of it 😃. We already took care of the listed To-dos. However, the link to our webpage at the top of the repository page is still pointing to the nldoc repository and it seems I don't have the user rights to edit the link (I don't see the edit button). We would also like to add some useful tags to the header of the repository. It would be great if you could set the user rights for me or @marcosci. @marinapapa and @thomasp85, if you don`t mind we would like to add you as reviewers in our package Description file. Are you fine with that? |
|
One last thing, I see you are using lifecycle badges (👍). However, at the minute you've got it set to Experimental which is defined as:
We'd expect an rOpenSci package to be at the very list at maturing stage:
Could you please confirm that you consider it to be at this stage? And if so could you please change the lifecycle to maturing? |
|
@annakrystalli that was just a remnant of the beginning of the package - I updated the badge, as we consider it quite stable now 👍 |
|
Thanks @marcosci . Well, that's a wrap! 😺 Thanks again all 👋 |
|
@nldoc more than fine with adding me to your Description file 😊 Thanks |
|
Did that, thanks again! |
|
Hello @nldoc and @marcosci and congratulations on passing peer review! Another one for you Marco. Are you interested in writing a blog post or tech note? This link will give you many examples of blog posts by authors of peer-reviewed packages so you can get an idea of the style and length you prefer: https://ropensci.org/tags/software-peer-review/. Technotes are here: https://ropensci.org/technotes/. Here are some technical and editorial guidelines for contributing a post: https://github.com/ropensci/roweb2#contributing-a-blog-post. Happy to answer any questions. |
Summary
nlrx enables you to run agent-based models in NetLogo from R.
This makes it possible to design reproducible experiments (which can be
zipped with nlrx and shared between persons). Furthermore, it
coerces the spatial data from NetLogo into raster and sf objects.
URL for the package (the development repository, not a stylized html page):
https://github.com/nldoc/nlrx
Please indicate which category or categories from our package fit policies this package falls under *and why(? (e.g., data retrieval, reproducibility. If you are unsure, we suggest you make a pre-submission inquiry.):
Who is the target audience and what are scientific applications of this package?
NetLogo. They now can use nlrx to parameterise NetLogo and scale their experiments up.
With nlrx it is the first also possible to run NetLogo experiments on computing clusters.
Are there other R packages that accomplish the same thing? If so, how does
yours differ or meet our criteria for best-in-category?
There is RNetLogo on CRAN, this package however is based on rJava (which is most often an obstacle) and has not as much auxiliary functions to run NetLogo and embed results in R objects.
Requirements
Confirm each of the following by checking the box. This package:
Publication options
paper.mdmatching JOSS's requirements with a high-level description in the package root or ininst/.Detail
Does
R CMD check(ordevtools::check()) succeed? Paste and describe any errors or warnings:Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:
If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:
The text was updated successfully, but these errors were encountered: