robotstxt

[petermeissner]

• 1. What does this package do? (explain in 50 words or less)

Web scraping allows to gather information of scientific value - mainly social science related in my experience. While scraping web pages one should respect permissions declared in robots.txt files.
The package provides functions to retrieve and parse robots.txt files. The core functionality is to check a bots/users permission for one or more resources (paths) for a given domain. To ease checking all functions have been bundled with relevant data into an R6 robotstxt class but everything works functional or object oriented depending on the users preferences.

• 2. Paste the full DESCRIPTION file inside a code block (bounded by ``` on either end).

Package: robotstxt
Type: Package
Title: A 'robots.txt' Parser and Webbot/Webspider/Webcrawler Permissions Checker
Version: 0.1.0
Author: Peter Meissner
Maintainer: Peter Meissner <retep.meissner@gmai.com>
Description: Class ('R6') and accompanying methods to
    parse and check 'robots.txt' files. Data fields are provided as
    data frames and vectors. Permissions can be checked by providing
    path character vectors and optional bot names.
License: MIT + file LICENSE
LazyData: TRUE
BugReports: https://github.com/petermeissner/robotstxt/issues
URL: https://github.com/petermeissner/robotstxt
Imports:
    R6 (>= 2.1.1),
    stringr (>= 1.0.0),
    httr (>= 1.0.0)
Suggests:
    knitr,
    rmarkdown,
    dplyr,
    testthat
Depends:
    R (>= 3.0.0)
VignetteBuilder: knitr
RoxygenNote: 5.0.1

• 3. URL for the package (the development repository, not a stylized html page)

https://github.com/petermeissner/robotstxt

• 4. What data source(s) does it work with (if applicable)?

robots.txt files like:

• https://en.wikipedia.org/robots.txt
• https://github.com/robots.txt
• 5. Who is the target audience?

Package developers and users that want an easy way to be nice while gathering data from the web.

• 6. Are there other R packages that accomplish the same thing? If so, what is different about yours?

None that I know of.

• 1. Check the box next to each policy below, confirming that you agree. These are mandatory.
• This package does not violate the Terms of Service of any service it interacts with
• The repository has continuous integration with Travis and/or another service: https://travis-ci.org/petermeissner/robotstxt
• The package contains a vignette:
• The package contains a reasonably complete readme with devtools install instructions
• The package contains unit tests
• The package only exports functions to the NAMESPACE that are intended for end users
• 1. Do you agree to follow the [rOpenSci packaging guidelines]

Yes, good guidelines!

• Are there any package dependencies not on CRAN?

No.

• Do you intend for this package to go on CRAN?

With or without ropensci.

• Does the package have a CRAN accepted license?

yes, MIT

• Did devtools::check() produce any errors or warnings? If so paste them below.

no:

* DONE
Status: OK

R CMD check succeeded

• 9. Please add explanations below for any exceptions to the above:

Does not apply.

• 10. If this is a resubmission following rejection, please explain the change in cirucmstances.

No, no resubmit.

[sckott]

Thanks for the submission 🚀 - seeking reviewers

[sckott]

Reviewers: @richfitz @Ironholds

[Ironholds]

1. Unless you own gmai.com there's a typo in your DESCRIPTION file ;p.
2. Do you need the alternate df implementations? They're fun but also a lot of extra code to replace stringsAsFactors = FALSE ;P.
3. The assignment here seems extraneous.
4. "did apply" to "applied" here

As you may notice, most of these quibbles are well, quibbles. On the whole the package is excellent :). I'll try to put time into a more detailed review later today or early tomorrow.

[petermeissner]

Thanks for the ropensci infrastructure and thanks for the feedback - much-much-much appreciated.

in response to @Ironholds

1. Unless you own gmai.com there's a typo in your DESCRIPTION file ;p.

• ... that is why devtools::build_win() never called back ... (Windows build only complains about qpdf missing link1 link2)
• fixed

2. Do you need the alternate df implementations? They're fun but also a lot of extra code to replace stringsAsFactors = FALSE ;P.

• fixed

3. The assignment here seems extraneous.

• after checking extranous here and here ...
• yes, developement relict
• fixed

4. "did apply" to "applied" here

• fixed

question of my own

path_allowed() (link) will response to unforseen permission constallations with a warning and return all data that led to the problem.

• Do you think that design decision is wise/ok ??
• Is it better to return a warning and FALSE for each path?
• Should it stop with an error? I think not - when used by headless programs this might break them also the problem is not fatal/severe

[Ironholds]

I think that's totally fine behaviour, but I'd caution you, in that scenario, to use more formal/explanatory description text, noting that problematic data will be returned.

(On the subject of formality, working on a more formal review now)

[Ironholds]

On the whole: solid work! Some thoughts:

1. There are a lot of functions, such as rt_get_fields, which are documented but not exported. What this means in practice is that the code can't be found or used but the documentation does appear within the package index, which is pretty confusing. I'd suggest adding the "internal" roxygen2 keyword to the docs for those functions so that you don't index them.
2. The vignette is still using "Vignette Title" for indexing, so that's what CRAN will list it as (do not feel bad, I have made this mistake 150m times and never notice until I've already submitted it to CRAN and they've accepted >.>)
3. The vignette opens "The package provides a class (‘R6’)". This makes it sound like the class is called R6, at least to me.
4. The vignette could really benefit from a brief explanation of what robots.txt files are and why they matter at the top, with some examples, rather than just links at the bottom.

[petermeissner]

in response to @Ironholds II

1. drop unexported from manual

• done

2. vignette title / index entry

• done

3. The vignette opens "The package ...

• done

4. The vignette could really benefit from a brief explanation of what robots.txt files are and why they matter ...

• done

[petermeissner]

in regard to how path_allowed() handles problems with permission deduction

• path_allowed() will now warn more informative describing what happened:

  robotstxt::path_allowed() says:
  Encountered problems while trying to determine bot permissions, returning NA instead of TRUE/FALSE

• furthermore, path_allowed() will return NA in cases where the permission cannot be determined

[petermeissner]

Dear Oliver (@Ironholds ),

thank you very much for taking the time to review my package and comment on it. I think the package gained a lot from your suggestions and I myself learned a lot as well. I really appreciate the effort. I would like to add you as contributor - what is the ropensci policy on that matter - within DESCRIPTION and README, if that's ok with you.

[Ironholds]

I feel like I should say "the package looks really good now, sorry for not mentioning it earlier, this week has been nuts" before accepting or rejecting ;p.