Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upVectorize `archivePackages()` and `pruneRepo()` #93
Conversation
| ##' This function is still undergoing development and polish and may | ||
| ##' change in subsequent versions. | ||
| ##' | ||
| ##' @title Move older copies of packages to an archive | ||
| ##' @importFrom stringr str_extract |
eddelbuettel
Jun 5, 2020
Owner
Please remove / replace.
Please remove / replace.
| # respective "type" directory is scraped | ||
| if (is.null(version)) { | ||
| if (grepl("binary", type)) { | ||
| r_versions <- stringr::str_extract(list.dirs(gsub( |
eddelbuettel
Jun 5, 2020
Owner
I appreciate your offer to rewrite this and not use extra dependencies for a function that is a bit more tangential to the package.
And if and when you rewite please avoid excess line breaks. 100 cols are fine. You can see in the rest of the file that we often use close to 100 cols (and the one at 111 cols could be considered too long ... but it works so no need for a PR or change of working code).
I appreciate your offer to rewrite this and not use extra dependencies for a function that is a bit more tangential to the package.
And if and when you rewite please avoid excess line breaks. 100 cols are fine. You can see in the rest of the file that we often use close to 100 cols (and the one at 111 cols could be considered too long ... but it works so no need for a PR or change of working code).
|
Thanks, looks better now. |
| "mac.binary.mavericks", "win.binary" | ||
| ), | ||
| pkg = NULL, | ||
| version = NULL) { |
eddelbuettel
Jun 7, 2020
Owner
The suggested change is now simpler and more focussed. One thing some people have learned to not be cavalier about is the change if existing interfaces. pkg and version defaulted to "possibly empty" with a test for is.missing. Having NULL is similar but not the same, and it is different. There may be users out there with an existing script calling this (I am actually not entirely sure if/how anybody uses archivePackages() but I cannot know). The use case of 'called with no argument' is preserved so I guess I can call that good enough.
The suggested change is now simpler and more focussed. One thing some people have learned to not be cavalier about is the change if existing interfaces. pkg and version defaulted to "possibly empty" with a test for is.missing. Having NULL is similar but not the same, and it is different. There may be users out there with an existing script calling this (I am actually not entirely sure if/how anybody uses archivePackages() but I cannot know). The use case of 'called with no argument' is preserved so I guess I can call that good enough.
| dirs = list.dirs(gsub( | ||
| "/[0-9].[0-9]$", "", contrib.url(repopath, type)), | ||
| recursive = FALSE) | ||
| r_versions = substring(dirs, regexpr("[0-9].[0-9]$", dirs)) | ||
| } |
eddelbuettel
Jun 7, 2020
Owner
Still not my style of writing but then I just like others think how R Core writes code matters most, and i happen to use the same editor they use (or at least a number of them). So whatevs. Thanks for adhering to four space indentation though, that is appreciated. (And this repo is so small that I never bothered with ,editoconfig which may be overkill.)
Still not my style of writing but then I just like others think how R Core writes code matters most, and i happen to use the same editor they use (or at least a number of them). So whatevs. Thanks for adhering to four space indentation though, that is appreciated. (And this repo is so small that I never bothered with ,editoconfig which may be overkill.)
| file.rename(file.path(repodir, old$file), file.path(repodir, "Archive", old$package, old$file)) | ||
|
|
||
| invisible(NULL) | ||
| return(invisible(NULL)) |
eddelbuettel
Jun 7, 2020
Owner
Adding invibsible is good style now that df may be larger.
Adding invibsible is good style now that df may be larger.
|
@jangorecki Any comments? |
|
Also, and that is of course a little out there and not a "formal" requirement: how would you go about testing this? You told use several times already that you use it for mlr3 repo or subrepo so is the best answer "in prod" ? |
|
not a functionality that I am after so not really following how it address requirements |
| for (f in rmfiles) { | ||
| fullfile <- file.path(repodir, f) | ||
| unlink(fullfile) | ||
| stop("Unknown package type. Valid values are 'source', 'mac.binary', 'mac.binary.el-capitan', 'mac.binary.mavericks' and 'win.binary'.", call. = FALSE) |
jangorecki
Jun 7, 2020
Contributor
make sense to support "binary" as well.
https://stat.ethz.ch/R-manual/R-devel/library/utils/html/install.packages.html
make sense to support "binary" as well.
https://stat.ethz.ch/R-manual/R-devel/library/utils/html/install.packages.html
FelixErnst
Jun 8, 2020
Contributor
However, binary is not specific. binary means in this case all binary cases, win.binary, mac.binary and so on.
A logic for this is now implemented including also the broadest term both.
However, binary is not specific. binary means in this case all binary cases, win.binary, mac.binary and so on.
A logic for this is now implemented including also the broadest term both.
|
The PR is also a roxygen run behind. I just added that. |
|
And just for completeness, the littler script dratInsert.r still works with this code and just up a new Rcpp dev version 1.0.4.12 so the core functionality is still there too... |
Yes, and given the complexity in testing this, it is sufficient for me here. One could of course replicate a small drat structure on the fly during tests and then reset it's state at the end of the tests. However, testing the results automatically is also not that easy. Feel free to make additional changes, including indention/coding style. |
|
@FelixErnst This PR changes some of the code you added recently and extends it. As you, if I remember correctly, use this "in prod" could you take a look and (if you have the time) a quick test drive of the branch / pr ? |
|
I think that the comments made in the issue discussion maybe were not considered to the full extent. The redundancies and and short cuts taken are unnecessary and I think the issue, which is a really good and valid issue, can be solved more elegantly. I suggested certain aspects of, what my ideas would be in the branch on my fork. Have a look at it here: FelixErnst/drat@master...FelixErnst:vectorize-archive-prune edit: just to summarise what I think is a good solution for solving the issue and not just flapping some
These changes are included in the branch on my fork edit2: the code in this PR works as intended and solves the issue |
| if (!file.exists(archive)) { | ||
| if (!dir.create(archive, recursive = TRUE)) { | ||
| stop("Archive directory not found and couldn't be created\n", call. = FALSE) | ||
| for (type in type) { |
FelixErnst
Jun 8, 2020
Contributor
This is not good style and I would argue, that there is an underlying issue.
I would argue the following:
for is ok to be used, if there is a connection between each iteration.
- That is not the case here, since path for different types should be independent.
- types are overlapping(e.g.
binary and win.binary), which makes iteration over them not ideal
This is not good style and I would argue, that there is an underlying issue.
I would argue the following:
foris ok to be used, if there is a connection between each iteration.- That is not the case here, since path for different types should be independent.
- types are overlapping(e.g.
binaryandwin.binary), which makes iteration over them not ideal
| } | ||
| } | ||
|
|
||
| mkArchive <- function(x) { |
FelixErnst
Jun 8, 2020
Contributor
The same as above. The function should not need to be redefined. I know, that repodir changes, but good style, would the refactoring of the mkArchive function for an additional parameter
The same as above. The function should not need to be redefined. I know, that repodir changes, but good style, would the refactoring of the mkArchive function for an additional parameter
| @@ -35,93 +35,108 @@ | |||
| ##' be removed. | |||
| ##' @author Dirk Eddelbuettel | |||
| pruneRepo <- function(repopath = getOption("dratRepo", "~/git/drat"), | |||
| type = "source", | |||
| pkg, | |||
| type = c( | |||
FelixErnst
Jun 8, 2020
Contributor
There is not check for enforcing correct type values before starting the loop. Therefore incorrect types leave the repo in an "undefined" state, if the second or later type produce a stop condition.
There is not check for enforcing correct type values before starting the loop. Therefore incorrect types leave the repo in an "undefined" state, if the second or later type produce a stop condition.
| if (is.null(version)) { | ||
| if (grepl("binary", type)) { | ||
| dirs = list.dirs(gsub( | ||
| "/[0-9].[0-9]$", "", contrib.url(repopath, type)), |
FelixErnst
Jun 8, 2020
Contributor
I am not sure, but this regex does not masks . It works here, but I think /336 would also match. No problem here, but /[0-9]\\.[0-9]$ would be better
I am not sure, but this regex does not masks . It works here, but I think /336 would also match. No problem here, but /[0-9]\\.[0-9]$ would be better
|
Felix, lot's can be made more sophisticated here, one could always add more assertions or tests. It works for the job that is should do and I also do not see much danger in wrong arguments inputs from users as the usage is pretty much tied to a properly structured drat repo anyways. I do not have more time budget to refine this PR substantially. Happy to give you push access to the fork if you want to do so.
I'm not sure what this means here - does it invalidate some of your previous comments? |
|
Hi Patrick @pat-s, The code in your PR, works as intended. So my comments are just about the logic in the code, which I think need to be improved, so that the code can be modified in incremental steps from this point forward, so that user contributions can be made easily. In R, in my opinion, and this just me talking, if you use the broad stroke functions I would be very happy to push the things I have in mind to your fork, if you like. |
|
to work on all R versions, |
|
I added fixed test packages in |
these test might break, when the R version gets bumped (probably April 2021)
|
@FelixErnst That is pretty awesome -- but it also broke the tests: Can you please look into restoring the package to passing tests. either by correcting what is currently broken (preferred) or removing broken tests (less ideal)? Also, can we slow down a little and maybe discuss what may get committed next and get agreement by all (or most) before just going off and doing it? We're almost there here and I really appreciate the input but the repo has gotten a little hyperactive. |
The error occurs because a certain version of the
Of course. I had the time and I was on a roll and really like the issues raised by @pat-s and the additional functionality if offers. I am done with the restructuring I had in mind. |
|
I see. The Travis setup uses the simple You can always cat a |
|
Slight thinko on my part. The existing |
|
hmm. the error persists. Can one get a hold of the status of the virtual machine from travis-CI? I encountered the same problem locally and fixed it, but without investigating what is going one travis-CI, I don't have any idea what might be going on here. I can of course remove the tests... |
|
I think we should remove the tests and table them for a later addition and get on with our lifes. I think you can come close to mimicking Travis CI by starting in a Ubuntu 18.04 "bionic" container, but it is not perfect. (Which is one of the reasons I went [in some larger repos] to fully Dockerized CI tests where Travis CI simply calls the Docker container I provide -- much better control over what is present and how. Overkill for |
| drat_.*\.tar\.gz$ |
eddelbuettel
Jun 9, 2020
Owner
Change may now not be needed but whatevs.
Change may now not be needed but whatevs.
| Version: 0.1.6 | ||
| Date: 2020-05-29 | ||
| Version: 0.1.6.4 | ||
| Date: 2020-06-09 |
eddelbuettel
Jun 9, 2020
Owner
I told @pat-s for "mucking" with Version and Date as that is what the release manager aka maintainer does. But at least you used the pattern I use which is a minimal delta increment (whereas he forcefully suggested that his changes should bump immediately to 0.2.0 which is .... something we agreed to reverse). This is fine.
I told @pat-s for "mucking" with Version and Date as that is what the release manager aka maintainer does. But at least you used the pattern I use which is a minimal delta increment (whereas he forcefully suggested that his changes should bump immediately to 0.2.0 which is .... something we agreed to reverse). This is fine.
| Author: Dirk Eddelbuettel with contributions by Carl Boettiger, Neal Fultz, | ||
| Sebastian Gibb, Colin Gillespie, Jan Górecki, Matt Jones, Thomas Leeper, | ||
| Steven Pav, Jan Schulz, Christoph Stepper and Felix G.M. Ernst. | ||
| Steven Pav, Jan Schulz, Christoph Stepper, Felix G.M. Ernst and Patrick | ||
| Schratz. |
eddelbuettel
Jun 9, 2020
Owner
Similar. My project. At the end of the day shouldn't I decide whom I anoint as a contributor? No need to change now, but please consider it next time.
Similar. My project. At the end of the day shouldn't I decide whom I anoint as a contributor? No need to change now, but please consider it next time.
FelixErnst
Jun 9, 2020
•
Contributor
I completely agree. However, I didn't want to highjack Patrick's idea. It was his and he did the hard work of coming up with it.
I completely agree. However, I didn't want to highjack Patrick's idea. It was his and he did the hard work of coming up with it.
| archivePackages) | ||
| pruneRepoForAllRversions, | ||
| archivePackages, | ||
| archivePackagesForAllRversions) |
eddelbuettel
Jun 9, 2020
Owner
I really want to be done with the PR but dropping new functions into the API without first discussing it is ... a little on the strong-willed side too. Here I am ok with your changes; the process was a little on the heavy side.
I really want to be done with the PR but dropping new functions into the API without first discussing it is ... a little on the strong-willed side too. Here I am ok with your changes; the process was a little on the heavy side.
FelixErnst
Jun 9, 2020
Contributor
Sorry for that. I didn't want to interfere with the expectation of current users, that more than R version would be targeted. I couldn't think of an alternative.
Sorry for that. I didn't want to interfere with the expectation of current users, that more than R version would be targeted. I couldn't think of an alternative.
eddelbuettel
Jun 9, 2020
Owner
I think the design is entirely defensible and possibly even good :) (I don't really use these archiver and pruner functions in my workflow so I am mostly 🤷♂️ here) but the process of dropping them unannounced is not one I prefer in my repos. That was all.
I think the design is entirely defensible and possibly even good :) (I don't really use these archiver and pruner functions in my workflow so I am mostly
FelixErnst
Jun 9, 2020
•
Contributor
Totally agree, However, I didn't drop anything, just slotted the additional functions in there at their appropriate places.
Totally agree, However, I didn't drop anything, just slotted the additional functions in there at their appropriate places.
|
See comments within. |
|
Lastly, I really like the code changes you brought. They help. We had bad luck with the test, and had to revert. Stuff happens. Now that we reverted, should these not be deleted? edd@rob:~/git/drat(vectorize-archive)$ tree inst/extdata/
inst/extdata/
├── 3.6
│ ├── foo_1.0.tgz
│ ├── foo_1.1.tgz
│ └── foo_1.2.tgz
├── 4.0
│ ├── foo_1.0.tgz
│ ├── foo_1.0.zip
│ ├── foo_1.1.tgz
│ ├── foo_1.1.zip
│ ├── foo_1.2.tgz
│ └── foo_1.2.zip
└── src
├── foo_1.0.tar.gz
├── foo_1.1.tar.gz
└── foo_1.2.tar.gz
3 directories, 12 files
edd@rob:~/git/drat(vectorize-archive)$ |
|
Nope. The CI problem was two fold.
The test works now as intended and the files listed above are used for testing. |
|
I will let this setlle for a bit and ponder, but it may now be ready for merge, maybe by tomorrow. Having an actual test is step up in rigour so thanks for putting it together! |
|
So after 65 comments and 21 commits we appear to be ready. |
5f7e03b
into
eddelbuettel:master
|
I haven't had time to follow up here. AFAICS there is now a new In addition, I tried |
|
In a week when I am back
…________________________________
From: Patrick Schratz <notifications@github.com>
Sent: Sunday, June 14, 2020 10:11:15 PM
To: eddelbuettel/drat <drat@noreply.github.com>
Cc: Felix Ernst <felix.gm.ernst@outlook.com>; Mention <mention@noreply.github.com>
Subject: Re: [eddelbuettel/drat] Vectorize `archivePackages()` and `pruneRepo()` (#93)
I haven't had time to follow up here.
Thanks for everyone pushing this forward.
AFAICS there is now a new archivePackagesForAllRversions(). Why is there an extra function needed to loop over different R versions? This is a quite unusual design decision.
In addition, I tried archivePackagesForAllRversions() on the example repo of the first post in this PR and the macOS binaries for R 4.0 did not get archived. @FelixErnst<https://github.com/FelixErnst> Could you have a look? Setting up the repo should only take a few seconds (see the snippet in the first post).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#93 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEH7UTCKH2TPUUUD7OGDBTTRWUVGHANCNFSM4NVAF4DA>.
|
|
So I looked into this
|
|
Thanks for taking the time.
The current repo state is ok but you are right, it had the wrong paths for the revision given in the first post. Seems to work as expected. |

Besides vectorizing both functions, I've also added the ability to archive packages for arbitrary R versions for every instance of "type".
By default (version = NULL), existing R versions within the given repopaths are checked and used.
Hence, by default all available "type"s are executed with all existing R version found in their respective directories.
I've already tested the approach on mlr3learners.drat in which we have 21 packages with binaries for different R versions.
So far it works fine.
I added two calls to {stringr} which could probably replaced by base R functions. If that's a blocker I can get rid of them. However, extracting substr with base R is not so nice because substr() does not support regexs.
If you want to try it out, you can do the following locally while having this branch installed (
remotes::install_github("pat-s/drat@vectorize-archive")):and then call
drat::archivePackages(".").