General problem uploading of package to drat repo

[RomanHornung]

Thank you for your help so far and, of course, for developing this important package.

I'm using Windows and drat version 0.1.8.1. I try to upload the R package BOLTSSIRR to a drat repo, because the new version of my CRAN R package uses this package and I want to reference the repo in the "Additional_repositories:" field of the DESCRIPTION file.

I started by forking the https://github.com/drat-base/drat page and enabled the docs subfolder there. Then I wrote "options(dratBranch="docs")" in R. Subsequently, in the folder "myfolder" I downloaded the drat repo using "git clone https://github.com/RomanHornung/drat.git". Then I tried to prune the repo using "drat::pruneRepo(repopath="myfolder/drat")" and then tried to add BOLTSSIRR using "drat::insertPackage("myfolder/BOLTSSIRR_0.1.0.tar.gz", "myfolder/drat/.git")", but unfortunately these last two commands did not seem to have changed the files.

What could I have missed here? Thank you very much!

[eddelbuettel]

Thanks for opening it here. This is better. This maybe lead to yet another bloody vignette :) with walkthrough. It was about time to switch from the (old, trusted, working -- but more cumbersome) gh-pages branch to using just docs/ as many of us do for other web-served content.

Using drat to support Additional_repositories: via Suggests: definitely works so let's get his worked out.

Cloning the new repo is great! I presume you clone drat-base/drat (you're the first, possibly, to do so!!) into RomanHornung/drat. One may to test that the 'run via docs/ subfolder is to attempt to access the new repo. If we browse by looking at it via the GitHub interface, we see for example the PACKAGES file in its full URL. Now, relative to docs/ this is course just src/contrib/PACKAGE (as drat/ is the top-level dir "served"). So let's try to see if we can wget that:

edd@rob:/tmp$ mkdir dratTest  # to make sure it is empty  
edd@rob:/tmp$ cd dratTest/
edd@rob:/tmp/dratTest$ wget https://github.com/RomanHornung/drat/src/contrib/PACKAGES
--2021-03-29 11:40:22--  https://github.com/RomanHornung/drat/src/contrib/PACKAGES                       
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-03-29 11:40:22 ERROR 404: Not Found.

edd@rob:/tmp/dratTest$ 

But if I check against one of my drat repos, here the one with test packages for Rcpp and friends, it works:

edd@rob:/tmp/dratTest$ wget https://RcppCore.github.io/drat/src/contrib/PACKAGES
--2021-03-29 11:45:30--  https://rcppcore.github.io/drat/src/contrib/PACKAGES
Resolving rcppcore.github.io (rcppcore.github.io)... 185.199.108.153, 185.199.109.153, 185.199.110.153, ...
Connecting to rcppcore.github.io (rcppcore.github.io)|185.199.108.153|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 746 [application/octet-stream]
Saving to: ‘PACKAGES’

PACKAGES                   100%[=====================================>]     746  --.-KB/s    in 0s      

2021-03-29 11:45:30 (83.9 MB/s) - ‘PACKAGES’ saved [746/746]

edd@rob:/tmp/dratTest$ 

So "something" is not quite right. If I look at the Settings for the RcppCore repo I see

Based on the failed wget above you do not have that working yet. We see a file PACKAGES yet GitHub does not serve it yet.

[RomanHornung]

Thanks for your fast reply! Yes, I also hope that you will be able to use this in your documentation, so it will be worth the effort beyond my specific use case.

So if I understand correctly, the first step would be to try to get the cloned version of the original repo working.

Actually, I had downloaded the repo to my local computer (using "git clone https://github.com/RomanHornung/drat.git"). I thought, I need to download (clone) the repo to my computer, prune it there, add the new package using insertPackage(), and upload the changes to GitHub.

The drat repo that is currently on my GitHub page is just a result of forking the https://github.com/drat-base/drat page and then setting up the docs subfolder under Settings -> GitHub pages -> Source.

Would I have to perform an additional step before the forked repo is ready?

[eddelbuettel]

We are talking past east other.

• My post demonstrates that the repo at https://github.com/RomanHornung/drat/ does not serve http (or https).
• You need a webserving site to serve content you arrange via drat. You do not have one.

How to set up GitHub pages is covered in a number of tutorials at GitHub. See if you can find one you like and follow it. I offered you a working litmus test to assert if you have it working. Until then, all work on drat is not yet meaningful.

[RomanHornung]

Ah, I hadn't understood that GitHub pages are something different than standard GitHub repos. Using a tutorial I set up a GitHub page without any content now: https://github.com/RomanHornung/RomanHornung.github.io

However, now I'm having difficulties forking the https://github.com/drat-base/drat page into the new GitHub page. When I go to https://github.com/drat-base/drat page and click "Fork" I get this https://github.com/RomanHornung/drat/. So the "Fork" does not go into the new Github page RomanHornung/RomanHornung.github.io.
How could this be changed?

[eddelbuettel]

Ah, I hadn't understood that GitHub pages are something different than standard GitHub repos.

Right. Our drat intro vignettes (which I rewrote / updated a little, so far only in the source repo, not the docs website) mention that but it requires that one "gets" that GitHub Pages is in effect website hosting. Hence drat use of it.

Also, error "There isn't a GitHub Pages site here." . You now have your own website at https://romanhornung.github.io/, I would put at least a one-liner "more coming soon" into index.html.

However, now I'm having difficulties forking the https://github.com/drat-base/drat page into the new GitHub page.

You cannot. A fork always does a complete copy of an existing repo and can hence not work with an existing repo at your end as it does add to a repo, it fully relaces. So you have to remove your existing repo drat, then fork, then turn on gh-pages for docs/ directory. (I think the last step is now and a "cost" of using docs/ rather than gh-pages so I need to document that clearly.)

[eddelbuettel]

Also, and just for completeness, you do not have to start with a fork. If you have a working GitHub repository (preferably named drat) you can call the initRepo() function on it. The forking method is a shortcut as people are impatient and don't always read (all) documentation (which is not a critique, I find myself doing the same -- we're all busy). It's just when taking short cuts you have to take the right ones, and possibly in the right order.

[RomanHornung]

Thank you! I now added text into the index.html file and it appears on https://romanhornung.github.io/.
I also activated the docs folder from "Settings -> GitHub Pages" and it says that the site is published at http://romanhornung.github.io/drat.

Subsequently, I successfully used "insertPackage(file="somepath1/BOLTSSIRR_0.1.0.tar.gz", repodir="somepath2/RomanHornung.github.io/.git")", because the package is now there in my local .git directory in the subdirectory "somepath2/RomanHornung.github.io/.git/src/contrib".

So I suppose the last step would be to upload these changes to the repo. However, unfortunately here I'm still stuck. I read in the vignette for package authors that the last steps are to add, commit, and push the changes. I'm using GitHub Desktop now. When I open GitHub Desktop it says "0 changes". So I cannot to commit the changes and push them to the repo.

How could I accomplish this? Thank you again, I think it will soon be done.

Maybe an issue is that GitHub Desktop does not recognize that there are new files in the ".git/src/contrib" subdirectory.

[eddelbuettel]

I don't use GitHub Desktop so you are on your own there. Most people use command-line git too, I recommend you become familiar with it.

You must commit locally. dratInsert does that. After that you need to push for which the remote repo has to be declared as a remote (and a fork gets that automatically). None of this is specific to drat so I recommend you just play a little with git.

Once the updated files (incl PACKAGES and PACKAGES.gz) are on the remote site (which now serves via https) you are done!

[RomanHornung]

I executed the script dratInsert.r now, but I'm not sure whether this changed something as the code ran very quickly.

After that I changed to the somepath2/RomanHornung.github.io directory in the Terminal of Rstudio and then used git push origin main, but it's telling me that everything would be up-to-date ("Everything up-to-date"). So I tried git commit, but this resulted in On branch main Your branch is up-to-date with 'origin/main'. nothing to commit, working tree clean.

I have no idea, how to continue here. Is there maybe a step I am missing between insertPackage() and trying to commit (using dratInsert.r or git commit)?

Or maybe I missed a step before, but if I'm not mistaken, I told you all steps I took so far.

[eddelbuettel]

I cannot teach you git over a sequence of messages in an issue ticket, that is something you need to do, and a (maybe implicit) requirement for use of drat. Learn how to assess the state of your local repo (git log and its options helps, as do GUIs) and do the same at the remote repo. If the files are at the remote, drat access should work.

There is not a lot more I can do for you here. I think we are at a point where we should consider closing the issue ticket as I don't really anything missing at drat the package --- apart from a "lower bar" for usage and that is not something I can add easily.

Sorry that this didn't yet work our for you,

[RomanHornung]

Thank you, nevertheless, for your time! I learned definitely something.

I understand that these are not issues specific to drat. However, drat seems to be the only possibility to let CRAN packages depend on non-CRAN packages. And I think this is an application that is probably very common. However, many authors of R packages coming from a purely statistical background likely have limited knowledge of informatics (like myself). For these it is probably hard to learn enough git to be able to do this on their own.

Given that the input "tar.gz file" and the output "package on drat repository" is always the same, I suppose that the process of putting an R package on a drat repo can be done by strictly following fixed steps that are always the same independent of the particular package considered. If such a step-by-step instruction "for dummies" would be available, I think this could have more R package developers use drat for letting their CRAN R packages depend on non-CRAN R packages.

[eddelbuettel]

However, drat seems to be the only possibility to let CRAN packages depend on non-CRAN packages.

Absolutely not. You aim to rely on a feature R offers: Additional_repositories and Suggests. None of that requires drat which "merely" helps to operationalise it. Now, if you are struggling with the help drat chances are doing it without drat will not be easier for you. But that is what it takes: at the end of the day you need to sit down and understand what a CRAN-accessible repository is (little, actually: an http-reachable director with subdirectories src/ and src/contrib with the files created by the R tools -- that drat calls for you). No more no less.

I sometimes joke that I wrote more documentation for drat than code. That still holds. Try and see what value you get out of five vignettes plus the associated ten-page R Journal paper about 'drat for data repositories'. If you find time to work through this, it would be terrific if you could crystalize bullet points into a short vignette. I, sadly, may not have the time to write this vignette for you.

PS For general git help people like https://happygitwithr.com/ which is already book-length. I can't do better than that in two paragraphs. Sorry.

[RomanHornung]

OK, I understand these points. I will try again and see if I can get the files on the repo consulting the documents you listed. If I succeed, I will definitely write down the steps I took and send that to you. If I don't get it done soon, I will put my package on GitHub for the moment, because I have to submit the paper that presents the method implemented in the paackage soon. But sooner or later I definitely want to manage to get the package on CRAN again. Thanks again a lot for your time and I will get in touch with you as soon as I managed to upload it to the drat repo.

[eddelbuettel]

One thought I had was that you could also

• play with a local webserver if you have one
• use netlify which people use for their blogs etc as, at the end of the day, you "just" need a https server for your files.

[RomanHornung]

Thanks for the tips! I don't think we have a local webserver. I will look into netlify as a plan B. But I will definitely try with drat again before. As I already have the files generated by drat locally, I just need to manage to upload them.

[RomanHornung]

I managed to upload the changes now... Sorry, I had made a stupid mistake before: I had written insertPackage(file="somepath1/BOLTSSIRR_0.1.0.tar.gz", repodir="somepath2/RomanHornung.github.io/.git") instead of insertPackage(file="somepath1/BOLTSSIRR_0.1.0.tar.gz", repodir="somepath2/RomanHornung.github.io"). For this reason the package got pushed into the hidden .git directory instead of into the romanhornung.github.io folder.

After fixing that I managed to add, commit and push the changes. They are now visible on RomanHornung/romanhornung.github.io in the same structure as on the https://github.com/drat-base/drat page:

But https://romanhornung.github.io/drat is still not there, https://github.com/RomanHornung/drat did not change and I was not able to install the package via install.packages("BOLTSSIRR", repos="https://romanhornung.github.io") or install.packages("BOLTSSIRR", repos="https://romanhornung.github.io/drat").

So in a nutshell, the files are on the romanhornung.github.io repo now, but there is no drat repo on https://romanhornung.github.io, although it says "Your site is published at https://romanhornung.github.io/drat/".

[eddelbuettel]

Why RomanHornung.github.io ? Now you always have to fully qualify the remote repo.

The one idea in drat is that for a user named 'abcde' at GitHub, if he/she forks the drat-base/drat repo creating a repo 'abcde/drat' we can then use if more directly to "autocompleting" to https://abcde.gitbub.io/drat.

That does not work with the structure you self-imposed, which is why you don't see them when you look for them. You committed and pushed to the wrong repo.

Please try to use the existing documentation.

[RomanHornung]

Thank you again. I hadn't pushed it to romanhornung.github.io on purpose. That was just, because I evidently used the git commands wrongly, but I will look git up more to get it to the right repo.

[eddelbuettel]

Sometimes it is easiest to just delete git repos and start over. You currently have no need for github.com/RomanHornung/romanhornung.github.io so let's just remove it from the conversation. For the same reason, delete the existing drat fork. Google how to delete a repo at GitHub. Run deletion for those two directories on your local machine.

Then start over: Fork drat-base/drat. It will be github.com/RomanHornung/drat. Ensure gh-pages serving is enabled.

Run git clone to bring the forked (remote) repository to you local machine.

Run dratInsert(...). Check the repo, it should have new files, git log should show a new transation in the local checkout.

Push from your local repo to your remote repo.

You should now be set.

[RomanHornung]

Thank you so much. Now the files are on https://github.com/RomanHornung/drat:

I was now able to install the package using install.packages("BOLTSSIRR", repos="https://romanhornung.github.io/drat"), so I guess it will pass the CRAN check.

I started writing down all (including minor) steps I did in consecutive order and will send this to you soon.

The only thing that still slightly worries me is that it says "404 File not found" on the pages https://romanhornung.github.io/drat/ and https://romanhornung.github.io/. But I guess, I will just have to add an index.html file to the drat repo for it to show something (but I don't think that's necessary, since the repo just exists to make it possible for CRAN users to install the package BOLTSSIRR that is required for my package). Also, this time I did not explicitly set up the romanhornung.github.io repo (GitHub page), but it seems to have been generated automatically even though it does not show up among my repos on the GitHub start page.

[eddelbuettel]

The only thing that still slightly worries me is that it says "404 File not found" on the pages

Yes, but that is "normal" or whatever GitHub Pages thinks is normal---it is a http server setting they enable to not permit "browsing" the directory content (which is the default on many sites). You can convince yourself that it is true by putting a (one-liner) index.html in either. Even a zero byte empty one is better than nothing because it gives you an empty screen rather a 404 error.

Very glad to hear you have it sorted out. Persistence is a usuful trait in dealing with computers, looks like you have it.

[eddelbuettel]

One useful thing to come out of all this is that you did test release 0.1.8.1 with support for docs/ 😀 So I think I should make that official soon-ish in a new 0.1.9 release given that I now also updated the pages at https://eddelbuettel.github.io/drat/ (which are of course served from drat's own docs/ directory).

[RomanHornung]

Cool, this sounds good that it was worth something beyond my specific problem! :)

I wrote down all steps I performed in detail. I don't know in how far you can publish what I've written... It probably contains many inaccuracies concerning the phrasings.

Here is the text (tried to upload it as an attachment, but that didn't work):

This step-by-step tutorial shows how to use drat to let a (to be) CRAN R package
depend on an R package only available on GitHub (or some other place other than
CRAN).

Requirements: GitHub account, R, RStudio, git

Preparing the R package for upload to the (not yet existing) new repository:

1. Download the R package that your (to be) CRAN package should depend on,
   but that is only available on GitHub (or some other place other than CRAN):
   Go to the GitHub page of the package, e.g. https://github.com/donaldduck/DUCKPACKAGE,
   and click "Code", then "Download ZIP".
   Extract the downloaded zip folder on your local computer.

2. In RStudio: Open the R project associated with DUCKPACKAGE:
   Click File -> Open Project , and then navigate into the unzipped folder
   from step 1 and open the Rproj file.

3. In RStudio: Build the source version and binary version of the package:

a) Build -> Build Source Package
b) Build -> Build Binary Package

This will generate two files "somepath2/DUCKPACKAGE_1.2.3.tar.gz" and
"somepath2/DUCKPACKAGE_1.2.3.zip" (the path "somepath2" will show in RStudio).

Making the new repository:

4. Make your own GitHub page:
   Log in to GitHub (https://github.com/): Click + -> New repository .
   Below "Repository" enter: "YourName.github.io", where you replace
   "YourName" with your user name shown below "Owner" on the left.
   Then click "Create repository" down on the page.

5. Go to https://github.com/drat-base/drat and fork that page by clicking
   the button "Fork".
   You will now have a copy of that repository named https://github.com/YourName/drat
   the content of which will below be replaced by your own content.

6. Go to "Settings" on https://github.com/YourName/drat and at
   "GitHub Pages" specify "master" below "Branch" and "docs" right of it and
   click "Save".

Uploading the R package to the new repository:

7. Go to the "Terminal" in RStudio (Next to the "Console" tab) and, using the
   "cd" command (google if you don't know it), navigate to an empty directory of
   choice you created before, e.g. "somepath1/GitHubFiles".

8. On the Terminal of RStudio execute:

git clone https://github.com/YourName/drat

Now you have the content of the forked repository on your computer in the
folder "somepath1/GitHubFiles".
NOTE: If the "git" command does not work, you have to install "git"
first, then restart RStudio and try again.

9. Delete all files in the folders
   somepath1/GitHubFiles/drat/docs/src/contrib
   somepath1/GitHubFiles/drat/docs/bin/windows/contrib/4.0
   These folders will later contain the files for the package that your (to be)
   CRAN package should depend on.

10. Install drat on the RStudio Console (!) by executing:

remotes::install_github("eddelbuettel/drat")

11. Execute on the Console:

options(dratBranch="docs")
drat::insertPackage(file="somepath2/DUCKPACKAGE_1.2.3.tar.gz", repodir="somepath1/GitHubFiles/drat")
drat::insertPackage(file="somepath2/DUCKPACKAGE_1.2.3.zip", repodir="somepath1/GitHubFiles/drat")

In the above "1.2.3" is a placeholder for the actual version number of the DUCKPACKAGE package.
This will add the DUCKPACKAGE package to the folders "somepath1/GitHubFiles/drat/docs/src/contrib"
and "somepath1/GitHubFiles/drat/docs/bin/windows/contrib/4.0".

12. Change to content of the file "somepath1/GitHubFiles/drat/README.md"
    to fit your purpose. The file can be deleted altogether alternatively.

13. In the Terminal of RStudio execute

cd drat

to get into the GitHub directory for drat.

14. If you use "git" for the first time, execute:

git config --global user.email "youremailaddressongithub"
git config --global user.name "YourName"

This will tell git your identity.

15. Then type:

git add .
git commit -m "Added DUCKPACKAGE"
git push origin master

This will upload the DUCKPACKAGE package to the repository on GitHub.

16. Test, whether the package can be installed from your new repository:
    On the R Console execute

install.packages("DUCKPACKAGE", repos="https://yourname.github.io/drat")

and verify that the package is installed successfully.

Prepare the DESCRIPTION file of your R package:

17. List the DUCKPACKAGE package under "Suggests:" and add the
    line "Additional_repositories: https://yourname.github.io/drat"

Now you are ready for submission to CRAN.

Optional:

(17.) Note that https://yourname.github.io/drat and https://yourname.github.io/
both will show "404 File not found". If you want to show empty pages there
instead, you should add an empty "index.html" to the repositories "drat" and "YourName.github.io":

a) Using an editor just make an empty file named "index.html" (file extension "html").

b) Copy "index.html" to the directory "somepath1/GitHubFiles/drat/docs".

c) On the Terminal of RStudio using "cd" go to the folder "somepath1/GitHubFiles/drat" and
execute:
git add .
git commit -m "Added empty index.html"
git push origin master

d) Still on the Terminal, using "cd", go to the directory "somepath1/GitHubFiles/" and execute:
git clone https://github.com/YourName/YourName.github.io

e) Copy "index.html" to the now new directory "somepath1/GitHubFiles/YourName.github.io".

f) Execute the following on the Terminal of RStudio:
cd YourName.github.io
git add .
git commit -m "Added empty index.html"
git push origin master

Now the websites https://yourname.github.io/drat and https://yourname.github.io/
should be empty instead of showing "404 File not found".

[eddelbuettel]

Thanks. I'll put this into an editable document and ping you. Is the ibe.med.uni-mue email the best to hit you with an invite to the document?

[RomanHornung]

Yes, this email is best, thank you for that and all your help!

[eddelbuettel]

Invite to a markdown document at https://hackmd.io (something I am experimenting with) sent. We can edit there. I backed it by a one-off git repo.

[eddelbuettel]

One or two things came up so this is not on the front burner but I will get to it. It is a really, really solid start which I think we should be able to use. We'll see.

[RomanHornung]

Thanks, that sounds good! :)

[eddelbuettel]

Please give it another read at the link I sent you, I made to rounds of edits. It is coming together.

[RomanHornung]

Looks really good!! I made minor changes (corrected typos) and added two comments.

[eddelbuettel]

Awesome. Just took and a snapshot, followed up there. One open edit issue. Do you want add it?

[RomanHornung]

I just looked at it, just removed the remark about "YourName.github.io" in "Additional optional step", and changed the status of the comments to "Resolved". For me, it looks good now, but let me know if you want to change it and I can double-check it!

[eddelbuettel]

Another good catch (also the one char type). Did you see the two lines I added re your previously opened issue?

I think we are done at our little 'skunkworks' and I will commit this here. Really nice work, and thanks for bootstrapping it all and the nice edits!

[RomanHornung]

Do you mean the new lines about the possibility of deleting drat in the cloned repository after adding the depended-upon R package? If yes, I saw these lines and they are fine from my point of view! Thank you also very much for your help and for taking my draft into account in this! I also think that it really looks nice now.

[RomanHornung]

Is the current (final) version of the vignette still on the HackMD page? I got the notification about your message about the (likely) final version, but on GitHub I can only read the html code of the vignette.

[eddelbuettel]

I made a PR so the actually-to-be-included version is in the branch the branch is based on. I did rename the file to match so it is this one now which I probably should have made more explicit (but the files changed view of a PR also helps).

[eddelbuettel]

I ended up merging it now just to move on -- it should be good (enough) to go and we can always fix as needed.

[RomanHornung]

Sorry for the delay, I looked at it again now and found
only these typos:

1. "This will generate are source file, say quacking_1.2.3.tar.gz."
   This should probably be "a source file" or "our source file".

2. "Optionally, change to content of the file drat/README.md"
   "the content"

3. "Note that you may have to say type="source" if your operating prefers"
   "operating system"

I wasn't able to make these changes on GitHub, so they
are not made yet.

From my point of view, apart from that, it should be done now :)

[eddelbuettel]

Thanks @RomanHornung and sorry I didn't wait for you which wasn't exactly polite (as I am trying to get 0.2.0 out of the door sooner rather than later). And thanks also for the typos. Pull requests are generally easier, but I will just add this now.

Now done in 65f35d5