Enhancement suggestion: RSS based updates

[doronbehar]

Hey,

I'm a relatively new NixOS user and one of my main motives to switch to it was this effort made by the community. You are a real hero @ryantm !

Ever since I started having my custom Arch Linux packages, I relied upon https://github.com/rss2email/rss2email along with GitHub's or GitLab's releases' atom feeds (e.g https://github.com/NixOS/nixpkgs/releases.atom).

My experience with this method of getting update notifications is rather good - I get automatic emails for new releases of packages I maintain and I edit the source code after every update.

I was wondering, was it ever considered to use these RSS urls to check for updates? BTW I'm not sure about automatic updates for Perl packages but cpan has RSS feeds for releases as well (e.g https://metacpan.org/feed/distribution/MIME-Tools).

[ryantm]

Cool! How did you learn about nixpkgs-update?

Currently, our only input source is Repology's API https://repology.org/, which I like because it provides a list of packages we are almost certainly out of date on. However, it lacks the ability to discover the latest version of package unless some other distribution or source has already updated. (It uses some interesting sources of versions, like Wikidata https://repology.org/repository/wikidata. )

The main input to the core update functionality is a triple of

${attrpath}, ${current/old version}, ${new version}

Any input source we can make like that would work with nixpkgs-update. Probably we can easily remove the requirement of the old version.

Thanks for pointing out that GitHub makes RSS feeds of projects. That might be simpler to use than GitHub's API. The project is already making use of GitHub's API, so I would probably still just try that first.

The CPAN link said "Please come back later".

Ryan

[doronbehar]

Cool! How did you learn about nixpkgs-update?

I've encountered that while browsing the forums..

The CPAN link said "Please come back later".

I've updated the example to a link that works.

The main input to the core update functionality is a triple of

${attrpath}, ${current/old version}, ${new version}

Any input source we can make like that would work with nixpkgs-update. Probably we can easily remove the requirement of the old version.

I've had an idea in the past to use rsstail (packaged) and write a crafty shell script to do on my own what nixpkgs-update does.. Do you think perhaps nixpkgs-update could leverage that? As opposed to implementing or using a Haskell RSS library to do it all in house..

Other external programs which may be of use are:

• https://github.com/skx/rss2hook - meant for Web hooks, not sure if arbitrary commands could be used instead..
• https://github.com/mackers/theyoke/ - seems to provide a very similar functionality to rsstail.
• https://castget.johndal.com/ - meant for automatically downloading artifacts of RSS feeds (audio / video files).

[ryantm]

Do you think perhaps nixpkgs-update could leverage that?

Maybe. My initial thought was to just use some XML parser library.

This discussion has got me more excited about this! If we could create a service that checks GitHub RSS feeds for all of the packages that use GitHub once a day, we could update a bunch of stuff immediately with almost no repository maintainer work.

[doronbehar]

This discussion has got me more excited about this! If we could create a service that checks GitHub RSS feeds for all of the packages that use GitHub once a day, we could update a bunch of stuff immediately with almost no repository maintainer work.

That's wonderful to hear :)

I'm no Haskell developer and I think I won't have enough time to learn it good enough for this project in the amount of time I have until I start university. I am however, rather apt as for writing a POC script that will use one of the external programs I listed above.

As for the actual design of it, I was thinking of either a daemon or a cronjob that could be deployed on a server and that will use either either castget (cron) or rsstail (daemon).

However, this script will need an external executable that will be based upon what nixpkgs-update does now.

Do you think you could write (perhaps in a seperate project / branch) a 'simpler' version of the current nixpkgs-update that will produce an executable that accepts as 2/3 arguments the attrpath and the new version? If you think this is a good path to go into, I'd be happy to write this script and use this executable.

[ryantm]

executable that accepts as 2/3 arguments the attrpath

It already basically works this way, but the instructions in the README could be better!

https://github.com/ryantm/nixpkgs-update has the instructions. Instead of following Step 2 make your own script that generates the packages-to-update.txt file. Here's an example content of the file:

zangband 2.7.4b 2.7.4c                                                                                                                                                                        
zgrviewer 0.9.0 0.10.0                                                                                                                                                                        
zxing 3.1.0 3.2.1

You also need to have a github_token.txt file in the same place that contains an API token (for Gists and PRs).

The PR will probably lie a little bit saying that the data came from Repology. Also, if you want, you could send me a list of packages that have new versions and I could run the list.

[doronbehar]

executable that accepts as 2/3 arguments the attrpath

It already basically works this way, but the instructions in the README could be better!

Boy then this can become really simple! I'd like to ask just a few more questions before I'll dive into this:

1. What if the program that updates the file packages-to-update.txt has multiple entries to a certain package, e.g:

zgrviewer 0.9.0 0.10.0
...
zgrviewer 0.10.0 0.11.0

Should it delete the old entry before adding the new one?

2. Is the process of nix run -c nixpkgs-update --update runs continuously or as a cronjob?
3. In any case of the above, do you thin it'll be best to implement my script which will replace your repology api script in a daemon or cronjob fashion?

[ryantm]

Should it delete the old entry before adding the new one?

Yes, because currently, it doesn't try to update a package if there is already an open PR to update it, so it wills kip the second one.

Is the process of nix run -c nixpkgs-update --update runs continuously or as a cronjob?

It runs through the whole packages-to-update.txt file and then stops. I usually run it manually so I can monitor the PRs that are being created.

daemon or cronjob fashion?

For now, you could probably make a cronjob that first generates the packages-to-update.txt file then calls nixpkgs-update.

[doronbehar]

Oh right, I think I get a better idea now as for how I'll implement this.. Since I will need to get a list of all packages that their src attribute is resulted from fetchFromGitHub, do you happen to have any advice on how to do that? I hope to avoid using an unreliable / overly complicated git grep command..

Oh and BTW, what does the r- in @r-ryantm stands for? Considering I might eventually deploy this script with my own bot account, I wonder how best to name it..

[ryantm]

do you happen to have any advice on how to do that?

Nothing really specific. I'd probably try to make a nix expression that generates a list of attrpaths that have src attributes that contain something specific to GitHub.

what does the r- in @r-ryantm stands for?

The r- stands for "robot". It's a reference to Asimov's convention to put R. at the beginning of the name of robots. For example, https://en.wikipedia.org/wiki/R._Daneel_Olivaw.

[ryantm]

Starting with something like https://raw.githubusercontent.com/NixOS/ofborg/released/ofborg/src/outpaths.nix from OfBorg https://github.com/nixos/ofborg might be good, since it can handle some of the complexities of the all packages attrset.

[doronbehar]

Thanks for the link, I'll look into that, I guess this part of my script will consume most of my time..

I'm trying to give a try to nix run -c nixpkgs-update with the flags --dry-run, just to get a feel of it and it fails with the following error:

exec: "getent": executable file not found in $PATH
nixpkgs-update: Received ExitFailure 1 when running
Raw command: hub clone nixpkgs /home/doron/.cache/nixpkgs

And I do have getent in my $PATH..

[ryantm]

Well, I just pushed a commit that adds getent to the runtime dependencies. Could you see if that helps?

[doronbehar]

Doesn't help.. same error. BTW, do you have to recompile it after every garbage collection? Or do you link a certain derivation to your /nix/var/nix/gcroots/per-user/$LOGNAME?

[ryantm]

What Operating System are you running it on? I'm running it on NixOS.

BTW, do you have to recompile it after every garbage collection?

I'm not sure. When I run it, it runs for so long that I haven't really been paying attention to how long it takes to start up.

[doronbehar]

I am using NixOS. getent is found here: /run/current-system/sw/bin/getent.

[doronbehar]

Are you using the command documented in the README? i.e nix run -c nixpkgs-update --dry-run --update? I tried also running the same command inside a nix-shell and it failed with:

error: unable to exec 'nixpkgs-update': No such file or directory

[ryantm]

I just tried doing it again from a different directory like this:

$ git clone git@github.com:ryantm/nixpkgs-update.git                                                                                                                          
Cloning into 'nixpkgs-update'...                                                                                                                                                              
Enter passphrase for key '/home/ryantm/.ssh/new':                                                                                                                                             
remote: Enumerating objects: 63, done.                                                                                                                                                        
remote: Counting objects: 100% (63/63), done.                                                                                                                                                 
remote: Compressing objects: 100% (45/45), done.                                                                                                                                              
remote: Total 1921 (delta 29), reused 39 (delta 15), pack-reused 1858                                                                                                                         
Receiving objects: 100% (1921/1921), 364.65 KiB | 2.12 MiB/s, done.                                                                                                                           
Resolving deltas: 100% (1328/1328), done.                                                                                                                                                     
$ cd nixpkgs-update/                                                                                                                                                          
$ ls                                                                                                                                                           
default.nix  LICENSE  nixpkgs-update.nix  package.yaml  README.md  src  test                                                                                                                  
$ nix run -c nixpkgs-update --dry-run --update                                                                                                                 
[3 built, 0.0 MiB DL]                                                                                                                                                                         
nixpkgs-update: github_token.txt: openFile: does not exist (No such file or directory)

Do you want to message me on IRC or something and we can try to work this out?

[ryantm]

Oh, I just realized your error message is about hub cloning nixpkgs. One possible theory is that it can't find your fork of nixpkgs. Can you try running GITHUB_TOKEN=XXXXXX nix-shell -p gitAndTools.hub --run 'hub clone nixpkgs /home/doron/.cache/nixpkgs' by itself?

[doronbehar]

I think the problem was that hub needed to ask an ssh passphrase somehow due to my git configuration and it failed because it didn't run inside a real terminal. I've cloned the repo manually to ~/.cache/nixpkgs and ran various commands that added and fetched the relevant remotes and now the nix run command at least doesn't fail.

I tried running it with the following packages-to-update.txt:

polybar 3.3.1 3.4.0

It's a real update BTW that needs to be done..

I don't want to actually create a PR with it so I used the --dry-run flag as follows:

nix run -c nixpkgs-update --dry-run --update

And it printed the following:

Cleaning up auto-update/polybar
Couldn't delete auto-update/polybar
Cleaning up auto-update/polybar
Couldn't delete auto-update/polybar

I won't mind moving the discussion to Riot or Telegram, as I'm too young for IRC 😎 but however, it's getting rather late here in my time zone and I'd like to keep working on this at another time. My Riot id is: @doronbehar:matrix.org.

[ryantm]

Okay, that makes sense; that cloning part needs to be improved, or dumped back on the user.

These "clean up" and "couldn't delete" messages are normal. Did you have any other issue after that?

I don't use Riot or Telgram. I am ryantm on Keybase. I also use Signal sorta, and Google Hangouts mostly.

[doronbehar]

I'll set this up tomorrow and send you a message.

[ryantm]

For this, we'll probably want to filter the updates by https://github.com/repology/repology-rules so we don't do development updates.

[doronbehar]

The main purpose for this issue was to enable automatic updates for GitHub and GitLab Repos. Now this is working (right? at least for GitHub I think...). As for other projects which may use a custom RSS feed for declaring releases, perhaps it'll be better to open a different issue for supporting automatic updates for such projects? I'm thinking it's not worth the effort, as such RSS feeds are an edge case I haven't encountered yet...