New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Atom feeds of per-maintainer outdated packages and problems #308

Closed
hartwork opened this Issue Aug 27, 2017 · 24 comments

Comments

Projects
None yet
3 participants
@hartwork

hartwork commented Aug 27, 2017

Hi!

Awesome project, thanks for that! It would be cool if there were Atom feed versions of

so that I could use my RSS-to-email bridge to get notified of things I should fix/update, automatically.
Are there plans to implement that?

Thanks and best, Sebastian

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Aug 29, 2017

That's actually the primary goal for repology development. However, it still requires some huge dependent tasks to be done first, such as implementing delta updates. Currently, whole repology database is rebuilt from scratch on each update, so there's no way to generate changes between "old" and "new" states, which could be then served in the form of feeds.

@hartwork

This comment has been minimized.

hartwork commented Aug 29, 2017

If the feed would contain entries in chronological order with stable IDs, the diffing could be done by the RSS client, as that is already doing that. So maybe the backend doesn't need diffing for that. What do you think?

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Aug 30, 2017

There's no chronology (as the whole database is created from scrach) and there are no stable IDs (the set of packages which comes from upstream is basically random, with arbitrary order, duplicates, renames etc.). Also such lists (thousands of packages or problems) would be too large for clients to handle.

@hartwork

This comment has been minimized.

hartwork commented Aug 30, 2017

I expect the per-maintainer lists to be less than thousand entries for humans, maybe even less than 50 in most cases. I would expect a feed reader to be able to handle that and consider it their problem (or a reasons to switch the client) if it does not. What do you think?

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Aug 30, 2017

Still too big.

  • You don't want 1000 items pop up when you add the feed.
  • You don't want to fetch hundreds of kilobytes each

Unfortunately, this will have to wait before the inner workings are refactored. After that we'll have a clean chronological feed of detailed events such as (package XXX max version bumped to YYY in repo ZZZ).

@hartwork

This comment has been minimized.

hartwork commented Aug 30, 2017

I don't follow in detail, but I'm looking forward to that feature. Good luck!

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 17, 2018

We are finally pretty close to implementing this. After #582 is done, all we need is a new state per maintainer+repo table, a corresponding events table, a couple of triggers and a couple of web views.

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 20, 2018

I'd like to hear opinions of how the feed should look like.

The mandatory part is like this, I guess:

  • "Package firefox is outdated"

But the options of how that can be extended are numerous:

  • "Package firefox is outdated by version 1.1"
  • "Package firefox version 1.0 is outdated by version 1.1"
  • "Package firefox version 1.0 is outdated by version 1.1 in repository AUR"

Should each version bump be reported?

  • "Package firefox is outdated by version 1.1"
  • "Package firefox is outdated by version 1.2"
  • "Package firefox is outdated by version 1.3"

Should updates be reported?

  • "Package firefox is at latest version"
  • "Package firefox is at latest version 1.1"

Should other events be reported?

  • "New package firefox 1.1"
  • "Package firefox removed"

Should extra details be reported (may be useful if you monitor multiple repos)?

  • "Package firefox in repository Gentoo is now outdated"
  • "Gentoo: package firefox is now outdated"

How should we handle multiple versions?
Should we handle other statuses (ignored, incorrect)?

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 21, 2018

I myself, as maintainer, would prefer all options, e.g. "Gentoo: package firefox 1.1 is outdated by 1.2 in AUR", added/removed events, problematic (ignored/incorrect) events, and repeating outdated events on each newest version bump. I'd aim at that, not yet sure if these all can be implemented easily wrt required SQL machinery.

We'll also see if that's too verbose, in which case it could be trimmed and/or optionized.

Btw, multiple versions is not really an issue and will be handled nicely automatically, as normally only one of them is newest/outdated, and others are legacy and are just skipped. However under certain circumstances (related to flavors mostly, e.g. when there are foo-client 1.0.0, foo-server 1.0.1 which are merged into the same metapackage, there can be multiple outdated versions which all will be reported, which is nice too.

Additional note: we probably want package (not metapackage) names in the feed. And even more likely, origins (e.g. repo-specific unique package names, see #439, #527) which we don't support yet. But it's not a blocker and can be improved later. The feed is also less sensible to format changes than history and can be painlessly reset, as what matters for it are recent events and not a backlog.

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 21, 2018

Yes, problems were also requested in this issue. I'd prefer to not touch them, as whole problems infrastructure should be rewritten, it's too clumsy and crutchy to touch now. Also, we'll have the same issue as #512 here (e.g. we'll need links to original packages). May fix it the same way, though I don't fully like that solution. A room for improvement here as well.

@AMDmi3 AMDmi3 self-assigned this Apr 22, 2018

@davidak

This comment has been minimized.

Contributor

davidak commented Apr 25, 2018

But the options of how that can be extended are numerous:

  • "Package firefox version 1.0 is outdated by version 1.1 in repository AUR"

Is the repository of the newer version really helpful for a maintainer? Also what repo should be displayed when multiple have the latest version?

Should each version bump be reported?

Yes. A maintainer probably always want to be up to date.

Should updates be reported?

  1. No, because the maintainer knows when he has updated a package. It's duplicate information.
  2. Yes, since other people could update the package. At least in NixOS not only the official maintainer updates packages...

Should other events be reported?

Again, the maintainer knows when he has added a new package, but might not know his package was removed. So probably yes. We still could remove some events when it is too much.

Should extra details be reported (may be useful if you monitor multiple repos)?

Yes, but keep it sane. The relevant information should be visible instantly and not hidden in a bunch of irrelevant stuff.

How should we handle multiple versions?

Some big projects maintain multiple stable versions, so 1.0.1 and 2.1.1 could both have the latest security fixes. But we don't have that information... We only could assume that newly updated packages have recent versions.

Example: https://repology.org/metapackage/linux/information
Compare to: https://www.kernel.org/

We could scrape the information from the website somehow, but i think there is no way to guess what version is LTS or EOL.

So i think the best we can do is to mark the latest version and show every other as outdated. "Outdated" can mean that that version is still maintained, so a hint is needed on the site about that.

Should we handle other statuses (ignored, incorrect)?

Like you see in the linux versions, some repos contain development versions marked with grey background. They shouldn't assumed to be latest. So a status is needed for them. Also there are strange version numbers sometimes that also need some status. So yes, something like that is needed to handle such situations.

I myself, as maintainer, would prefer all options
We'll also see if that's too verbose, in which case it could be trimmed and/or optionized.

Yes, do it :)

I think it's also helpful to see if it's a major, minor or patch level update. Stable or LTS channels often don't do major version bumps.
Sure it can't be always detected, for example with date-based release numbers. Then only say "... is outdated by update to..."

Gentoo: package firefox 1.1 is outdated by minor version update to 1.2 in AUR

--

Also the entry should have a link to more informations or multiple links.

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 26, 2018

Is the repository of the newer version really helpful for a maintainer?

Depends. Maybe someone is less trustful for some repos, or is just curious.

Also what repo should be displayed when multiple have the latest version?

All of them obviously.

No, because the maintainer knows when he has updated a package. It's duplicate information.

Still useful to make sure it got propagated. There were multiple times I've forgot to actually do the commit of a prepared update, or I've missed svn fail with files out of date by switching to another work.

Yes, since other people could update the package. At least in NixOS not only the official maintainer updates packages...

True.

Yes, but keep it sane. The relevant information should be visible instantly and not hidden in a bunch of irrelevant stuff.

Well each feed entry has the title which should be as short as possible to fit into notificatios and phone screens without wrapping and trimming. But there's also a content part which can be as large as required. However, if we use text format (which I prefer for compatibility) it's not too readable, since there's no way to highlight names and versions.

Anyway, I haven't implemented any additional info in the first version.

Some big projects maintain multiple stable versions

Well branches were planned to solve this. They are still not fully implemented (just in the marginal form of a single devel branch) but it may be time to complete them. There are not too many projects with multiple supported branches, so I don't see to many problems with maintaining them manually, at least at start.

For now though just the highest version in the repository is counted.

I think it's also helpful to see if it's a major, minor or patch level update

That's not possible, since not all project adhere to semantic versioning, and there's no telling which actually do.

So, the initial version is ready for testing. I plan to test it on myself for a week and then add links to the feeds from maintainer page.

For you that'd be:

no events yet though.

@davidak

This comment has been minimized.

Contributor

davidak commented Apr 27, 2018

if we use text format (which I prefer for compatibility) it's not too readable, since there's no way to highlight names and versions

most readers should be capable of basic markup, right?

So, the initial version is ready for testing.

Amazing!

The Atom feed link contains "rss", what is kind of misleading. Is a RSS feed planned? I think more people know and use them.

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 27, 2018

most readers should be capable of basic markup, right?

Yes, but if they don't, people will probably see tags, which is ugly.

The Atom feed link contains "rss", what is kind of misleading. Is a RSS feed planned? I think more people know and use them.

My understanding is that regardless of actual feed format, rss term is used for them all and more widely known. For the format itself, atom is recommended as newer and more consistent. I don't see any need to add actual rss feed in parallel.

@davidak

This comment has been minimized.

Contributor

davidak commented Apr 27, 2018

My understanding is that regardless of actual feed format, rss term is used for them all

No, RSS and Atom are two different formats.

Both RSS and Atom are widely supported and are compatible with all major consumer feed readers. RSS gained wider use because of early feed reader support. Technically, Atom has several advantages

https://en.wikipedia.org/wiki/RSS#RSS_compared_with_Atom

Most software like blogging software just support both. Should be easy with some library.

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented Apr 27, 2018

No, RSS and Atom are two different formats.

I know that. The point is that it's common to just call them both rss.

Most software like blogging software just support both. Should be easy with some library.

It lists atom advantages. Since it's well supported, I see no point to support RSS.

@davidak

This comment has been minimized.

Contributor

davidak commented Apr 29, 2018

The point is that it's common to just call them both rss.

Really? I have never noticed that. Any examples?

@hartwork

This comment has been minimized.

hartwork commented Apr 29, 2018

I checkout out "my" feed https://repology.org/maintainer/sping%40gentoo.org/feed-for-repo/gentoo/rss now. My vote for mentioning the outdating upstream version 0.4.0 somewhere and to go with /atom or /feed for the URL rather than /rss.

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented May 3, 2018

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented May 3, 2018

and to go with /atom or /feed for the URL rather than /rss.

Done.

AMDmi3 added a commit that referenced this issue May 3, 2018

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented May 4, 2018

Package version feed is complete. Problems may be added to it later for as a part of large problems refactoring (related #196).

@AMDmi3 AMDmi3 closed this May 4, 2018

@hartwork

This comment has been minimized.

hartwork commented May 5, 2018

Thank you!

@hartwork

This comment has been minimized.

hartwork commented May 22, 2018

The feed has started to be my fastest source of outstanding package bumps, including some that I would have missed through all other channels. So this feature helps reduce delays on my end already. Pretty awesome, thanks again!

@AMDmi3

This comment has been minimized.

Member

AMDmi3 commented May 23, 2018

Cool, I'm happy it's useful! It'll become even more useful when we start looking for new versions at upstream directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment