Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need help for changing the feed sort order from lastSeen to the publish date #2083

Open
hanan-cohen opened this issue Oct 27, 2018 · 17 comments
Labels
Documentation 📚 Feature Request ideas for new features FR: sort articles Sort feeds by published Date

Comments

@hanan-cohen
Copy link

Hello,

I am deploying a public feed aggregator and I will frequently add new feeds. The current sorting of the feed by lastSeen will look strange to the visitors.

I would like to change the sort order of the feed to be by the publishing date of the post (date) instead of by the date it was seen by the aggregator (lastSeen).

I searched the code and didn't find where to do that. Help will be appreciated.

Thanks.

@Alkarex
Copy link
Member

Alkarex commented Oct 27, 2018

Hello,
This is a topic that has been often times addressed, e.g. #1771 (comment)
The short version is:

  • If you let run FreshRSS for a few days with a refresh cron job (e.g. every 30min), the sorting will look better
  • Advertised date is not reliable enough to be useful, and using it creates many problems
  • Most other systems (including larger news aggregators but also e-mail clients etc.) are all using the same approach
  • Visitors are accustomed to items not strictly sorted by their declared / publication date, as it is also the case on popular systems such as Facebook, LinkedIn, Twitter, etc.
  • For our APIs (mobile clients) and for paging during scrolling, the guaranties given by the fact that no out-of-sequence article will be inserted allow for some important optimisations.

What we could easily do is to make an option to replace in the UI the advertised date by the discovery date, but I am not so found of the idea.

Feel free to post a link and/or screenshot, so that we can have a better idea of your precise situation, which depends on the feeds you are following, whether they support push (PubSubHubbub / WebSub) and if not, at which frequency you retrieve them.

@Alkarex
Copy link
Member

Alkarex commented Oct 27, 2018

P.S. the articles are sorted by the date they were first added to the database, not by the last seen date (which would not make sense).

@hanan-cohen
Copy link
Author

Thanks.

I have read #1771 and this is the reason I explained why I need the change.

If it was only for my personal use, that would be OK. Since it's a public website, the users will be confused by the sorting order when I add a new blog.

The live site is at https://blogim.info/

It's a mix of Enslish/LTR and Hebrew/RTL. Work in progress.

As for lastSeen, this is the name of the value in the DB for "date added". I will loot at the code at try to figure out how to sort by date (publish date).

@Alkarex
Copy link
Member

Alkarex commented Oct 27, 2018

No, lastSeen is the last seen date (at its name implies), not the date when the article was added.
Changing the SQL requests to modify the sort order is trivial, but that will break features such as scrolling / paging, mark all as read, etc.

I invite you to check the other systems that your users might be familiar with (e-mails, social media platforms, other RSS aggregators, Planet systems, social links...). E-mail clients (Outlook, GMail...) are using received date. Other RSS aggregators (including popular - current or past - ones such as Google Reader, Digg, Feedly, etc.), social media are using a multi-criteria ranking, so users are rarely exposed to a strict sort by publication date.

You can simply hide this publication date if it is a problem for you:
image
Or show the received date instead (like in an e-mail client) with a simple change in the code (we might make that an option).

We have a work-in-progress Hebrew translation, in which contributions would be welcomed https://github.com/FreshRSS/FreshRSS/tree/dev/app/i18n/he

@tetsuhou
Copy link

tetsuhou commented Dec 2, 2020

Hello, I also would like to be able to sort articles by the publishing date, but it seems difficult.

Would it be possible to add an option that allows me to add a new feed without getting the current item, but just the new article?

@ghost
Copy link

ghost commented Apr 21, 2022

Since this issue is still open, I'd like to add my support for this.

Sorting by date fetched tends to cause articles from the same source to get grouped together instead of having organic variety in feed content, unless you fetch at a higher frequency. That's mostly why I'd want this. It's not a huge deal obviously, but would be a nice improvement.

Couldn't something like this be put into place to at least make a best attempt at sorting the content by when it was published?

Sort By: If PublishedDate is determined to be well-structured/reliable, use PublishedDate else use FetchedDate

It doesn't have to be perfect, but I think it would help to distribute the feed content.

@olinorwell
Copy link

olinorwell commented May 28, 2022

I have to agree with @0x199x - sorting by PublishedDate then FetchedDate if PublishedDate is empty/'unreliable' would be a massive improvement. Even if it was optional, I think many people would immediately turn it on.

Currently I have a situation where I add a new personal blog for somebody, then at the top of my feed are effectively all the stuff they have ever written for the last few years, all listed as today, while I can clearly see the correct published date next to the title.

I think the issue here is that it depends on what type of RSS feeds you are subscribing to. If you are subscribed to hundreds and they are very frequently updated, then this issue wouldn't be so obvious due to the sheer number of items.

I subscribe to say 10 programmer blogs, who post every couple of months. All posts have a correct published date. If I add a new blog, I want the new blog's articles to sit in my 'timeline' on the correct dates they were published. Instead I get 'How to do XYZ, published May 2021' under 28 May 2022, 'How to do ABC, published Feb 2021', also under 28 May 2022, and so on, for typically 20+ items.

This seems like an obvious fix - when importing a post, take at face value the published date, and store that as the main date in the inbox. If it is empty, by all means use today.

@sjthespian
Copy link

For another use case for sorting by publication date, take a look at the feed at https://www.elephant.town/comic/rss. The author releases a new chapter every week of 25 pages or so. Since the pubDate is ignored, I'm ending up with a scrambled list of pages every week. For example, this is what I have after this week's update:
Screen Shot 2022-09-08 at 14 35 55

I basically end up ignoring FreshRSS, except as a notification of a new chapter, and have to read it on her web site.

@Alkarex
Copy link
Member

Alkarex commented Sep 9, 2022

For another use case for sorting by publication date, take a look at the feed at https://www.elephant.town/comic/rss. The author releases a new chapter every week of 25 pages or so. Since the pubDate is ignored, I'm ending up with a scrambled list of pages every week.

Interesting example @sjthespian . Actually, it should work just fine because FreshRSS does consider the publication date behinds the scenes when retrieving a batch of new articles. I have just tried and it worked fine for me.

image

I am not sure of where the problem comes from in your case, but I will try to keep an eye on it. It could be that the publications dates in the RSS feed were first wrong when you got them, and then later manually fixed. Or it might be that some chapters were published in a random order before some others.

With exactly one second of difference between the dates Sun, 28 Aug 2022 22:00:24 -0400, Sun, 28 Aug 2022 22:00:23 -0400, Sun, 28 Aug 2022 22:00:22 -0400 it looks like there is a manual process somewhere.

Anyway, I plan to add more sorting criteria in the near future, including alpha-numeric.

@Frenzie
Copy link
Member

Frenzie commented Sep 9, 2022

@Alkarex Actually 1.19.x and edge don't behave the same here. @sjthespian's screenshot is from 1.19.x or older, yours is from upcoming 1.20.

Edit: put another way, whatever that issue is, it's already fixed. ;-)

@limdingwen
Copy link

limdingwen commented Dec 28, 2022

EDIT: This does not work. See below for a better patch.

Here is a quick hack to enable published-date sorting in the web UI. Change the line in app/Models/Entry.php from dateAdded to date.

Or if you're up for some git apply:

From de80b56e3a5eff74c4fd49fe9dd917085a883452 Mon Sep 17 00:00:00 2001
From: Lim Ding Wen <limdingwen@gmail.com>
Date: Wed, 28 Dec 2022 17:33:49 +0800
Subject: [PATCH] HACK: Sort by published date

I have no idea why this works.
---
 app/Models/Entry.php | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/app/Models/Entry.php b/app/Models/Entry.php
index 47fcf3b4..14a24c5b 100644
--- a/app/Models/Entry.php
+++ b/app/Models/Entry.php
@@ -507,7 +507,7 @@ class FreshRSS_Entry extends Minz_Model {
 	}
 
 	public function isDay(int $day, int $today): bool {
-		$date = $this->dateAdded(true);
+		$date = $this->date(true);
 		switch ($day) {
 		case FreshRSS_Days::TODAY:
 			$tomorrow = $today + 86400;
-- 
2.30.2

0001-HACK-Sort-by-published-date.patch

@limdingwen
Copy link

@Alkarex Actually 1.19.x and edge don't behave the same here. @sjthespian's screenshot is from 1.19.x or older, yours is from upcoming 1.20.

Edit: put another way, whatever that issue is, it's already fixed. ;-)

From my testing it seems that Normal view sorts by Fetch Date first, followed by Publish Date if the Fetch Date is the same.

@limdingwen
Copy link

limdingwen commented Dec 30, 2022

Here's a better patch, however, no options menu for this yet because I don't need an options menu right now. This means this is not PR-ready, but hopefully this is useful to whoever wants this change.

This patch permanently changes the sorting method to published date. Tested with the web app and SQLite. Might not work with mobile apps.

  1. Make a backup of FreshRSS.
  2. In your copy of FreshRSS, type git apply and press enter.
  3. Copy and paste below.
  4. Press Ctrl-D.
From 03136e7e8890040c5539bfae55134b4205606104 Mon Sep 17 00:00:00 2001
From: Lim Ding Wen <limdingwen@gmail.com>
Date: Fri, 30 Dec 2022 17:50:15 +0800
Subject: [PATCH] Sort by published date instead of fetch date

Does not add an option for this.
---
 app/Controllers/indexController.php | 2 +-
 app/Models/Entry.php                | 2 +-
 app/Models/EntryDAO.php             | 6 +++---
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/app/Controllers/indexController.php b/app/Controllers/indexController.php
index 7fced48a..918235ec 100755
--- a/app/Controllers/indexController.php
+++ b/app/Controllers/indexController.php
@@ -87,7 +87,7 @@ class FreshRSS_index_Controller extends FreshRSS_ActionController {
                        if ($nbEntries >= FreshRSS_Context::$number) {
                                //We have enough entries: we discard the last one to use it for the next articles' page
                                ob_clean();
-                               FreshRSS_Context::$next_id = $lastEntry->id();
+                               FreshRSS_Context::$next_id = $lastEntry->date(true);
                        }
                        ob_end_flush();
                };
diff --git a/app/Models/Entry.php b/app/Models/Entry.php
index 47fcf3b4..14a24c5b 100644
--- a/app/Models/Entry.php
+++ b/app/Models/Entry.php
@@ -507,7 +507,7 @@ class FreshRSS_Entry extends Minz_Model {
        }
 
        public function isDay(int $day, int $today): bool {
-               $date = $this->dateAdded(true);
+               $date = $this->date(true);
                switch ($day) {
                case FreshRSS_Days::TODAY:
                        $tomorrow = $today + 86400;
diff --git a/app/Models/EntryDAO.php b/app/Models/EntryDAO.php
index cda51e5b..59d4065b 100644
--- a/app/Models/EntryDAO.php
+++ b/app/Models/EntryDAO.php
@@ -1050,7 +1050,7 @@ SQL;
                                throw new FreshRSS_EntriesGetter_Exception('Bad order in Entry->listByType: [' . $order . ']!');
                }
                if ($firstId !== '') {
-                       $search .= 'AND ' . $alias . 'id ' . ($order === 'DESC' ? '<=' : '>=') . ' ? ';
+                       $search .= 'AND ' . $alias . 'date ' . ($order === 'DESC' ? '<=' : '>=') . ' ? ';
                        $values[] = $firstId;
                }
                if ($date_min > 0) {
@@ -1123,7 +1123,7 @@ SQL;
                        . ($type === 't' || $type === 'T' ? 'INNER JOIN `_entrytag` et ON et.id_entry = e.id ' : '')
                        . 'WHERE ' . $where
                        . $search
-                       . 'ORDER BY e.id ' . $order
+                       . 'ORDER BY e.date ' . $order
                        . ($limit > 0 ? ' LIMIT ' . intval($limit) : ''));      //TODO: See http://explainextended.com/2009/10/23/mysql-order-by-limit-performance-late-row-lookups/
        }
 
@@ -1138,7 +1138,7 @@ SQL;
                        . 'INNER JOIN ('
                        . $sql
                        . ') e2 ON e2.id=e0.id '
-                       . 'ORDER BY e0.id ' . $order;
+                       . 'ORDER BY e0.date ' . $order;
 
                $stm = $this->pdo->prepare($sql);
                if ($stm && $stm->execute($values)) {
-- 
5.30.2

If the copy-paste doesn't work, try git apply <file name> from this instead:

0001-Sort-by-published-date-instead-of-fetch-date.patch

@redactedscribe
Copy link

Just installed FreshRSS. First thing I wanted was my imported feeds to be sorted by published date as a whole (not by published date and grouped based upon the time the feed was added). As a first impression, it makes the program seem like it doesn't order things correctly (even though I assume it does once FreshRSS starts to pull the feeds in daily).

@aofilhox
Copy link

aofilhox commented May 1, 2023

Hello everybody!
I made things easy for us:

Hugs to all!

Antonio Filho
A guy who simplifies problems.


Thanks to @limdingwen - I used your code/solution. :)

@limdingwen
Copy link

Thanks for simplifying things, @antonioof! 😉 I’m glad to have helped.

@ThallesLazaro
Copy link

Here's a better patch, however, no options menu for this yet because I don't need an options menu right now. This means this is not PR-ready, but hopefully this is useful to whoever wants this change.

This patch permanently changes the sorting method to published date. Tested with the web app and SQLite. Might not work with mobile apps.

  1. Make a backup of FreshRSS.
  2. In your copy of FreshRSS, type git apply and press enter.
  3. Copy and paste below.
  4. Press Ctrl-D.
From 03136e7e8890040c5539bfae55134b4205606104 Mon Sep 17 00:00:00 2001
From: Lim Ding Wen <limdingwen@gmail.com>
Date: Fri, 30 Dec 2022 17:50:15 +0800
Subject: [PATCH] Sort by published date instead of fetch date

Does not add an option for this.
---
 app/Controllers/indexController.php | 2 +-
 app/Models/Entry.php                | 2 +-
 app/Models/EntryDAO.php             | 6 +++---
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/app/Controllers/indexController.php b/app/Controllers/indexController.php
index 7fced48a..918235ec 100755
--- a/app/Controllers/indexController.php
+++ b/app/Controllers/indexController.php
@@ -87,7 +87,7 @@ class FreshRSS_index_Controller extends FreshRSS_ActionController {
                        if ($nbEntries >= FreshRSS_Context::$number) {
                                //We have enough entries: we discard the last one to use it for the next articles' page
                                ob_clean();
-                               FreshRSS_Context::$next_id = $lastEntry->id();
+                               FreshRSS_Context::$next_id = $lastEntry->date(true);
                        }
                        ob_end_flush();
                };
diff --git a/app/Models/Entry.php b/app/Models/Entry.php
index 47fcf3b4..14a24c5b 100644
--- a/app/Models/Entry.php
+++ b/app/Models/Entry.php
@@ -507,7 +507,7 @@ class FreshRSS_Entry extends Minz_Model {
        }
 
        public function isDay(int $day, int $today): bool {
-               $date = $this->dateAdded(true);
+               $date = $this->date(true);
                switch ($day) {
                case FreshRSS_Days::TODAY:
                        $tomorrow = $today + 86400;
diff --git a/app/Models/EntryDAO.php b/app/Models/EntryDAO.php
index cda51e5b..59d4065b 100644
--- a/app/Models/EntryDAO.php
+++ b/app/Models/EntryDAO.php
@@ -1050,7 +1050,7 @@ SQL;
                                throw new FreshRSS_EntriesGetter_Exception('Bad order in Entry->listByType: [' . $order . ']!');
                }
                if ($firstId !== '') {
-                       $search .= 'AND ' . $alias . 'id ' . ($order === 'DESC' ? '<=' : '>=') . ' ? ';
+                       $search .= 'AND ' . $alias . 'date ' . ($order === 'DESC' ? '<=' : '>=') . ' ? ';
                        $values[] = $firstId;
                }
                if ($date_min > 0) {
@@ -1123,7 +1123,7 @@ SQL;
                        . ($type === 't' || $type === 'T' ? 'INNER JOIN `_entrytag` et ON et.id_entry = e.id ' : '')
                        . 'WHERE ' . $where
                        . $search
-                       . 'ORDER BY e.id ' . $order
+                       . 'ORDER BY e.date ' . $order
                        . ($limit > 0 ? ' LIMIT ' . intval($limit) : ''));      //TODO: See http://explainextended.com/2009/10/23/mysql-order-by-limit-performance-late-row-lookups/
        }
 
@@ -1138,7 +1138,7 @@ SQL;
                        . 'INNER JOIN ('
                        . $sql
                        . ') e2 ON e2.id=e0.id '
-                       . 'ORDER BY e0.id ' . $order;
+                       . 'ORDER BY e0.date ' . $order;
 
                $stm = $this->pdo->prepare($sql);
                if ($stm && $stm->execute($values)) {
-- 
5.30.2

If the copy-paste doesn't work, try git apply <file name> from this instead:

0001-Sort-by-published-date-instead-of-fetch-date.patch

Hello everybody! I made things easy for us:

Hugs to all!

Antonio Filho A guy who simplifies problems.

Thanks to @limdingwen - I used your code/solution. :)

A fix in PHP 8

Anyone who has a problem with the error: PHP Fatal error: Uncaught TypeError: Cannot assign int to property FreshRSS_Context::$next_id of type string in /home3/gravad70/*****/app/Controllers/indexController.php:97

In the part you change it to: FreshRSS_Context::$next_id = $lastEntry->date(true);

Put strval() in $lastEntry->date(true) and it will work perfectly, see below how it will look:

FreshRSS_Context::$next_id = strval($lastEntry->date(true));

It has :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation 📚 Feature Request ideas for new features FR: sort articles Sort feeds by published Date
Projects
None yet
Development

No branches or pull requests