[PP-3498] Add sort option to crawlable feed by dbernstein · Pull Request #3002 · ThePalaceProject/circulation

dbernstein · 2026-01-22T19:45:51Z

Description

This PR introduces a bug fix and a new feature.

Before the fix, the work.last_update_time was being updated whenever circulation data was imported, regardless of whether any data changed as a result. With this fix, we are updating that field only when a license pool's data has changed. Additionally I've added another field "last_updated" to the license pool itself. When data changes, this flag is updated. I've also added a new order facet to the crawlable search interface.

By specifying order=license_pool_last_updated in the crawlable url, the results will be order in reverse chronological order according to the licensepool.last_updated field. The new facet will avoid using the computationally intensive custom opensearch script that is used when ordering by the default order ("last_update"). In this case, the most recently updated licensepool associated with a work is used to determine the sort. Since the work.last_update_time is always updated with the same value as license_pool.last_updated flag when the license pool changes, the xml element (which maps to the Work.last_update_time in the feed should always reflect a timestamp that is at or after, but never before, the license pool update. (In the case that the bibliographic data is updated after the license pool, the element will reflect a date that is after the LP.last_updated.)

This PR is intended to address the needs of Aspen as well as reduce load

Sort in reverse chron order of changed license pools
Avoid using the costly opensearch custom sort script

As a side note: I confirmed that whenever availability is updated - whether by import or by checking in or out, license pool's last updated (as well as the work's) will be updated. No changes were required to fulfill that requirement.

Motivation and Context

https://ebce-lyrasis.atlassian.net/browse/PP-3498

How Has This Been Tested?

I've added tests, updated tests, and ran it locally both in my IDE as well as using compose.

Checklist

I have updated the documentation accordingly.
All new and existing tests passed.

codecov · 2026-01-22T21:29:39Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.99%. Comparing base (d995b16) to head (0b0f9d2).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3002   +/-   ##
=======================================
  Coverage   92.99%   92.99%           
=======================================
  Files         468      469    +1     
  Lines       43384    43417   +33     
  Branches     6034     6038    +4     
=======================================
+ Hits        40344    40376   +32     
  Misses       1967     1967           
- Partials     1073     1074    +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jonathangreen

@dbernstein, you set the ticket to code review in JIRA, so I started reviewing it. I then noticed it's still in draft and you haven't requested a review in GitHub yet. I'd already added some comments before I realized, so I figured I'd submit a review anyway.

jonathangreen · 2026-01-23T13:18:04Z

src/palace/manager/search/revision_directory.py

+from palace.manager.search.v7 import SearchV7

-REVISIONS = [SearchV5()]
+REVISIONS = [SearchV5(), SearchV6(), SearchV7()]


This is going to cause us to do a complete search re-index everywhere. That is okay, but you'll have to take care when rolling this out. Since it creates a completely new index it will use 2x the storage while the complete index is happening. Depending on our storage usage in each OS cluster this might pose some issues.

Also I don't think we automatically drop the old search revision, so after you roll this out and the index is complete you will also have to go and drop the old index to free up space in the opensearch index.

You'll have to keep all this in mind while you roll out this change.

jonathangreen · 2026-01-23T13:25:10Z

src/palace/manager/api/lanes.py

+        Facets.ORDER_FACET_GROUP_NAME: [
+            Facets.ORDER_LAST_UPDATE,
+            Facets.ORDER_LICENSE_POOL_LAST_UPDATED,
+        ],


After applying this PR. I don't see the new facet exposed in the crawlable feed, so clients have no way to discover it.

<link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=All&collectionName=All" rel="http://opds-spec.org/facet" title="Last Update" opds:facetGroup="Sort by" opds:activeFacet="true" defaultFacet="true"/> <link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=All&collectionName=All" rel="http://opds-spec.org/facet" title="All" opds:facetGroup="Distributor" opds:activeFacet="true" defaultFacet="true"/> <link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=Minotaur+Palace+Marketplace&collectionName=All" rel="http://opds-spec.org/facet" title="Minotaur Palace Marketplace" opds:facetGroup="Distributor"/> <link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=Blackstone+Unlimited&collectionName=All" rel="http://opds-spec.org/facet" title="Blackstone Unlimited" opds:facetGroup="Distributor"/> <link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=All&collectionName=All" rel="http://opds-spec.org/facet" title="All" opds:facetGroup="Collection Name" opds:activeFacet="true" defaultFacet="true"/> <link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=All&collectionName=TEST+Minotaur+Test+Library+Palace+Marketplace" rel="http://opds-spec.org/facet" title="TEST Minotaur Test Library Palace Marketplace" opds:facetGroup="Collection Name"/> <link href="http://localhost:6500/test/crawlable?entrypoint=All&order=last_update&available=all&distributor=All&collectionName=Blackstone+Unlimited" rel="http://opds-spec.org/facet" title="Blackstone Unlimited" opds:facetGroup="Collection Name"/>

jonathangreen · 2026-01-23T13:30:13Z

tests/manager/api/controller/test_crawlfeed.py

            if l["rel"] == "http://opds-spec.org/facet"
        }
-        assert facet_groups == {"Collection Name", "Distributor"}
+        assert facet_groups == {"Collection Name", "Distributor", "Sort by"}


We also need to test that the Sort By facet group has the members we expect

dbernstein · 2026-01-23T18:17:34Z

@jonathangreen : thanks for the review. I took care of the missing link and added a test to confirm. I also manually tested it to make sure everything looks good.

jonathangreen

Looks good!

…se and search index. Add ability to sort search results by that field.

…ue is being populated when works are indexed.

…hanged only when the data changes. Also add support license_pool_last_updated order facet value.

…feed.

dbernstein added bug Something isn't working feature New feature DB migration This PR contains a DB migration and removed bug Something isn't working labels Jan 22, 2026

dbernstein force-pushed the PP-3498-add-sort-option-to-crawlable-feed-take-2 branch from 2aa4c42 to 8bb3e8f Compare January 22, 2026 20:33

jonathangreen reviewed Jan 23, 2026

View reviewed changes

jonathangreen changed the title ~~[PP-3498] Add sort option to crawlable feed take~~ [PP-3498] Add sort option to crawlable feed Jan 23, 2026

dbernstein force-pushed the PP-3498-add-sort-option-to-crawlable-feed-take-2 branch from f6d1f2d to a296d48 Compare January 23, 2026 17:04

dbernstein marked this pull request as ready for review January 23, 2026 17:50

jonathangreen approved these changes Jan 23, 2026

View reviewed changes

dbernstein added 8 commits January 23, 2026 14:33

[PP-3498] Add support for LicensePool.last_updated in both the databa…

957d09d

…se and search index. Add ability to sort search results by that field.

Add last_updated field to search index schema and ensure that the val…

36e902d

…ue is being populated when works are indexed.

Add new sort order and tests.

6c0ad20

Ensure that licensepools.last_updated and work.last_update_time are c…

20f6f5c

…hanged only when the data changes. Also add support license_pool_last_updated order facet value.

Fix broken test.

bc662ba

Fix mypy

e6a6e37

Ensure that licensepool.last_updated is set on creation.

8140a47

Ensure the 'License Pool Last Updated' facet is present in crawlable …

e1dbb33

…feed.

dbernstein force-pushed the PP-3498-add-sort-option-to-crawlable-feed-take-2 branch from 7a80e0c to e1dbb33 Compare January 23, 2026 22:52

Fix formatting.

0b0f9d2

dbernstein enabled auto-merge (squash) January 23, 2026 23:40

dbernstein disabled auto-merge January 23, 2026 23:47

dbernstein merged commit 25287e5 into main Jan 23, 2026
19 checks passed

dbernstein deleted the PP-3498-add-sort-option-to-crawlable-feed-take-2 branch January 23, 2026 23:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PP-3498] Add sort option to crawlable feed#3002

[PP-3498] Add sort option to crawlable feed#3002
dbernstein merged 9 commits intomainfrom
PP-3498-add-sort-option-to-crawlable-feed-take-2

dbernstein commented Jan 22, 2026 •

edited by jonathangreen

Loading

Uh oh!

codecov bot commented Jan 22, 2026 •

edited

Loading

Uh oh!

jonathangreen left a comment

Uh oh!

jonathangreen Jan 23, 2026

Uh oh!

jonathangreen Jan 23, 2026

Uh oh!

jonathangreen Jan 23, 2026

Uh oh!

dbernstein commented Jan 23, 2026

Uh oh!

jonathangreen left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dbernstein commented Jan 22, 2026 • edited by jonathangreen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Checklist

Uh oh!

codecov bot commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jonathangreen left a comment

Choose a reason for hiding this comment

Uh oh!

jonathangreen Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

jonathangreen Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

jonathangreen Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

dbernstein commented Jan 23, 2026

Uh oh!

jonathangreen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dbernstein commented Jan 22, 2026 •

edited by jonathangreen

Loading

codecov bot commented Jan 22, 2026 •

edited

Loading