add support to API for params: "sort_field", "sort_order"

[warren-bank]

very minor patch that adds some much-needed functionality to the API.

please test this before merging..
my PHP is very rusty..
nearly 20 years rusty..
but it seems like a very straight-forward update.

[notartom]

So I'm trying to test this with the following URL (on a staging server):

https://librivox.org/api/feed/audiobooks?author=twain&limit=2&sort_field=id&sort_order=desc

That doesn't seem to change anything. Whatever I put as the sort_order, I get ID 59 first, followed by ID 141.

If you're willing to put some time into this, the Ansible playbooks that we use (https://github.com/LibriVox/librivox-ansible) can be used to set up local development environments (so that you can test and iterate locally), and there's an anonymized database dump available to play around with. It's not perfect, but others have done it before.

[warren-bank]

Shoot. Thanks for testing. I don't miss PHP.

I'll try to figure out how to run this thing in Laragon.
If I can get a portable version going that can simply be unzipped and run.. I'll let you know.

update:

• the server requires several config files.. but I can't find templates, which could be renamed and then configured.. do any such templates exist? The config files that I'm referring to include (but might not be limited to):
  • ./application/config/development/config.php
  • ./application/config/development/database.php
• ultimately, I just want to configure the server to use an instance of mysql running at localhost

update:

• found them

[warren-bank]

so.. I got a server running at localhost.
I migrated a minimal subset of the apache mod-rewrite rules to nginx.
The paths that redirect to wordpress aren't currently working.. probably because I didn't bother to configure wordpress.
More importantly, the paths that redirect to the catalog do work.

And.. drum roll.. it worked perfectly for me.
Meaning that sort field and sort order produced the expected search results.

1. limit 5, ascending
   • query
   • output:

       {
         "books": [{
           "id": "59",
           "title": "Adventures of Huckleberry Finn",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "141",
           "title": "Connecticut Yankee in King Arthur's Court",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "160",
           "title": "Chapters from my Autobiography",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "276",
           "title": "Eve\u2019s Diary",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "367",
           "title": "Prince and the Pauper",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }]
       }

2. limit 5, descending
   • query
   • output:

       {
         "books": [{
           "id": "18424",
           "title": "Meine Reise um die Welt. Erste Abteilung",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "18065",
           "title": "Querkopf Wilson",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "18035",
           "title": "Prince and the Pauper (Version x - Dramatic Reading)",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "17637",
           "title": "Im Gold- und Silberland",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }, {
           "id": "17423",
           "title": "Mark Twain's Letters from Hawaii",
           "authors": [{
             "id": "9",
             "first_name": "Mark",
             "last_name": "Twain",
             "dob": "1835",
             "dod": "1910"
           }]
         }]
       }

Rather than dumping a bunch of scripts and config files here..
I think I'll just make a new repo..
that way it'll be easier to replicate..
and maybe somebody more interested in the wordpress side of things can fix that and make a PR.
I'll post a link shortly..

[notartom]

Thanks for doing the legwork! Can you post what URL you hit to get those results? I can double check on my end as well, this is very strange indeed.

[warren-bank]

The "query" links point to my local server;
Laragon automagically modifies the "hosts" file to handle local DNS of its test servers.

But with regards to the querystrings.. and the PHP code that produces the output.. nothing has changed in the PR.
I'm just working with a snapshot.

[garethsime]

Sorry about the drive-by comment, but I just thought I'd leave a comment to say that this seems to be working on my local as well, I tried a bunch of variations on the link @notartom posted and it all seemed reasonable:

https://librivox.org/api/feed/audiobooks?author=twain&limit=2&sort_field=id&sort_order=desc

(Note that my localdev is not very standard either though - I'm jamming the ansible stuff into Docker, and there are a few rough edges.)

[notartom]

OK, not sure what was going I before (I suspect weirdness between Firefox, /etc/hosts, and/or cache), but trying with curl everything works as expected.

I think this is fine. I'm a bit weary of changing the API, but as this is purely additive, I think we're OK and are not going to break any existing code that's out there in the wild that's consuming the API. The default sort order is unchanged (it is currently the MariaDB default of ASC); because there's a hardcoded limit of 50 there's no possibility of unintentional DOS by asking to sort all of the books by the un-indexed text field title... I think this is good.

[garethsime]

because there's a hardcoded limit of 50 there's no possibility of unintentional DOS by asking to sort all of the books by the un-indexed text field title

That's a good point though, whether the limit is 50 or 10,000, the database will still need to compare all the rows to figure out the sort order if it's not indexed. I don't think any of the new sort fields are indexed

[warren-bank]

OK.. looks like I'm late to the party 😃

• here is a repo that has an unzip-and-run local server for Win64
  • the Wordpress side of things got absolutely no love
  • the catalog works
• I had also considered the potential performance impact of sorting on a field that isn't indexed
  • since I'm not aware of the schema, or which fields are (or are not) indexed..
    • figured I'll leave the decision to you..
    • adding or removing permitted sort fields is simple enough..
      there's an array of field names for that exact purpose
  • personally, I only care about sorting by id.. so anything else is gravy

[notartom]

OK, do you mind if for now I push a small change that limits sorting to just the id, because we do have an index on that? I'll leave a comment explaining why there's only id for now, and how we can go about adding more sort_field possibilities.

Edit: c9e6c00

[warren-bank]

I'd absolutely understand and agree.. my only question is whether date_catalog should be allowed to remain.. since it is already used as a sort field here.. to get a list of the most recent books.

The flip side.. is that if there isn't already an index of this field, then there absolutely should be.

[warren-bank]

off topic, but just wondering..
regarding my little standalone server for localhost:

• I see what I did wrong with Wordpress.. easy enough to fix
• my question is whether there exists a database dump of the librivox_blog table?
  • even an old one would be adequate for a local test environment
  • without one.. I don't see much point in installing Wordpress

[notartom]

When I put some effort into making localdev kinda work, my assumption was that WordPress would be initialized with an empty database. There isn't a lot (or any, really) dev work to be done on the WordPress side. The only custom stuff there is the theme, and we haven't touched that in years.

[warren-bank]

gotcha. In that case, I think I'll remove it from the main install script.. and add an optional install script specifically for Wordpress + theme.. same as is the case for PhpMyAdmin. (not that anyone will likely ever use it)

update: FWIW, that's all done, and releases has an updated prebuilt server that includes all of the options: PhpMyAdmin, WordPress, and the Librivox Wordpress theme. Seems to work quite well.