dockets aren't indexed for date_modified

[saizai]

E.g. https://www.courtlistener.com/api/rest/v3/dockets/?date_modified__gte=2014-10-30T07%3A30%3A44.561840Z&fields%21=resource_uri&order_by=date_modified%2Cid&page=9 responds at ~11 records per minute — slower even than 26 rec/min for a high page index, eg https://www.courtlistener.com/api/rest/v3/dockets/?fields%21=resource_uri&order_by=date_modified%2Cid&page=9120).

Contrast e.g. clusters & fjc-integrated-database, which load quickly for the same query (approx. 300-400 rec/min, and not slower with date_modified__gte). And opinions runs at about 60-75 rec/min either way (maybe slightly faster with the gte constraint).

I think it would also be helpful for performance to increase the page size. A query returning 100 records is probably less of a hit than 5 returning 20, given the high search penalty. I'd suggest the max page size be based on the payload size & RAM, i.e. such that it isn't so large that the server runs out of memory or the payload gets cut off due to network i/o loss. Not sure what exactly that number would be, but pretty sure it's not as low as 20.

[mlissner]

I don't think we can change the page size without doing a major new version of the API (though we could add a parameter for it, somehow, I suppose).

Performance of date_modified is probably more to do with the size of the table than with the absence of an index:

https://github.com/freelawproject/courtlistener/blob/master/cl/search/models.py#L376-L383

Not sure where that leaves us though.

[mlissner]

This is just an issue with the size of the table.