add an option to make page indexing more efficient

[rlskoeser]

we often need to reindex all page content in ppa, but when we hit an error that causes the script to crash we have to reindex everything

would be helpful for developer sanity to have an opt-in / non-default way to index only the content that needs it - maybe we can determine by comparing page counts in the database and solr? or maybe we can use modification time to figure out which works need pages indexed? or some combination of the two?

[quadrismegistus]

Would this general pattern work?

1. Get a map of {work_id} to its [{page_id}, {page_id}, ...] from django db. That gives us # pages per work too.
2. Query for # of pages per work_id in solr.
3. For those with mismatches, reindex. Or if --force or something set, reindex all.

If it's fast from solr to get a list of all page ids per work id, then you could do an even more careful comparison than # of pages by comparing sets of page ids to make sure they perfectly overlap.

[rlskoeser]

We can easily get a count of total pages / total works; should be able to do a facet query in solr to get number of page items per source id (group id?). Could filter by modification time as well so we only count recently indexed pages and index works that don't have enough recent pages

[rlskoeser]

Was able to use this script in qa to do partial reindex after the full page index crashed part way through. We still need to figure out why there's a discrepancy in page counts between db and solr, but we're tracking that on #567