Skip to content

Wildbook 10.12: faster re-ID, fewer duplicates

Latest

Choose a tag to compare

@JasonWildMe JasonWildMe released this 18 Aug 18:09
· 21 commits to main since this release

Wildbook 10.12 is about the matching workflow: getting bulk re-identification through much faster and making sure that confirming a match does what you asked. Bulk re-identification can now run encounters in parallel, controlled by a new iaMatchThreads setting (the default of 1 keeps the old serial behavior). On a warm 50,000-candidate test pool this took throughput from about 9 to about 46 matches per second. On the correctness side, two old bugs that quietly created duplicate individuals are gone: confirming a positive match to an individual with a human-readable ID (like TZ-173) no longer clones it, and typing an existing ID into the encounter page's "Add to existing individual ID" field now attaches the encounter instead of minting a second individual with the same name.

Search got attention too. Quicksearch ranks by relevance again, individuals now inherit their species from their encounters (so quicksearch and individual search stop showing blank species lines), and filtering individual search by a subspecies returns that subspecies instead of every individual in the database. Bulk imports index encounters after their thumbnails exist, so newly imported encounters no longer show broken images in search results. Field reporters get their free-text locality box back on the React report page, entered dates and times are stored exactly as typed instead of shifting with the server's timezone, and logged-in researchers can once again edit encounters submitted by the public.

Under the hood, 10.12 fixes the database connection pool (a stock install was silently capped at 8 connections; the setting is now honored and defaults to 100), plugs several long-standing connection leaks, keeps Wildbook and WBIA annotations reconciled with a recurring sweep, makes shared search-query IDs survive restarts, and updates the default Docker memory settings to match what we run in production.

What's Changed

  • Allow logged-in users to edit public encounters by @JasonWildMe in #1655
  • Fix no-op single-id updateStandardChildrenBackground overload by @JasonWildMe in #1674
  • Shut down bulkOpensearchIndex executor after its single task by @JasonWildMe in #1675
  • Vector-match throughput: batch prospect loads, stateless match search, configurable IA concurrency by @JasonWildMe in #1676
  • Batch prospect encounter loads in match-result serialization (follow-up to #1676) by @JasonWildMe in #1677
  • Parallel bulk re-identification (iaMatchThreads, default 1) by @JasonWildMe in #1684
  • Match repo to optimized production settings by @JasonWildMe in #1670
  • Enable GET /api/v3/individuals/{id} with proper view ACL by @JasonWildMe in #1633
  • Migrate Social Diagram reads to secured /api/v3/individuals/info/social-data by @JasonWildMe in #1635
  • Fix bulk import index ordering after media derivatives by @JasonWildMe in #1673
  • Apply Search ID: durable DB-backed stored queries + frontend fixes (#1500) by @JasonWildMe in #1686
  • Validate search pagination against index max_result_window; retry queryPit only on stale PIT by @JasonWildMe in #1682
  • Restore matchability of spot-crop (unity) annotations on the React Encounter page by @JasonWildMe in #1656
  • Fix Shepherd/JDO connection leaks causing stuck dbconnections entries by @JasonWildMe in #1553
  • Fix legacy user accounts with email-as-UUID blocking project membership (#1545) by @JasonWildMe in #1546
  • fix(report): preserve civil encounter date-times by @JasonWildMe in #1688
  • Fix COCO export timeouts with temp file approach by @JasonWildMe in #1529
  • Add verbatimLocality field to React report encounter page by @JasonWildMe in #1687
  • fix(match): link to existing individual by primary key, not just UUID (#1305) by @JasonWildMe in #1689
  • fix(ia): exclude subspecies from Manual Annotation IA-class dropdown (#1298) by @JasonWildMe in #1690
  • Fix quicksearch ordering: rank by relevance, not id sort by @JasonWildMe in #1697
  • fix: resolve existing individuals by name when assigning IDs to encounters (#1318) by @JasonWildMe in #1698
  • Fix OpenSearch index staleness when encounters assigned to individuals by @JasonWildMe in #1547
  • fix(ia): restore species→iaClass mapping for spot-map annotations by @JasonWildMe in #1699
  • Fix silent no-op deep reindex for newly created individuals in assignFromIAAPI by @JasonWildMe in #1704
  • fix(config): size the DBCP2 connection pool with maxActive by @JasonWildMe in #1708
  • fix(individuals): inherit taxonomy from encounters so quicksearch shows a species (#1113) by @JasonWildMe in #1709
  • feat(ia): WBIA annotation reconciliation sweep by @JasonWildMe in #1710
  • fix(encounter): sort Individual ID typeahead results by name descending by @JasonWildMe in #1716
  • bugfix: allow for 2 or more taxonomy words (sub-species) in Individual search by @naknomum in #1719
  • tiny tweak to get sort order of names correct by @naknomum in #1725
  • Fix taxonomy-backfill JSP: DataNucleus misparses POSIX regex classes as named SQL parameters by @JasonWildMe in #1726