v0.12.2

Release notes for pkdb 0.12.2

pkdb 0.12.2 collects the fixes of the study format 2 cutover of pkdb_data. pkdb migrate converts reproducibly with many worker processes and handles more images and registry entries, pkdb issues sync survives dropped GitHub connections and closes legacy issues reliably, pkdb review assign names a reviewer without approving the study, and the curator roster names the GitHub logins of the curators. The backend scopes its advisory locks to the current schema; stop every backend process of a deployment before the new one starts, because the lock keys change. The database needs no migration.

Changes

• pkdb review assign FOLDER USER names a reviewer of a study in review.json without changing its status (#901). Before, a reviewer was named only by approving the study, so a study under review had no reviewer and its issue no reviewer as assignee.

• pkdb issues sync sends a read or an edit of an issue again when GitHub drops the connection, after 2, 8 and 30 seconds (#900). The sync of the 849 converted pkdb_data studies stopped twice on such drops (RemoteProtocolError, ReadTimeout) and needed reruns; a new issue or comment is still never sent twice.

• The backend scopes its advisory locks to the current schema (#899). PostgreSQL advisory locks belong to the whole database, so the parallel backend tests, each in a schema of one test database, blocked each other: test_same_key_distinct_publications_overlap failed at random when the vocabulary bootstrap of another test took the exclusive vocabulary lock between its two publications. With one schema, as in production, the locks behave as before; their keys change, so stop every backend process of a deployment before the new one starts.

• The curator roster names the GitHub logins of 13 more accounts and fixes three, so that pkdb issues sync can assign the issues of their studies (#898): FlorBar (was the typo FloBar), michelle-elias for mii-halina, beastemmt for stemllb, KathleenGreen for kgreen, and the logins of drosdekv, linafrolova, jonathanspeh and six accounts of a 2022 student group. The group account xresearch is no longer in the roster or in the curation app; pkdb_data no longer names it. pkdb-server import-users --apply now also corrects a GitHub login or ORCID that an earlier import set, so a corrected roster reaches existing accounts; a value the user edited stays.

• The worker processes of pkdb migrate space their PubMed and DOI requests together (#895). Each worker kept the interval of 0.36 seconds that PubMed allows only for itself, so a run with many workers sent PubMed many times more requests than it allows; studies whose reference.json had to be created or replaced then failed at random with pubmed unavailable (HTTP 429) as not_converted.

• The local curation app scans large workspaces with far less CPU (#893). The scan each second lists the entries of a folder only after the folder changed, and once every 30 seconds otherwise, and still reads the size and modification time of every source file. On Linux, a scan of 1600 study format 2 folders takes about 35 ms of processor time instead of about 480 ms, and one of a pkdb_data checkout with format 1 folders about 12 ms instead of 160 ms. Where adding, removing or renaming a file changes the modification time of its folder, as on the local disks of Linux, macOS and Windows, the app notices the same changes as before; on file systems that do not, such as some network drives, such a change can show up to 30 seconds late until the app notices this and lists every folder on each scan.

• Make pkdb issues sync close a legacy issue again when it cannot close it as completed, and report each problem once and truthfully (#892). An issue closed with another reason than completed is reopened together with its other changes and then closed as completed; when that close fails, the sync closes the issue again with its former reason and names the state that GitHub then reports, or says that the issue may be left open when it cannot tell, also when the run stops or is interrupted in between. An issue that a study names together with a study that cannot be read is no longer changed but reported. A new issue whose labels and assignees GitHub dropped is reported once and listed with the labels and assignees it got; the warning about an issue with a similar title asks for a rename only when no new issue was created for the study, and otherwise names the new issue so that you can close the similar one; a rerun after a failed close of a duplicate does not repeat the comment "Duplicate of #N"; and an unexpected error during the sync exits with 1 instead of 2, the code of usage errors.

• Harden the images and registry of pkdb migrate (#891): study names with glob characters such as [ find their images, JPG images are turned upright by their EXIF orientation and keep their ICC profile as PNG, other image formats named .png become PNG, corrupt PNG images refuse the study as image_unreadable, a failed write is reported as image_write instead of an unreadable image, a malformed registry names its file and entry instead of raising a raw error, and the registry indexes its locations once.

• Follow-ups of pkdb new and pkdb check (#890). pkdb new removes the hidden folder .NAME.new that a killed run left, says so, and creates the study, when the folder holds only the unfinished study folder, the study does not exist and nothing in the folder changed for 10 minutes, so that a running pkdb new is never disturbed; a younger folder is refused as before. pkdb check says that a PATH does not exist, prints usage errors as {"ok": false, "error": "..."} on stderr with --format json, reports an unexpected error of one study as an unreadable_study problem and still checks the others, turns an unreadable studies folder into a usage error, leaves an untracked file with a name that is not valid UTF-8 out of --staged like other untracked files, and counts every study for the identifier and issue checks when the checkout lies inside another git work tree. The timing tests of the study format accept a larger ratio of CPU times under load, still far below the ratio of quadratic work.