Isocenter 1.0.0

Highlights

1.0.0 is the first stable release of Isocenter. These highlights are for a reader who upgrades from 0.9.x. They give the main changes of 1.0.0rc1 to 1.0.0rc22 and of this release. Each item names its issue. The full entry is in CHANGELOG.md, in the section of the candidate that made the change. The entries of this release follow the highlights.

What 1.0 promises

The API has three tiers. The API stability page names each one (#26, #787).

• Frozen (tier 1). These names keep their spelling and behaviour for every 1.x release. They are Session, its methods and their parameters, the shapes they return, and the other callables in the frozen tables. A change to them is a 2.0.
• Documented but internal (tier 2). These names are safe to call. A 1.x release can change them, with a changelog entry that names the old spelling and the new one.
• Private (tier 3). Names with a leading underscore, and every module that the page does not list, can change without notice.
• The output. For the same input, configuration and project secret, a 1.x release exports what the previous release exported, or its changelog says what changed. Each release's export of a fixed cohort is fingerprinted, and a release stops on a difference that no changelog entry names (#717).
• The configuration. Every 1.x release loads a file that 1.0 loads. A 1.x release that applies an unchanged file differently raises the schema's minor version (#762, #782). A privacy profile name is pinned to its PS3.15 edition: basic means basic@2026c in every 1.x release (#714).
• Plugins. The exporter registry is provisional until 1.1. In 1.0, an export through a third-party exporter grades REVIEW_REQUIRED (#527).

Before you upgrade from 0.9.x

Isocenter migrates nothing from 0.9.x (owner ruling on #790).

• Identity tokens. Releases before 1.0 wrote the identity token into the wrong element of the Encrypted Attributes item. 1.x refuses a token in that layout by name and does not decrypt it. To recover such an identity, use Isocenter 0.9.x and the key that locked it (#790).
• Stores. A 0.9.x store opens in 1.0. On open, 1.0 adds the columns it records into, empty, and converts no value that the store holds.
  • Its PHI statuses have no recorded policy. An export of them grades REVIEW_REQUIRED until you run audit() and anonymize() under your configuration, then save() (#555).
  • Its instances keep their source UIDs until the next audit() and anonymize() replace them (#544). Files exported before 1.0 keep their source UIDs, and 1.0 exports of the same data do not link to them by UID.
  • 0.9.x grouped files with no Patient ID under one patient, '' or UnknownPatient. Such a store keeps that grouping, because a split would give dates already shifted a second offset. While a patient's ID in the store is still '' or UnknownPatient, every open writes one WARNING row that gives only a count. Every report over the store then grades REVIEW_REQUIRED. audit(), anonymize() and save() do not remove a row already written. A group that anonymize() gave a pseudonym, in 0.9.x or in 1.0, is not counted. So a store that was anonymized before 1.0 can hold such a group with no row. To separate the subjects, ingest their source files into a new store (#584).
  • 0.9.x does not write the new columns. A status that 0.9.x saves into a store that 1.0 opened then sits beside the policy of 1.0's last scan, and 1.0 cannot detect this. Before you open a 0.9.x store in 1.0, keep a copy of it. Do not save into the 1.0 store from 0.9.x. If you go back to 0.9.x, use the copy.
• Configuration files. A file that 0.9.x's create_config() or auto-save wrote loads unchanged, with one exception. Before 0.9.8, set_phi_tag(replacement=) saved a phi_tags rule with a replacement key. The loader refuses that key. Rename it to value (#538).
  • The loader refuses an unknown key, a value of the wrong type, and a version that this library does not read (#711, #712, #713).
  • 0.9.x cannot load privacy_profile: basic@2026c, which 1.0 writes. Write basic in a file that both must load (#714).
  • A change to the configuration no longer writes the file. Call configuration.save(), or set configuration.auto_save (#715).
• Renamed and removed names.
  • start_patient(id=) is now patient_id=. add_rule(model=, zones=) is now model_name= and redaction_zones=. set_phi_tag(replacement=) is now value=. The old keywords raise TypeError (#26).
  • write_project_secret() and load_project_secret() are removed. A project secret stays in the store that made it, and a copy of the store is the same project (#716).
  • ConfigLoader.load_redaction_rules(), ConfigLoader.load_phi_config() and the config_path parameter of PhiInspector are removed (#729).
  • export_dataframe() requires its output path (#812).
• Selections. A bare str as patient_ids raises TypeError. Write patient_ids=["X"] (#696). An ID or UID that names nothing is counted, and an export with one grades REVIEW_REQUIRED (#686, #725).
• A stricter grade. A run that graded PASS in 0.9.x can grade REVIEW_REQUIRED in 1.0. The analytics page lists every condition. These are the main new ones:
  • A finding raised under the policy that nothing acted on (#573).
  • An exported status recorded under another policy (#555).
  • An entity edited after its last scan (#767).
  • An entity that no scan read, in a store that holds a recorded status (#1064).
• Changed output under basic@2026c and the floor.
  • Each UID that Table E.1-1 codes U is replaced by a UID derived from the value and the store's project secret. One UID gets one replacement everywhere in the store, so references between exported files still resolve (#544).
  • An element that Table E.1-1 codes D gets a dummy value of its VR (#557).
  • Retired curve groups, whole overlay groups, Content Sequence, Graphic Annotation Sequence and Referenced File ID are removed (#556, #840, #848, #1133).
  • Every exported DICOM file that is not JPEG 2000 is Explicit VR Little Endian, so each element states its VR (#1107).
• Other changed behaviour.
  • At ingest, files with no Patient ID are no longer merged into one patient. Each study that they belong to is its own patient. A store that 0.9.x wrote keeps its grouping (see Stores, above) (#584).
  • A machine rule with serial_number: "*" now also covers series with no Device Serial Number (#1108).
  • A private-creator key matches its creator in any case and spacing. Two keys for one creator with different rules are refused (#1147).

New in 1.0

• De-identification markers. An exported DICOM file that the audit found clean, or that anonymize() de-identified, says how (#554, #978, #1011).
  • Patient Identity Removed (0012,0062) is YES, unless the file says Burned In Annotation (0028,0301) YES.
  • De-identification Method (0012,0063) names the version and the policy.
  • Longitudinal Temporal Information Modified (0028,0303) is MODIFIED when this store shifted a date in the file. It is REMOVED when every date in the file is absent, empty or its VR's dummy. Otherwise the file keeps the source's value. But a source's UNMODIFIED is not kept when this store took a top-level date out of the file.
  • A rule of any action on one of these three tags stops that stamp.
• Burned-in text. A burned_in_text policy makes redact() read the images, then redact, report or withhold the text that it finds (#1108). The policy has three more settings (#1121):
  • scope: which images to read.
  • target: which kinds of text to act on (names, dates, IDs).
  • detector: how to find text. Use tesseract, or a text-detection model that ships with Isocenter and runs under OpenCV (the textdetect extra).
• SCRUB. A SCRUB rule removes the identifiers inside a text element and keeps the rest of the text (#1130, #1146). It removes the kinds that you name: names, dates, IDs, organizations, phone numbers, addresses, sites and initials. With SCRUB, you can keep a structured report's content and scrub its text values (#1131).
• More precise rules. A private-element rule can name its creator, as in '0009,"GEMS_IDEN_01",27' (#1132). A machine rule can select the series with no Device Serial Number (serial_number: ""), or select by device (match:) (#1108).
• Waveforms. A WFDB export names coded ECG leads I, II, aVR, V1 and so on (#828, #832).
• Documentation. The site has one folder for each minor version (#866), and it is built with Sphinx (#1155). Its tutorials run as part of the test suite (#27). A new page gives Isocenter's scores on TCIA's MIDI-B de-identification benchmark (docs/midi-b.md).

Security fixes since 0.9.x

• Export path (GHSA-2rc2-r9r5-x7hm). Every release from 0.7.0 to 1.0.0rc15 could write a file outside the export folder when a SOP Instance UID held a path separator. Fixed in 1.0.0rc16. Its entry in CHANGELOG.md says which earlier exports to check.
• Free text in structured reports and presentation states. Under basic and the floor, this text was exported as written. Fixed in 1.0.0rc4 and 1.0.0rc5 (#840, #848).
• Two redaction rules on one image. Burned-in text inside a redaction zone could be exported unredacted when two rules covered one image. Fixed in 1.0.0rc10. Its entry in CHANGELOG.md says to run redact() again and export again (#908).

Added

• docs: a MIDI-B benchmark page. docs/midi-b.md (/1.0/midi-b/) gives Isocenter 1.0.0rc22's scores on TCIA's MIDI-B de-identification benchmark, scored with the benchmark's validation script at commit ccc0939: 99.22% of the Validation set's checks and 99.23% of the Test set's, against 72.51% and 68.45% for the default configuration. It sets out the configuration part by part with what each part is worth, what is left and why, the choices a site may want to change, how to reproduce the run, and the whole configuration. The configuration was tuned on both sets, so neither figure is a held-out estimate. Example values are synthetic: MIDI-B's own where an issue already quotes them, invented values of the same shape otherwise. It takes the place "Upgrading from 0.9.x" held in the site's navigation and the README.

Changed

• docs: the overview pages and the MIDI-B page follow Simplified Technical English (ASD-STE100). docs/index.md, docs/for-institutions.md, docs/architecture.md, docs/performance.md and docs/midi-b.md now split long sentences and replace semicolons with full stops, in STE-flavored form. Their step lists and the MIDI-B "Reproducing these results" steps use strict form. No heading, link target, code block, figure or example value changed, and every hedge keeps its strength, so each published anchor and each number on these pages is as before. docs/about/index.md and docs/guides/index.md already complied and are unchanged.

• docs: the procedure pages and the README in Simplified Technical English (ASD-STE100), and a glossary. docs/installation.md, docs/quickstart.md, docs/tutorials/index.md, the six tutorials, docs/ctp-import.md and README.md are rewritten in shorter sentences with no semicolons, one action per step, and one term for each concept. No heading, code block or link target changes, so every heading id is the same. No fact, condition or hedge changes, except two sentences that were no longer true and are corrected: the installation page said no CI job installs the spaCy model, which CI installs on every entry that is not free-threaded, and the redaction tutorial said redact() reads every image that holds pixels under a burned_in_text policy, where since #1121 it reads only the images in the policy's scope. A new page, docs/glossary.md (/1.0/glossary/, under About), names the one term the documentation uses for each concept and the words it does not use for it. The README's tagline and the home page's hero are now two sentences, and they stay identical: "De-identify a DICOM cohort without touching the source files. Hand compliance a report that names anything the run could not do." The README reaches PyPI only at the next upload.

• docs: the configuration guide follows Simplified Technical English (ASD-STE100). docs/configuration.md now splits long sentences and replaces semicolons with full stops, in STE-flavored form, and its template, method and auto-discovery steps use strict form. No heading, link target, code block, number or quoted standard or runtime text changed, and every hedge keeps its strength. Four sentences that were no longer true are corrected: the burned-in names paragraph said no CI job runs the model, which CI runs on every entry that is not free-threaded; basic@2026c replaces some dates with the dummy rather than removing or emptying all of them; a REPLACE with a value: on Study or Series Instance UID is refused when it loads (#877); and a machine rule can reach a series with no Device Serial Number (serial_number: "" or match).

• docs: the reference pages for the report, export output, codecs, OCR, waveforms, environment variables and development follow Simplified Technical English (ASD-STE100). docs/analytics.md, docs/export-output.md, docs/codecs.md, docs/ocr.md, docs/waveforms.md, docs/environment.md and docs/developer_guide.md now split long sentences, replace semicolons with full stops, and put long series into lists, in STE-flavored form. The OCR prerequisites and the developer guide's command steps use strict form. No heading, link target, code block, quoted standard text, quoted runtime text or example value changed, so every published anchor is as before, and every hedge keeps its strength. Four sentences that were no longer true are corrected: the environment page said one test fails rather than skips under ISOCENTER_REQUIRE_TEXT_DETECTOR, where several do; the developer guide called the API-page check the second command of its fence, where it is the third; the export page said DicomExporter.write_tree() applies "the same CT Type 1 check", where since #879 the check covers CT, MR and PET; and the OCR page said the spaCy model runs in no CI job, where CI runs the names arm's name finder with the real model on every entry that is not free-threaded (the redact() and export(check_burned_in=True) paths through it are tested with a stand-in model, as docs/configuration.md says).

• docs: the API pages follow Simplified Technical English (ASD-STE100). docs/api/stability.md, docs/api/session.md, docs/api/entities.md, docs/api/exporters.md, docs/api/results.md, docs/api/configuration.md, docs/api/persistence.md and docs/api/index.md now split long sentences and replace semicolons with full stops, in STE-flavored form, and the exporter-author rules use strict form. No heading, table row or code block changed, one link target changed (the transfer-syntax correction below), and every hedge keeps its strength. The 1.0 promise on the stability page means what it meant: six sentences that tests/test_frozen_surface.py pins word for word are reworded (semicolons made full stops, a passive with a known actor made active), and the test's literals change with them, in wording only. Three sentences that were not true are corrected: the exporter registry page said the Quick Start lists the transfer syntaxes ingest reads, which it never did and Codec support does, and that the two built-in formats register when isocenter is imported, where they register when isocenter.exporters is imported, which session.export() does; and the entities page said the export writes any Decimal as a float, where a non-finite one, or one past a float's range, stays a Decimal.

• docs: the docstrings that the API pages render follow Simplified Technical English (ASD-STE100). The rendered docstrings and #: comments of isocenter/session.py, entities.py, io_handlers.py (IngestSummary, ExportSummary, ExportError), persistence.py, configuration.py, privacy.py, verification.py, automation.py and exporters/__init__.py now split long sentences, replace semicolons with full stops, and put long series into lists, in STE-flavored form. The Args:, Returns:, Raises: and Attributes: items keep their shape, and no code, code comment or runtime message changed. Every hedge keeps its strength, and no tier-1 promise changes in meaning. The docstrings of services.py, discovery.py and pixel_analysis.py are not in this change. Twenty-two sentences that were not true are corrected: RedactionVerifier.verify_instance said a rule is selected by serial number alone, where it takes every rule that covers the equipment (services.rules_matching), so with no equipment the "" and "*" rules still apply; set_phi_tag said a tag is stored lower-case, where a private creator's name keeps its case and has its surrounding spaces stripped; Equipment.from_parts said every machine rule matches on the serial, where a match: rule does not; scan_pixel_content said a series with no Device Serial Number is never scanned, where a "", "*" or match: rule covers it as redact() reads it; redact named only the serial as the way a rule selects a series, where match: selects one too; auto_remediate_config said it returns the number of rules updated, where it returns the number of changes applied, one per zone added or grown; reconcile_private_tags and redact(force=) each pointed to the "Upgrading from 0.9.x" page, which no longer exists; four sentences in redact(), export() and _export_dicom() said OcrUnavailableError is raised when the ocr extra or the tesseract binary is unavailable, where it is raised when a reader that the burned_in_text policy needs cannot run, tesseract is not needed under detector: db with target: any, and its subclasses TextDetectorUnavailableError and NlpUnavailableError are raised for the text detector and the names model (#1164); five sentences in the same three said the policy's read is by OCR, where under detector: db the text detector reads; redact() said its REDACTION row names the tesseract version, where under detector: db it names the detector model and the OpenCV version; redact() and redact_by_machine() called the drain that runs a failed background save again the entry drain, where it comes after redact() reads its burned_in_text policy (as the stability page says since #1172); set_phi_tag's value said only an empty value is no value, where any value that is false in Python is (#1172); and lock_identities did not say that, for Patient's Name and Patient ID, the lock reads the patient's value when an instance holds none (#1172).

Removed

• docs: the "Upgrading from 0.9.x" page (docs/migration.md). Nothing before 1.0 is accommodated (owner ruling on #790), and each change it listed is recorded in this changelog's 0.9.x and 1.0.0rc sections. Its URLs, /1.0/migration/ and /latest/migration/, and their anchors (#configuration-files, #hollow-waveform-multiplex-items and the rest) no longer resolve once the site is next deployed. The links to it from the index, analytics, configuration, for-institutions and waveforms pages, and from the README, are removed, not repointed.

Fixed

• docs: sentences on the API pages that the code did not bear out (#1172). Found during the ASD-STE100 rewrite (#1170). They were already on main before it. No behaviour changes.
  • docs/api/stability.md: redact() and the two lock methods do not run a failed background save again "on entry, before anything else". redact() first reads its burned_in_text policy, and returns 0 with no rules and no policy. The locks first check that reversible anonymization is enabled, and the key. The single form also checks the Patient ID. The batch form also checks its selection and the refusals of every patient. The page now lists these checks, and the Behaviours bullet says the same for redact().
  • The RuntimeError for no key file while the session holds a token of ours is the lock's, not enable_reversible_anonymization()'s. The lock methods' "not enabled" RuntimeError is now listed.
  • recover_patient_identity() also raises RuntimeError when reversible anonymization is not enabled, ValueError for a malformed key file, and RuntimeError for a token from a later release. Nothing on the patient is restored after any of the errors that the page lists.
  • set_attr() stores a tuple that holds a numpy value as a new list, so the value held does not equal the tuple set (also docs/api/entities.md). #1173 is to keep the tuple.
  • For the tags_to_lock refusal, the lock reads Patient's Name and Patient ID from the patient when an instance holds none. So under the default tags this refusal does not occur for a patient whose Patient ID is a str. An empty tags_to_lock is also refused.
  • set_phi_tag() stores no value key for None, "" or another value that is false in Python, such as 0 or False. #1174 is to refuse 0 and False.
  • docs/api/entities.md: under a standard AT element, export() writes a tuple of exactly two integers or two str as one tag (#1009). A pair that set_attr() stored as a list (a tuple that holds a numpy number), or that a reopened store holds as a list (#1016), is written as two tags. set_attr() does not raise for a value of another type, but Instance.set_attr() can raise ValueError for an edit to a pixel descriptor.
  • docs/api/exporters.md: after a third-party export, the report shows the "generated before any export" note only when the store holds no EXPORT row from an earlier export.
• docs: three API-page entries, and eight comments and docstrings, made true to the code (#1178). Found in the review of #1177. No behaviour changes. docs/api/stability.md's recover_patient_identity() entry did not list the RuntimeError that restore=True raises, before anything is written, when patients that would share a Patient ID after the restore were de-identified under different date-offset schemes. The shared-scheme entry on the same page already named the call. Its redact() burned_in_text entry did not name the ValueError that redact() raises, before its drain, for a policy that load_config would refuse. The same entry said redact() and export(check_burned_in=True) raise RuntimeError whenever the ocr extra or the tesseract binary is unavailable. Under detector: db with target: any, the policy does not need tesseract, so neither call raises when tesseract is missing. The entry now names the readers that each detector and target needs. docs/api/exporters.md said export() only resolves the format and dispatches. Before it dispatches, it also clears the last export's counts, and for a third-party exporter it writes the WARNING row and logs the same text. Four code comments in isocenter/session.py and three test docstrings said redact() drains on entry, where it drains after it reads its burned_in_text policy and before it writes. The DicomItem.set_attr docstring said a list or tuple is converted when it holds a numpy number, where any numpy value in it, np.str_ included, makes it a new list.