Diffs and Checksums

[jamlung-ri]

Here’s the full checksum package from the perspective of CIEL (which is a good place to start):

• Imports to CIEL only create new resource versions when there is a change (using standard checksums)
• Resource comparison of 2 versions of a CIEl concept leverages standard and smart checksums to help users quickly understand meaningful changes
• Repo diffs can be performed using both standard & smart checksums as part of the import validation process
• Changelog for a repo can be generated automatically (using both standard & smart checksums)
• Old un-changed/duplicate resource versions have been squashed

[jamlung-ri]

User Stories: (note that this assumes that the API-level capabilities have been built into some TermBrowser workflows)

• v3
  • I can see a summary list of changes to concepts and mappings from one CIEL version to another next e.g. "X retired, X new concepts, X new mappings"
  • When I look at a concept's version history, it's very clear when a concept has changed over the course of several CIEL releases.
  • I can see an itemized list of concepts that changed (i.e. are not the same in a meaningful way) from an old version to the current version (even if the old version is several versions ago).
  • When I see that a concept has changed since my last check-in with CIEL, I want to dive deeper to explore the exact changes to see if that will affect how I currently use it.
• Unsure
  • ?
• After v3
  • ?

[paynejd]

Here’s the full checksum package from the perspective of CIEL (which is a good place to start):

• Preventing duplicate resources: Imports to CIEL only create new resource versions when there is a change (using standard checksums)
  • Current: Implemented for both concepts & mappings, deployed everywhere behind a toggle (enabled on QA)
  • Next Steps:
    • Finalize HTTP response code to use for a duplicate resource that did not result in a change
    • Validate the new duplicate version behavior with a bulk import on QA
• Local repo diff: Repo diff can be performed outside of OCL e.g. downloading an export that contains checksums
  • Current: Checksums are available in the exports; checksum operation available in the operations panel
  • Next steps:
    • Create Javascript or python method to calculate checksum locally
    • Perform repo diff locally between import script and a repo export – note that the bulk import JSON will not have default values applied (but the export will)
• Repo diff API: Repo diffs can be performed using both standard & smart checksums as part of the import validation process
  • Current: API implemented at /sources/compare/ (includes concepts and hierarchy as an attribute of concepts; does not include the repo/repo version metadata or mappings at all)
  • Next Steps:
    • Incorporate mappings into the diff
    • Define the generic $compare operation that can diff 2 arbitrary sets of resources (repo version, list of concepts, etc.)
    • Decide how to also diff repo metadata and incorporate into a single repo comparison view
• Changelog: Changelog for a repo can be generated automatically
  • Next steps:
    • Define requirements and implement MVP to generate change log between 2 repo versions (Generate changelog between 2 repo versions #1814)
    • Incorporate new changelog capability into the TBv3 release workflow
• Squash existing content: Old un-changed/duplicate resource versions have been squashed
  • Next steps:
    • Initial research: How deep of a hole are we in?
    • Define the requirements for a $squash operation??
• Resource diff: Comparison of 2 versions of a CIEl concept leverages standard and smart checksums to help users quickly understand meaningful changes
  • Current: TB comparison tool does not currently leverage checksums
  • Next steps:
    • Design checksum-enabled resource comparison into the TBv3 comparison tool (not planning to do this in v2)

[paynejd]

User story: Bulk import completed, and fetch the checksums for a repo version or a specific list of resources that I know were updated to verify that the import completed successfully without waiting for a full export to be available.