Generate changelog between 2 repo versions

[paynejd]

We want OCL to be able to use the results of OCL's compare operation to generate a change log between 2 versions of a repo. Here is a proposal for an MVP that closely matches our

This version of a change log matches the structure of the compare output, with the hope that it is straightforward to generate. It does not, however, provide a diff analysis that groups a concept with its mappings. But this still could be a great place to start.

• Concepts
  • New Concepts: list ID, display name + synonyms, description(s)
  • Removed Concepts (if there are any): list ID, display name
  • Major Concept Updates (smart changed): list ID, display name, update_comment, changes
  • Minor Concept Updates (standard changed): list ID, display name, update_comment, changes
  • Retired concepts: list ID, display name, update_comment, concept that replaced it (if relevant mapping created)
• Mappings
  • New Mappings: list ID, from-source+concept, to-source+concept
  • Removed Mappings: list ID, from-source+concept, to-source+concept
  • Major Mapping Updates: list ID, from-source+concept, to-source+concept, update_comment, changes
  • Minor Mapping Updates: list ID, from-source+concept, to-source+concept, update_comment, changes
  • Retired Mappings: list ID, from-source+concept, to-source+concept, update_comment

Expected output:

• API endpoint that includes the above content as structured JSON, omitting fields that have no data (i.e. no need to include a section for "removed mappings" or "removed concepts" if there are none

Resources:

• Andy’s talk post with a CIEL change log: https://talk.openmrs.org/t/new-version-of-ciel-concept-dictionary-added-to-dropbox/1749/170?u=paynejd
• Spreadsheet with a CIEL change log: https://docs.google.com/spreadsheets/d/1SHVUPBsrn4Py5VRhPdJ295zMgFbatCx6KeXhoEj1JTo/edit#gid=928744150

[paynejd]

Updated changelog approach based on discussion with Andy and the OpenMRS squad:

For all concepts in one of the repo version(s), changes to mappings should be grouped together as concept changes (using mapping.from_concept). Only if mapping.from_concept is not a member of one of the repo versions should it be separated out into the Mapping Change Log. Note: This approach means that the change log may need to retrieve concept summaries for concepts that did not change, because their mappings changed.

Concept Change Log (includes all concept changes plus mapping changes to any concepts in one of the repos, e.g. mapping.from_concept == concept.id)

• New Concepts: Concept Summary + changed Mappings
• Removed Concepts: Concept Summary + changed mappings
• Major Concept Updates (change based on concept.smart_checksum): Concept Summary + changed mappings
• Minor Concept Updates (change based on concept.standard_checksum): Concept Summary + changed mappings
• Concepts in the repo with only mapping changes: Concept Summary (mapping.from_concept) + Mapping Changes
• Retired concepts: Concept Summary + changed mappings

Mapping Change Log (where mapping.from_concept is not associated with a concept in the repos)

• New Mappings: Mapping summary
• Removed Mappings: Mapping summary
• Major Mapping Updates (mapping.smart_checksum): Mapping summary
• Minor Mapping Updates (mapping.standard_checksum): Mapping summary
• Retired Mappings: Mapping summary

[paynejd]

@snyaggarwal This is ready for your review

[paynejd]

• Also want to add diff of additional metadata that collections and expansions offer: References, and the reference(s) involved in bringing a specific resource into a collection
• Finding "equivalent" resources between two different repos -- new Concept added to CIEL that is equivalent to one in my source (i.e. same smart checksums?) where I might want to add a "same-as" map
• "smart review of upstream changes" -- user might unintentionally bring in a duplicate concept when rebuilding an expansion
• "health check" on your dictionary - find duplicates; highlight references that the user may want to model differently

[paynejd]

Note: The current diff/changelog approach is applicable to source-based repo types (source, code system, concept map). It does not yet cover collection-based repo-types (e.g. collection, value set, dictionary with linked source). Until then, a diff or changelog for a collection-based repo type will only look at concepts and mappings, not references/expansions.

[paynejd]

• concepts with smart checksum changed should show up in smart-changed category and not show up in the changed category
• smart_same_with_mapping_changes is actually the same as changed - so we don't need this one
• also some changed mappings are showing up in the mappings top-level category even though their from-concept is part of the CIEL source -- these should go under concepts -> same_with_mapping_changes

[snyaggarwal]

Version Diff APIs:

1. Versions diff (just numbers) -- default ?verbosity=0

Request

POST /sources/$compare/
{
    "version1": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
    "version2": "/orgs/CIEL/sources/CIEL/v2023-10-27/"
}

Response

{
    "meta": {
        "version1": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
            "concepts": 54584,
            "mappings": 229492
        },
        "version2": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-10-27/",
            "concepts": 54517,
            "mappings": 228957
        }
    },
    "concepts": {
        "new": 76,
        "removed": 0,
        "retired": 9,
        "changed": 769,
        "smart_changed": 90
    },
    "mappings": {
        "new": 644,
        "removed": 0,
        "retired": 109,
        "changed": 0,
        "smart_changed": 0
    }
}

2. Versions diff (just numbers) -- includes same ?verbosity=1

Request

POST /sources/$compare/
{
    "version1": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
    "version2": "/orgs/CIEL/sources/CIEL/v2023-10-27/"
}

Response

{
    "meta": {
        "version1": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
            "concepts": 54584,
            "mappings": 229492
        },
        "version2": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-10-27/",
            "concepts": 54517,
            "mappings": 228957
        }
    },
    "concepts": {
        "new": 76,
        "removed": 0,
        "retired": 9,
        "changed": 769,
        "smart_changed": 90,
        "same": 53739,
        "smart_same": 54418
    },
    "mappings": {
        "new": 644,
        "removed": 0,
        "retired": 109,
        "changed": 0,
        "smart_changed": 0,
        "same": 228848,
        "smart_same": 228848
    }
}

3. Versions diff -- includes same and IDs ?verbosity=2

Request

POST /sources/$compare/
{
    "version1": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
    "version2": "/orgs/CIEL/sources/CIEL/v2023-10-27/"
}

Response

{
    "meta": {
        "version1": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
            "concepts": 54584,
            "mappings": 229492
        },
        "version2": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-10-27/",
            "concepts": 54517,
            "mappings": 228957
        }
    },
    "concepts": {
        "new": {"total": 76, "mnemonic": [...]},
        "removed": 0,
        "retired": {"total": 9, "mnemonic": [...]},
        "changed": {"total": 769, "mnemonic": [...]},
        "smart_changed": {"total": 90, "mnemonic": [...]},
        "same": {"total": 53739, "mnemonic": [...]},
        "smart_same": {"total": 54418, "mnemonic": [...]}
    },
    "mappings": {
        "new": {"total": 44, "mnemonic": [...]},
        "removed": 0,
        "retired": {"total": 109, "mnemonic": [...]},
        "changed": 0,
        "smart_changed": 0,
        "same": {"total": 228848, "mnemonic": [...]},
        "smart_same": {"total": 228848, "mnemonic": [...]}
    }
}

Version Changelog APIs:

1. Versions changelog

Request

POST /sources/$changelog/
{
    "version1": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
    "version2": "/orgs/CIEL/sources/CIEL/v2023-10-27/"
}

Response

{
    "meta": {
        "version1": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
            "concepts": 54584,
            "mappings": 229492
        },
        "version2": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-10-27/",
            "concepts": 54517,
            "mappings": 228957
        }
    },
    "concepts": {
        "new": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "retired": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "removed": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "changed": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "same_with_mapping_changes": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
    },
    "mappings": {
        "new": {},
        "removed": {},
        "retired": {},
        "changed": {}
    }
}

2. Versions changelog with diff

Request

POST /sources/$changelog/?verbosity=1
{
    "version1": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
    "version2": "/orgs/CIEL/sources/CIEL/v2023-10-27/"
}

Response

{
    "meta": {
        "version1": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-12-15/",
            "concepts": 54584,
            "mappings": 229492
        },
        "version2": {
            "uri": "/orgs/CIEL/sources/CIEL/v2023-10-27/",
            "concepts": 54517,
            "mappings": 228957
        },
       "diff": {...}
    },
    "concepts": {
        "new": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "retired": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "removed": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "changed": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
        "same_with_mapping_changes": {
            "<str:id>": {
                 "id": "<str:id>",
                 "display_name": "<str>",
                 "mappings": [
                     "new": [],
                     "retired": [],                     
                     "changed": [],                     
                     "removed": [],                     
                 ],                 
             },
    },
    "mappings": {
        "new": {},
        "removed": {},
        "retired": {},
        "changed": {}
    }
}

[snyaggarwal]

@paynejd this is deployed on QA

[snyaggarwal]

@paynejd This is also on Staging

[paynejd]

Changelog Fixes

• Mappings that are bundled under concepts are sometimes duplicated in the changelog results

Proposed changes to the diff output

• Change field names:
  • retired --> changed-retired
  • changed --> changed-minor
  • smart-changed --> changed-major
• Add a changed-total count that is the sum of changed-retired, changed-major and changed-minor

{
  "meta": {
    "version1": {
      "uri": "/orgs/CIEL/sources/CIEL/v2024-05-01/",
      "concepts": 55037,
      "mappings": 231154
    },
    "version2": {
      "uri": "/orgs/CIEL/sources/CIEL/v2024-03-14/",
      "concepts": 54904,
      "mappings": 230711
    }
  },
  "concepts": {
    "new": 133,
    "removed": 0,
    "changed-total": 24,
    "changed-retired": 3,
    "changed-major": 19,
    "changed-minor": 2
  },
  "mappings": {
    "new": 450,
    "removed": 0,
    "changed-total": 7,
    "changed-retired": 7,
    "changed-major": 0,
    "changed-minor": 0
  }
}

Proposed changelog output

Use the same structure as the diff with 2 exceptions:

• Remove concepts.changed-total and mappings.changed-total fields (so that we don't duplicate resources)
• Add concepts.changed-mappings-only
• Include the diff summary (when verbose == 1 under meta)

{
  "meta": {
    "version1": {
      "uri": "/orgs/CIEL/sources/CIEL/v2024-05-01/",
      "concepts": 55037,
      "mappings": 231154
    },
    "version2": {
      "uri": "/orgs/CIEL/sources/CIEL/v2024-03-14/",
      "concepts": 54904,
      "mappings": 230711
    },
    "diff": {
      "concepts": {
      "new": 133,
      "removed": 0,
      "changed-total": 24,
      "changed-retired": 3,
      "changed-major": 19,
      "changed-minor": 2
    },
    "mappings": {
      "new": 450,
      "removed": 0,
      "changed-total": 7,
      "changed-retired": 7,
      "changed-major": 0,
      "changed-minor": 0
    }
  },
  "concepts": {
    "new": {},
    "removed": {},
    "changed-retired": {},
    "changed-major": {},
    "changed-minor": {},
    "changed-mappings-only": {}
  },
  "mappings": {
    "new": {},
    "removed": {},
    "changed-retired": {},
    "changed-major": {},
    "changed-minor": {}
  }
}

Questions

• For the diff operation, add a concepts.changed-mappings-only count?
• The changelog does a great job categorizing and listing the resources that have changed, but it doesn't include what has changed. Basically- the "changelog" may actually be what we want for a "diff" output. The "Changelog" should actually include what has changed. Let's discuss.

[snyaggarwal]

@paynejd These changes are deployed on QA/Staging

[snyaggarwal]

@paynejd Updated changelog with diff https://json.link/NTyBdEbTWa.json
Async Request:
POST https://api.staging.openconceptlab.org/sources/$changelog/?verbosity=1
{
"version1": "/orgs/CIEL/sources/CIEL/v2024-05-01/",
"version2": "/orgs/CIEL/sources/CIEL/v2024-03-14/"
}

[paynejd]

Definitions

• new - resource in version2 and not in version1
• removed - resource in version1 and not in version2
• changed_total - total number of changed resources -- this is only included when listing the counts
• changed_retired - resource active in version1 and retired in version2
• changed_major - resource with a "smart checksum" change between version1 and version2
• changed_minor - resource with only a "standard checksum" change between version1 and version2

Example:

{
  "meta": {
    "version1": {
      "uri": "/orgs/CIEL/sources/CIEL/v2024-05-01/",
      "concepts": 55037,
      "mappings": 231154
    },
    "version2": {
      "uri": "/orgs/CIEL/sources/CIEL/v2024-03-14/",
      "concepts": 54904,
      "mappings": 230711
    },
    "diff": {
      "concepts": {
        "new": 133,
        "removed": 0,
        "changed_total": 21,
        "changed_retired": 0,
        "changed_major": 19,
        "changed_minor": 2
      },
      "mappings": {
        "new": 450,
        "removed": 0,
        "changed_total": 7,
        "changed_retired": 7,
        "changed_major": 0,
        "changed_minor": 0
      }
    }
  },
  "concepts": {
        "new": [],
        "changed_major": [],
        "changed_minor": []
  },
  "mappings":{
  }

[snyaggarwal]

@paynejd This is deployed everywhere. Anything else should be recorded on new ticket, closing this