records: DOI versioning integration checklist

[slint]

Currently adding versioning to Zenodo requires a variety of changes throughout Deposit API, DOI/Datacite registration, management tasks (deleting/moving records), etc. Here's a list of such cases and their affecting components:

Deposits/Records API

• New records should also be coupled with creation of a new Concept PID
  • Concept PIDs should get their Recid minted first, to keep a logical order (Concept PID > Version PID)
• Creation of a new version:
  • Create copy of metadata - Strip/regenerate certain fields ('_deposit', 'doi', '_oai', '_files', '_buckets', '$schema')
  • Create unlocked snapshot of bucket
• Record metadata (JSON Schema) may need additional fields:
  • "conceptdoi" (Concept DOI) - manually added to the record metadata at minting
  • "conceptrecid" (Concept recid) - same as "conceptdoi"
• Editing older versions:
  • Disallow editing communities
• Disallow creating new versions of Custom DOIs
• Disallow creating a new version if the files have not been modified or if the bucket contents is the same as one of the previous versions (e.g. adding a file in one version, then removing it the same as creating a duplicate record). (Prod, 2h)
• Creating new version requires opening the last one in draft mode (in which it stays). Perhaps new version should be done from the Record view also?
  • Discard "current" deposit draft when clicking "New Version" from the deposit form (uses modal prompt to inform the user)
• Create a new "New Version" button on the Record page, so it's only kept in draft if user went through "Edit" first (which is fine). (Prod, 0.5h)**
• Deleting record in versioning
  • Deleting a last version should update Datacite information.
• Non-versioned deposit should create new versioning info on publish/save actions (lazy upon action call).
• Non-versioned records should create new versioning info on new version.
• Non-Zenodo DOI
  • Adding custom DOI new version of a versioned record shouldn't be allowed - On deposit.newversion pre-reserve Zenodo DOI immediately
• Serialized result contains empty relations key when the record is not versioned (relations: {})

UI/UX

• Search facets - All Versions Toggle
• Fixing deposit-ui issues
  • Generation of links from ES (search) and DB (single record serialization) (Prod)
  • bulk-index all children upon publishing so that the draft_child and draft_child_deposit information is reindexed properly for all siblings. (0.5h, Prod)
  • Filter/Search by Concept-only/All-Versions. Integrate search facets in the same way they appear in regular record search (QA, 4h)
  • Custom indexer function which indexes all non-versioned records as "latest" version (QA, 2h)
  • Clicking "Discard Changes" does not redirect properly.
• Upload page redesign for deposit results list. Users should be able to:
  • Be able to clearly see in what status each record/deposit is, and generally get more information about each entry (Prod)
• Record view page
  • Make blue alert box on top of page similar to the wide screen alert of flashed messages
  • Change phrasing of alert box to "There are other versions of this record. The latest is available here"
  • Make version dropdown more intuitive about current/latest record versions
  • Make version dropdown handle many versions (eg. add scrollbar)
  • New version dialog on deposit should not mention Concept DOI, but state that:
    • It's intended only for updating files, and that use should return back to the form is he intends to update the metadata only
    • A continuing with New Version will create a new deposit, which upon publishing will have a new DOI
  • Lower-case the labels and buttons text "New Version" -> "New version", "Record Versions" -> "Record versions", etc.
  • Yellow warning color on the flash box on "There exists new version", both on record page and deposit.
  • Change badges on the search result Version 2 (latest) -> DATE (v2) (merge with date badge)
  • Deposit sort by Versions
  • Badge caching issue (github: badges should contain no caching headers #912) (doesn't affect DOI badges atm)

DOI/Datacite

• Registration of Concept DOI to DataCite (same as registration of regular one).
• Decide on what kind of metadata the Concept DOI will hold (copy of the latest version)
  • What happens when the latest Version PID is modified - the metadata should be updated for the Concept DOI in DataCite as well
  • Fill-in Related Identifiers metadata on Zenodo at versioning
    • isNewVersionOf/isPreviousVersionOf: If we serialize these we have to update the metadata for previous/next objects in the linked list upon creation/deletion of new versions.
    • Comment this out for now because of semantic versioning records (e.g. GitHub)
    • hasVersions/isVersionOf - DataCite 4.x sprint LATER
    • isPartOf/hasPart: Has similar issues with the above, but not at such a large scale. (QA)
• Add single-record serialization of relations in DublinCore and Marc21 in OAI-PMH

Elasticsearch/Indexing

• New mappings for records and deposits, with necessary fields to make searches possible

REST API

• Version-aware /api/records endpoint
  • Filters
  • Links generation
• /api/deposit/newversion?recid=134 vs /api/deposit/123/actions/newversion
• 'concept_doi' nor 'concept_record_id' are returned in the JSON response (see value of data here: https://github.com/zenodo/zenodo/blob/master/tests/unit/deposit/test_api_editflow.py#L402). We should return those as well

Communities

• Are communities listing only the last version of the record? If so, what if a community is removed at some point in the version-history - does it disappear? Solutions:
  • Propagate "community" metadata changes to all version.
    • Resolving any InclusionRequest (of any version) is propagated for all versions
    • Checking if IR exists should be done for any of siblings
  • Lock editing of communities to only the latest version (both agular schema form and REST API validation). (Not required anymore)
  • Propagate changes to the new version draft as well (currently propagating across published versions only) (QA, 1h)

GitHub

• Subsequent repository releases should be in a versioning scheme on Zenodo. (Prod, 2d)
• Allow to manually publish new versions of GitHub records
  • Insert new GitHub release while allowing for the new version draft to be open (circumvent the limitation) OR fail and leave a notification on the Release object. (Prod, 1d)
    • IF going for failing - implement a retry button (Prod, 2h)
• Condition on creating New version to be an owner of the latest_version of the record in the versioning scheme (old GitHub repo owners shouldn't be able to manually create new record versions). If the record is GitHub-originated, check for the newest owner(Prod, 2h)
• Migrate old GitHub records/deposit into a versioning-scheme. (Prod, 1-2d)
  • Migrated GitHub records need to be synchronize their community membership (union of all communities), and propagated to deposits too.

Misc

• Profile performance and DB usage throughout views and endpoints
• Management of versioned records as part of support (deleting, changing files, etc)
• Citation generation for Concept PID
• Also include date of version in the "Record Versions" dropdown
• Documentation/Guides/Blogpost

Migration of Zenodo data

• Decide if we mint Concept recid for each record, and prepare a migration script (No/Not needed - we will not migrate all of old records - they will get conceptdoi/recid information per request or per first new version).

1. Versioned vs Non-versioned records.
   1. All of the old records stay as they are. No batch migration will be done.
   2. All new records will already be "versioned". Versioned records will have a registered Concept DOI and Concept recid.
   • Each non-versioned record will display a "Convert to a versioned Record" button on the record page and deposit item list. Conversion will supplement the existing records with those, while all new records will get those at first publishing.

Testing

• Unit tests:

  • Write unit testing for Zenodo-specific versioning API parts
  • Write unit test for new-version cases (create new version, delete draft, edit old metadata)
    • Some of it might be already done, but needs checking.
  • Fix communities merging unit tests

• Manual testing:

  • Test GitHub releases edge cases (changing repository owners, manual new versions, etc)
  • UI Tests, either manually or write up some simple template-redering tests
    • Migrated GitHub records should display one InclusionRequest/Community

• Test Non-versioned deposits actions

• Non-Zenodo DOI records actions

  • Versioned deposits
  • Non-versioned deposits

• Missing tests:

  • Related identifier serialization
  • Test eager vs bulk_indexing of siblings (whether they are actually indexed in bulk, use mock)