Executive summary of 23.1 Backend Goals

[martenson]

this list is currently work in progress

high priority

• Pulsar container scheduling (AWS batch, TES, k8s)
  • Pulsar extended metadata hardening
• Workflow conditionals, step input expressions

standard priority

• Work on SqlAlchemy 2.0
  • SQLAlchemy 2.0 compatibility #12541
  • Beta1 is out
• Hard removal of items from database
  • Hard removal of items in the database (scalability discussion) #13992
• History, Invocation export / import
  • [WIP] Model Store Workflow Invocation Import/Export #9024
  • Add task-based history export tracking #14839
  • First class RO-crate support
    • Export Histories into Research Objects #3088
  • Locked, archived histories
    • Archiving Histories #14771
• Workflow Invocations
  • Make invocation errors more transparent (Marius)
  • Make invocation / invocation step job progress more transparent (Marius)
• Tool state modernization (important, big project, hackathon)
  • https://docs.google.com/document/d/1HQOLpLN54CjrB-wbD463XqzvUm-dNB8vTXUFBExh_2o/edit?usp=sharing
• Object store selection
  • Empower Users to Select Storage Destination #14073
• API client generator improvements (Marius, Dannon)
  • With UI/UX, consume in testing
• Async (avoid timeouts, make it feel snappy) (John D)
  • Tool submission
    • Need to figure out status updates
    • SSE to keep hand on status
  • History copy + import (good starting issue)
  • Copy from celery issue with more tasks
• Support Graph-based view of history/invocation (Marius)
  • Figure out data structure and efficient queries
• Final push for gxformat2
  • Promote gxformat2 to default workflow export format #13584
  • Additional download button - John C
  • Fix gxformat2 import on dockstore - John C

low priority

• Tool Shed partial rewrite (low priority)
  • Tool Shed partial rewrite #14887
  • Serving tool state
• Workflow lineages ? (Marius, low priority, Martin ?)
  • Enable updating TRS workflows
    • Advanced stuff:
      • To support modifying subworkflows in the parent context ?
      • Stored Workflow modifications are lost currently
        • How to get updates to customized workflow