Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Form 526 Document Upload - Lighthouse status polling system #15964

Closed
wants to merge 19 commits into from

Commits on Feb 14, 2024

  1. Create LighthouseDocumentUpload table

    Creates a new model, LighthouseDocumentUpload, responsible for persisiting a record of a document that was submitted to Lighthouse's Benefits Document API. We will use this to track the status of a document as it makes its way through the VA's systems.
    
    Document Uploads must be attached to a Form 526 submission, and if they reference a Veteran-submitted document that was forwarded on to Lighthouse, they must be attached to the FormAttachment record created for that upload as well.
    
    A lighthouse_document_request_id is also required; this is returned from the Lighthouse Benefits Documents API as requestId in their response body. We will need this identifier to track the upload as it passes through Lighthouse's processing, since their status endpoint requires we pass it to look up the status of a document.
    NB28VT committed Feb 14, 2024
    Configuration menu
    Copy the full SHA
    5aad486 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. Add state machine to LighthouseDocumentUpload

    Adds an AASM state machine to LighthouseDocumentUpload that accounts for the various potential lifecycle events once a document reaches Lighthouse. Adds transition guards to ensure the proper metadata is stored before making certain transitions (e.g. if there is a failure makes sure we have saved an error_message)
    
    Updates factories for LighthouseDocumentUpload and FormAttachment to support tests
    NB28VT committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    d774425 View commit details
    Browse the repository at this point in the history
  2. Add event transition logging to LighthouseDocumentUpload

    Capture events metadata on LighthouseDocumentUpload state transitions and persist the actual response metadata from the Lighthouse /uploads/status API endpoint
    
    These logs will drive DataDog alert events when there are submission failures
    NB28VT committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    87b4214 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. Create class for parsing Lighthouse Document Status response

    Creates a Plain Old Ruby Object to abstract out the parsing of the complex nested response from the uploads/status endpoint of the Lighthouse Benefits Documents API.
    
    Provides convenience methods for checking whether a document has completed all processing, has failed processing, or has progressed to another step in the process
    NB28VT committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    e9039f0 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. WORKING COMMIT AMEND

    Begin Update documents status service
    NB28VT committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    9f944d6 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Implement UploadStatusUpdater

    Implements a service class that encapsulates all of the parsing of a Lighthouse document status data structure, updates a supplied LighthouseDocumentUpload accordingly
    NB28VT committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    de4558d View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Configuration menu
    Copy the full SHA
    90752a5 View commit details
    Browse the repository at this point in the history
  2. Create Lighthouse526DocumentUploads table

    Create table for tracking the status of a Form 526 supplemental document we have uploaded to Lighthouse
    NB28VT committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d079b55 View commit details
    Browse the repository at this point in the history
  3. Overhaul and simplify document model

    Simplifies state tracking and validaitons in the model we use to track Form526 document submissions to Lighthouse
    NB28VT committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    eb3eb55 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Refactor Status Updater

    Overhauls the service class for parsing a Lighthouse document status response and updating a Lighthouse526DocumentUpload record accordingly.
    
    Simplifies the API for this class to reflect the simplified infrastructure of the service object that will be using this
    NB28VT committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    108091b View commit details
    Browse the repository at this point in the history
  2. Refactor UpdateDocumentsStatusService for new logging approach

    Simplifies the UpdateDocumentsStatusService to use the simplified API of the UploadStatusUpdater. Makes the service responsible for passing the Lighthouse response metadata to the updater class, and logging to statsd metrics based on the outcome
    NB28VT committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    4d6e21d View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Implement Lighthouse::Form526DocumentUploadPollingJob

    Implements the actual job that orchestrates polling of the Lighthouse uploads/status endpoint to get the current status of a document we have uploaded to Lighthouse. Includes Sidekiq retry exhaustion handling
    NB28VT committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    7f922e6 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Refactor UpdateDocumentsStatusService for new flow

    Update service and tests to take Lighthouse benefits response from the job
    NB28VT committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    7baac10 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. Freeze development at good stopping place for feedback

    Freezing development for early draft PR feedback; latest changes:
    - Account for documents that have not been polled for updates yet (status_last_polled_at is nil)
    - Begin end to end testing using the Lighthouse QA domain (still debugging with Lighthouse)
    NB28VT committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    aaa88e5 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. Cleanup for draft review

    Couple loose ends to clean up ahead of draft review
    NB28VT committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    a2476c1 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Complete e2e testing and debugging

    Adds live VCR tests with the status polling endpoint using documents provided by Lighthouse in their QA environment.
    
    Makes necessary changes across service components and in tests to reflect bugs that were found when testing with the actual endpoint
    NB28VT committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    c0dac15 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. Handle failure responses from status endpoint

    Adds logging and a StatsD metric (for alert monitoring) when the call to the status endpoint returns a non 200 response
    NB28VT committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    b4d118b View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. Configuration menu
    Copy the full SHA
    0bce3ee View commit details
    Browse the repository at this point in the history
  2. oops

    tblackwe committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    9d1b157 View commit details
    Browse the repository at this point in the history