Manifest Config Updates and Minor App Observability

• Update for ABCs
• Update for ED3N
• Internal /version endpoint showing git commit short sha and latest release tag

v2.2.0 Redis File Locker

DEX Upload Release Notes - Version 2.2.0

Release Date: 2024-06-24

Version Number: Version 2.2.0

Release Changelog

API Swagger Documentation

DEX Upload API Github Releases

DEX Upload API Github Tags

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 2.2.0 is to release an improvement to how the Upload API prevents file corruption when the service horizontally scales. This improvement is implemented in this release via a distributed file locker that uses a Redis cache to keep track of which Tus instance is currently working on a file, and makes sure another Tus instance doesn't work on the same file. Before this improvement, we were relying on Azure's internal blob storage implementation to prevent clusters of tus servers from corrupting files. This implementation is obscured from us and completely out of our control. This custom distributed lock with Redis gives us full control, and has even been adopted by the Tusd maintainers.

Enhancements

• Improved reliability of horizontal scaling via a custom file locking mechanism that uses an external Redis cache.

Reporting Team

• DEX Upload Team Distribution List - dexuploadapi@cdc.gov

v2.1.1

use websocket for upload processor service bus

v1.6.3

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.6.3 is to apply a hotfix to not include appended tguid values to filenames copied to EDAV storage accounts.

New Features

• Remove appended tguid values from filenames routed to EDAV storage accounts.

v1.6.2

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.6.2 is to update upload configurations for data streams, replace filename suffix clock ticks with upload id (tguid), hotfix for Routine Immunization v1 sender manifest config file, and adjust upload filename restrictions.

New Features

• Update upload filename character restrictions to only disallow forward slashes (/)
• Update Routine Immunization v1 sender manifest configuration file to remove incorrect metadata fields
• Add v2 sender manifest configuration files for NRSS and EHDI data streams
• Replace the filename suffix clock tick value with file upload id (tus guid) value

v1.5.0

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.5.0 is to update upload configurations for data streams, add version 1 sender manifest metadata values to version 2 metadata fields, add the version 2 metadata configuration folder structure, add file copy functionality to routing storage, add Processing Status API integration, add Retry functionality, and add Replay functionality.

New Features

• Version 2 sender manifest configuration folder structure
• Version 1 sender manifest metadata values to version 2 metadata fields
• File copy functionality to routing storage accounts
• Processing Status API integration
• Retry functionality
• Replay functionality
• Upload configurations for the DAART data stream
• Remove file copy EDAV target from the CELR data stream

v1.4.1

Overview

The purpose of Enterprise Data Exchange (DEX) Upload release version 1.4.1 is to enhance telemetry through integration with the Processing Status API, enhance resiliency through retry/replay capabilities for our post-processing component, and publish configurations for onboarding the AIMS CELR use case.

Infrastructure Changes

• Renaming of Upload Processor function app
• Infrastructure to support retry/replay functionality
• Application settings to support connection to Processing Status API
• Deployment of blue/green slots to the Upload Processor function app

v1.2.3

Added descriptions for the functions in the repo
Routing Integration: copy to routing changes
Configured test event to send files to routing
Metadata update for NDLP accepted values
Added integration tests suite to upload repo
Metadata configuration changes for routing
Integrate app insight Bulk Upload Processor
Updated log level from error to information
Unit test for bulk upload
Metadata definitions added to summary table ( tus/file-hooks/metadata-verify/definitions/readme.md )

1.2.2

• NDLP APL historical data configuration (#159)
• Changing all upload configs of NDLP to not append clock ticks to the filename (#153)
• Implementation of printing exception stack trace for increased visibility into error details (#162)
• Updates to OpenAPI specifications (#160)
• Workflow refactoring for upload configurations (CI/CD) (#158)
• Addition of workflow for upload configurations (CI/CD) (#156)

1.2.1

• Fix for ndlp sending data without filename in metadata ( TUS post-receive hook )
• Create upoad root folder in edav ( bulk upload processor )
• Add new NDLP metadata and upload configs ( TUS file hooks: metadata-verify, upload-configs )
• aims-celr configurations ( TUS file hooks: metadata-verify, upload-configs )