All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Updated Processing Status Report schema to new consolidated version
- Updated docker file to use a smaller image that does not require ssh
- Updated docker file to perform multistage builds to build code + create image.
- Configured port 8080 on Dockerfile
- Minor changes to Pom.xml to support deployment of both K8s and serverless
- Updated DAART profiles
- Enhanced all FN health checks to check dependencies.
- Improved Redactor when redacting fields with multiple values
- Separated PHLIP into two data streams: PHLIP_FLU and VPD
- SImplified redactor profile loading by data_stream_id
- Added version to Routing Metadata
- Created DockerFile for all Functions
- Added Fields to PID up to 35
- Included Fortify scans for new modules
- Dockerized two functions for AKS migration path
- Removing binary data from HL7 message and saving it as a separate file.
- Removed Process Status Traces from functions
- Added sender_id and received_filename to metadata
- Keep supporting metadata when file sink is copying files
- Throw unknown exceptions on all functions for Azure retries
- Saving partitioned folders by date using new dex_ingest_datetime
- Moved AzureBlobProxy from receiver-debatcher to lib-dex-commons (to be reused by other Fns)
- Created bash scripts to deploy Azure Functions with Zip deploy instead of maven plugin.
- Modified file-sink to create blobs and metadata in single call.
- Added retries on file sink when upload fails
- Improved Structure Validator status when critical failures happens (as when no profiles match message)
- Fixed issues with NRSS validation profile
- Added health checks for resource dependencies on lib-commons
- Added retries on Receiver/Debatcher for loading metadata appropriately
- Added retries to read metadata out of blob files (Azure SDK bug?)
- Bug fixes - fixed Debatcher report on counts
- Added eventhub health check on lib-dex-commons
- Added configuration to file-sink to customize output directories.
- Fixed some typos on PhinGuideProfile used to generate HL7 Json
- Implemented Metadata V2 on all functions.
- Changed Receiver Fn to Read events from Storage Queue instead of EH to avoid duplication
- All functions now provide the actual stage.version instead of hardcoded
- All PS Reports have unique schema names (PS Requirement)
- Improved Receiver/Debatcher Reporting schema to be standard with other stages.
- Setting batchCheckpointFrequency to 1 on all Pipeline functions
- Refactored all functions to reuse azure resource connections - event hub, service bus, cosmos db, storage account, etc.
- Added health check to all functions and sidecar functions and configured Health Checks on DEV
- Removed transformation Jsons from Processing status report (hl7-json and lake of segments)
- HL7-Json now no longer serialize attributes with no values
- Replaced processes array on metadata for a single Stage attribute (remove duplication of information on payloads)
- Refactored Receiver debatcher to be called from EventGrid events to avoid the Event hub duplication (Receiver cannot afford to receive duplicate events, otherwise the data will be fully duplicated across the pipeline with a new message-uuid.
- Improved EventHub sender to bypass messages that are too big (and Report on them accordingly
- Moved transformation output from "report" node to "output" node in JSON
- Upgrade azure-functions maven plugin due to strange bug
- Upgraded azure Bom to 1.2.20 due to NPE bug
- Fixed logging on file-sink functions
- Added extra metadata to Debatcher report
- Updated HL7 Json profile for
○ multiple patient ids
○ Changed file_separator to field_separator for MSH-1
- Created a Covid19 specific config for redaction
- Added /health endpoint to Debatcher Fn
- Created ability to Redact based on a expression for Covid requirements
- PHLIP-VPD fixes
- Improved Structure validator error messages when missing MD
- File-sink bug fix on date folders.
○ Removed content from payload.
- Using Static Event hub clients for better connection management.
- Updated Validation API documentation to include support for DAART
- Added validation profiles for NRSS
- Bug fixes on PHLIP FLU profiles
- Fixed a bug on lib-bumblebee for primitive types and cardinaltiy > 1
- Included meta_ext_uploadid as part of metadata
- Updated the Debatching report for Receiver-Debatcher Function with appropriate metadata and deployed file-sink sidecar function to provision the report to programs.
- Deployed Processing Status Reports sidecar to submit report to PS API
- Improved Lake of Segments schema to have a segment_id based on HL7 Path, and parent_segments to reference segment_id ordered from parent to grand-parent all the way to root.
- Upgraded Azure bom to 1.2.18
- Creating sidecar function to send appropriate Reports to Processing Status API
- Fixed metadata to remove duplicate config/configs and remove serialized eventHubMetadata
- Including DEX-Upload metadata on HL7 metadata (upload_id, destination_id, event)
- Deployed DAART profiles to Structure Validator
- Updated File-sink sidecar function to support sending data to Routing, using config to know folder name and default to partition files by yy/mm/dd subfolders.
- Modified Redaction Config of PID-5 to not reject messages that send data on PID-5[1] instead of PID-5[2]
- Performed bug fixes on PHLIP FLU Profiles
- Generating a Report for Receiver Debatcher indicating number of messages received per file.
- Implemented a CosmosDB client on lib-dex-commons used by cosmos-sink sidecar
- Configured Elastic functions to scale beyond 20 instances
- Created File sink side car to save events to storage account
- Created function to move files between containers (for performance testing)
- Deployed Branching sidecar fn and reconfigure pipeline functions to output events to a single eventhub.
- Added Constants.kt to lib-dex-commons
- Cleaned out CosmosDBOut annotations on all Functions.
- Improved Open API documentation for Validation API
- Added segment_id to Lake of segments and listing parentage using segment_id from parent to Grandparent order.
- Add Error Response JSON to Validation API
- Updated Validator API Postman collections
- Created sidecar function to handle branching of good/bad messages
- Created a CosmoDB Proxy on Lib-dex-commons for standardization on using cosmos db
- PHLIP_FLU bug fixes:
- MSH-21[1] missing.
- Performance Testing:
- Moved all Functions to Elastic Plans
- Changed Batch sizes (64, 32) for processing Event Hub messages.
- Added /health and /heartbeat endpoints to validator API and transport service
- Configured batch size and prefetch on all functions for better performance
- Created and Sidecar function to save event to cosmos DB
- Added CI/CD for Validator API and Replay API
- Creating Open API documentation for our APIs -Replay, Validation API
- HL7Transform: Added extra fields on OBX, and handling OBX children of SPM, fixed some property name typos
- Deployed our Validation API in DEV to fully support up to 100 msgs
○ Designed and implemented a summary report for multiple messages
○ Added Postman scripts for Happy Path tests
○ Created first draft of our Open API documentation
- Bug Fixes:
○ Fixed couple of issues with PHLIP-FLU profile
- Modified CI/CD Process to build artifact only once and deploy to TST with a single approval.
- Added private end points to all missing resources in Azure DEV RG
- Run performance tests to gauge CosmosDB and found some improvements due to high level of indexing done out-of-the-box
- Wrangling the replay API to deploy a first endpoint (/message_uuid) to DEV
- Migrated Structure validator to Elastic Plan in TST environment.
- Updated Structure Validator to load profiles with lazy-load approach
- Changed all Functions to post events to Event Hub via annotation (@EventHubOutput)
- Improved error handling on all functions to avoid losing messages
- Added HTTP endpoint for HL7 Json for quick transformations
- Updated local-run tool with all changes from last and this sprint (cosmosDB, etc)
- All pipeline functions syncing events to Cosmos DB
- All pipeline functions sending Events using annotation
- All pipeline functions migrated to Java 17 (Need DevOps support)
- Clean up DEV and TST environments (remove Redis and deprecated functions)
- Updated TST terraform scripts for current resources.
- Partitioning data from PHINMS and Mercury into year/month/day for Bulk load and daily load.
- Created an initial Validation API to expose our validation to States.
- Created an initial Processing Monitor API to query status of a single message against our data lake.
- Worked with Microsoft on Event Hubs Missing messages.
- Recovered Lake of segments unit tests
- Introduced Azure BOM on projects.
- Finalize and test logback for fns-hl7-pipeline functions
- Enhance DEX Metadata with "Unknown" source Metadata
- Identify if PHINMS has extra metadata
- Delete all MMG related functions from the HL7 Pipeline.
- Delete dependencies on Redis.
- Added Event Code logic to Structure Validator.
- Added handling of unexpected metadata in the incoming message.
- Updated pipeline to align ELR and Case messages to a single pipeline.
- Created Local Run tool allowing developers to run the HL7 pipeline on their local machines (Path: tools/local-run).
- Created SVC Validator for API driven validation.
- Started CosmosDB Writer, still in progress.
- Enhanced unit tests across project libraries ( lib-dex-commons, lib-nist-validator, lib-bumblebee ) and HL7 functions.
- Completed Performance testing and fixed some bugs found during testing.
- Fixed logging of Functions to show logs in Application Insights.
- Completed the local-run tool to run the pipeline locally without dependency on Azure.
- Completed Documentation for Programs.
- Fixed svc-transport by adding a "ping" endpoint to make Azure happy.
- Separated Unit test and Integration test for MMG Validator.
- Updated Structure Validator to use only Structure Error status for errors.
- Parameterized container name on Receiver/Debatcher
- Added Ping method to svc-cloud-transport for Azure requirements.
- Improved redactor to load different configuration for different message types/routes.
- Value set errors on Structure Validator set to warning.
- Updated summary metadata on all functions
- Adding local_record_id to message_info metadata.
- Fixed Original_file_name metadata
- Updated the upload_messages.py script to use multiple threads
- Implementing CI/CD for continuous delivery to DEV/TST.
- Added Hepatitis 2.0 MMG
- Added PHLIP FLU and PHLIP VPD Profiles
- Bug fixes on PHIN Spec 3.x
- Enhancements on Function for Performance purposes
- Improved HL7 Transform to not populate fully empty objects
- Unit test and coverage for MMG-Validator
- Added redaction fields for PHLIP
- Adding CI/CD process to deploy code changes after PRs.
- Created method to convert HL7 message into JSON, based on HL7-PET profiles.
- Created AZ function to process hl7 messages into JSON model based on method above.
- Updated HL7 Usage in Gen v1 MMG to "R" for all fields whose Cardinality is [1..1]
- Removed ZLR segments from Covid19 ELR Nist Profiles
- Improving logging statements of Receiver/Debatcher
- Added Code coverage for Lake Of Segments
- Improved Redis Proxy to use Connection pooling
- Removed constraint for HD fields that requires a valid OID
- Changed Unescaped delimiter errors from Error to Warning
- Using Class level configuration for Fn to improve performance - read configuration only once on class lifecycle
- Handling Lab Templates for MMG-Based and MMG-in-SQL
- Created Plan for Performance testing and ran some baselines in DEV env.
- Added Unit test for Receiver, Redactor and Structure Validator.
- Improvements on MMG-In-SQL configuration to reduce null values
- Added logic to MMG-Based and MMG-In-SQL to handle only EPI Observation
- Ignoring empty lines when ingesting HL7 message
- Removed validation of NND value set against OBR-31 (event code)
- Handling subcomponents on MMG-Based schema
- Properly validating different OBR fields on GenV1
- Added support for new conditions:
- Cryptosporidiosis
- Cyclosporiasis
- Cholera and Vibriosis
- Campylobacteriosis
- Salmonellosis
- STEC
- Shigellosis
- S Typhi and S Paratyphi
- FoodNet
- Candida Auris
- CP-CRE
- Updated Receiver to be able to handle files in sub folders.
- Updated MMG-Based and MMG-SQL to support new feature of the new conditions
- Elements mapped more than in one block
- Handling properly complex fields that are not populated
- Added check for empty single collections.
- Create sub tables for repeating elements in repeating groups for MMG-in-SQL
- Added unit test and code coverage to Receiver/Debatcher and Redactor functions.
- Added support for file upload to cloud-transport service and support for up to 10Mb files.
- Renamed MSH-21 column names to be one based instead of zero based.
- Removed rule that validates OBR-22 against OBR-7
- Added validation for MMWR week.
- Updated PHIN Spec profiles to match discrepancies finding testing against MVPS
- Fixed fortify scan Critical and High issues.
- Updated Readme files with latest changes and features
- Updated all process names and process status on all functions to be consistent.
- Redacting ELR messages
- Updated NIST Profiles to remove rules not present in MVPS
- Updated Legacy MMGs to not include subject/patient name type
- Lake of Segments now processing ELR Messages
- Added more MMG support - STD, congenital syphilis, Rubella, Congenital Rubella, Mumps, Malaria, Tuberculosis
- Bug Fixes
- Properly handling Repeating elements when creating inner lists in repeating table
- Repeating elements of type ST not properly mapped
- Changed repeat table to null instead of empty array in JSON output
- Handling null repeating elements in repeating groups
- Properly mapping primitive repeating elements
-
Added config information to event metadata (configuration file names)
-
Added MUMPS, Pertussis, Tuberculosis, Babesiosis, Trichinellosis MMGs and Event Codes
-
Updates on PHIN Spec profiles
- Replaced predicate statements with conformance statements for better uer messages.
- Removed content validation of dates from IGAMT profiles into MMG validation.
-
Updated structure validator to validate HL7 delimiters
-
Creted tool to update IGAMT profile XML files with friendly error messages and deployed new updated profiles.
-
Improved performance on MMG Validator by refactoring Redis lookups
-
Metadata bug fixes
- Redacted did not update summary
- Process metadata missing when exceptions were thrown on several functions.
-
Dev Ops
- Moved Redis to Premium in Dev and TST with automated backups
- Terraformed cloud service app and deployed in DEV, TST and STG (waiting on firewall for private end points)
- Started process to create ADF for AWS S3 Data migration.
- Created PhinMS Security Architecture diagram.
- Rerun Fortify Scans
- Set up STG environment and pipelines
- Updated PHIN Specs with findings on discrepacies with MVPS validation
- Shrotened the name of columns and repeating groups due to size limitations
- Included extra fields to be redacted
- Created redaction config for lab messages
- Added config names to Processes for clarity on which configuration was used.
- Several bug fixes.
- Set up TST environment and pipelines
- Develop transport service to upload files with Metadata (pending deployment)
- Designed and Develop Transport metadata to identify messages upon upload
- Flaggged cardinality over as warnings
- Added vocabulary check for OBR-31/GenV2 messages
- Added COVID ELR profiles for structure validation
- Bug fixes and Enhancements
- Fixed NK1 Redaction paths
- Routing unknwon messages properly
- Fixed MMG-Based and MMG-In-SQL transform to support non-GenV2 messages
- Created new Function to Redact messages
- Adding MMG for Varicella and TB Messages
- Started JADs for incorporating ELR messages on the pipeline.
- Created Lake of Segments transformations
- Modified Validation rule for CARDINALITY_OVER to always be an WARNING (pipeline will pick first value if multiple are present)
- Added EventHub metadata to Receiver/Debatcher function
- Fixed Fortify Vulnerability scans issues
- Enhanced MMG Validation with following rules:
- OBX 3 + OBX-4 must be unique
- OBX-4 must be present for grouped elements
- Added HTTP endpoints for structure and mmg-validation for QA
- Parameterized Azure environment specific variables on pom files of Azure function projects to help with CI/CD
- Finalized upgrade of NIST library for v. 1.5.6 and scala 2.13
- Added message_info to metadata payload with relevant information from message, including routing.
- Updated mmg-based-transformer with exception handling for bad format event hub messages or transformations exceptions.
- Updated mmg-sql-transformer with exception handling for bad format event hub messages or transformations exceptions.
- Updated mmg-sql-transformer for messsage_profile_identifier changed from tables to flat model.
- Updated all functions for dev for function names based on new teraform config and added sample reports in all functions.
- Upgraded Nist Libraries for scala 2.13 and retrofitted upgrade to our own lib-nist-validator.
- Created first draft of Lab Template MMG configuration.
- Fixed bug where JSON was supressing elements with NULL values in lib-dex-commons.
- Renamed mmg-processor module to mmg-based-transformer
- Created function to load Condition2MMGMapping table
- Created new function fn-mmg-sql-transformer which transforms the output of the fn-mmg-based-transformer, which includes all MMG elements: singles, singles repeated, and repeated blocks.
- Designed Data consumption routing and implemented changes on metadata. (Need to implement receiver-debatcher)
- Updated Receiver/Debatcher to throw invalid messages into the Error queue (hl7_recdeb_err)
- Created Readme to several modules - lib-dex-common, lib-nist-validator, Receiver/Debatcher, Structure Validator
- Updated getMMGs() to throw error if condition is not present on table and created method to retrieve a list of MMG names used by a given scenario (event_code, profile, reporting_jursidiction)
- Created Hepatitis MMGs
- Created PhinSpec 3.2 profiles for NIST Structure Validation
- Updated MMG-Based Transformer (fn-message-processor) to generate HL7-specific field elements and added PHIN-Vads concept name and preferred concept names.
- Bug Fixes
- Fixed Debatcher Code broken due an previous update
- MMGReport status was printing the entire report instead of just the status.
- Missing entries in Redis for Value Set
- Added new features to lib-dex-commons: RedisProxy & String normalization function.
- Added ability to load legacy MMGs into Redis.
- Added method to retrieve appropriate MMGs based on HL7 message for Validation and transformation.
- Updated AZ Functions to properly populate summary metadata.
- Added Hepatitis support for structure and MMG validation.
- Created Message Processor to transform HL7 into MMG-Based schema.
- Created Databricks notebooks to ingest Event Hub messages.
- Updated all functions (fn-structure-validator, fn-mmg-validator, fn-receiver-debatcher) to be deployed and running on AZ
- Updated metadata on all functions (see above) based on design - #105 .
- Created lib-dex-commons library for sharing code among functions/services.
- Updated fn-structure-validator to be kotlin (instead of scala)
- Replaced hl7-structure-validator (scala) with lib-nist-validator (Kotlin)
- Updated PhinVads for error management.
- Created Azure Data Explorer to ingest Event Hub messages for monitoring and debugging.
- Updated Legacy Arbo MMG.
- Created Unit tests for structure and mmg validation.
- Added Vocabulary validation to MMG Validator.
- Added all Legacy MMGs and both GenV1.
- Added Unit test for GenV1 messages.
- Changed build and configuration files for Receiver/Debatcher for Deployment.
- Changed Structure Validator to use basic Json Objects instead of POJOs.
- Created MMG Validator Function.
- Created tool to generate MMG configurations
- Created GenV1 MMG configurations ( Case and Summary).
- Fixed Structure Validation Report serialization.
- HL7 Structural Validator: added PHIN 2.0 and 3.0 profiles, updated to validate based on profile requested
- Added Phin Spec 2.0 and 3.0 IGAMT Profiles.
- HL7 validation function to structure validate messages
- Started on HL7 Message Processor function
- Updated Phin VADS function to be based on schedule.
- Started on MMG Loader function.
- Developed Receiver-Debatcher function
- Developed phin-vads function
- First Graphana dashboard with basic infrastructure metrics
- Case-based surveillance data pipeline proof-of-concept for Spark DataBricks
- HL7 Messages structural validator
- Docs: How to create development environment for Azure functions project
- Tools: Various HL7 development tools or experiments
- Folders for future tools: mmg, phin-vocab, and fns-hl7-pipeline
- Phin-vocab function for fetching the value sets from phinvads api.