Develop#169
Merged
MattPCollins merged 22 commits intomainfrom Jun 12, 2025
Merged
Conversation
Forward merge to keep develop in sync
* Update packages * Published functionapp with updated NuGet packages * Update .gitignore
* Overhaul PowerShell and Bicep 1-click deployment functionality * Update Bicep configurations * Update Databricks Deployment options * Update bicep deployment logic - tidy up namings and small bug fix with transform * Update gitignore for Functions App
* Adding linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Updating linkedService: Ingest_LS_Jira_UserAuth * Adding JiraObject: Ingest_DS_Jira_UserAuth * Updating JiraObject: Ingest_DS_Jira_UserAuth * Adding pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Updating pipeline: Ingest_PL_Jira * Update DefaultConnectionTypes.sql
* Adding linkedService: Ingest_LS_Oracle_V2_SIDAuth * Adding integrationRuntime: integrationRuntimeCumulusDataSources02 * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating linkedService: Ingest_LS_Oracle_V2_SIDAuth * Updating linkedService: Ingest_LS_Oracle_V2_SIDAuth * Adding OracleTable: Ingest_DS_Oracle_V2_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Updating pipeline: Ingest_PL_Oracle * Updating pipeline: Ingest_PL_Oracle * Update GetDatasetPayload.sql removed PSQL from the clause that adds a semicolon * Updating linkedService: Ingest_LS_Oracle_V2_SIDAuth * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updated gitignore to exclude integrationRuntime folder --------- Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
* Databricks Module restructure * refactor notebooks to functions * Update file references * Update Ingest to reference new python files rather than notebooks * refactor pep8 naming * refactor pep8 naming convention * refactor pep8 naming conventions * integration test changes for pep8 formatting * refactor transform python pep8 naming conventions * Add python file references to Transform notebooks * Integration tests new Transform Managed Notebooks * Add further ingest tests and utils.HelperFunctions final tests * Add unit tests for remaining CheckPayloadFunctions * Update automated testing for python unit tests * update unit test workflow * amend pytest file path * amend pytest file path * Update pytest runner for correct paths --------- Co-authored-by: MattPCollins <MattPCollins>
* Additional utilities unit tests * update automated testing for python unit tests
* Adding linkedService: Common_LS_Databricks_Cluster_MIAuth * Updating linkedService: Common_LS_Databricks_Cluster_MIAuth * Updating pipeline: Ingest_PL_Merge * Updating pipeline: Transform_PL_Managed * Updating pipeline: Transform_PL_Unmanaged * Updating pipeline: Transform_PL_Unmanaged * Updating pipeline: Transform_PL_Managed * Adding linkedService: Common_LS_Databricks_JobCluster_MIAuth * Updating linkedService: Common_LS_Databricks_JobCluster_MIAuth * Updating pipeline: Transform_PL_Managed * Updating pipeline: Transform_PL_Unmanaged * Updating pipeline: Transform_PL_Managed * Updating pipeline: Ingest_PL_Merge * Update MetadataAsCodeCommon Script for updated LS names * Deleting linkedService: Transform_LS_Databricks_Cluster_MIAuth * Deleting linkedService: Ingest_LS_Databricks_Cluster_MIAuth * Deleting linkedService: Ingest_LS_Databricks_JobCluster_MIAuth --------- Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
* Update Databricks CICD process * Update Databricks CICD process2 * Test DAB profile config * Test DAB profile config with script correction * Test DAB profile config with script correction * Extend DAB logic to test/prod and rename WS file refernces for databricks in ADF pipelines and metadata. * Databricks complete deploy in PS
* Update adf-cicd.yml adding additional parameters * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Updating linkedService: Ingest_LS_SQLServer_WinAuth * Updating linkedService: Ingest_LS_SQLServer_SQLAuth * Updating linkedService: Ingest_LS_SQLDB_SQLAuth * Update adf-cicd.yml adding additional parameters * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating pipeline: Ingest_PL_Oracle * Updating linkedService: Ingest_LS_SQLServer_WinAuth * Updating linkedService: Ingest_LS_SQLServer_SQLAuth * Updating linkedService: Ingest_LS_SQLDB_SQLAuth * Updating pipeline: Ingest_PL_MSSQL * Update functions-cicd.yml * Adding integrationRuntime: SHIRcumulusDataSources02 * Updating linkedService: Ingest_LS_Oracle_SIDAuth * Updating linkedService: Ingest_LS_Oracle_SIDAuth
* Update SQLCMD variables * update keyvault secret name * Added demo connection variables * Update sql-server-common-cicd.yml * Update Control SQLCMDs and MetadataAsCode * updates to ingest cicd * remove unused notebook and update transform cicd * Creating Metadata.core and metadata.data projs * create yml files for core, data and master * Update sql-server-master-cicd.yml * Add Pipeline dependencies and move yml files to folder * Add transform BL Notebooks, metadata as code and integration execution in dev. * Add transform BL Notebooks, metadata as code and integration execution in dev * rename transform pipelines * Pipeline name tidy-up --------- Co-authored-by: MattPCollins <mc12338@my.bristol.ac.uk> Co-authored-by: MattPCollins <34279269+MattPCollins@users.noreply.github.com>
* Update notebook path * Update dim products notebook content.
* Update dimdate typing * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files * Updating DAB config for historic files
Collaborator
Author
|
HL Summary:
|
CFToby-Andrew
approved these changes
Jun 12, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.