Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDM on Synapse dedicated pool #155

Merged
merged 8 commits into from
Jun 26, 2023
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,13 @@ Some of the OHDSI projects included:

## CDM Version

This setup is based on the [CDM v5.4.0 for PostgreSQL](https://github.com/OHDSI/CommonDataModel/tree/main/inst/ddl/5.4/postgresql).
This setup is based on the [CDM v5.4.0](https://github.com/OHDSI/CommonDataModel/tree/main/inst/ddl/5.4) and supports both PostgreSQL and Azure Synapse.

## Getting Started

To get started, click on deploy to Azure button.
To get more detailed instructions, please refer to the [Deployment Guide](./docs/DeploymentGuide.md).

[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2Fmicrosoft%2FOHDSIonAzure%2Fv2%2Finfra%2Farm_output%2Fmain.json)

## Contributing
Expand Down
11 changes: 6 additions & 5 deletions docs/DeploymentGuide.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,18 @@ In order to deploy OHDSI on Azure, you will need the following prerequisites:
| Region | The region of the resource group. | As the resource group |
| Location | The location of the resources. | Resource group location |
| Suffix | The suffix for the resource names. | Unique string is being generated |
| CDM Container URL | The URL prefix or Blob container where CDM files can be downloaded from. | <https://omoppublic.blob.core.windows.net/shared/synthea1k/> |
| CDM SAS Token | The SAS token for accessing the CDM container. Leave empty if the files are available anonymously. | Blank |
| Postgres OMOP CDM Database Name| The name of the PostgreSQL OMOP CDM database. | Synthea1k |
| CDM Container URL | The URL prefix or Blob container where CDM files can be downloaded from. | <https://omoppublic.blob.core.windows.net/shared/synthea1k/> |
| CDM SAS Token | The SAS token for accessing the CDM container. Leave empty if the files are available anonymously. | Blank |
| OMOP CDM Database Type | The type of database to host the CDM on. Valid values are: `PostgreSQL`, `Synapse Dedicated Pool` | PostgreSQL |
| OMOP CDM Database Name | The name of the OMOP CDM database. | synthea1k |
| OMOP CDM Password | The admin password for the OMOP CDM database. | Unique password is being generated |
| App Plan SKU | The SKU for the app plan. | S1 |
| Postgres SKU | The SKU for the PostgreSQL database. | Standard_D2s_v3 |
| Postgres Storage Size | The storage size for the PostgreSQL database. | 32 |
| Postgres Admin Password | The password for the PostgreSQL admin user. | Unique password is being generated |
| Postgres WebAPI Admin Password| The password for the PostgreSQL WebAPI admin user. | Unique password is being generated |
| Postgres WebAPI App Password | The password for the PostgreSQL WebAPI app user. | Unique password is being generated |
| Postgres OMOP CDM Password | The password for the PostgreSQL OMOP CDM user. | Unique password is being generated |
| Atlas Security Admin Password | The password for the Atlas security admin user. | Unique password is being generated |
| Atlas Security Admin Password | The password for the Atlas security admin user. | Unique password is being generated |
| Atlas Users List | Atlas Users List | The list of users for the Atlas system should be provided in the following format: username1,password1,username2,password2' and so on. Ensure usernames and passwords are in the correct order and do not include any additional spaces or characters between the credentials. Note that this format allows you to specify multiple sets of username-password pairs for different users. Make sure each pair is properly formatted and separated by commas. | None |None |
| Local Debug | Enable local debugging mode. If enabled, it will create a Firewall rule that will enable unrestricted access to the PostgreSQL database from any location. | false |

Expand Down
391 changes: 376 additions & 15 deletions infra/arm_output/main.json

Large diffs are not rendered by default.

48 changes: 37 additions & 11 deletions infra/main.bicep
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ param cdmContainerUrl string = 'https://omoppublic.blob.core.windows.net/shared/
param cdmSasToken string = ''

@description('The name of the database to create for the OMOP CDM')
param postgresOMOPCDMDatabaseName string = 'synthea1k'
param OMOPCDMDatabaseName string = 'synthea1k'

@description('The app service plan sku')
@allowed([
Expand Down Expand Up @@ -78,7 +78,7 @@ param postgresWebapiAppPassword string = uniqueString(newGuid())

@secure()
@description('The password for the postgres OMOP CDM user')
tamirkamara marked this conversation as resolved.
Show resolved Hide resolved
param postgresOMOPCDMPassword string = uniqueString(newGuid())
param OMOPCDMPassword string = uniqueString(newGuid())

@secure()
@description('The password for atlas security admin user')
Expand All @@ -91,6 +91,14 @@ param atlasUsersList string
@description('Enables local access for debugging.')
param localDebug bool = false

@description('OMOP CDM database type')
@allowed([
'PostgreSQL'
'Synapse Dedicated Pool'
]
)
param cdmDbType string = 'PostgreSQL'

var tenantId = subscription().tenantId

@description('Creates the app service plan')
Expand Down Expand Up @@ -182,29 +190,46 @@ module ohdsiWebApiWebapp 'ohdsi_webapi.bicep' = {
]
}

@description('Creates OMOP CDM database')
module omopCDM 'omop_cdm.bicep' = {
name: 'omopCDM'
@description('Creates OMOP CDM database on Postgres')
module omopCDMPostgres 'omop_cdm_postgres.bicep' = if (cdmDbType == 'PostgreSQL') {
name: 'omopCDMPostgres'
params: {
location: location
keyVaultName: keyVault.name
cdmContainerUrl: cdmContainerUrl
cdmSasToken: cdmSasToken
postgresAtlasDatabaseName: atlasDatabase.outputs.postgresWebApiDatabaseName
postgresOMOPCDMDatabaseName: postgresOMOPCDMDatabaseName
postgresOMOPCDMDatabaseName: OMOPCDMDatabaseName
postgresAdminPassword: postgresAdminPassword
postgresWebapiAdminPassword: postgresWebapiAdminPassword
postgresOMOPCDMPassword: postgresOMOPCDMPassword
postgresOMOPCDMPassword: OMOPCDMPassword
postgresServerName: atlasDatabase.outputs.postgresServerName
ohdsiWebapiUrl: ohdsiWebApiWebapp.outputs.ohdsiWebapiUrl
}

dependsOn: [
ohdsiWebApiWebapp
atlasDatabase
]
}

@description('Creates OMOP CDM database on Synapse')
module omopCDMSynapse 'omop_cdm_synapse.bicep' = if (cdmDbType == 'Synapse Dedicated Pool') {
name: 'omopCDMSynapse'
params: {
location: location
suffix: suffix
keyVaultName: keyVault.name
cdmContainerUrl: cdmContainerUrl
cdmSasToken: cdmSasToken
databaseName: OMOPCDMDatabaseName
sqlAdminPassword: OMOPCDMPassword
ohdsiWebapiUrl: ohdsiWebApiWebapp.outputs.ohdsiWebapiUrl
}
dependsOn: [
ohdsiWebApiWebapp
]
}

@description('Creates the ohdsi atlas UI')
module atlasUI 'ohdsi_atlas_ui.bicep' = {
name: 'atlasUI'
Expand Down Expand Up @@ -294,7 +319,7 @@ resource deplymentAddDataSource 'Microsoft.Resources/deploymentScripts@2020-10-0
environmentVariables: [
{
name: 'CONNECTION_STRING'
secureValue: 'jdbc:postgresql://${atlasDatabase.outputs.postgresServerFullyQualifiedDomainName}:5432/${postgresOMOPCDMDatabaseName}?user=postgres_admin&password=${postgresOMOPCDMPassword}&sslmode=require'
secureValue: cdmDbType == 'PostgreSQL' ? 'jdbc:postgresql://${atlasDatabase.outputs.postgresServerFullyQualifiedDomainName}:5432/${OMOPCDMDatabaseName}?user=postgres_admin&password=${OMOPCDMPassword}&sslmode=require' : omopCDMSynapse.outputs.OmopCdmJdbcConnectionString
}
{
name: 'OHDSI_WEBAPI_PASSWORD'
Expand All @@ -310,7 +335,7 @@ resource deplymentAddDataSource 'Microsoft.Resources/deploymentScripts@2020-10-0
}
{
name: 'DIALECT'
value: 'postgresql'
value: cdmDbType == 'PostgreSQL' ? 'postgresql' : 'synapse'
}
{
name: 'SOURCE_NAME'
Expand Down Expand Up @@ -346,6 +371,7 @@ resource deplymentAddDataSource 'Microsoft.Resources/deploymentScripts@2020-10-0
dependsOn: [
deploymentAtlasSecurity
ohdsiWebApiWebapp
omopCDM
omopCDMPostgres
omopCDMSynapse
]
}
15 changes: 11 additions & 4 deletions infra/omop_cdm.bicep → infra/omop_cdm_postgres.bicep
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ var postgresWebapiAdminUsername = 'ohdsi_admin_user'
var postgresAdminUsername = 'postgres_admin'
var postgresOMOPCDMJDBCConnectionString = 'jdbc:postgresql://${postgresServer.properties.fullyQualifiedDomainName}:5432/${postgresOMOPCDMDatabaseName}?user=${postgresOMOPCDMUsername}&password=${postgresOMOPCDMPassword}&sslmode=require'


// Get the postgres server
resource postgresServer 'Microsoft.DBforPostgreSQL/flexibleServers@2022-12-01' existing = {
name: postgresServerName
Expand Down Expand Up @@ -152,16 +153,20 @@ resource deploymentScript 'Microsoft.Resources/deploymentScripts@2020-10-01' = {
name: 'POSTGRES_OMOP_TEMP_SCHEMA_NAME'
value: postgresOMOPTempSchemaName
}
{
name: 'OHDSI_WEBAPI_URL'
value: ohdsiWebapiUrl
}
{
name: 'SQL_create_omop_schemas'
value: loadTextContent('sql/create_omop_schemas.sql')
value: loadTextContent('sql/create_omop_schemas_postgres.sql')
}
{
name: 'SQL_create_achilles_schema'
value: loadTextContent('sql/create_achilles_schema.sql')
name: 'SQL_create_achilles_tables'
value: loadTextContent('sql/create_achilles_tables_postgres.sql')
}
]
scriptContent: loadTextContent('scripts/create_omop_cdm.sh')
scriptContent: loadTextContent('scripts/create_omop_cdm_postgres.sh')
supportingScriptUris: [
'https://raw.githubusercontent.com/OHDSI/CommonDataModel/main/inst/ddl/5.4/postgresql/OMOPCDM_postgresql_5.4_ddl.sql'
'https://raw.githubusercontent.com/OHDSI/CommonDataModel/main/inst/ddl/5.4/postgresql/OMOPCDM_postgresql_5.4_constraints.sql'
Expand All @@ -175,3 +180,5 @@ resource deploymentScript 'Microsoft.Resources/deploymentScripts@2020-10-01' = {
}
}
}

output OmopCdmJdbcConnectionString string = postgresOMOPCDMJDBCConnectionString
Loading