From f0762a74b8cd647bfcdb257a3701abe3964c6b3e Mon Sep 17 00:00:00 2001 From: Thomas Rittson Date: Tue, 7 Oct 2025 18:57:47 +1000 Subject: [PATCH 1/6] Add mariadb instructions and other tweaks --- .../server/database/ef/index.mdx | 132 +++++++++++++----- 1 file changed, 96 insertions(+), 36 deletions(-) diff --git a/docs/getting-started/server/database/ef/index.mdx b/docs/getting-started/server/database/ef/index.mdx index 1267411e2..7e892b4e8 100644 --- a/docs/getting-started/server/database/ef/index.mdx +++ b/docs/getting-started/server/database/ef/index.mdx @@ -12,6 +12,10 @@ export const providers = [ label: "MySQL", value: "mysql", }, + { + label: "MariaDB", + value: "mariadb", + }, { label: "SQLite", value: "sqlite", @@ -80,7 +84,20 @@ sure you update the existing values instead of creating new ones Be sure to change information like root password as needed. If you already have these secrets, make -sure you update the existing values instead of creating new ones +sure you update the existing values instead of creating new ones. + +```json +"globalSettings:databaseProvider": "mysql", +"globalSettings:mySql:connectionString": "server=localhost;uid=root;pwd=example;database=vault_dev", +``` + + + + +MariaDB is generally a drop-in alternative to MySQL, so it shares the `mysql` configuration keys. + +Be sure to change information like root password as needed. If you already have these secrets, make +sure you update the existing values instead of creating new ones. ```json "globalSettings:databaseProvider": "mysql", @@ -109,7 +126,7 @@ that the changes take effect. ::: -### Updating the database +### Start the database server + + +1. Confirm that `MYSQL_ROOT_PASSWORD` in `dev/.env` matches the password in `dev/secrets.json`. + +2. In the `dev` folder of your server repository, run + +```bash +docker compose --profile mariadb up +``` + +:::tip[Confirm your database connection!] + +If you run into connection errors, double check that your `.env` and `secrets.json` files have +matching passwords. If they do, you may have initialized your database incorrectly. Delete the +Docker storage volume and initialize the database from scratch. + +::: + +No additional step is required for SQLite. The migrator will create the database file if it doesn't +exist. + :::tip[Confirm your database path!] -The migrator creates the database file if it doesn't exist, but it does not create folders. If you -get an error that the path doesn't exist, double check that the path exists and that the folder -containing the sqlite database has write and/or create permissions. +The migrator does not create folders. If you get an error that the path doesn't exist, double check +that the path exists and that the folder containing the sqlite database has write and/or create +permissions. ::: -### Migrations +:::tip[Start all containers at once] + +Use the `ef` profile to start all EntityFramework database containers at once: + +```bash +docker compose --profile ef up +``` + +::: + +### Run Migrations + +In the `dev` folder, run the following command to update the database to the latest migration: -In the `dev` folder run the following to update the database to the latest migration - ```bash pwsh migrate.ps1 -postgres ``` -The `-postgres` flag on `migrate.ps1` runs `dotnet ef` commands to perform the migrations. - -In the `dev` folder run the following to update the database to the latest migration - ```bash pwsh migrate.ps1 -mysql ``` -The `-mysql` flag on `migrate.ps1` runs `dotnet ef` commands to perform the migrations. + + + +```bash +pwsh migrate.ps1 -mariadb +``` -In the `dev` folder run the following to update the database to the latest migration - ```bash pwsh migrate.ps1 -sqlite ``` -The `-sqlite` flag on `migrate.ps1` runs `dotnet ef` commands to perform the migrations. - :::note The migrator creates the database file if it doesn't exist, but it does not create folders. If you @@ -212,13 +258,7 @@ get an error that the path doesn't exist, it's referring to missing folders. -You can also run migrations for all database providers at once using - -```bash -pwsh migrate.ps1 -all -``` - -### Verifying changes +### Verify changes If you would like to verify that everything worked correctly: @@ -227,10 +267,14 @@ If you would like to verify that everything worked correctly: - Note: this requires a configured MSSQL database. You may also need to set up other EF providers for tests to pass. -## Testing changes +## Database integration tests -In your `server/dev/secrets.json` file find or add this block of secrets in the root of the json -structure: +Database integration tests run for each database provider for both Dapper (MSSQL) and Entity +Framework. Developers are not expected to manually test each database provider. Instead, use +integration tests to ensure correctness across all supported databases. + +1. In your user secrets, find or add this block of secrets in the root of the json structure + (**not** in `GlobalSettings`): ``` "databases:0:type": "Postgres", @@ -245,6 +289,9 @@ structure: "databases:3:type": "SqlServer", "databases:3:connectionString": "Server=localhost;Database=ef_test;User Id=SA;Password=_________;Encrypt=True;TrustServerCertificate=True;", "databases:3:enabled": "true" +"databases:4:type": "MySql", +"databases:4:connectionString": "server=localhost;port=4306;uid=maria;pwd=_________;database=ef_test;AllowUserVariables=true", +"databases:4:enabled": "true", ``` :::info @@ -255,14 +302,27 @@ multiple versions of the same database running tests at the same time. ::: This block is used for test databases for each supported provider type. These are what integration -tests will connect to. You should update the password for these connection strings to match your -existing databases if you have not already. If these settings are not present at all in your -`server/dev/secrets.json` file just add them to the bottom. These settings _do not_ go in -`globalSettings`. Then run `pwsh setup_secrets.ps1 -clear` to apply them to your local projects. - -With connection strings applied to your projects: ensure your databases are all migrated using -`pwsh server/dev/migrate.ps1 --all`. Then you can run EF tests from the -`test/Infrastructure.IntegrationTest` folder using `dotnet test`. +tests will connect to. Make sure that you fill in the password for each connection string. It is +recommended that you use a different database for your integration tests as they will produce a lot +of test data. + +2. Run `pwsh setup_secrets.ps1 -clear` to apply the updated user secrets to your local projects. + +3. Ensure your databases are all migrated (see instructions above). You can use the `-test` flag to + only migrate the integration test databases, or `-all` to migrate everything. + + ``` + # Migrate integration test databases only + pwsh migrate.ps1 -test + + # Migrate a specific integration test database + pwsh migrate.ps1 -postgres -test + + # Migrate a specific development database + pwsh migrate.ps1 -postgres + ``` + +4. Run integration tests from the `test/Infrastructure.IntegrationTest` folder using `dotnet test`. # Modifying the database From ff64d7323c8d40e93ed086842f5aceae4006105f Mon Sep 17 00:00:00 2001 From: Thomas Rittson Date: Tue, 7 Oct 2025 19:07:05 +1000 Subject: [PATCH 2/6] Tweak separate testing page --- docs/contributing/testing/database/index.md | 41 ++++++++++----------- 1 file changed, 20 insertions(+), 21 deletions(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index 1f62c5d13..a89bca43b 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -7,20 +7,21 @@ pipeline. ## Creating a new test -To create a new database test, add the `[DatabaseTheory]` and `[DatabaseData]` attributes to test. -Then, use the parameters of the test to inject any repository layer services you need. The test will -run for every database that is [configured in the current environment](#configure-the-tests). Since -you inject the interface of the service, some runs of the test will use the Dapper-based repository +To create a new database test, add the `[Theory]` and `[DatabaseData]` attributes to test. Then, use +the parameters of the test to inject any repository layer services you need. The test will run for +every database that is [configured in the current environment](#configure-the-tests). Since you +inject the interface of the service, some runs of the test will use the Dapper-based repository implementation targeting Microsoft SQL Server and others will use the Entity Framework Core based -implementations (which we use for MySql, Postgres, and SQLite). +implementations (which we use for all other supported databases). The goal of database tests is to test the business logic that is encapsulated in a given method. For -example, if a stored procedure in SQL Server calls another procedure to update the -`User.AccountRevisionDate` then the corresponding EF implementation should do that as well. By -running the test against all variants, we are ensuring all the variants are feature-equal. to only -run the SQL Server tests along with one EF implementation; SQLite is often the easiest in that -regard. The other supported EF database providers will still run in the pipeline to catch any -differences between them. +example, if a stored procedure in SQL Server updates the `User.AccountRevisionDate` then the +corresponding EF implementation should do that as well. By running the test against all variants, we +are ensuring all the variants are feature-equal. + +During development, you may choose to only run the SQL Server tests along with one EF +implementation; SQLite is often the easiest in that regard. The other supported EF database +providers will still run in the pipeline to catch any differences between them. ## Configure the tests @@ -51,14 +52,11 @@ will be configured with the Entity Framework Core repositories. `Enabled` allows disable one database but not delete the entry; it can be helpful if you are encountering a problem with just a single database type and want to run the tests just for it instead of for all of them. -### Locally - -To set the tests up locally you may want to add the configuration to your `server/dev/secrets.json` -file. You may have already done this during setup and can just run the tests with `dotnet test`. If -not, please refer to -[the getting started guide](/getting-started/server/database/ef/#testing-ef-changes). +## Locally -You can also configure the tests just like the pipeline. +To run the tests locally during development (highly recommended), see +[Getting Started - Entity Framework](../../../getting-started/server/database/ef/index.mdx). Once +configured, you can run the tests with `dotnet test` or using your IDE. ### Pipeline @@ -82,9 +80,10 @@ across all configured database providers. :::note -This is meant for testing data migrations only. It assumes your database schema is already fully -up-to-date. After setting up your test data, it re-runs the specified migration to verify how it -transforms the data. It will not work for schema-only migrations. +This is meant for testing data migrations only (i.e. migrations that transform data already present +in the database). It assumes your database schema is already fully up-to-date. After setting up your +test data, it re-runs the specified migration to verify how it transforms the data. It will not work +and is not required for schema-only migrations. ::: From 233180f358a02eca1aff1ed74432a277306c64c9 Mon Sep 17 00:00:00 2001 From: Thomas Rittson Date: Tue, 7 Oct 2025 19:17:29 +1000 Subject: [PATCH 3/6] headings and cross references --- .../server/database/ef/index.mdx | 34 ++++++++++++++----- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/getting-started/server/database/ef/index.mdx b/docs/getting-started/server/database/ef/index.mdx index 7e892b4e8..ccfcf10a6 100644 --- a/docs/getting-started/server/database/ef/index.mdx +++ b/docs/getting-started/server/database/ef/index.mdx @@ -273,6 +273,12 @@ Database integration tests run for each database provider for both Dapper (MSSQL Framework. Developers are not expected to manually test each database provider. Instead, use integration tests to ensure correctness across all supported databases. +Integration tests have their own connection strings, so that you can use separate databases to the +one used by your local development server. This is recommended because integration tests produce a +lot of test data over time. + +### Configuring test databases + 1. In your user secrets, find or add this block of secrets in the root of the json structure (**not** in `GlobalSettings`): @@ -294,6 +300,12 @@ integration tests to ensure correctness across all supported databases. "databases:4:enabled": "true", ``` +:::note + +The second MySql entry refers to MariaDB. + +::: + :::info The example database index + type combinations are required for the tooling to work, and to support @@ -302,9 +314,7 @@ multiple versions of the same database running tests at the same time. ::: This block is used for test databases for each supported provider type. These are what integration -tests will connect to. Make sure that you fill in the password for each connection string. It is -recommended that you use a different database for your integration tests as they will produce a lot -of test data. +tests will connect to. Make sure that you fill in the password for each connection string. 2. Run `pwsh setup_secrets.ps1 -clear` to apply the updated user secrets to your local projects. @@ -312,19 +322,25 @@ of test data. only migrate the integration test databases, or `-all` to migrate everything. ``` - # Migrate integration test databases only - pwsh migrate.ps1 -test - - # Migrate a specific integration test database + # Migrate a specific integration test database (used by tests) pwsh migrate.ps1 -postgres -test - # Migrate a specific development database + # Migrate a specific development database (used by your local server) pwsh migrate.ps1 -postgres + + # Migrate all local databases + pwsh migrate.ps1 -all ``` 4. Run integration tests from the `test/Infrastructure.IntegrationTest` folder using `dotnet test`. -# Modifying the database +### Writing integration tests + +See +[Contributing - Database Integration Testing](../../../../contributing/testing/database/index.md) +for more information on testing utilities and patterns. + +## Modifying the database The process for modifying the database is described in [Migrations](./../../../../contributing/database-migrations/ef.md). From da1c0081e79ad1b8208b179ae5822c3f2af643a3 Mon Sep 17 00:00:00 2001 From: Thomas Rittson Date: Tue, 7 Oct 2025 19:33:17 +1000 Subject: [PATCH 4/6] clarify --- docs/getting-started/server/database/ef/index.mdx | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/getting-started/server/database/ef/index.mdx b/docs/getting-started/server/database/ef/index.mdx index ccfcf10a6..e8f352f04 100644 --- a/docs/getting-started/server/database/ef/index.mdx +++ b/docs/getting-started/server/database/ef/index.mdx @@ -275,7 +275,8 @@ integration tests to ensure correctness across all supported databases. Integration tests have their own connection strings, so that you can use separate databases to the one used by your local development server. This is recommended because integration tests produce a -lot of test data over time. +lot of test data over time. In the example below, this is done by using `vault_test` as the database +name. ### Configuring test databases @@ -284,19 +285,19 @@ lot of test data over time. ``` "databases:0:type": "Postgres", -"databases:0:connectionString": "Host=localhost;Username=postgres;Password=_________;Database=ef_test", +"databases:0:connectionString": "Host=localhost;Username=postgres;Password=_________;Database=vault_test", "databases:0:enabled": "true", "databases:1:type": "Sqlite", "databases:1:enabled": "true", "databases:1:connectionString": "Data Source=_________", "databases:2:type": "MySql", -"databases:2:connectionString": "server=localhost;uid=root;pwd=_________;database=ef_test", +"databases:2:connectionString": "server=localhost;uid=root;pwd=_________;database=vault_test", "databases:2:enabled": "true", "databases:3:type": "SqlServer", -"databases:3:connectionString": "Server=localhost;Database=ef_test;User Id=SA;Password=_________;Encrypt=True;TrustServerCertificate=True;", +"databases:3:connectionString": "Server=localhost;Database=vault_test;User Id=SA;Password=_________;Encrypt=True;TrustServerCertificate=True;", "databases:3:enabled": "true" "databases:4:type": "MySql", -"databases:4:connectionString": "server=localhost;port=4306;uid=maria;pwd=_________;database=ef_test;AllowUserVariables=true", +"databases:4:connectionString": "server=localhost;port=4306;uid=maria;pwd=_________;database=vault_test;AllowUserVariables=true", "databases:4:enabled": "true", ``` From b1b8eda4821c33a3010f3bac667965d999508c10 Mon Sep 17 00:00:00 2001 From: Thomas Rittson Date: Tue, 7 Oct 2025 19:42:19 +1000 Subject: [PATCH 5/6] remove refs to DatabaseTheory --- docs/contributing/testing/database/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/testing/database/index.md b/docs/contributing/testing/database/index.md index a89bca43b..b016cc5a8 100644 --- a/docs/contributing/testing/database/index.md +++ b/docs/contributing/testing/database/index.md @@ -93,7 +93,7 @@ To test a migration, set the `MigrationName` property on the `[DatabaseData]` at `IMigrationTesterService`: ```csharp -[DatabaseTheory, DatabaseData(MigrationName = "ExampleDataMigration")] +[Theory, DatabaseData(MigrationName = "ExampleDataMigration")] public async Task TestExampleDataMigration( IMigrationTesterService migrationTester, IOrganizationRepository organizationRepository) From abbc4e0a34a6e48f0eab86f419ab829ea5d3d4e2 Mon Sep 17 00:00:00 2001 From: Thomas Rittson Date: Wed, 8 Oct 2025 08:18:45 +1000 Subject: [PATCH 6/6] sentence case for headings --- docs/getting-started/server/database/ef/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started/server/database/ef/index.mdx b/docs/getting-started/server/database/ef/index.mdx index e8f352f04..c337595f1 100644 --- a/docs/getting-started/server/database/ef/index.mdx +++ b/docs/getting-started/server/database/ef/index.mdx @@ -214,7 +214,7 @@ docker compose --profile ef up ::: -### Run Migrations +### Run migrations In the `dev` folder, run the following command to update the database to the latest migration: