feat: persisting the database

.dockerignore

@@ -1,13 +1,15 @@
 /Dockerfile
 **/Dockerfile
 docker-compose.yml
+**/docker-compose.yml
 .dockerignore
 **/node_modules
 npm-debug.log
 .vscode
 .next
 .git
 .env
+**/.env
 **/dist
 # Rush temporary files
 common/deploy/
@@ -16,3 +18,11 @@ common/autoinstallers/*/.npmrc
 **/.rush/temp/
 
 **/*.log
+
+*.db
+*.db-shm
+*.db-wal
+
+**/*.db
+**/*.db-shm
+**/*.db-wal

.gitignore

@@ -71,3 +71,6 @@ common/autoinstallers/*/.npmrc
 dist
 .vscode/
 docker-compose.yml
+*.db
+*.db-shm
+*.db-wal

README.md

@@ -122,6 +122,166 @@ To run tests across all apps and libraries in one command, a rush script has bee
 $ rush test
 ```
 
+## Multiple database engines support
+VC-API backend supports:
+- in-memory SQLite (default)
+- SQLite persisted in a file
+- Postgres (needs to be started separately)
+
+### In-memory SQLite
+**This mode is not intended to be used by multiple load balanced VC-API service instances.**
+
+This is the default mode of operation. The database in this mode is purged and initiated everytime process is started.
+No special
+steps are required when upgrading versions of the VP-API. Every instance of the VC-API uses its own instance of this
+database. 
+
+Start your local VC-API instance with the following:
+```shell
+$ cd apps/vc-api
+$ npm run start:dev
+```
+
+### SQLite
+**This mode is not intended to be used by multiple load balanced VC-API service instances.**
+
+To activate data persistence for SQLite, set the following env variables:
+```dotenv
+DB_TYPE=SQLITE
+SQLITE_FILE=<path/to/your/file/sqlite.db>
+```
+
+Start the VC-API:
+```shell
+$ cd apps/vc-api
+$ npm run start:dev
+```
+
+### Postgres
+To use Postgres database, start your local postgres server, for example with docker and `docker-compose.full.yaml` 
+located in the `apps/vc-api` folder:
+```shell
+$ cd apps/vc-api
+$ docker compose -f docker-compose.full.yaml up
+```
+
+Set the following env variables:
+```dotenv
+DB_TYPE=POSTGRES
+POSTGRES_DB_HOST=127.0.0.1
+POSTGRES_DB_PORT=5432
+POSTGRES_DB_USER=postgres
+POSTGRES_DB_PASSWORD=postgres
+POSTGRES_DB_NAME=vc-api
+```
+
+Start the VC-API:
+```shell
+$ cd apps/vc-api
+$ npm run start:dev
+```
+
+## Database migrations
+
+Having database persisted on production makes it necessary to have database schema migration process in place when
+upgrading VC-API version running. Thanks to the migrations feature of TypeORM, this can be automated.
+
+### Executing migrations after VC-API code updated
+
+#### WARNING!!!
+
+The following settings are recommended on production. These are also default application settings if not provided:
+```dotenv
+# below, if set to `true`, erases all data on every application start
+DB_DROP_ON_START=false
+# below is intended to be set to `true` only when developing locally
+DB_SYNC_SCHEMA_ON_START=false
+```
+
+Execution of the database schema migrations is automated, so that only required migrations are executed to upgrade the 
+database schema. This can be done in two ways:
+
+#### By the application
+
+A VC-API service is able to execute migrations by itself when starting. To enable this feature the following env 
+variable is needed:
+```dotenv
+DB_RUN_MIGRATIONS=true
+```
+
+#### With npm script
+
+If you decide to disable execution of migrations by application, they can be executed manually:
+```shell
+$ cd apps/vc-api
+$ npm run migration:run
+```
+
+### Creating new migrations after development
+
+The reccommended settings for development are:
+```dotenv
+DB_DROP_ON_START=true
+DB_SYNC_SCHEMA_ON_START=true
+DB_RUN_MIGRATIONS=true
+```
+
+This means that database schema will be dropped on every application restart, migrations executed and all entities 
+changes not reflected in migrations will be synchronised to the DB schema. This allows faster development.
+
+After completing chunk of work, for example when preparing a pull request to be reviewed and merged, another db 
+schema migration needs to be created and committed.
+
+**Warning!** Two sets of migrations need to be added. For SQLite and Postgres.
+
+To create migrations, you will need to:
+
+1. [drop db schemas](#dropping-schemas) for both Postgres and SQLite
+2. stop your application instances if running
+3. ensure your `.env` file contains valid settings for both Postgres and SQLite persisted in file
+4. [execute](#executing-existing-migrations) existing migrations
+5. [generate](#generating-new-migration-files) migration files
+6. validate migration files - this should include executing all migrations (including new ones) and E2E tests for both
+   database engines SQLite and Postgres
+7. commit migration files
+
+#### Dropping schemas
+1. Remove your SQLite file. By default, it should be `apps/vc-api/sqlite.db`.
+2. Start your Postgres instance from scratch. If you started it with `docker compose -f docker-compose.full.yaml up` 
+execute the following:
+    ```shell
+    $ cd apps/vc-api
+    $ docker compose -f docker-compose.full.yaml down -v
+    ```
+    `-v` means volumes are to be removed - **this is important option here**
+
+#### Executing existing migrations
+```shell
+$ npm run migration:run:pg
+```
+```shell
+$ npm run migration:run:sqlite
+```
+
+#### Generating new migration files
+```shell
+npm run migration:generate:pg <migration-name>
+```
+this should add a new migration file for Postgres to the `apps/vc-api/src/migrations/pg` folder
+
+and
+```shell
+npm run migration:generate:sqlite <migration-name>
+```
+
+this should add a new migration file for SQLite to the `apps/vc-api/src/migrations/sqlite` folder
+
+#### Warning
+Migrations are executed in order of files sorted by name. Migration files start with a timestamp, for instance 
+`1671212337326-initial.ts`. If you are planning to merge your PR to the base branch, keep in mind that in some 
+situations you will need to delete and regenerate your migrations again to keep them in order with migrations added 
+in meantime to the base branch.
+
 ## Contributing Guidelines 
 See [contributing.md](./contributing.md)
 

apps/vc-api/.env.example

@@ -0,0 +1,16 @@
+DB_TYPE=SQLITE_IN_MEMORY
+
+#DB_TYPE=SQLITE
+SQLITE_FILE=sqlite.db
+
+#DB_TYPE=POSTGRES
+POSTGRES_DB_HOST=127.0.0.1
+POSTGRES_DB_PORT=5432
+POSTGRES_DB_USER=postgres
+POSTGRES_DB_PASSWORD=postgres
+POSTGRES_DB_NAME=vc-api
+
+#for DB_TYPE=POSTGRES or DB_TYPE=SQLITE
+DB_DROP_ON_START=false
+DB_SYNC_SCHEMA_ON_START=false
+DB_RUN_MIGRATIONS=true

apps/vc-api/Dockerfile

@@ -28,9 +28,15 @@ RUN cp -R ./apps/vc-api/dist ./common/deploy/apps/vc-api
 
 FROM base as final
 ENV NODE_ENV=production
+
 EXPOSE 3000
 
 COPY --from=builder /app/common/deploy /app
 COPY license-header.txt /app
 
+ENV DB_TYPE=SQLITE_IN_MEMORY
+
+VOLUME /data
+ENV SQLITE_FILE=/data/sqlite.db
+
 CMD ["node", "apps/vc-api/dist/src/main.js"]

apps/vc-api/docker-compose.dev.yaml

@@ -0,0 +1,22 @@
+version: '3.8'
+
+services:
+  postgres:
+    image: postgres:14-alpine
+    restart: unless-stopped
+    environment:
+      POSTGRES_PASSWORD: $POSTGRES_DB_PASSWORD
+      POSTGRES_USER: $POSTGRES_DB_USER
+      POSTGRES_DB: $POSTGRES_DB_NAME
+    ports:
+      - '127.0.0.1:5432:5432'
+    volumes:
+      - dev-postgres-data:/var/lib/postgresql/data
+    networks:
+      main:
+
+networks:
+  main:
+
+volumes:
+  dev-postgres-data:

apps/vc-api/package.json

@@ -22,7 +22,12 @@
     "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
     "test:e2e": "jest --config ./test/jest-e2e.json",
     "write-openapi": "ts-node ./scripts/write-open-api-json.ts && prettier -w docs/openapi.json",
-    "update-openapi": "npm run write-openapi; git add docs/openapi.json"
+    "update-openapi": "npm run write-openapi; git add docs/openapi.json",
+    "migration:generate:pg": "export DB_TYPE=POSTGRES; npm run migration:run && cd src/migrations/pg && typeorm-ts-node-commonjs -d ../../config/ormconfig.ts migration:generate -p",
+    "migration:generate:sqlite": "export DB_TYPE=SQLITE; npm run migration:run && cd src/migrations/sqlite && typeorm-ts-node-commonjs -d ../../config/ormconfig.ts migration:generate -p",
+    "migration:run": "typeorm-ts-node-commonjs -d src/config/ormconfig.ts migration:run",
+    "migration:run:pg": "export DB_TYPE=POSTGRES; typeorm-ts-node-commonjs -d src/config/ormconfig.ts migration:run",
+    "migration:run:sqlite": "export DB_TYPE=SQLITE; typeorm-ts-node-commonjs -d src/config/ormconfig.ts migration:run"
   },
   "dependencies": {
     "@energyweb/ssi-did": "0.0.1",
@@ -49,7 +54,9 @@
     "@nestjs/axios": "^3.0.0",
     "@nestjs/serve-static": "^4.0.0",
     "joi": "^17.9.2",
-    "axios": "~1.4.0"
+    "axios": "~1.4.0",
+    "pg": "^8.0.0",
+    "dotenv": "^16.3.1"
   },
   "devDependencies": {
     "@nestjs/cli": "^10.1.0",

apps/vc-api/src/app.module.ts

@@ -16,15 +16,16 @@
  */
 
 import { DynamicModule, Module } from '@nestjs/common';
-import { ConfigModule } from '@nestjs/config';
+import { ConfigModule, ConfigService } from '@nestjs/config';
 import { KeyModule } from './key/key.module';
 import { DidModule } from './did/did.module';
 import { VcApiModule } from './vc-api/vc-api.module';
-import { TypeOrmSQLiteModule } from './in-memory-db';
+import { typeOrmConfigFactory } from './config/db';
 import { ServeStaticModule } from '@nestjs/serve-static';
 import { join } from 'path';
 import { SeederModule } from './seeder/seeder.module';
 import { envVarsValidationSchema } from './config/env-vars-validation-schema';
+import { TypeOrmModule } from '@nestjs/typeorm';
 
 let config: DynamicModule;
 
@@ -46,7 +47,13 @@ try {
 
 @Module({
   imports: [
-    TypeOrmSQLiteModule(),
+    TypeOrmModule.forRootAsync({
+      imports: [ConfigModule],
+      inject: [ConfigService],
+      useFactory: (config) => ({
+        ...typeOrmConfigFactory(config)
+      })
+    }),
     KeyModule,
     DidModule,
     VcApiModule,