Integrate the broker's extensions (#921)

* Add accidentally deleted SUMMARY

* chore: Apply new config style

* Code review

* Merge docs

* Merge trivial templates changes

* Merge more conflicting files

* Merge more conflicting files

* Merge more conflicting files and cleanup old versions

* Add connector submodule

* Merge more conflicting files

* Move broker CHANGELOG

* Move the broker's README

* More cleaning

* Fix links

* Prepare docker build to merge the broker

* Merge the docker build

* Fix links and format

* Add connector type info

* self review

* Convert MD table to html

* Only use test containers for the Broker

* Add broker NPM delivery

* code review

* Add broker local dev doc

* Merge deployment READMEs and change healthcheck header

* Fix links

* Cleanup

* Fix javadocs and add -sources and -javadocs artefacts

* Rename .env.extensions to .env.connector

* Rename IDS broker -> broker

* Remove redundant .editorconfig setting

* Change connector-type -> deployment-type in workflow config

* NPM clean install

* Removed mention of the broker in the main readme

* Dry postgres image name

* Remove test DB deprecation

* Move broker group to toml

* Fix missing broker's groups

* More versions cleanup

* Change MD for HTML

* Merge the broker compose file

* #921 (comment)

* Merge local quick start for broker

* Align postgres versions

* Stop annoying me with DB not ready messages on new startup

* Wait for the DBs

* Code review

* remove docs task. The repo is now public

* Avoid star imports

* Remove reference to the merged doc

.dockerignore

@@ -38,7 +38,8 @@ build
 *.hprof
 
 .env
-!launchers/.env
+!launchers/.env.connector
+!launchers/.env.broker
 
 # Log files
 *.log
@@ -48,3 +49,6 @@ build
 **/*.jks
 
 docs/secrets
+
+node_modules
+npm-debug.log

.editorconfig

@@ -8,3 +8,23 @@ end_of_line = lf
 charset = utf-8
 trim_trailing_whitespace = true
 insert_final_newline = true
+indent_size = 4
+
+[*.ts]
+quote_type = single
+
+[*.md]
+max_line_length = off
+trim_trailing_whitespace = false
+indent_size = 2
+
+[*.{yml,yaml}]
+indent_size = 2
+
+[{*.kt,*.kts}]
+ij_kotlin_imports_layout = *, java.**, javax.**, kotlin.**, ^
+ij_kotlin_packages_to_use_import_on_demand = explicitly-none
+ij_kotlin_name_count_to_use_star_import = 2147483647
+ij_kotlin_name_count_to_use_star_import_for_members = 2147483647
+ij_continuation_indent_size = 4
+

.env

@@ -3,3 +3,4 @@ EDC_IMAGE=ghcr.io/sovity/edc-dev:7.4.2
 TEST_BACKEND_IMAGE=ghcr.io/sovity/test-backend:7.4.2
 EDC_UI_IMAGE=ghcr.io/sovity/edc-ui:3.2.2
 EDC_UI_ACTIVE_PROFILE=sovity-open-source
+BROKER_IMAGE=ghcr.io/sovity/broker-server-dev:4.2.0

.env.dev

@@ -3,4 +3,4 @@ EDC_IMAGE=ghcr.io/sovity/edc-dev:latest
 TEST_BACKEND_IMAGE=ghcr.io/sovity/test-backend:latest
 EDC_UI_IMAGE=ghcr.io/sovity/edc-ui:latest
 EDC_UI_ACTIVE_PROFILE=sovity-open-source
-
+BROKER_IMAGE=ghcr.io/sovity/broker-server-dev:latest

.github/ISSUE_TEMPLATE/feature_request_mds.md

@@ -26,6 +26,14 @@ _What problems does that user face that existing functionalities do solve?_
 
 _Describe whether this request is related to an existing workflow, feature, or otherwise something in the product today. Or, does this open us up to new innovative ideas?_
 
+## MDS Scope check
+
+_Is this feature part of the contracted scope?_
+- [ ] It is part of the contracted scope
+- [ ] It is not part of the contracted scope
+
+If not, please add the label "mds/future-scope"
+
 ## (For sovity Team to complete) Stakeholders
 
 _Add more on who asked for this, i.e. company, person, how much they pay us, what their tier is, are they a strategic account, etc. Who needs to be kept up-to-date about this feature?_

.github/actions/build-connector-image/action.yml

@@ -19,6 +19,9 @@ inputs:
   connector-name:
     required: true
     description: "EDC Connector Name in launchers/connectors/{connector-name}"
+  deployment-type:
+    required: true
+    description: "Type of deployment: 'connector' or 'broker'"
   title:
     required: true
     description: "Docker Image Title"
@@ -62,7 +65,7 @@ runs:
           type=raw,value=latest,enable={{is_default_branch}}
           type=raw,value=release,enable=${{ startsWith(github.ref, 'refs/tags/') }}
     - name: "Docker: Build and Push"
-      uses: docker/build-push-action@v4
+      uses: docker/build-push-action@v5
       with:
         file: launchers/Dockerfile
         context: .
@@ -71,5 +74,6 @@ runs:
         labels: ${{ steps.meta.outputs.labels }}
         build-args: |
           CONNECTOR_NAME=${{ inputs.connector-name }}
+          CONNECTOR_TYPE=${{ inputs.deployment-type }}
           "EDC_LAST_COMMIT_INFO_ARG=${{ env.LAST_COMMIT_INFO }}"
           EDC_BUILD_DATE_ARG=${{ env.BUILD_DATE }}

.github/workflows/ci.yml

@@ -65,6 +65,7 @@ jobs:
           image-base-name: ${{ env.IMAGE_BASE_NAME }}
           image-name: "edc-dev"
           connector-name: "sovity-dev"
+          deployment-type: "connector"
           title: "sovity Dev EDC Connector"
           description: "Extended EDC Connector built by sovity. This dev version contains no dataspace auth and can be used to quickly start a locally running EDC + EDC UI."
       - name: "Docker Image: edc-ce"
@@ -76,6 +77,7 @@ jobs:
           image-base-name: ${{ env.IMAGE_BASE_NAME }}
           image-name: "edc-ce"
           connector-name: "sovity-ce"
+          deployment-type: "connector"
           title: "sovity Community Edition EDC Connector"
           description: "EDC Connector built by sovity. Contains sovity's Community Edition EDC extensions and requires dataspace credentials to join an existing dataspace."
       - name: "Docker Image: edc-ce-mds"
@@ -87,6 +89,7 @@ jobs:
           image-base-name: ${{ env.IMAGE_BASE_NAME }}
           image-name: "edc-ce-mds"
           connector-name: "mds-ce"
+          deployment-type: "connector"
           title: "MDS Community Edition EDC Connector"
           description: "EDC Connector built by sovity and configured for compatibility with the Mobility Data Space (MDS). This EDC requires dataspace credentials, and additional MDS Services such as a Clearing House."
       - name: "Docker Image: test-backend"
@@ -98,8 +101,33 @@ jobs:
           image-base-name: ${{ env.IMAGE_BASE_NAME }}
           image-name: "test-backend"
           connector-name: "test-backend"
+          deployment-type: "connector"
           title: "Test Data Source / Data Sink"
           description: "Provides a minimal data source / data sink for E2E tests."
+      - name: "Docker Image: broker-server-dev"
+        uses: ./.github/actions/build-connector-image
+        with:
+          registry-url: ${{ env.REGISTRY_URL }}
+          registry-user: ${{ env.REGISTRY_USER }}
+          registry-password: ${{ secrets.GITHUB_TOKEN }}
+          image-base-name: ${{ env.IMAGE_BASE_NAME }}
+          image-name: "broker-server-dev"
+          connector-name: "broker-server-dev"
+          deployment-type: "broker"
+          title: "Broker Server (Dev)"
+          description: "sovity EDC Broker Server. This dev version contains no auth and can be used to quickly start a locally running Broker Server + Broker UI."
+      - name: "Docker Image: broker-server-ce"
+        uses: ./.github/actions/build-connector-image
+        with:
+          registry-url: ${{ env.REGISTRY_URL }}
+          registry-user: ${{ env.REGISTRY_USER }}
+          registry-password: ${{ secrets.GITHUB_TOKEN }}
+          image-base-name: ${{ env.IMAGE_BASE_NAME }}
+          image-name: "broker-server-ce"
+          connector-name: "broker-server-ce"
+          deployment-type: "broker"
+          title: "Broker Server (Community Edition)"
+          description: "sovity EDC Broker Server. Requires dataspace credentials to join an existing dataspace."
   ts-api-client-library:
     name: TS API Client Library
     runs-on: ubuntu-latest
@@ -153,6 +181,60 @@ jobs:
         env:
           NODE_USER: richardtreier-sovity
           NODE_AUTH_TOKEN: ${{ secrets.SOVITY_EDC_CLIENT_NPM_AUTH }}
+  ts-broker-api-client-library:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    permissions:
+      contents: read
+      packages: write
+
+    steps:
+      - uses: FranzDiebold/github-env-vars-action@v2
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v3
+        with:
+          distribution: 'temurin'
+          java-version: '17'
+          cache: 'gradle'
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 16
+          cache: 'npm'
+          registry-url: 'https://registry.npmjs.org'
+          cache-dependency-path: ./extensions/broker-server-api/client-ts/package-lock.json
+      - name: Generate openapi.yaml & Client Code
+        run: |
+          ./gradlew :extensions:broker-server-api:api:clean :extensions:broker-server-api:api:build -x test --no-daemon
+        env:
+          USERNAME: ${{ github.actor }}
+          TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: NPM Package Dist Tag & Version
+        working-directory: ./extensions/broker-server-api/client-ts
+        run: |
+          if [ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]; then
+            # Full Release
+            VERSION="${GITHUB_REF#refs/tags/v}"
+            DIST_TAG=latest
+          else
+            VERSION="0.$(date '+%Y%m%d.%H%M%S')-main-$CI_SHA_SHORT"
+            DIST_TAG=main
+          fi
+          npm version $VERSION
+          echo "DIST_TAG=$DIST_TAG" >> $GITHUB_ENV
+      - name: Build NPM Library
+        working-directory: ./extensions/broker-server-api/client-ts
+        run: |
+          npm ci && npm run build
+      - name: Publish NPM Library
+        if: github.ref == 'refs/heads/main' || github.ref == 'refs/tags/v*'
+        working-directory: ./extensions/broker-server-api/client-ts
+        run: |
+          npm set //registry.npmjs.org/:_authToken $NODE_AUTH_TOKEN
+          npm set //registry.npmjs.org/:username $NODE_USER
+          npm publish --access public --tag "${{ env.DIST_TAG }}"
+        env:
+          NODE_USER: richardtreier-sovity
+          NODE_AUTH_TOKEN: ${{ secrets.SOVITY_BROKER_SERVER_CLIENT_NPM_AUTH }}
   markdown-link-checks:
     name: Markdown Link Checks
     runs-on: ubuntu-latest

.github/workflows/code_analysis.yml

@@ -17,7 +17,7 @@ jobs:
       spotbugs_active: ${{ steps.check_spotbugs.outputs.spotbugs_active }}
     steps:
       - name: Checkout code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Check file existence
         id: check_files
         uses: andstor/file-existence-action@v2
@@ -34,7 +34,7 @@ jobs:
     if: needs.is_java_project.outputs.pom_exists == 'true' && needs.is_java_project.outputs.checkstyle_active == 'true'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up JDK ${{ matrix.java }}
         uses: actions/setup-java@v3
         with:
@@ -48,7 +48,7 @@ jobs:
     if: needs.is_java_project.outputs.pom_exists == 'true' && needs.is_java_project.outputs.spotbugs_active == 'true'
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up JDK ${{ matrix.java }}
         uses: actions/setup-java@v3
         with:

.github/workflows/license_scan.yml

@@ -28,9 +28,11 @@ jobs:
       - name: Checkout code
         uses: actions/checkout@v3
       - name: npm install (typescript-client)
-        run: cd extensions/wrapper/clients/typescript-client && npm install
+        run: cd extensions/wrapper/clients/typescript-client && npm clean-install
       - name: npm install (typescript-client-example)
-        run: cd extensions/wrapper/clients/typescript-client-example && npm install
+        run: cd extensions/wrapper/clients/typescript-client-example && npm clean-install
+      - name: npm install (client-ts)
+        run: cd extensions/broker-server-api/client-ts && npm clean-install
       - name: Run license scanner
         uses: aquasecurity/trivy-action@master
         with:

.github/workflows/secret_scan.yml

@@ -15,7 +15,7 @@ jobs:
       fail-fast: false
     steps:
       - name: Checkout code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Run vulnerability scanner
         uses: aquasecurity/trivy-action@master
         with:

.gitignore

@@ -39,8 +39,8 @@ build
 
 **/.env
 !.env
-!launchers/.env
-!connector/.env
+!launchers/.env.connector
+!launchers/.env.broker
 
 # Log files
 *.log

CHANGELOG.md

@@ -7,9 +7,17 @@ please see [changelog_updates.md](docs/dev/changelog_updates.md).
 
 ### Overview
 
+Starting from version `8`, the Broker has been merged with the Community edition.
+
+[The former changelog](https://github.com/sovity/edc-broker-server-extension/blob/main/CHANGELOG.md) for the Broker is still available but will not be updated anymore.
+
+The Broker's version therefore jumps from version 4 to version 8.
+
+The functionalities of each part, Broker and Extensions, on this release, is the same as before the change.
+
 ### EDC UI
 
-### EDC Extensions
+### EDC Extensions and Broker
 
 #### Major Changes
 

README.md

@@ -66,7 +66,7 @@ Check out our [Getting Started Section](#getting-started) on how to run a local
 Our sovity Community Edition EDC takes available Open Source EDC Extensions and combines them with our own
 open source EDC Extensions from this repository to build ready-to-use EDC Docker Images.
 
-See [here](launchers/README.md) for a list of our sovity Community Edition EDC Docker Images.
+See [here](launchers/README.md) for a list of our sovity Community Edition EDC Docker images.
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 

SECURITY.md

@@ -29,4 +29,5 @@ This information will help us triage your report more quickly.
 We prefer all communications to be in English.
 
 ## Attribution
-This file is adapted from [eclipse-edc](https://github.com/eclipse-edc/DataDashboard) project.
+
+This file is adapted from [eclipse-edc/DataDashboard](https://github.com/eclipse-edc/DataDashboard) project.

SUMMARY.md

@@ -0,0 +1,48 @@
+# Summary
+
+* [Start](./README.md)
+* [Connector Versions](./launchers/README.md)
+* [Changelog](./CHANGELOG.md)
+
+
+## User Documentation
+
+* [Getting Started](./docs/getting-started/README.md)
+* [Data Transfer Modes](./docs/getting-started/documentation/data-transfer-methods.md)
+* [API Wrapper](./docs/getting-started/documentation/api_wrapper.md)
+* [OAuth Data Address](./docs/getting-started/documentation/oauth-data-address.md)
+* [Parameterized Assets via UI](./docs/getting-started/documentation/parameterized_assets_via_ui.md)
+* [Parameterized Assets via Managment API](./docs/getting-started/documentation/parameterized_assets.md)
+* [Pull Data Transfer](./docs/getting-started/documentation/pull-data-transfer.md)
+
+## Deployment Documentation
+* [Deployment Goal: Local Demo](./docs/deployment-guide/goals/local-demo)
+* [Deployment Goal: Development](./docs/deployment-guide/goals/development)
+* [Deployment Goal: Production](./docs/deployment-guide/goals/production)
+    * [Productive Deployment Guide](./docs/deployment-guide/goals/production)
+    * [Productive Deployment Guide 4.2.0 / MS8 / MDS 1.2](docs/deployment-guide/goals/production/4.2.0/README.md)
+
+## Developer Documentation
+
+* [Code of Conduct](./CODE_OF_CONDUCT.md)
+* [Contribution Guide](./CONTRIBUTING.md)
+* [Code-Style Guide](./STYLEGUIDE.md)
+* [Security Guide](./SECURITY.md)
+
+
+## Extensions
+
+* [API Wrapper](./extensions/wrapper/README.md)
+    * [Community Edition API](./extensions/wrapper/wrapper-api/README.md)
+    * [Enterprise Edition API](./extensions/wrapper/wrapper-ee-api/README.md)
+    * [Java API Client Library](./extensions/wrapper/clients/java-client/README.md)
+    * [Java API Client Library Example](./extensions/wrapper/clients/java-client-example/README.md)
+    * [TypeScript API Client Library](./extensions/wrapper/clients/typescript-client/README.md)
+    * [TypeScript API Client Library Example](./extensions/wrapper/clients/typescript-client-example/README.md)
+* Policies
+    * [Always True](./extensions/policy-always-true/README.md)
+    * [Referring Connector](./extensions/policy-referring-connector/README.md)
+    * [Time Interval](./extensions/policy-time-interval/README.md)
+* [Database Migration](./extensions/postgres-flyway/README.md)
+* [EDC UI Config](./extensions/edc-ui-config/README.md)
+* [Last Commit Info](./extensions/last-commit-info/README.md)