Project specific S3 storages

[boal]

Handling

Handling of files over S3 storages is currently restricted to the DigiWF specific storage. Its not possible to handle files in project specific storages.

Within a project specific process, it should be possible to handle files with project specific S3 storages.

Authentication/authorization

All project specific S3 storages expose REST endpoints which will be secured with project specific ROLEs/RIGHTs using Oauth2.

I.e. the call to project specific S3 storages in the Cosys/Mail integration will be handled by Oauth2 client credentials flow. For each call to a project-specific S3 storage, the corresponding ClientID and the corresponding ClientSecret are then required in the Cosys/Mail integration.

Currently, this cannot be handled in the Cosys and Mail services. As a result the current images cannot be used for this purpose.

Concept

• Every process can use a specific S3 service (the s3 service is configured in the process config)
• Every S3 service has its own kafka topic (developers of the S3 service are responsible to claim the topic)

Todos

DigiWF Engine

• Save process config as isntance variable when starting a process
  • adjust processConfigFunction bean to use variable as default and load from db as fallback
• Adjust Task Listener to extract specific s3 from process config and add it as input parameter
• Adjust Rest-Template to add Token by default
• Adjust TaskFileService to use specific S3 URL
• Adjust ProcessDefintionStart Service to use specific S3 URL

S3 Service

• Add async interface for creating presigned urls
• Provide element templates for creating presigned urls

Mail Service

• Adjust async interface to expect a list of presigned urls for loading documents
• adjust element template and add own template process

Cosys Service

• Adjust async interface to expect a presigned url for saving documents
• adjust element template and add own template process

[dominikhorn93]

@boal What about passing the presigned URLs to the integration services instead of a s3 path?
Each S3 Service would need an async interface... Also, async could cause difficulties when the URLs expire.
But I think we should discuss this anyway.

[boal]

@dominikhorn93

Yes, this is a third valid solution and maybe in terms of simplicity and security the best one.

One drawback ist the possible expiration of the presigned URL.

In case of an expiration, an response sould be send to the caller to signal the expiration of the presigned url.

[boal]

1.Project-specific S3 service requires asynchronous interfaces analogous to the synchronous interface (own project-specific S3 topic) -> basic modules, element templates

https://github.com/it-at-m/digiwf-s3-integration/issues/73

2. Synchronous/asynchronous new interface for lists of PresignedURL to load files.

https://github.com/it-at-m/digiwf-s3-integration/issues/72

3. Integration components like Mail or Cosys expecting the presignedURLs instaead of a path and document storage urls.

https://github.com/it-at-m/digiwf-cosys-integration/issues/5

it-at-m/digiwf-email-integration#20

4. DigiWF core services getting authorization for synchronous access to the project-specific S3 service.

[boal]

@dominikhorn93

Which services are still affected?

[lmoesle]

Topics claimed: https://git.muenchen.de/openshift/kafka/-/issues/83

Todo: We need Topics for prod before we can relase the new feature

[lmoesle]

What's working

• determine s3 storage by adding a process config. Each processdefinition can use a different s3 storage.
• create presigned urls for any s3 storage
• send emails with files attached from presigned urls
• multifile input can use any s3 storage

What's still missing

• S3 storage authentication
• Cosys Integration
• Multifile Input does not work on process start

DigiWF Engine

• Added element templates to create presigned urls and send emails
• Created a small demo process that
  • shows the s3 specific process config
  • creates presigned urls for all files in a test directory
  • sends a mail with the files from the test directory attached
• Process Config
  • Save process config as instance variable when starting a process
  • Added a tasklistener that adds the s3 config as an input parameter to every task
  • As fallback use the s3 configs saved in application.properties (default s3 config)
• Save s3 config for sync (http Endpoint for multifile input) and async usage (kafka topic)
• Refactoring to reduce code

Pull Request: https://git.muenchen.de/digitalisierung/digiwf-engine/-/merge_requests/77

S3 Service

• Added asyncapi to the s3 integration service to create presigned urls
• S3 integration service will create presigned urls for all files in the specified path
• Presigned urls are sent as correlate message event back to the engine
• Refactored logic to create presigned urls to simplify it
• Created an element template for creating presigned urls (digiwf-engine repo)

Pull Request: https://github.com/it-at-m/digiwf-s3-integration/pull/77

Email Integration

• Pass file attachments as input parameter to email integration service
• Email Integration Service will download files from presigned urls and attaches them to an email before sending it
• Refactored validation using javax validation. Added validation rule that presigned urls must use the GET action. All other actions don't make sense in this context.
• Refactored email element template

Pull Request: it-at-m/digiwf-email-integration#25