prepare Ditto 3.4.0 release notes

* and update SECURITY.md to support 3.3 and 3.4 with security patches

Signed-off-by: Thomas Jäckle <thomas.jaeckle@beyonnex.io>

SECURITY.md

@@ -9,9 +9,9 @@ These versions of Eclipse Ditto are currently being supported with security upda
 
 | Version | Supported          |
 |---------| ------------------ |
+| 3.4.x   | :white_check_mark: |
 | 3.3.x   | :white_check_mark: |
-| 3.2.x   | :white_check_mark: |
-| < 3.2.0 | :x:                |
+| < 3.3.0 | :x:                |
 
 ## Reporting a Vulnerability
 

documentation/src/main/resources/_config.yml

@@ -114,6 +114,8 @@ plugins:
 docVersions:
   - label: "development"
     basePath: ""
+  - label: "3.4"
+    basePath: "3.4"
   - label: "3.3"
     basePath: "3.3"
   - label: "3.2"

documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml

@@ -23,6 +23,9 @@ entries:
       - title: Release Notes
         output: web
         folderitems:
+          - title: 3.4.0
+            url: /release_notes_340.html
+            output: web
           - title: 3.3.7
             url: /release_notes_337.html
             output: web
@@ -44,17 +47,17 @@ entries:
           - title: 3.3.0
             url: /release_notes_330.html
             output: web
-          - title: 3.2.1
-            url: /release_notes_321.html
-            output: web
-          - title: 3.2.0
-            url: /release_notes_320.html
-            output: web
 
             subfolders:
               - title: Archive
                 output: web
                 subfolderitems:
+                  - title: 3.2.1
+                    url: /release_notes_321.html
+                    output: web
+                  - title: 3.2.0
+                    url: /release_notes_320.html
+                    output: web
                   - title: 3.1.2
                     url: /release_notes_312.html
                     output: web

documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md

@@ -1,14 +1,14 @@
 ---
 title: "Eclipse Ditto Benchmark"
 published: true
+keywords: benchmark, load, loadtest, performance, scalability
 permalink: 2023-10-09-ditto-benchmark.html
 layout: post
 author: vasil_vasilev
 tags: [blog]
 hide_sidebar: true
 sidebar: false
 toc: true
-
 ---
 
 This blog post is presenting a benchmark of Eclipse Ditto. It consists of a few scenarios to cover most important ditto functionalities, test the performance and provide a tuning guide. This benchmark is done via the [benchmark-tool](https://github.com/eclipse/ditto/tree/master/benchmark-tool), based on [k6](https://k6.io/) load testing tool
@@ -27,8 +27,7 @@ This blog post is presenting a benchmark of Eclipse Ditto. It consists of a few
 
 - Deployed [k6-operator](https://github.com/grafana/k6-operator) - Kubernetes operator for running distributed k6 tests.
 
-- [MongoDB](https://cloud.mongodb.com/) instance of type [M50](https://www.mongodb.
-  com/docs/atlas/manage-clusters/#nvme-considerations), version 5.0.21
+- [MongoDB](https://cloud.mongodb.com/) instance of type [M50](https://www.mongodb.com/docs/atlas/manage-clusters/#nvme-considerations), version 5.0.21
 
 ## Scenarios
 

documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md

@@ -0,0 +1,84 @@
+---
+title: "Announcing Eclipse Ditto Release 3.4.0"
+published: true
+permalink: 2023-10-12-release-announcement-340.html
+layout: post
+author: thomas_jaeckle
+tags: [blog]
+hide_sidebar: true
+sidebar: false
+toc: false
+---
+
+The Eclipse Ditto teams is proud to announce the availability of Eclipse Ditto 
+[3.4.0](https://projects.eclipse.org/projects/iot.ditto/releases/3.4.0).
+
+Version 3.4.0 mainly concentrates on exchanging the use of the [Akka toolkit](https://akka.io) 
+(due to a change in licensing) with its fork [Apache Pekko](https://pekko.apache.org/) which remains Apache 2.0 licensed.  
+Apart from that, several improvements are also included which can be found in the changelog.
+
+
+## Adoption
+
+Companies are willing to show their adoption of Eclipse Ditto publicly: 
+[https://iot.eclipse.org/adopters/?#iot.ditto](https://iot.eclipse.org/adopters/?#iot.ditto)
+
+When you use Eclipse Ditto it would be great to support the project by putting your logo there.  
+
+
+## Changelog
+
+The main improvements and additions of Ditto 3.4.0 are:
+
+Eclipse Ditto 3.4.0 focuses on the following areas:
+
+* Supporting **HTTP `POST`** for performing **searches** with a very **long query**
+* Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses
+* New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding**
+* Configure **MQTT message expiry interval for published messages** via a header
+
+The following non-functional work is also included:
+
+* **Swapping the [Akka toolkit](https://akka.io)** (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x)
+  **with its fork [Apache Pekko](https://pekko.apache.org/)** which remains Apache 2.0 licensed.
+* Support for using AWS DocumentDB as a replacement for MongoDB
+* Improve logging by adding the W3C traceparent header as MDC field to logs
+* Adjust handling of special MQTT headers in MQTT 5
+* Optimize docker files
+* Migration of Ditto UI to TypeScript
+* There now is an official [Eclipse Ditto Benchmark](2023-10-09-ditto-benchmark.html) which shows how Ditto is able
+  to scale horizontally and provides some tuning tips
+* Addition of a benchmark tooling to run own Ditto benchmarks
+
+The following notable fixes are included:
+
+* Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out"
+
+Please have a look at the [3.4.0 release notes](release_notes_340.html) for a more detailed information on the release.
+
+
+## Artifacts
+
+The new Java artifacts have been published at the [Eclipse Maven repository](https://repo.eclipse.org/content/repositories/ditto/)
+as well as [Maven central](https://repo1.maven.org/maven2/org/eclipse/ditto/).
+
+The Ditto JavaScript client release was published on [npmjs.com](https://www.npmjs.com/~eclipse_ditto):
+* [@eclipse-ditto/ditto-javascript-client-dom](https://www.npmjs.com/package/@eclipse-ditto/ditto-javascript-client-dom)
+* [@eclipse-ditto/ditto-javascript-client-node](https://www.npmjs.com/package/@eclipse-ditto/ditto-javascript-client-node)
+
+
+The Docker images have been pushed to Docker Hub:
+* [eclipse/ditto-policies](https://hub.docker.com/r/eclipse/ditto-policies/)
+* [eclipse/ditto-things](https://hub.docker.com/r/eclipse/ditto-things/)
+* [eclipse/ditto-things-search](https://hub.docker.com/r/eclipse/ditto-things-search/)
+* [eclipse/ditto-gateway](https://hub.docker.com/r/eclipse/ditto-gateway/)
+* [eclipse/ditto-connectivity](https://hub.docker.com/r/eclipse/ditto-connectivity/)
+
+The Ditto Helm chart has been published to Docker Hub:
+* [eclipse/ditto](https://hub.docker.com/r/eclipse/ditto/)
+
+<br/>
+<br/>
+{% include image.html file="ditto.svg" alt="Ditto" max-width=500 %}
+--<br/>
+The Eclipse Ditto team

documentation/src/main/resources/pages/ditto/installation-running.md

@@ -21,7 +21,7 @@ In order to start Ditto, you'll need:
      * 4.4
      * 5.0
    * Alternatively, [Amazon DocumentDB (with MongoDB compatibility)](https://aws.amazon.com/documentdb/) may also be used,
-     however with some limitations, see the [section about DocumentDB below](#managed-amazon-documentdb--with-mongodb-compatibility-)
+     however with some limitations, see the [section about DocumentDB below](#managed-amazon-documentdb-with-mongodb-compatibility)
 * the built Docker images of Ditto
     * either by building them as described in [Building Ditto](installation-building.html),
     * or by using the pre-built [Ditto images on Docker Hub](https://hub.docker.com/u/eclipse/).

documentation/src/main/resources/pages/ditto/release_notes_340.md

@@ -3,7 +3,7 @@ title: Release notes 3.4.0
 tags: [release_notes]
 published: true
 keywords: release notes, announcements, changelog
-summary: "Version 3.4.0 of Eclipse Ditto, released on 13.09.2023"
+summary: "Version 3.4.0 of Eclipse Ditto, released on 12.10.2023"
 permalink: release_notes_340.html
 ---
 
@@ -19,14 +19,181 @@ investigated."
 
 Eclipse Ditto 3.4.0 focuses on the following areas:
 
-* Swapping the Akka framework (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x)
-  with its fork [Apache Pekko](https://pekko.apache.org/docs/pekko/current/index.html) which is Apache 2.0 licensed. [#1477](https://github.com/eclipse-ditto/ditto/issues/1477)
-* Improve logging by adding the W3C traceparent header as MDC field to logs [#1739](https://github.com/eclipse-ditto/ditto/issues/1739)
-* Optimize docker files [#1744](https://github.com/eclipse-ditto/ditto/issues/1744)
+* Supporting **HTTP `POST`** for performing **searches** with a very **long query**
+* Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses
+* New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding**
+* Configure **MQTT message expiry interval for published messages** via a header
+
+The following non-functional work is also included:
+
+* **Swapping the [Akka toolkit](https://akka.io)** (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x)
+  **with its fork [Apache Pekko](https://pekko.apache.org/)** which remains Apache 2.0 licensed.
+* Support for using AWS DocumentDB as a replacement for MongoDB
+* Improve logging by adding the W3C traceparent header as MDC field to logs
+* Adjust handling of special MQTT headers in MQTT 5
+* Optimize docker files
+* Migration of Ditto UI to TypeScript
+* There now is an official [Eclipse Ditto Benchmark](2023-10-09-ditto-benchmark.html) which shows how Ditto is able
+  to scale horizontally and provides some tuning tips
+* Addition of a benchmark tooling to run own Ditto benchmarks
+
+The following notable fixes are included:
+
+* Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out"
+
+### New features
+
+#### Supporting HTTP `POST` for performing searches with a very long query
+
+In [#1706](https://github.com/eclipse-ditto/ditto/pull/1706) support for the additional HTTP verb `POST` on the 
+search HTTP API `/api/2/search/things` and `/api/2/search/things/count` was added.  
+This is beneficial if the passed in [RQL search query](basic-search.html#rql) would get too long to send via query 
+parameter of the `GET` verb.  
+Documentation was added [here](httpapi-search.html#post) and in the [OpenAPI documentation](http-api-doc.html#/Things-Search/post_api_2_search_things).
+
+#### Addition of a new placeholder to use in connections to use payload of the thing JSON e.g. in headers or addresses
+
+For Ditto managed [connections](basic-connections.html) a new ["thing-json" placeholder](basic-placeholders.html#thing-json-placeholder)
+was added, resolving issue [#1727](https://github.com/eclipse-ditto/ditto/issues/1727).  
+With the `thing:json` placeholder it is possible to access arbitrary thing payload as a placeholder, for example in order
+to use it as part of an outbound HTTP call for a managed [HTTP-push connection](connectivity-protocol-bindings-http.html) (WebHook).
+
+Example: call a foreign weather service with the location being a part of the thing's attributes:
+```
+"address": "GET:/weather?longitude={%raw%}{{thing-json:attributes/location/lon}}&latitude={{thing-json:attributes/location/lat}}{%endraw%}"
+```
+
+#### New placeholder functions for joining multiple elements into a single string and doing URL-encoding and -decoding
+
+With [#1754](https://github.com/eclipse-ditto/ditto/pull/1754) there is a new [placeholder function](basic-placeholders.html#function-library), 
+`fn:join('delimiter')`, which can be used in order to join a pipeline element, containing multiple values, into a single
+string.
+
+As part of [#1727](https://github.com/eclipse-ditto/ditto/issues/1727) several placeholders were also added in order to
+be able to apply URL-encoding and -decoding plus also Base64-encoding and -decoding: 
+* `fn:url-encode()`
+* `fn:url-decode()`
+* `fn:base64-encode()`
+* `fn:base64-decode()`
+
+#### Configure MQTT message expiry interval for published messages via a header
+
+Resolving issue [#1729](https://github.com/eclipse-ditto/ditto/issues/1729), a functionality was added to add a special
+header `mqtt.message-expiry-interval` as part of a [MQTT5 target header mapping](connectivity-protocol-bindings-mqtt5.html#target-header-mapping),
+dynamically influencing the MQTT message expiry interval, e.g. as part of a payload mapper for certain to-be-published 
+messages, or as a header mapping for all published messages.
+
+
+### Changes
+
+#### Swapping the Akka toolkit with its fork, Apache Pekko
+
+The biggest change of Ditto 3.4.0 is surely the switch from the [Akka toolkit](https://akka.io) to its OpenSource-friendly 
+fork, [Apache Pekko](https://pekko.apache.org), tracked via [#1477](https://github.com/eclipse-ditto/ditto/issues/1477).  
+To read about the Akka license switch to the BSL (Business Source License), please
+[visit the Lightbend FAQ on that topic](https://www.lightbend.com/akka/license-faq).
+
+Eclipse Ditto will use Apache Pekko, starting with Ditto 3.4.0.
+
+The required [migration steps are documented as part of the release notes](#migrating-to-ditto-34x) and upgrading requires
+a full cluster restart, no rolling update from prior versions to Ditto 3.4.0 is possible.
+
+#### Support for using AWS DocumentDB as a replacement for MongoDB
+
+Adding support for using [Amazon DocumentDB (with MongoDB compatibility)](https://aws.amazon.com/documentdb/) with
+some [documented limitations](installation-running.html#managed-amazon-documentdb-with-mongodb-compatibility).
+
+#### Improve logging by adding the W3C traceparent header as MDC field to logs
+
+In [#1739](https://github.com/eclipse-ditto/ditto/issues/1739) Ditto adds support to log a [W3C Trace Context](https://www.w3.org/TR/trace-context/) 
+`traceparent` header passed into Ditto (e.g. as HTTP header or as part of a [Connection's header mapping](connectivity-header-mapping.html)) 
+to the MDC.  
+If [tracing is enabled](installation-operating.html#tracing), Ditto will even produce traceparents for (up to) each API
+invocation.
+
+#### Adjust handling of special MQTT headers in MQTT 5
+
+Resolving [#1758](https://github.com/eclipse-ditto/ditto/issues/1758), a feature toggle
+([configuration](https://github.com/eclipse-ditto/ditto/blob/7ee1a778ffc6f254f59a9097d9c44372f069e897/internal/utils/config/src/main/resources/ditto-devops.conf#L24-L26))
+was added to configure whether to preserve "special" MQTT properties (like `mqtt.topic`, `mqtt.qos`, ...) as headers or not.  
+The default is to preserve them (as this was the default until now).
+
+#### Optimize docker files
+
+The Docker files were improved in [#1744](https://github.com/eclipse-ditto/ditto/issues/1744) in order to reduce image
+size and follow best-practices.
+
+#### Addition of a benchmark tooling to run own Ditto benchmarks
+
+As part of the [Eclipse Ditto Benchmark blogpost](2023-10-09-ditto-benchmark.html) a benchmark-tool was developed and
+is now part of the Eclipse Ditto [Git repository](https://github.com/eclipse-ditto/ditto/tree/master/benchmark-tool).  
+If one needs/wants to run own benchmarks, this tool can be a good starting point.
+
+#### Migration of Ditto UI to TypeScript
+
+With [#1688](https://github.com/eclipse-ditto/ditto/pull/1688), the Ditto UI has been migrated from JavaScript codebase
+to TypeScript, introducing `npm` to build the UI.
+
+
+### Bugfixes
+
+#### Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out"
+
+A bug [#1703](https://github.com/eclipse-ditto/ditto/issues/1703) was fixed, where the search index for things which 
+could not fetch an updated policy via a cache-loader, was basically dropped.  
+This could e.g. happen if a single policy is used by a lot of things and this was updated.
+
+### Helm Chart
+
+#### Allow BASIC authentication for devops/status users while using Helm deployment and Ingress Controller
+
+In [#1760](https://github.com/eclipse-ditto/ditto/pull/1760) the Ditto Helm chart was enhanced to use "Basic Authentication"
+when using the Ditto Helm chart together with an Ingress Controller for authenticating at the DevOps APIs.
+
+
+## Migration notes
+
 
 ### Migrating to Ditto 3.4.x
 
-To migrate a running system with live data there are few configurations that should be overridden with Java system properties in the following services.
+{% include warning.html content="Updating to Ditto 3.4.0 from versions < 3.4.0 is only possible with a full cluster recreate.
+  Rolling update is not supported as there are changes in the management urls and ports." %}
+
+To migrate a Ditto < 3.4.0 to Ditto 3.4.0, the renaming of an index name in Ditto's MongoDB persistence has to be done.  
+The following section describes, how.
+
+Apart from that, the transition to 3.4.0 should be smooth.
+
+#### Renaming an index name
+
+As part of the migration from Akka to Pekko and from [akka-persistence-mongo](https://github.com/scullxbones/akka-persistence-mongo)
+persistence plugin to [pekko-persistence-mongo](https://github.com/scullxbones/pekko-persistence-mongo) a previously
+named index `akka_persistence_metadata_pid` was renamed to `pekko_persistence_metadata_pid`. 
+
+When starting Ditto 3.4.0 and not having adjusted the index name, the MongoDB will respond with errors as `pekko-persistence-mongo`
+will try to create the same index with a different name.
+
+To migrate, there are two options:
+1. dropping the old indexes before upgrading to Ditto 3.4.0
+2. adjusting the configuration with system properties to still keep using the old index name
+
+##### Option 1: dropping the old indexes
+
+* Connect to your MongoDB with the permissions to alter/drop indexes
+* Drop the indexes using the following commands (via MongoDB shell):
+  ```
+  db.policies_metadata.dropIndex("akka_persistence_metadata_pid")
+  db.things_metadata.dropIndex("akka_persistence_metadata_pid")
+  db.connection_metadata.dropIndex("akka_persistence_metadata_pid")
+  db.connection_remember_metadata.dropIndex("akka_persistence_metadata_pid")
+  ```
+* Depending on whether you use a single MongoDB "database" or several, you need to switch to the correct database before
+  each drop
+
+
+##### Option 2: adjusting the configuration with system properties
+
+Configure the following system properties for the described Ditto services.
 
 Policies:
 ```markdown
@@ -47,8 +214,17 @@ Connectivity:
 * -Dpekko-contrib-mongodb-persistence-connection-remember-journal.overrides.metadata-index=akka_persistence_metadata_pid
 ```
 
-Full cluster recreate is required, rolling update is not supported as there are changes in the management
-urls and ports.
 
-Other than that the transition should be smooth.
+## Roadmap
+
+Looking forward, the (current) ideas for Ditto 3.5.0 are:
+
+* [#1650](https://github.com/eclipse-ditto/ditto/issues/1650) Enforcing linked WoT ThingModels in Things/Features by validating JsonSchema of model elements 
+  * Ensuring that a Ditto Thing is ensured to always follow its WoT ThingModel and also message payloads are always
+    provided in the specified format
+* [#1521](https://github.com/eclipse-ditto/ditto/issues/1521) Configure on a namespace basis the fields to index in the thing-search
+* [#1700](https://github.com/eclipse-ditto/ditto/issues/1700) Show policy imports in Ditto explorer UI
+* [#1637](https://github.com/eclipse-ditto/ditto/issues/1637) Let Policies declare to be applicable only for certain namespaces
 
+Apart from those ideas we are open to contributions of the community, improving or fixing areas which are important for
+their use of Ditto.