diff --git a/SECURITY.md b/SECURITY.md
index 14caa8332f..17c5c026df 100644
--- a/SECURITY.md
+++ b/SECURITY.md
@@ -9,9 +9,9 @@ These versions of Eclipse Ditto are currently being supported with security upda
| Version | Supported |
|---------| ------------------ |
+| 3.4.x | :white_check_mark: |
| 3.3.x | :white_check_mark: |
-| 3.2.x | :white_check_mark: |
-| < 3.2.0 | :x: |
+| < 3.3.0 | :x: |
## Reporting a Vulnerability
diff --git a/documentation/src/main/resources/_config.yml b/documentation/src/main/resources/_config.yml
index 5e51ab227d..7758f82345 100644
--- a/documentation/src/main/resources/_config.yml
+++ b/documentation/src/main/resources/_config.yml
@@ -114,6 +114,8 @@ plugins:
docVersions:
- label: "development"
basePath: ""
+ - label: "3.4"
+ basePath: "3.4"
- label: "3.3"
basePath: "3.3"
- label: "3.2"
diff --git a/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml b/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml
index 4006e2c861..3580bdc687 100644
--- a/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml
+++ b/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml
@@ -23,6 +23,9 @@ entries:
- title: Release Notes
output: web
folderitems:
+ - title: 3.4.0
+ url: /release_notes_340.html
+ output: web
- title: 3.3.7
url: /release_notes_337.html
output: web
@@ -44,17 +47,17 @@ entries:
- title: 3.3.0
url: /release_notes_330.html
output: web
- - title: 3.2.1
- url: /release_notes_321.html
- output: web
- - title: 3.2.0
- url: /release_notes_320.html
- output: web
subfolders:
- title: Archive
output: web
subfolderitems:
+ - title: 3.2.1
+ url: /release_notes_321.html
+ output: web
+ - title: 3.2.0
+ url: /release_notes_320.html
+ output: web
- title: 3.1.2
url: /release_notes_312.html
output: web
diff --git a/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md b/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md
index f077b7e116..d66b930b61 100644
--- a/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md
+++ b/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md
@@ -1,6 +1,7 @@
---
title: "Eclipse Ditto Benchmark"
published: true
+keywords: benchmark, load, loadtest, performance, scalability
permalink: 2023-10-09-ditto-benchmark.html
layout: post
author: vasil_vasilev
@@ -8,7 +9,6 @@ tags: [blog]
hide_sidebar: true
sidebar: false
toc: true
-
---
This blog post is presenting a benchmark of Eclipse Ditto. It consists of a few scenarios to cover most important ditto functionalities, test the performance and provide a tuning guide. This benchmark is done via the [benchmark-tool](https://github.com/eclipse/ditto/tree/master/benchmark-tool), based on [k6](https://k6.io/) load testing tool
@@ -27,8 +27,7 @@ This blog post is presenting a benchmark of Eclipse Ditto. It consists of a few
- Deployed [k6-operator](https://github.com/grafana/k6-operator) - Kubernetes operator for running distributed k6 tests.
-- [MongoDB](https://cloud.mongodb.com/) instance of type [M50](https://www.mongodb.
- com/docs/atlas/manage-clusters/#nvme-considerations), version 5.0.21
+- [MongoDB](https://cloud.mongodb.com/) instance of type [M50](https://www.mongodb.com/docs/atlas/manage-clusters/#nvme-considerations), version 5.0.21
## Scenarios
diff --git a/documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md b/documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md
new file mode 100644
index 0000000000..cb09aceaa5
--- /dev/null
+++ b/documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md
@@ -0,0 +1,84 @@
+---
+title: "Announcing Eclipse Ditto Release 3.4.0"
+published: true
+permalink: 2023-10-12-release-announcement-340.html
+layout: post
+author: thomas_jaeckle
+tags: [blog]
+hide_sidebar: true
+sidebar: false
+toc: false
+---
+
+The Eclipse Ditto teams is proud to announce the availability of Eclipse Ditto
+[3.4.0](https://projects.eclipse.org/projects/iot.ditto/releases/3.4.0).
+
+Version 3.4.0 mainly concentrates on exchanging the use of the [Akka toolkit](https://akka.io)
+(due to a change in licensing) with its fork [Apache Pekko](https://pekko.apache.org/) which remains Apache 2.0 licensed.
+Apart from that, several improvements are also included which can be found in the changelog.
+
+
+## Adoption
+
+Companies are willing to show their adoption of Eclipse Ditto publicly:
+[https://iot.eclipse.org/adopters/?#iot.ditto](https://iot.eclipse.org/adopters/?#iot.ditto)
+
+When you use Eclipse Ditto it would be great to support the project by putting your logo there.
+
+
+## Changelog
+
+The main improvements and additions of Ditto 3.4.0 are:
+
+Eclipse Ditto 3.4.0 focuses on the following areas:
+
+* Supporting **HTTP `POST`** for performing **searches** with a very **long query**
+* Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses
+* New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding**
+* Configure **MQTT message expiry interval for published messages** via a header
+
+The following non-functional work is also included:
+
+* **Swapping the [Akka toolkit](https://akka.io)** (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x)
+ **with its fork [Apache Pekko](https://pekko.apache.org/)** which remains Apache 2.0 licensed.
+* Support for using AWS DocumentDB as a replacement for MongoDB
+* Improve logging by adding the W3C traceparent header as MDC field to logs
+* Adjust handling of special MQTT headers in MQTT 5
+* Optimize docker files
+* Migration of Ditto UI to TypeScript
+* There now is an official [Eclipse Ditto Benchmark](2023-10-09-ditto-benchmark.html) which shows how Ditto is able
+ to scale horizontally and provides some tuning tips
+* Addition of a benchmark tooling to run own Ditto benchmarks
+
+The following notable fixes are included:
+
+* Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out"
+
+Please have a look at the [3.4.0 release notes](release_notes_340.html) for a more detailed information on the release.
+
+
+## Artifacts
+
+The new Java artifacts have been published at the [Eclipse Maven repository](https://repo.eclipse.org/content/repositories/ditto/)
+as well as [Maven central](https://repo1.maven.org/maven2/org/eclipse/ditto/).
+
+The Ditto JavaScript client release was published on [npmjs.com](https://www.npmjs.com/~eclipse_ditto):
+* [@eclipse-ditto/ditto-javascript-client-dom](https://www.npmjs.com/package/@eclipse-ditto/ditto-javascript-client-dom)
+* [@eclipse-ditto/ditto-javascript-client-node](https://www.npmjs.com/package/@eclipse-ditto/ditto-javascript-client-node)
+
+
+The Docker images have been pushed to Docker Hub:
+* [eclipse/ditto-policies](https://hub.docker.com/r/eclipse/ditto-policies/)
+* [eclipse/ditto-things](https://hub.docker.com/r/eclipse/ditto-things/)
+* [eclipse/ditto-things-search](https://hub.docker.com/r/eclipse/ditto-things-search/)
+* [eclipse/ditto-gateway](https://hub.docker.com/r/eclipse/ditto-gateway/)
+* [eclipse/ditto-connectivity](https://hub.docker.com/r/eclipse/ditto-connectivity/)
+
+The Ditto Helm chart has been published to Docker Hub:
+* [eclipse/ditto](https://hub.docker.com/r/eclipse/ditto/)
+
+
+
+{% include image.html file="ditto.svg" alt="Ditto" max-width=500 %}
+--
+The Eclipse Ditto team
diff --git a/documentation/src/main/resources/pages/ditto/installation-running.md b/documentation/src/main/resources/pages/ditto/installation-running.md
index 27a474b8a5..6ae1bafb7e 100644
--- a/documentation/src/main/resources/pages/ditto/installation-running.md
+++ b/documentation/src/main/resources/pages/ditto/installation-running.md
@@ -21,7 +21,7 @@ In order to start Ditto, you'll need:
* 4.4
* 5.0
* Alternatively, [Amazon DocumentDB (with MongoDB compatibility)](https://aws.amazon.com/documentdb/) may also be used,
- however with some limitations, see the [section about DocumentDB below](#managed-amazon-documentdb--with-mongodb-compatibility-)
+ however with some limitations, see the [section about DocumentDB below](#managed-amazon-documentdb-with-mongodb-compatibility)
* the built Docker images of Ditto
* either by building them as described in [Building Ditto](installation-building.html),
* or by using the pre-built [Ditto images on Docker Hub](https://hub.docker.com/u/eclipse/).
diff --git a/documentation/src/main/resources/pages/ditto/release_notes_340.md b/documentation/src/main/resources/pages/ditto/release_notes_340.md
index 55ed449113..a02f664c2e 100644
--- a/documentation/src/main/resources/pages/ditto/release_notes_340.md
+++ b/documentation/src/main/resources/pages/ditto/release_notes_340.md
@@ -3,7 +3,7 @@ title: Release notes 3.4.0
tags: [release_notes]
published: true
keywords: release notes, announcements, changelog
-summary: "Version 3.4.0 of Eclipse Ditto, released on 13.09.2023"
+summary: "Version 3.4.0 of Eclipse Ditto, released on 12.10.2023"
permalink: release_notes_340.html
---
@@ -19,14 +19,181 @@ investigated."
Eclipse Ditto 3.4.0 focuses on the following areas:
-* Swapping the Akka framework (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x)
- with its fork [Apache Pekko](https://pekko.apache.org/docs/pekko/current/index.html) which is Apache 2.0 licensed. [#1477](https://github.com/eclipse-ditto/ditto/issues/1477)
-* Improve logging by adding the W3C traceparent header as MDC field to logs [#1739](https://github.com/eclipse-ditto/ditto/issues/1739)
-* Optimize docker files [#1744](https://github.com/eclipse-ditto/ditto/issues/1744)
+* Supporting **HTTP `POST`** for performing **searches** with a very **long query**
+* Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses
+* New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding**
+* Configure **MQTT message expiry interval for published messages** via a header
+
+The following non-functional work is also included:
+
+* **Swapping the [Akka toolkit](https://akka.io)** (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x)
+ **with its fork [Apache Pekko](https://pekko.apache.org/)** which remains Apache 2.0 licensed.
+* Support for using AWS DocumentDB as a replacement for MongoDB
+* Improve logging by adding the W3C traceparent header as MDC field to logs
+* Adjust handling of special MQTT headers in MQTT 5
+* Optimize docker files
+* Migration of Ditto UI to TypeScript
+* There now is an official [Eclipse Ditto Benchmark](2023-10-09-ditto-benchmark.html) which shows how Ditto is able
+ to scale horizontally and provides some tuning tips
+* Addition of a benchmark tooling to run own Ditto benchmarks
+
+The following notable fixes are included:
+
+* Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out"
+
+### New features
+
+#### Supporting HTTP `POST` for performing searches with a very long query
+
+In [#1706](https://github.com/eclipse-ditto/ditto/pull/1706) support for the additional HTTP verb `POST` on the
+search HTTP API `/api/2/search/things` and `/api/2/search/things/count` was added.
+This is beneficial if the passed in [RQL search query](basic-search.html#rql) would get too long to send via query
+parameter of the `GET` verb.
+Documentation was added [here](httpapi-search.html#post) and in the [OpenAPI documentation](http-api-doc.html#/Things-Search/post_api_2_search_things).
+
+#### Addition of a new placeholder to use in connections to use payload of the thing JSON e.g. in headers or addresses
+
+For Ditto managed [connections](basic-connections.html) a new ["thing-json" placeholder](basic-placeholders.html#thing-json-placeholder)
+was added, resolving issue [#1727](https://github.com/eclipse-ditto/ditto/issues/1727).
+With the `thing:json` placeholder it is possible to access arbitrary thing payload as a placeholder, for example in order
+to use it as part of an outbound HTTP call for a managed [HTTP-push connection](connectivity-protocol-bindings-http.html) (WebHook).
+
+Example: call a foreign weather service with the location being a part of the thing's attributes:
+```
+"address": "GET:/weather?longitude={%raw%}{{thing-json:attributes/location/lon}}&latitude={{thing-json:attributes/location/lat}}{%endraw%}"
+```
+
+#### New placeholder functions for joining multiple elements into a single string and doing URL-encoding and -decoding
+
+With [#1754](https://github.com/eclipse-ditto/ditto/pull/1754) there is a new [placeholder function](basic-placeholders.html#function-library),
+`fn:join('delimiter')`, which can be used in order to join a pipeline element, containing multiple values, into a single
+string.
+
+As part of [#1727](https://github.com/eclipse-ditto/ditto/issues/1727) several placeholders were also added in order to
+be able to apply URL-encoding and -decoding plus also Base64-encoding and -decoding:
+* `fn:url-encode()`
+* `fn:url-decode()`
+* `fn:base64-encode()`
+* `fn:base64-decode()`
+
+#### Configure MQTT message expiry interval for published messages via a header
+
+Resolving issue [#1729](https://github.com/eclipse-ditto/ditto/issues/1729), a functionality was added to add a special
+header `mqtt.message-expiry-interval` as part of a [MQTT5 target header mapping](connectivity-protocol-bindings-mqtt5.html#target-header-mapping),
+dynamically influencing the MQTT message expiry interval, e.g. as part of a payload mapper for certain to-be-published
+messages, or as a header mapping for all published messages.
+
+
+### Changes
+
+#### Swapping the Akka toolkit with its fork, Apache Pekko
+
+The biggest change of Ditto 3.4.0 is surely the switch from the [Akka toolkit](https://akka.io) to its OpenSource-friendly
+fork, [Apache Pekko](https://pekko.apache.org), tracked via [#1477](https://github.com/eclipse-ditto/ditto/issues/1477).
+To read about the Akka license switch to the BSL (Business Source License), please
+[visit the Lightbend FAQ on that topic](https://www.lightbend.com/akka/license-faq).
+
+Eclipse Ditto will use Apache Pekko, starting with Ditto 3.4.0.
+
+The required [migration steps are documented as part of the release notes](#migrating-to-ditto-34x) and upgrading requires
+a full cluster restart, no rolling update from prior versions to Ditto 3.4.0 is possible.
+
+#### Support for using AWS DocumentDB as a replacement for MongoDB
+
+Adding support for using [Amazon DocumentDB (with MongoDB compatibility)](https://aws.amazon.com/documentdb/) with
+some [documented limitations](installation-running.html#managed-amazon-documentdb-with-mongodb-compatibility).
+
+#### Improve logging by adding the W3C traceparent header as MDC field to logs
+
+In [#1739](https://github.com/eclipse-ditto/ditto/issues/1739) Ditto adds support to log a [W3C Trace Context](https://www.w3.org/TR/trace-context/)
+`traceparent` header passed into Ditto (e.g. as HTTP header or as part of a [Connection's header mapping](connectivity-header-mapping.html))
+to the MDC.
+If [tracing is enabled](installation-operating.html#tracing), Ditto will even produce traceparents for (up to) each API
+invocation.
+
+#### Adjust handling of special MQTT headers in MQTT 5
+
+Resolving [#1758](https://github.com/eclipse-ditto/ditto/issues/1758), a feature toggle
+([configuration](https://github.com/eclipse-ditto/ditto/blob/7ee1a778ffc6f254f59a9097d9c44372f069e897/internal/utils/config/src/main/resources/ditto-devops.conf#L24-L26))
+was added to configure whether to preserve "special" MQTT properties (like `mqtt.topic`, `mqtt.qos`, ...) as headers or not.
+The default is to preserve them (as this was the default until now).
+
+#### Optimize docker files
+
+The Docker files were improved in [#1744](https://github.com/eclipse-ditto/ditto/issues/1744) in order to reduce image
+size and follow best-practices.
+
+#### Addition of a benchmark tooling to run own Ditto benchmarks
+
+As part of the [Eclipse Ditto Benchmark blogpost](2023-10-09-ditto-benchmark.html) a benchmark-tool was developed and
+is now part of the Eclipse Ditto [Git repository](https://github.com/eclipse-ditto/ditto/tree/master/benchmark-tool).
+If one needs/wants to run own benchmarks, this tool can be a good starting point.
+
+#### Migration of Ditto UI to TypeScript
+
+With [#1688](https://github.com/eclipse-ditto/ditto/pull/1688), the Ditto UI has been migrated from JavaScript codebase
+to TypeScript, introducing `npm` to build the UI.
+
+
+### Bugfixes
+
+#### Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out"
+
+A bug [#1703](https://github.com/eclipse-ditto/ditto/issues/1703) was fixed, where the search index for things which
+could not fetch an updated policy via a cache-loader, was basically dropped.
+This could e.g. happen if a single policy is used by a lot of things and this was updated.
+
+### Helm Chart
+
+#### Allow BASIC authentication for devops/status users while using Helm deployment and Ingress Controller
+
+In [#1760](https://github.com/eclipse-ditto/ditto/pull/1760) the Ditto Helm chart was enhanced to use "Basic Authentication"
+when using the Ditto Helm chart together with an Ingress Controller for authenticating at the DevOps APIs.
+
+
+## Migration notes
+
### Migrating to Ditto 3.4.x
-To migrate a running system with live data there are few configurations that should be overridden with Java system properties in the following services.
+{% include warning.html content="Updating to Ditto 3.4.0 from versions < 3.4.0 is only possible with a full cluster recreate.
+ Rolling update is not supported as there are changes in the management urls and ports." %}
+
+To migrate a Ditto < 3.4.0 to Ditto 3.4.0, the renaming of an index name in Ditto's MongoDB persistence has to be done.
+The following section describes, how.
+
+Apart from that, the transition to 3.4.0 should be smooth.
+
+#### Renaming an index name
+
+As part of the migration from Akka to Pekko and from [akka-persistence-mongo](https://github.com/scullxbones/akka-persistence-mongo)
+persistence plugin to [pekko-persistence-mongo](https://github.com/scullxbones/pekko-persistence-mongo) a previously
+named index `akka_persistence_metadata_pid` was renamed to `pekko_persistence_metadata_pid`.
+
+When starting Ditto 3.4.0 and not having adjusted the index name, the MongoDB will respond with errors as `pekko-persistence-mongo`
+will try to create the same index with a different name.
+
+To migrate, there are two options:
+1. dropping the old indexes before upgrading to Ditto 3.4.0
+2. adjusting the configuration with system properties to still keep using the old index name
+
+##### Option 1: dropping the old indexes
+
+* Connect to your MongoDB with the permissions to alter/drop indexes
+* Drop the indexes using the following commands (via MongoDB shell):
+ ```
+ db.policies_metadata.dropIndex("akka_persistence_metadata_pid")
+ db.things_metadata.dropIndex("akka_persistence_metadata_pid")
+ db.connection_metadata.dropIndex("akka_persistence_metadata_pid")
+ db.connection_remember_metadata.dropIndex("akka_persistence_metadata_pid")
+ ```
+* Depending on whether you use a single MongoDB "database" or several, you need to switch to the correct database before
+ each drop
+
+
+##### Option 2: adjusting the configuration with system properties
+
+Configure the following system properties for the described Ditto services.
Policies:
```markdown
@@ -47,8 +214,17 @@ Connectivity:
* -Dpekko-contrib-mongodb-persistence-connection-remember-journal.overrides.metadata-index=akka_persistence_metadata_pid
```
-Full cluster recreate is required, rolling update is not supported as there are changes in the management
-urls and ports.
-Other than that the transition should be smooth.
+## Roadmap
+
+Looking forward, the (current) ideas for Ditto 3.5.0 are:
+
+* [#1650](https://github.com/eclipse-ditto/ditto/issues/1650) Enforcing linked WoT ThingModels in Things/Features by validating JsonSchema of model elements
+ * Ensuring that a Ditto Thing is ensured to always follow its WoT ThingModel and also message payloads are always
+ provided in the specified format
+* [#1521](https://github.com/eclipse-ditto/ditto/issues/1521) Configure on a namespace basis the fields to index in the thing-search
+* [#1700](https://github.com/eclipse-ditto/ditto/issues/1700) Show policy imports in Ditto explorer UI
+* [#1637](https://github.com/eclipse-ditto/ditto/issues/1637) Let Policies declare to be applicable only for certain namespaces
+Apart from those ideas we are open to contributions of the community, improving or fixing areas which are important for
+their use of Ditto.