From 795f5748a101661fcb523b051e4d1bd67061340b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Thomas=20J=C3=A4ckle?= Date: Tue, 10 Oct 2023 15:31:51 +0200 Subject: [PATCH] prepare Ditto 3.4.0 release notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * and update SECURITY.md to support 3.3 and 3.4 with security patches Signed-off-by: Thomas Jäckle --- SECURITY.md | 4 +- documentation/src/main/resources/_config.yml | 2 + .../_data/sidebars/ditto_sidebar.yml | 15 +- .../_posts/2023-10-09-ditto-benchmark.md | 5 +- .../2023-10-12-release-announcement-340.md | 84 ++++++++ .../pages/ditto/installation-running.md | 2 +- .../pages/ditto/release_notes_340.md | 194 +++++++++++++++++- 7 files changed, 285 insertions(+), 21 deletions(-) create mode 100644 documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md diff --git a/SECURITY.md b/SECURITY.md index 14caa8332f..17c5c026df 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,9 +9,9 @@ These versions of Eclipse Ditto are currently being supported with security upda | Version | Supported | |---------| ------------------ | +| 3.4.x | :white_check_mark: | | 3.3.x | :white_check_mark: | -| 3.2.x | :white_check_mark: | -| < 3.2.0 | :x: | +| < 3.3.0 | :x: | ## Reporting a Vulnerability diff --git a/documentation/src/main/resources/_config.yml b/documentation/src/main/resources/_config.yml index 5e51ab227d..7758f82345 100644 --- a/documentation/src/main/resources/_config.yml +++ b/documentation/src/main/resources/_config.yml @@ -114,6 +114,8 @@ plugins: docVersions: - label: "development" basePath: "" + - label: "3.4" + basePath: "3.4" - label: "3.3" basePath: "3.3" - label: "3.2" diff --git a/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml b/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml index 4006e2c861..3580bdc687 100644 --- a/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml +++ b/documentation/src/main/resources/_data/sidebars/ditto_sidebar.yml @@ -23,6 +23,9 @@ entries: - title: Release Notes output: web folderitems: + - title: 3.4.0 + url: /release_notes_340.html + output: web - title: 3.3.7 url: /release_notes_337.html output: web @@ -44,17 +47,17 @@ entries: - title: 3.3.0 url: /release_notes_330.html output: web - - title: 3.2.1 - url: /release_notes_321.html - output: web - - title: 3.2.0 - url: /release_notes_320.html - output: web subfolders: - title: Archive output: web subfolderitems: + - title: 3.2.1 + url: /release_notes_321.html + output: web + - title: 3.2.0 + url: /release_notes_320.html + output: web - title: 3.1.2 url: /release_notes_312.html output: web diff --git a/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md b/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md index f077b7e116..d66b930b61 100644 --- a/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md +++ b/documentation/src/main/resources/_posts/2023-10-09-ditto-benchmark.md @@ -1,6 +1,7 @@ --- title: "Eclipse Ditto Benchmark" published: true +keywords: benchmark, load, loadtest, performance, scalability permalink: 2023-10-09-ditto-benchmark.html layout: post author: vasil_vasilev @@ -8,7 +9,6 @@ tags: [blog] hide_sidebar: true sidebar: false toc: true - --- This blog post is presenting a benchmark of Eclipse Ditto. It consists of a few scenarios to cover most important ditto functionalities, test the performance and provide a tuning guide. This benchmark is done via the [benchmark-tool](https://github.com/eclipse/ditto/tree/master/benchmark-tool), based on [k6](https://k6.io/) load testing tool @@ -27,8 +27,7 @@ This blog post is presenting a benchmark of Eclipse Ditto. It consists of a few - Deployed [k6-operator](https://github.com/grafana/k6-operator) - Kubernetes operator for running distributed k6 tests. -- [MongoDB](https://cloud.mongodb.com/) instance of type [M50](https://www.mongodb. - com/docs/atlas/manage-clusters/#nvme-considerations), version 5.0.21 +- [MongoDB](https://cloud.mongodb.com/) instance of type [M50](https://www.mongodb.com/docs/atlas/manage-clusters/#nvme-considerations), version 5.0.21 ## Scenarios diff --git a/documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md b/documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md new file mode 100644 index 0000000000..cb09aceaa5 --- /dev/null +++ b/documentation/src/main/resources/_posts/2023-10-12-release-announcement-340.md @@ -0,0 +1,84 @@ +--- +title: "Announcing Eclipse Ditto Release 3.4.0" +published: true +permalink: 2023-10-12-release-announcement-340.html +layout: post +author: thomas_jaeckle +tags: [blog] +hide_sidebar: true +sidebar: false +toc: false +--- + +The Eclipse Ditto teams is proud to announce the availability of Eclipse Ditto +[3.4.0](https://projects.eclipse.org/projects/iot.ditto/releases/3.4.0). + +Version 3.4.0 mainly concentrates on exchanging the use of the [Akka toolkit](https://akka.io) +(due to a change in licensing) with its fork [Apache Pekko](https://pekko.apache.org/) which remains Apache 2.0 licensed. +Apart from that, several improvements are also included which can be found in the changelog. + + +## Adoption + +Companies are willing to show their adoption of Eclipse Ditto publicly: +[https://iot.eclipse.org/adopters/?#iot.ditto](https://iot.eclipse.org/adopters/?#iot.ditto) + +When you use Eclipse Ditto it would be great to support the project by putting your logo there. + + +## Changelog + +The main improvements and additions of Ditto 3.4.0 are: + +Eclipse Ditto 3.4.0 focuses on the following areas: + +* Supporting **HTTP `POST`** for performing **searches** with a very **long query** +* Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses +* New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding** +* Configure **MQTT message expiry interval for published messages** via a header + +The following non-functional work is also included: + +* **Swapping the [Akka toolkit](https://akka.io)** (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x) + **with its fork [Apache Pekko](https://pekko.apache.org/)** which remains Apache 2.0 licensed. +* Support for using AWS DocumentDB as a replacement for MongoDB +* Improve logging by adding the W3C traceparent header as MDC field to logs +* Adjust handling of special MQTT headers in MQTT 5 +* Optimize docker files +* Migration of Ditto UI to TypeScript +* There now is an official [Eclipse Ditto Benchmark](2023-10-09-ditto-benchmark.html) which shows how Ditto is able + to scale horizontally and provides some tuning tips +* Addition of a benchmark tooling to run own Ditto benchmarks + +The following notable fixes are included: + +* Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out" + +Please have a look at the [3.4.0 release notes](release_notes_340.html) for a more detailed information on the release. + + +## Artifacts + +The new Java artifacts have been published at the [Eclipse Maven repository](https://repo.eclipse.org/content/repositories/ditto/) +as well as [Maven central](https://repo1.maven.org/maven2/org/eclipse/ditto/). + +The Ditto JavaScript client release was published on [npmjs.com](https://www.npmjs.com/~eclipse_ditto): +* [@eclipse-ditto/ditto-javascript-client-dom](https://www.npmjs.com/package/@eclipse-ditto/ditto-javascript-client-dom) +* [@eclipse-ditto/ditto-javascript-client-node](https://www.npmjs.com/package/@eclipse-ditto/ditto-javascript-client-node) + + +The Docker images have been pushed to Docker Hub: +* [eclipse/ditto-policies](https://hub.docker.com/r/eclipse/ditto-policies/) +* [eclipse/ditto-things](https://hub.docker.com/r/eclipse/ditto-things/) +* [eclipse/ditto-things-search](https://hub.docker.com/r/eclipse/ditto-things-search/) +* [eclipse/ditto-gateway](https://hub.docker.com/r/eclipse/ditto-gateway/) +* [eclipse/ditto-connectivity](https://hub.docker.com/r/eclipse/ditto-connectivity/) + +The Ditto Helm chart has been published to Docker Hub: +* [eclipse/ditto](https://hub.docker.com/r/eclipse/ditto/) + +
+
+{% include image.html file="ditto.svg" alt="Ditto" max-width=500 %} +--
+The Eclipse Ditto team diff --git a/documentation/src/main/resources/pages/ditto/installation-running.md b/documentation/src/main/resources/pages/ditto/installation-running.md index 27a474b8a5..6ae1bafb7e 100644 --- a/documentation/src/main/resources/pages/ditto/installation-running.md +++ b/documentation/src/main/resources/pages/ditto/installation-running.md @@ -21,7 +21,7 @@ In order to start Ditto, you'll need: * 4.4 * 5.0 * Alternatively, [Amazon DocumentDB (with MongoDB compatibility)](https://aws.amazon.com/documentdb/) may also be used, - however with some limitations, see the [section about DocumentDB below](#managed-amazon-documentdb--with-mongodb-compatibility-) + however with some limitations, see the [section about DocumentDB below](#managed-amazon-documentdb-with-mongodb-compatibility) * the built Docker images of Ditto * either by building them as described in [Building Ditto](installation-building.html), * or by using the pre-built [Ditto images on Docker Hub](https://hub.docker.com/u/eclipse/). diff --git a/documentation/src/main/resources/pages/ditto/release_notes_340.md b/documentation/src/main/resources/pages/ditto/release_notes_340.md index 55ed449113..a02f664c2e 100644 --- a/documentation/src/main/resources/pages/ditto/release_notes_340.md +++ b/documentation/src/main/resources/pages/ditto/release_notes_340.md @@ -3,7 +3,7 @@ title: Release notes 3.4.0 tags: [release_notes] published: true keywords: release notes, announcements, changelog -summary: "Version 3.4.0 of Eclipse Ditto, released on 13.09.2023" +summary: "Version 3.4.0 of Eclipse Ditto, released on 12.10.2023" permalink: release_notes_340.html --- @@ -19,14 +19,181 @@ investigated." Eclipse Ditto 3.4.0 focuses on the following areas: -* Swapping the Akka framework (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x) - with its fork [Apache Pekko](https://pekko.apache.org/docs/pekko/current/index.html) which is Apache 2.0 licensed. [#1477](https://github.com/eclipse-ditto/ditto/issues/1477) -* Improve logging by adding the W3C traceparent header as MDC field to logs [#1739](https://github.com/eclipse-ditto/ditto/issues/1739) -* Optimize docker files [#1744](https://github.com/eclipse-ditto/ditto/issues/1744) +* Supporting **HTTP `POST`** for performing **searches** with a very **long query** +* Addition of a **new placeholder** to use **in connections** to use **payload of the thing JSON** e.g. in headers or addresses +* New **placeholder functions** for **joining** multiple elements into a single string and doing **URL-encoding and -decoding** +* Configure **MQTT message expiry interval for published messages** via a header + +The following non-functional work is also included: + +* **Swapping the [Akka toolkit](https://akka.io)** (because of its switch of license to [BSL License](https://www.lightbend.com/akka/license-faq) after Akka v2.6.x) + **with its fork [Apache Pekko](https://pekko.apache.org/)** which remains Apache 2.0 licensed. +* Support for using AWS DocumentDB as a replacement for MongoDB +* Improve logging by adding the W3C traceparent header as MDC field to logs +* Adjust handling of special MQTT headers in MQTT 5 +* Optimize docker files +* Migration of Ditto UI to TypeScript +* There now is an official [Eclipse Ditto Benchmark](2023-10-09-ditto-benchmark.html) which shows how Ditto is able + to scale horizontally and provides some tuning tips +* Addition of a benchmark tooling to run own Ditto benchmarks + +The following notable fixes are included: + +* Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out" + +### New features + +#### Supporting HTTP `POST` for performing searches with a very long query + +In [#1706](https://github.com/eclipse-ditto/ditto/pull/1706) support for the additional HTTP verb `POST` on the +search HTTP API `/api/2/search/things` and `/api/2/search/things/count` was added. +This is beneficial if the passed in [RQL search query](basic-search.html#rql) would get too long to send via query +parameter of the `GET` verb. +Documentation was added [here](httpapi-search.html#post) and in the [OpenAPI documentation](http-api-doc.html#/Things-Search/post_api_2_search_things). + +#### Addition of a new placeholder to use in connections to use payload of the thing JSON e.g. in headers or addresses + +For Ditto managed [connections](basic-connections.html) a new ["thing-json" placeholder](basic-placeholders.html#thing-json-placeholder) +was added, resolving issue [#1727](https://github.com/eclipse-ditto/ditto/issues/1727). +With the `thing:json` placeholder it is possible to access arbitrary thing payload as a placeholder, for example in order +to use it as part of an outbound HTTP call for a managed [HTTP-push connection](connectivity-protocol-bindings-http.html) (WebHook). + +Example: call a foreign weather service with the location being a part of the thing's attributes: +``` +"address": "GET:/weather?longitude={%raw%}{{thing-json:attributes/location/lon}}&latitude={{thing-json:attributes/location/lat}}{%endraw%}" +``` + +#### New placeholder functions for joining multiple elements into a single string and doing URL-encoding and -decoding + +With [#1754](https://github.com/eclipse-ditto/ditto/pull/1754) there is a new [placeholder function](basic-placeholders.html#function-library), +`fn:join('delimiter')`, which can be used in order to join a pipeline element, containing multiple values, into a single +string. + +As part of [#1727](https://github.com/eclipse-ditto/ditto/issues/1727) several placeholders were also added in order to +be able to apply URL-encoding and -decoding plus also Base64-encoding and -decoding: +* `fn:url-encode()` +* `fn:url-decode()` +* `fn:base64-encode()` +* `fn:base64-decode()` + +#### Configure MQTT message expiry interval for published messages via a header + +Resolving issue [#1729](https://github.com/eclipse-ditto/ditto/issues/1729), a functionality was added to add a special +header `mqtt.message-expiry-interval` as part of a [MQTT5 target header mapping](connectivity-protocol-bindings-mqtt5.html#target-header-mapping), +dynamically influencing the MQTT message expiry interval, e.g. as part of a payload mapper for certain to-be-published +messages, or as a header mapping for all published messages. + + +### Changes + +#### Swapping the Akka toolkit with its fork, Apache Pekko + +The biggest change of Ditto 3.4.0 is surely the switch from the [Akka toolkit](https://akka.io) to its OpenSource-friendly +fork, [Apache Pekko](https://pekko.apache.org), tracked via [#1477](https://github.com/eclipse-ditto/ditto/issues/1477). +To read about the Akka license switch to the BSL (Business Source License), please +[visit the Lightbend FAQ on that topic](https://www.lightbend.com/akka/license-faq). + +Eclipse Ditto will use Apache Pekko, starting with Ditto 3.4.0. + +The required [migration steps are documented as part of the release notes](#migrating-to-ditto-34x) and upgrading requires +a full cluster restart, no rolling update from prior versions to Ditto 3.4.0 is possible. + +#### Support for using AWS DocumentDB as a replacement for MongoDB + +Adding support for using [Amazon DocumentDB (with MongoDB compatibility)](https://aws.amazon.com/documentdb/) with +some [documented limitations](installation-running.html#managed-amazon-documentdb-with-mongodb-compatibility). + +#### Improve logging by adding the W3C traceparent header as MDC field to logs + +In [#1739](https://github.com/eclipse-ditto/ditto/issues/1739) Ditto adds support to log a [W3C Trace Context](https://www.w3.org/TR/trace-context/) +`traceparent` header passed into Ditto (e.g. as HTTP header or as part of a [Connection's header mapping](connectivity-header-mapping.html)) +to the MDC. +If [tracing is enabled](installation-operating.html#tracing), Ditto will even produce traceparents for (up to) each API +invocation. + +#### Adjust handling of special MQTT headers in MQTT 5 + +Resolving [#1758](https://github.com/eclipse-ditto/ditto/issues/1758), a feature toggle +([configuration](https://github.com/eclipse-ditto/ditto/blob/7ee1a778ffc6f254f59a9097d9c44372f069e897/internal/utils/config/src/main/resources/ditto-devops.conf#L24-L26)) +was added to configure whether to preserve "special" MQTT properties (like `mqtt.topic`, `mqtt.qos`, ...) as headers or not. +The default is to preserve them (as this was the default until now). + +#### Optimize docker files + +The Docker files were improved in [#1744](https://github.com/eclipse-ditto/ditto/issues/1744) in order to reduce image +size and follow best-practices. + +#### Addition of a benchmark tooling to run own Ditto benchmarks + +As part of the [Eclipse Ditto Benchmark blogpost](2023-10-09-ditto-benchmark.html) a benchmark-tool was developed and +is now part of the Eclipse Ditto [Git repository](https://github.com/eclipse-ditto/ditto/tree/master/benchmark-tool). +If one needs/wants to run own benchmarks, this tool can be a good starting point. + +#### Migration of Ditto UI to TypeScript + +With [#1688](https://github.com/eclipse-ditto/ditto/pull/1688), the Ditto UI has been migrated from JavaScript codebase +to TypeScript, introducing `npm` to build the UI. + + +### Bugfixes + +#### Fixed that failed retrieval of a policy (e.g. after policy change) leads to search index being "emptied out" + +A bug [#1703](https://github.com/eclipse-ditto/ditto/issues/1703) was fixed, where the search index for things which +could not fetch an updated policy via a cache-loader, was basically dropped. +This could e.g. happen if a single policy is used by a lot of things and this was updated. + +### Helm Chart + +#### Allow BASIC authentication for devops/status users while using Helm deployment and Ingress Controller + +In [#1760](https://github.com/eclipse-ditto/ditto/pull/1760) the Ditto Helm chart was enhanced to use "Basic Authentication" +when using the Ditto Helm chart together with an Ingress Controller for authenticating at the DevOps APIs. + + +## Migration notes + ### Migrating to Ditto 3.4.x -To migrate a running system with live data there are few configurations that should be overridden with Java system properties in the following services. +{% include warning.html content="Updating to Ditto 3.4.0 from versions < 3.4.0 is only possible with a full cluster recreate. + Rolling update is not supported as there are changes in the management urls and ports." %} + +To migrate a Ditto < 3.4.0 to Ditto 3.4.0, the renaming of an index name in Ditto's MongoDB persistence has to be done. +The following section describes, how. + +Apart from that, the transition to 3.4.0 should be smooth. + +#### Renaming an index name + +As part of the migration from Akka to Pekko and from [akka-persistence-mongo](https://github.com/scullxbones/akka-persistence-mongo) +persistence plugin to [pekko-persistence-mongo](https://github.com/scullxbones/pekko-persistence-mongo) a previously +named index `akka_persistence_metadata_pid` was renamed to `pekko_persistence_metadata_pid`. + +When starting Ditto 3.4.0 and not having adjusted the index name, the MongoDB will respond with errors as `pekko-persistence-mongo` +will try to create the same index with a different name. + +To migrate, there are two options: +1. dropping the old indexes before upgrading to Ditto 3.4.0 +2. adjusting the configuration with system properties to still keep using the old index name + +##### Option 1: dropping the old indexes + +* Connect to your MongoDB with the permissions to alter/drop indexes +* Drop the indexes using the following commands (via MongoDB shell): + ``` + db.policies_metadata.dropIndex("akka_persistence_metadata_pid") + db.things_metadata.dropIndex("akka_persistence_metadata_pid") + db.connection_metadata.dropIndex("akka_persistence_metadata_pid") + db.connection_remember_metadata.dropIndex("akka_persistence_metadata_pid") + ``` +* Depending on whether you use a single MongoDB "database" or several, you need to switch to the correct database before + each drop + + +##### Option 2: adjusting the configuration with system properties + +Configure the following system properties for the described Ditto services. Policies: ```markdown @@ -47,8 +214,17 @@ Connectivity: * -Dpekko-contrib-mongodb-persistence-connection-remember-journal.overrides.metadata-index=akka_persistence_metadata_pid ``` -Full cluster recreate is required, rolling update is not supported as there are changes in the management -urls and ports. -Other than that the transition should be smooth. +## Roadmap + +Looking forward, the (current) ideas for Ditto 3.5.0 are: + +* [#1650](https://github.com/eclipse-ditto/ditto/issues/1650) Enforcing linked WoT ThingModels in Things/Features by validating JsonSchema of model elements + * Ensuring that a Ditto Thing is ensured to always follow its WoT ThingModel and also message payloads are always + provided in the specified format +* [#1521](https://github.com/eclipse-ditto/ditto/issues/1521) Configure on a namespace basis the fields to index in the thing-search +* [#1700](https://github.com/eclipse-ditto/ditto/issues/1700) Show policy imports in Ditto explorer UI +* [#1637](https://github.com/eclipse-ditto/ditto/issues/1637) Let Policies declare to be applicable only for certain namespaces +Apart from those ideas we are open to contributions of the community, improving or fixing areas which are important for +their use of Ditto.