-
Notifications
You must be signed in to change notification settings - Fork 3.9k
/
3.9.0.md
242 lines (116 loc) · 13.7 KB
/
3.9.0.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
The most important changes are [Streams](https://blog.rabbitmq.com/posts/2021/07/rabbitmq-streams-overview/), support for [Erlang 24](https://blog.rabbitmq.com/posts/2021/03/erlang-24-support-roadmap/), and improved integration with our [RabbitMQ Kubernetes Operator](https://blog.rabbitmq.com/posts/2020/11/rabbitmq-kubernetes-operator-reaches-1-0/). Other changes worth mentioning are JSON logging, per-protocol and queue type message rates as Prometheus metrics and faster initial cluster formation on Kubernetes.
Find blog posts that expand on the previously mentioned topics: https://blog.rabbitmq.com/tags/v3.9.x/
## Highlights
### Streams
[Streams](http://rabbitmq.com/streams.html) are a new persistent and replicated data structure ("queue type") in RabbitMQ which models an append-only log with non-destructive consumer semantics.
They can be used as a regular AMQP 0.9.1 queue or through a new [binary protocol plugin](http://rabbitmq.com/stream.html) and associated client(s).
Streams enable messaging patterns that were not possible or practical before, for example, repeatable non-destructive consumption and mass fanouts to a very large number of consumers.
### Erlang 24 Support
[Erlang 24](https://blog.rabbitmq.com/posts/2021/03/erlang-24-support-roadmap/) provides 20%-50% higher throughput for many workloads tested by the RabbitMQ core team and community members. Supporting the latest runtime version also allowed RabbitMQ to replace a few external dependencies with libraries now provided by the runtime.
Erlang 24 is now used by default in the [community RabbitMQ Docker image](https://github.com/docker-library/rabbitmq).
### Kubernetes
[RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html) automates provisioning, management, and operations of RabbitMQ clusters running on Kubernetes. It is [open source](https://github.com/rabbitmq/cluster-operator) and is developed by the RabbitMQ Core team at VMware, Inc. This Operator is not specific to 3.9 and can be used with latest 3.8.x release series.
[Messaging Topology Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html#topology-operator) makes it possible to define RabbitMQ resources (virtual hosts, users, permissions, topologies, policies, etc) as Kubernetes objects. This Operator is also [open source](https://github.com/rabbitmq/messaging-topology-operator) and can be used with latest 3.8.x release series.
### Logging in JSON
[Switch from Lager to the new Erlang Logger API for logging](https://github.com/rabbitmq/rabbitmq-server/pull/2861) introduces an option for structured logging in JSON. This is relevant for the Prometheus-based alerting feature that we have been investing in this year: [Notify me when RabbitMQ has a problem](https://blog.rabbitmq.com/posts/2021/05/alerting/).
## Obtaining Packages
RabbitMQ releases are distributed via [GitHub](https://github.com/rabbitmq/rabbitmq-server/releases), [Cloudsmith](https://cloudsmith.io/~rabbitmq/repos/), and [PackageCloud](https://packagecloud.io/rabbitmq).
### Erlang/OTP Compatibility Notes
This release [requires Erlang 23.2](https://www.rabbitmq.com/which-erlang.html) or later. Erlang 24 is recommended as it [offers 20%-50% throughput improvements](https://blog.rabbitmq.com/posts/2021/03/erlang-24-support-roadmap/) for many workloads.
[Provisioning Latest Erlang Releases](https://www.rabbitmq.com/which-erlang.html#erlang-repositories) explains what package repositories and tools can be used to provision latest patch versions of Erlang 23.3.x and 24.x.
## Upgrading to 3.9
See the [Upgrading guide](https://www.rabbitmq.com/upgrade.html) for documentation on upgrades and [RabbitMQ change log](https://www.rabbitmq.com/changelog.html) for release notes of other releases.
RabbitMQ 3.9.0 nodes can run alongside `3.8.x` nodes. However, `3.8.16+` as well as `3.9.x` both require Erlang 23.2 or later (and support 24). `3.9.x`-specific features can only be made available when all nodes in the cluster upgrade to 3.9.0 or any other patch release in the series.
While operating in mixed version mode, some aspects of the system may not behave as expected. The list of known behavior changes is covered below. Once all nodes are upgraded to 3.9.0, these irregularities will go away.
Mixed version clusters are a mechanism that allows rolling upgrade and are not meant to be run for extended periods of time (no more than a few hours).
### Quorum Queue Declaration
In a mixed version cluster, declaration of new quorum queues will fail if executed on a node in the version minority. For example, in a five node cluster with two nodes running 3.8, the three 3.9 nodes are in the version majority and the remaining are in the version minority.
Existing quorum queues will operate just like before.
### Classic Queue Metrics
HTTP API responses will provide a [limited number of metrics](https://github.com/rabbitmq/rabbitmq-server/issues/3080) or metrics for a limited number of queues (for operations such as `GET /api/queues/{vhost}`) when operating in mixed version mode.
## Compatibility Notes
### Client Library Compatibility
Client libraries that were compatible with RabbitMQ `3.8.x` will be compatible with `3.9.0`.
### Upgrading to Erlang 24.x
RabbitMQ `3.8.16` and later require Erlang `23.2` and support Erlang `24` (recommended). Users of RabbitMQ versions that [do not support Erlang 23.2](https://www.rabbitmq.com/which-erlang.html) should first upgrade to the latest `3.8.x` patch release on Erlang 23.2, then upgrade to `3.9.0`, and ideally Erlang 24.
### Getting Help
Any questions about this release, upgrades or RabbitMQ in general are welcome on the [RabbitMQ mailing list](https://groups.google.com/forum/#!forum/rabbitmq-users).
## Changes Worth Mentioning
Release notes are kept under [rabbitmq-server/release-notes](https://github.com/rabbitmq/rabbitmq-server/tree/v3.9.x/release-notes).
Contributors are encouraged to update them together with their changes. This helps with release automation and more
consistent release schedule.
### Core Server
#### Enhancements
* [Streams](http://rabbitmq.com/streams.html), a new durable replicated data structure that allows for repeatable (non-destructive) consumption. RabbitMQ 3.9 introduces a new binary protocol for working with streams. While streams can be used by AMQP 0-9-1 clients, in most cases it will be used via a [RabbitMQ stream binary protocol](http://rabbitmq.com/stream.html) client. The new protocol exposes all the operations specific to this queue type and is heavily optimized for them, resulting in much higher read and write throughput.
[Java](https://github.com/rabbitmq/rabbitmq-stream-java-client) and [Go](https://github.com/rabbitmq/rabbitmq-stream-go-client) clients
* Structural logging in JSON. Nodes can now log in JSON format.
GitHub issue: [#2861](https://github.com/rabbitmq/rabbitmq-server/pull/2861)
* [Peer discovery](https://www.rabbitmq.com/cluster-formation.html) no longer uses randomized startup delay to ensure only one node starts as a seed one during initial (parallel) cluster formation. Instead it relies on an internal runtime locking library.
GitHub issue: [#3075](https://github.com/rabbitmq/rabbitmq-server/pull/3075)
* Binding recovery on node startup is now more efficient. This means in clusters with a lot of queues and/or bindings, nodes start quicker.
GitHub issue: [#3137](https://github.com/rabbitmq/rabbitmq-server/pull/3137)
* Classic mirrored queue replcas are now started synchronously.
GitHub issue: [#2751](https://github.com/rabbitmq/rabbitmq-server/pull/2751)
* `journald` notifications are now performed using a library and not an external tool (`socat`)
GitHub issue: [#2940](https://github.com/rabbitmq/rabbitmq-server/pull/2940)
* Startup banner now provides more information some of which can be customised.
GitHub issues: [#2247](https://github.com/rabbitmq/rabbitmq-server/pull/2247), [#2777](https://github.com/rabbitmq/rabbitmq-server/pull/2777)
* Per-protocol and queue type message rates are now available as Prometheus metrics, as well as Streams-specific errors. This also addresses a long-standing 3.8.x counters issue for most metrics of this type, as captured in [#2783](https://github.com/rabbitmq/rabbitmq-server/issues/2783).
GitHub issues: [#3127](https://github.com/rabbitmq/rabbitmq-server/pull/3127), [#3136](https://github.com/rabbitmq/rabbitmq-server/pull/3136), [#3157](https://github.com/rabbitmq/rabbitmq-server/pull/3157)
#### Internal API and Other Changes
* Standard runtime logger is now used for logging instead of Lager, a 3rd party dependency. In modern Erlang versions, the module offers at least a comparable feature set and efficiency of Lager.
Users who relied on Lager-specific configuration, e.g. custom extensions or sinks, would have to rebuild them on top of the standard Erlang `logger` module.
GitHub issue: [#2861](https://github.com/rabbitmq/rabbitmq-server/pull/2861)
* Different Raft-based features now can have different defaults and settings, with more configurability coming at a later point. For example, client ID tracking in the MQTT plugin can use settings for lower memory footprint and more frequent compaction while stream and quorum queues use settings more optimal for higher throughput on the same node.
GitHub issue: [#2909](https://github.com/rabbitmq/rabbitmq-server/pull/2909)
* Continuous integration of open source RabbitMQ has switched to Bazel, GitHub Actions and [BuildBuddy](https://buildbuddy.io), resulting in much faster and incremental test runs.
[Bazel support for Erlang](https://github.com/rabbitmq/bazel-erlang) is a new project open sourced by the RabbitMQ Core team as a result.
* Process group membership now uses `pg`.
GitHub issue: [#2850](https://github.com/rabbitmq/rabbitmq-server/pull/2850)
#### Bug Fixes
* `powershell.exe` now can run without profile
GitHub issue: [#2201](https://github.com/rabbitmq/rabbitmq-server/pull/2201)
* Queue index recovery now can continue after a forced node shutdown during recovery (e.g. killed)
GitHub issue: [#3099](https://github.com/rabbitmq/rabbitmq-server/pull/3099)
* Correct an issue where the `num_acceptors.ssl` configuration value was applied incorrectly
GitHub issue: [#3180](https://github.com/rabbitmq/rabbitmq-server/pull/3180)
* Correct an issue where the `--tags` flag was ignored if `--description` was not specified in the `rabbitmqctl add_vhost` command
### Kubernetes
#### Enhancements
* [RabbitMQ Cluster Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html) automates provisioning, management, and operations of RabbitMQ clusters running on Kubernetes.
* [Messaging Topology Operator](https://www.rabbitmq.com/kubernetes/operator/operator-overview.html#topology-operator) makes it possible to define RabbitMQ resources (virtual hosts, users, permissions, topologies, policies, etc) as Kubernetes objects.
* Kubernetes [peer discovery](https://www.rabbitmq.com/cluster-formation.html) mechanism no longer uses randomized delays to avoid the inherent race condition during [initial cluster formation](https://www.rabbitmq.com/cluster-formation.html#initial-formation-race-condition). Instead it relies on an internal distributed locking mechanism available in modern Erlang releases.
While randomized startup delay was reasonably effective at ensuring only one cluster was formed even during parallel node startup, it by definitely assumes that with some low probability more than one node can still start as seed nodes, and be joined by different groups of cluster peers. The locking mechanism does not have this downside. Now that RabbitMQ requires Erlang 23.2, the internal locking library becomes an option for peer discovery.
`cluster_formation.randomized_startup_delay_range.min` and `cluster_formation.randomized_startup_delay_range.max` configuration keys will no longer have any effect.
GitHub issue: [#3075](https://github.com/rabbitmq/rabbitmq-server/pull/3075)
### AWS Peer Discovery Plugin
#### Enhancements
* [AWS peer discovery mechanism](https://www.rabbitmq.com/cluster-formation.html#peer-discovery-aws) now supports [Instance Metadata Service v2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html). In case it is not available, requests will fall back to the original metadata service endpoint.
The v2 endpoint offers substantial security improvements and is one of the [AWS best practices](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-standards-fsbp-controls.html#ec2-8-remediation).
Contributed by @thuandb (AWS).
GitHub issue: [#2952](https://github.com/rabbitmq/rabbitmq-server/pull/2952)
### Federation Plugin
#### Enhancements
* Quorum queues now can be federated.
GitHub issue: [#2804](https://github.com/rabbitmq/rabbitmq-server/pull/2804)
### Management Plugin
#### Bug Fixes
* Management UI now honors queue name length limits
GitHub issue: [#3202](https://github.com/rabbitmq/rabbitmq-server/pull/3202)
* Management UI now sets a correct www-authenticate response header on authentication failures
GitHub issue: [#3205](https://github.com/rabbitmq/rabbitmq-server/pull/3205)
### Dependency Changes
* cowboy has been upgraded from `2.6.1` to `2.8.0`
* cowlib has been upgraded from `2.7.0` to `2.9.1`
* lager has been removed
* jsx has been upgraded from `2.9.0` to `3.1.0`
* observer_cli has been upgraded from `1.5.2` to `1.6.2`
* osiris had been introduced at `0.1.0`
* prometheus has been upgraded from `4.6.0` to `4.8.1`
* ra has been upgraded from `1.1.9` to `2.0.0`
* ranch has been upgraded from `1.7.1` to `2.0.0`
* recon has been upgraded from `2.5.0` to `2.5.1`
* seshat had been introduced at `0.1.0`
* socat has been removed
* syslog has been upgraded from `3.4.5` to `4.0.0`
* systemd had been introduced at `0.6.1`