Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Erlang/OTP 21 Compatibility #1616

Closed
45 tasks done
michaelklishin opened this issue May 30, 2018 · 7 comments
Closed
45 tasks done

Erlang/OTP 21 Compatibility #1616

michaelklishin opened this issue May 30, 2018 · 7 comments
Assignees
@michaelklishin @dumbbell
Labels
effort-high
Milestone
3.7.7

Comments

@michaelklishin
Copy link
Member

michaelklishin commented May 30, 2018 •
edited
Loading

Overview

Erlang/OTP 21 has plenty of breaking changes, some of which affect RabbitMQ, our build system and dependencies. This issue is a publicly available "umbrella issue" that can be used to monitor progress on the OTP 21 compatibility track.

UPD: RabbitMQ 3.7.7 is the first release to support Erlang/OTP 21. Earlier RabbitMQ versions will start on Erlang 21.0 but won't accept client connections and CLI tools will fail with exceptions.

Notable Changes and Regressions

Note that Elixir is only OTP 21-compatible starting with 1.6.6, although 1.6.5 was "compatible enough" for us to make progress with it.

Known incompatibilities and other new features that affect us discovered so far:

  • OTP-14692 breaks compilation without extra compiler options that silence warnings
  • New logging subsystem breaks certain assumptions in Lager
  • We've observed issues with client connection handshake timeouts that we cannot reproduce on OTP 20
  • Regression that breaks erl_call: ERL-634
  • Regression that breaks all tests that use httpc: ERL-636
  • Major regression in os:getenv/1 on Windows: ERL-644 addressed in 21.0.1 (republished as 21.0)
  • Support for some (insecure) cipher suites in TLS has been removed or disabled by default
  • OTP-14370 may affect CLI tools test suite and likely all multi-node test suites
  • OTP-14928 may require changes to server scripts
  • OTP-14346 may require changes to RabbitMQ server scripts
  • OTP-13295 seems to cause extra logging in CLI tools which may need silencing
  • OTP-15001 may have an effect on connection error handling
  • OTP-14961 is potentially useful for more detailed memory allocators info
  • OTP-14902 may affect the Erlang client
  • OTP-14594 may have positive effects: requires benchmarking
  • OTP-15087 (the OTP_RELEASE macro) can be potentially helpful

Tasks

Specific contained tasks:

  • rabbit_common compiles
  • core server compiles
  • rabbitmq-cli compiles
  • rabbitmq-management compiles
  • rabbitmq-amqp1.0 compiles
  • rabbitmq-stomp compiles
  • rabbitmq-mqtt compiles
  • Erlang AMQP 0-9-1 client compiles
  • Erlang AMQP 1.0 client compiles
  • rabbitmq-federation compiles
  • rabbitmq-shovel compiles
  • rabbitmq-trust-store compiles
  • All other plugins in the umbrella compile
  • rabbit_common passes tests
  • Nodes can start and stop, including using CT helpers
  • Nodes can accept client connections and complete AMQP 0-9-1 handshake
  • gmake ct-fast in core server passes
  • AMQP 0-9-1 Erlang client passes tests
  • rabbitmq-cli passes tests
  • rabbitmq-management passes tests
  • core server with necessary plugins passes client test suites
  • rabbitmq-amqp1.0 passes tests
  • rabbitmq-stomp passes tests
  • rabbitmq-mqtt passes tests
  • rabbitmq-shovel passes tests
  • rabbitmq-federation passes tests
  • rabbitmq-peer-discovery-k8s passes tests
  • rabbitmq-peer-discovery-etcd passes tests
  • rabbitmq-peer-discovery-consul passes tests
  • rabbitmq-peer-discovery-aws passes tests
  • rabbitmq-trust-store passes tests
  • rabbitmq-top is updated to handle new ETS table info items
  • Lager supports OTP 21 (as of 3.6.3)
  • All core server suites pass
  • Upgrade tests are updated to include upgrades from OTP 21 and pass
  • All other plugins in the umbrella pass their tests
  • New OTP 21 Concourse pipeline version is all green (core server, tier 1 plugins, packages, upgrades, clients)
  • Server scripts are updated to use the same effective I/O subsystem defaults
  • Windows build on OTP 21 is comparable to earlier versions in terms of efficiency
  • Zero dependency Erlang RPM is updated to OTP 21
  • Standalone MacOS package bundles OTP 21
  • Supported Erlang versions guide is updated for OTP 21
  • Release notes explain how to upgrade to 3.7.7 and upgrade Erlang to 21.x at the same time.

Things that may not be functionally important but still would be nice to have for us:
This was referenced May 30, 2018
Merged
Merged
Merged
Merged
michaelklishin added a commit to rabbitmq/rabbitmq-ct-helpers that referenced this issue Jun 4, 2018
@michaelklishin
Log exit code when the test rabbitmqctl invocation fails
f525ad1 
Per discussion with @dumbbell.

This helped us find out the root cause of test suite failures on OTP 21.
References rabbitmq/rabbitmq-server#1616.

[#157964874]
michaelklishin added a commit that referenced this issue Jun 4, 2018
@michaelklishin
Be more defensive about what output rabbitmqctl eval produces here
4d6fd8f 
OTP 21 has a new logging subsystem and extra logging added to the kernel app
to accompany the asynchronous automatic node connection feature. This breaks
this eval command that asks the node for some of its effective paths (e.g.
the plugin dir).

Since we know the set of lines we are interested in, we can filter out
everything else.

Pair: @dumbbell.

Part of #1616.

[#157964874]
dumbbell pushed a commit to rabbitmq/rabbitmq-ct-helpers that referenced this issue Jun 4, 2018
@michaelklishin @dumbbell
Log exit code when the test rabbitmqctl invocation fails
62fd843 
Per discussion with @dumbbell.

This helped us find out the root cause of test suite failures on OTP 21.
References rabbitmq/rabbitmq-server#1616.

[#157964874]

(cherry picked from commit f525ad1)
dumbbell pushed a commit to rabbitmq/rabbitmq-ct-helpers that referenced this issue Jun 4, 2018
@michaelklishin @dumbbell
Log exit code when the test rabbitmqctl invocation fails
7fa94d9 
Per discussion with @dumbbell.

This helped us find out the root cause of test suite failures on OTP 21.
References rabbitmq/rabbitmq-server#1616.

[#157964874]

(cherry picked from commit f525ad1)
@michaelklishin michaelklishin mentioned this issue Jun 5, 2018
Closed
dumbbell pushed a commit that referenced this issue Jun 6, 2018
@michaelklishin @dumbbell
Be more defensive about what output rabbitmqctl eval produces here
1b167ed 
OTP 21 has a new logging subsystem and extra logging added to the kernel app
to accompany the asynchronous automatic node connection feature. This breaks
this eval command that asks the node for some of its effective paths (e.g.
the plugin dir).

Since we know the set of lines we are interested in, we can filter out
everything else.

Pair: @dumbbell.

Part of #1616.

[#157964874]
@michaelklishin michaelklishin mentioned this issue Jun 6, 2018
Merged
dumbbell pushed a commit that referenced this issue Jun 7, 2018
@michaelklishin @dumbbell
Be more defensive about what output rabbitmqctl eval produces here
dae785a 
OTP 21 has a new logging subsystem and extra logging added to the kernel app
to accompany the asynchronous automatic node connection feature. This breaks
this eval command that asks the node for some of its effective paths (e.g.
the plugin dir).

Since we know the set of lines we are interested in, we can filter out
everything else.

Pair: @dumbbell.

Part of #1616.

[#157964874]
@michaelklishin michaelklishin added this to the 3.7.7 milestone Jun 7, 2018
michaelklishin added a commit to rabbitmq/rabbitmq-management-agent that referenced this issue Jun 7, 2018
@michaelklishin
Compile on OTP 21 from scratch
7583f6f 
Part of rabbitmq/rabbitmq-server#1616.

[#157964874]
michaelklishin added a commit to rabbitmq/rabbitmq-trust-store that referenced this issue Jun 7, 2018
@michaelklishin
Compile on OTP 21 from scratch
7bdcb20 
Part of rabbitmq/rabbitmq-server#1616.

[#157964874]
@dumbbell dumbbell mentioned this issue Jun 13, 2018
Merged
dumbbell pushed a commit that referenced this issue Jun 13, 2018
@michaelklishin @dumbbell
Be more defensive about what output rabbitmqctl eval produces here
924819c 
OTP 21 has a new logging subsystem and extra logging added to the kernel app
to accompany the asynchronous automatic node connection feature. This breaks
this eval command that asks the node for some of its effective paths (e.g.
the plugin dir).

Since we know the set of lines we are interested in, we can filter out
everything else.

Pair: @dumbbell.

Part of #1616.

[#157964874]

(cherry picked from commit dae785a)
This was referenced Jun 13, 2018
Merged
Merged
Merged
Merged
Merged
Merged
Closed
dumbbell pushed a commit to rabbitmq/rabbitmq-management-agent that referenced this issue Jun 13, 2018
@michaelklishin @dumbbell
Compile on OTP 21 from scratch
70bd63c 
Part of rabbitmq/rabbitmq-server#1616.

[#157964874]

(cherry picked from commit 7583f6f)
@dumbbell dumbbell mentioned this issue Jun 13, 2018
Merged
dumbbell pushed a commit to rabbitmq/rabbitmq-trust-store that referenced this issue Jun 13, 2018
@michaelklishin @dumbbell
Compile on OTP 21 from scratch
1953aae 
Part of rabbitmq/rabbitmq-server#1616.

[#157964874]

(cherry picked from commit 7bdcb20)
This was referenced Jun 13, 2018
Merged
Merged
michaelklishin added a commit to rabbitmq/erlang-rpm that referenced this issue Jun 20, 2018
@michaelklishin
Update patches for OTP 21
fa267ea 
References rabbitmq/rabbitmq-server#1616.
@ilovezfs ilovezfs mentioned this issue Jun 20, 2018
Closed
4 tasks
@josevalim
Copy link

Elixir v1.6.6 is out and fully supports Erlang/OTP 21.

@michaelklishin
Copy link
Member Author

@josevalim thank you for the update!

@croensch
Copy link

Why do we need pinning in the installation routine? (https://www.rabbitmq.com/install-debian.html)
Could we not set the min/max erlang version for the dependency to be stable*? AFAIK apt also allows for installing "unstable" but i don't exactly know how it works.
apt show is

Package: rabbitmq-server
Version: 3.7.6-1
Depends: init-system-helpers (>= 1.13~), erlang-nox (>= 1:19.3) | esl-erlang (>= 1:19.3), adduser, logrotate, socat`

*so like my six-step ansible script can install a stable server, or containers that get rebuild often... pinning a version there just looks overly complicated and locks that down to break later just my 2 cents
Or am i wrong?

@michaelklishin
Copy link
Member Author

@croensch this is not a discussion forum. There can be plenty of reasons for pinning that have nothing to do with OTP 21. In fact the pinning section long predates this issue or OTP 21 RCs.

@rabbitmq rabbitmq locked and limited conversation to collaborators Jun 28, 2018
@michaelklishin
Copy link
Member Author

All discussions, questions, opinions, suggestions, ideas belong to the mailing list. This issue is about making RabbitMQ compatible with OTP 21 and getting OTP 21 test coverage to the same level as other supported series.

@gerhard
Copy link
Contributor

gerhard commented Jul 2, 2018

Worth keeping an eye on this: Node not responding to init:stop() - Erlang/OTP 21

@michaelklishin
Copy link
Member Author

There will be a couple of additional OTP 21 compatibility improvements in 3.7.8 but they hopefully don't affect more than a low single digit % of users.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@michaelklishin michaelklishin

@dumbbell dumbbell

Labels
effort-high
Projects
None yet
Milestone
3.7.7
Development

No branches or pull requests
5 participants
@michaelklishin @gerhard @josevalim @dumbbell @croensch