`mix local.hex`fails with TLS 1.3. middlebox error

[kenny-evitt]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1]

Elixir 1.18.2 (compiled with Erlang/OTP 27)

Operating system

Ubuntu 16.04.2 LTS

Current behavior

I'm setting up a new CI/CD server for this project and the shell command in the CI/CD pipeline that's failing is:

time mix local.hex --force --if-missing

From the log for that command:

$ time mix local.hex --force --if-missing
...
** (Mix) httpc request failed with: {:failed_connect, [{:to_address, {~c"builds.hex.pm", 443}}, {:inet, [:inet], {:tls_alert, {:unexpected_message, ~c"TLS client: In state hello_retry_middlebox_assert at ssl_gen_statem.erl:781 generated CLIENT ALERT: Fatal - Unexpected Message\n {unexpected_msg,\n     {internal,\n         {server_hello,\n             {3,3},\n             <<4,116,235,26,246,203,137,210,46,130,57,151,...>>,\n             <<86,181,141,85,247,100,159,84,31,29,125,...>>,\n             <<19,1>>,\n             \#{server_hello_selected_version =>\n                   {server_hello_selected_version,{3,4}},\n               pre_shared_key => undefined,\n               key_share =>\n                   {key_share_server_hello,\n                       {key_share_entry,secp384r1,<<4,201,167,...>>}}}}}}"}}}]}
Could not install Hex because Mix could not download metadata at https://builds.hex.pm/installs/hex-1.x.csv.
Alternatively, you can compile and install Hex directly with this command:
    $ mix archive.install github hexpm/hex branch latest

The server isn't behind a proxy and I was able to download the file https://builds.hex.pm/installs/hex-1.x.csv on the server using curl.

What seem to be relevant issues for Erlang/OTP:

• TLS 1.3 middlebox incompatibilities · Issue #6772 · erlang/otp
• Failed to assert middlebox server message in ssl:connect on OTP 26 and 27 · Issue #8470 · erlang/otp

Expected behavior

mix local.hex should successfully install Hex.

[josevalim]

@ruslandoga do you have suggestions here? Is there a flag, such as middlebox_comp_mode, we should enable/disable by default?

@kenny-evitt in any case, you can use the command mix archive.install github hexpm/hex branch latest at the end to install it.

[ruslandoga]

👋

I think middlebox_comp_mode: false is worth a try :)
I think after we started using it in Plausible CE (for self-hosters), we got fewer reports about this error.

---

Maybe it would be possible to run these commands on the CI box to make sure it works before changing the defaults?

iex> :ssl.start
#==> :ok
iex> :ssl.connect(~c"builds.hex.pm", 443, cacerts: :public_key.cacerts_get)
#==> {:ok, {:sslsocket, {:gen_tcp, #Port<0.6>, :tls_connection, :undefined}, [#PID<0.140.0>, #PID<0.139.0>]}}
iex> :ssl.connect(~c"builds.hex.pm", 443, cacerts: :public_key.cacerts_get, middlebox_comp_mode: false)
#==> {:ok, {:sslsocket, {:gen_tcp, #Port<0.7>, :tls_connection, :undefined}, [#PID<0.144.0>, #PID<0.143.0>]}}

🙏 @kenny-evitt ? :)

[kenny-evitt]

@ruslandoga Seems to work!:

iex(1)> :ssl.start
:ok
iex(2)> :ssl.connect(~c"builds.hex.pm", 443, cacerts: :public_key.cacerts_get)

18:21:20.872 [warning] Description: ~c"Failed to assert middlebox server message"
     Reason: [missing: {:change_cipher_spec, 1}]


18:21:20.877 [notice] TLS :client: In state :hello_retry_middlebox_assert at ssl_gen_statem.erl:781 generated CLIENT ALERT: Fatal - Unexpected Message
 - {:unexpected_msg,
 {:internal,
  {:server_hello, {3, 3},
   <<129, 139, 238, 191, 67, 70, 95, 139, 152, 132, 244, 14, 4, 236, 234, 141,
     21, 23, 251, 253, 103, 253, 202, 187, 3, 46, 236, 196, 131, 52, 153, 14>>,
   <<173, 124, 180, 161, 142, 160, 239, 29, 230, 217, 133, 11, 5, 242, 151, 17,
     62, 215, 74, 157, 176, 6, 13, 222, 127, 175, 99, 170, 185, 89, 179, 83>>,
   <<19, 1>>,
   %{
     server_hello_selected_version: {:server_hello_selected_version, {3, 4}},
     pre_shared_key: :undefined,
     key_share: {:key_share_server_hello,
      {:key_share_entry, :secp384r1,
       <<4, 247, 61, 95, 81, 0, 61, 82, 129, 121, 71, 117, 72, 234, 213, 116,
         63, 52, 69, 192, 7, 242, 131, 36, 0, 54, 105, 111, 228, 175, 69, 156,
         ...>>}}
   }}}}
{:error,
 {:tls_alert,
  {:unexpected_message,
   ~c"TLS client: In state hello_retry_middlebox_assert at ssl_gen_statem.erl:781 generated CLIENT ALERT: Fatal - Unexpected Message\n {unexpected_msg,\n     {internal,\n         {server_hello,\n             {3,3},\n             <<129,139,238,191,67,70,95,139,152,132,244,14,...>>,\n             <<173,124,180,161,142,160,239,29,230,217,133,...>>,\n             <<19,1>>,\n             \#{server_hello_selected_version =>\n                   {server_hello_selected_version,{3,4}},\n               pre_shared_key => undefined,\n               key_share =>\n                   {key_share_server_hello,\n                       {key_share_entry,secp384r1,<<4,247,61,...>>}}}}}}"}}}
iex(3)> :ssl.connect(~c"builds.hex.pm", 443, cacerts: :public_key.cacerts_get, middlebox_comp_mode: false)
{:ok,
 {:sslsocket, {:gen_tcp, #Port<0.6>, :tls_connection, :undefined},
  [#PID<0.142.0>, #PID<0.141.0>]}}

[josevalim]

@ruslandoga would you like to send a PR to fix httpc on our end? Thank you 🙌

[kenny-evitt]

Unsurprisingly, mix local.rebar also fails with the same error but, unfortunately, mix archive.install ... doesn't completely workaround it:

$ time mix archive.install github erlang/rebar3 --force
...
* Getting new package (https://github.com/erlang/rebar3.git)
remote: Enumerating objects: 23962, done.        
remote: Counting objects: 100% (234/234), done.        
remote: Compressing objects: 100% (145/145), done.        
remote: Total 23962 (delta 100), reused 156 (delta 82), pack-reused 23728 (from 4)        
Receiving objects: 100% (23962/23962), 7.98 MiB | 0 bytes/s, done.
Resolving deltas: 100% (16023/16023), done.
origin/HEAD set to main
** (Mix) httpc request failed with: {:failed_connect, [{:to_address, {~c"builds.hex.pm", 443}}, {:inet, [:inet], {:tls_alert, {:unexpected_message, ~c"TLS client: In state hello_retry_middlebox_assert at ssl_gen_statem.erl:781 generated CLIENT ALERT: Fatal - Unexpected Message\n {unexpected_msg,\n     {internal,\n         {server_hello,\n             {3,3},\n             <<173,235,99,239,222,55,73,35,192,7,21,35,...>>,\n             <<118,189,165,76,98,166,149,68,242,255,150,...>>,\n             <<19,1>>,\n             \#{server_hello_selected_version =>\n                   {server_hello_selected_version,{3,4}},\n               pre_shared_key => undefined,\n               key_share =>\n                   {key_share_server_hello,\n                       {key_share_entry,secp384r1,<<4,141,226,...>>}}}}}}"}}}]}
Could not install Rebar because Mix could not download metadata at https://builds.hex.pm/installs/rebar3-1.x.csv.