Compiler hang

[dmorneau]

Environment

• Elixir & Erlang/OTP versions (elixir --version):

Erlang/OTP 22 [erts-10.7.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
Elixir 1.10.3 (compiled with Erlang/OTP 22)

• Operating system: macOS Catalina

Behavior

While doing a clean build of an app (via mix test) after upgrading to Elixir 1.10.3 (from 1.10.2), I hit a hang in the compiler. It printed the "taking longer than 15s..." warning for many files, then stopped making any progress.

After 10 ~ 15 min I took a crash dump using SIGUSR1. At that point, the only process with messages in its queue was code_server (MsgQ=7). The code server is inside do_purge, waiting on a reply from erts_code_purge, which is itself inside a do_purge function, purging a module called elixir_compiler_18.

The request looks like it came from another process that was compiling some
LiveComponent part of the app:

I can't share the crash dump, but is there something I could check that might help?

This didn't reproduce after I killed the compiler and tried again.

[lukaszsamson]

I've also noticed similar hangs and slowdowns lately. Those happen both on local MacOS and CI (Debian Linux) and started happening before 10.3 was released.

Unfortunately I don't have anything that can help in pinpointing the issue but CI logs like

==> ecto
Compiling 55 files (.ex)
Compiling lib/ecto/repo/queryable.ex (it's taking more than 15s)
Compiling lib/ecto/query/inspect.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/filter.ex (it's taking more than 15s)
Compiling lib/ecto/repo/preloader.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/windows.ex (it's taking more than 15s)
Compiling lib/ecto/schema.ex (it's taking more than 15s)
Compiling lib/ecto/repo/schema.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/preload.ex (it's taking more than 15s)
Compiling lib/ecto/embedded.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/order_by.ex (it's taking more than 15s)
Compiling lib/ecto/type.ex (it's taking more than 15s)
Compiling lib/ecto/uuid.ex (it's taking more than 15s)
Compiling lib/ecto/association.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/join.ex (it's taking more than 15s)
Compiling lib/ecto/log_entry.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/cte.ex (it's taking more than 15s)
Compiling lib/ecto/query/planner.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/select.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/distinct.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/limit_offset.ex (it's taking more than 15s)
Compiling lib/ecto/changeset.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/group_by.ex (it's taking more than 15s)
Compiling lib/ecto/changeset/relation.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/dynamic.ex (it's taking more than 15s)
Compiling lib/ecto/multi.ex (it's taking more than 15s)
Compiling lib/ecto.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder.ex (it's taking more than 15s)
Compiling lib/ecto/query/builder/update.ex (it's taking more than 15s)

Too long with no output (exceeded 10m0s): context deadline exceeded

[josevalim]

Thanks @dmorneau. We have recently heard similar reports. Can you check if there is any process waiting on the code_server/code:purge call? If so, what is the callsite?

@lukaszsamson, do you remember if this happened with v1.9? Or did it start with v1.10? Tracking this down would help us a lot.

[lukaszsamson]

@josevalim IIRC it started happening 2-3 weeks ago and since over a month all of our build pipelines run on 1.10. The first CI failure from our servers is from 19 days ago. If I had to guess I'd blame it on otp 22.3.2 released 21 days ago.

[josevalim]

@lukaszsamson can you please use OTP 22.2 on CI and see if it addresses the issue?

[wojtekmach]

If your CI doesn't allow you to choose a particular OTP version, but you can use a custom Docker image, you can try one of these: https://hub.docker.com/r/hexpm/elixir/tags?name=1.10.3-erlang-22.2.8&page=1

[dmorneau]

Can you check if there is any process waiting on the code_server/code:purge call? If so, what is the callsite?

@josevalim Yes, it's the second screenshot I pasted. It looks like this is the call:

https://github.com/elixir-lang/elixir/blob/v1.10.3/lib/elixir/src/elixir_compiler.erl#L77
https://github.com/erlang/otp/blob/OTP-22.3.2/lib/kernel/src/code.erl#L199

[josevalim]

Thanks! It seems there was a bugfix in the code purger on 22.3.0 to 22.3.1, so I would recommend those running into it to revert to 22.2 and see if the issue persists.

[lukaszsamson]

@lukaszsamson can you please use OTP 22.2 on CI and see if it addresses the issue?

We can but it's not going to be a drop in replacement as we currently use circleci/elixir images which do not tag otp releases

[ericmj]

We can but it's not going to be a drop in replacement as we currently use circleci/elixir images which do not tag otp releases

As @wojtekmach said you can use these docker images which do tag the OTP version: https://hub.docker.com/r/hexpm/elixir/tags?name=1.10.3-erlang-22.2.8&page=1

[adrienmo]

On our side we have seen the problem happening when we switched from 22.3 to 22.3.2. Currently trying to downgrade to see if it will fix it.

[jesse-c]

We use CircleCI images as well and we're unable to use the Hex images due to them lacking git (and possibly other things)

[josevalim]

If anyone runs into this issue, please do:

1. If you are on Linux, do kill -ABRT <pid of beam.smp>
2. If you are on elsewhere, press Ctrl+C, then in the break menu type "A" (uppercase) and press enter

I have opened up an issue with Erlang/OTP here. @dmorneau, if it is an option, please consider sending the crash dump privately to them.

[dimitarvp]

Just upgraded to Erlang 22.3.3, the hang happened still (and on Elixir 1.10.3), and finally got an erl_crash.dump file.

Will I be able to upload it to the Erlang issue tracker if I sign up there?

[josevalim]

@dimitarvp yes, you should. If not, feel free to email me (email is on my profile), and I can make sure it is delivered.

[dimitarvp]

@josevalim Done: https://bugs.erlang.org/browse/ERL-1236?focusedCommentId=17807&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17807