Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix crashes on OTP 27 by disabling ETS table compression #796

Merged
merged 3 commits into from
Jul 23, 2024

Conversation

zachallaun
Copy link
Collaborator

Fixes #788.

We've had mysterious OTP 27 crashes for quite some time now, but I was finally able to track it down to what I believe is a bug in :ets. When the :compressed table option is used, the system either segfaults or crashes with a memory-related error on any call that traverses the entire table. In our case, it was a call to :ets.foldl/3, but other full traversals like :ets.tab2list/1 or :ets.tab2file/2 also crash.

The temporary fix to get things working is to just not use :compressed on OTP 27. Unfortunately, this has a very sizeable impact on memory:

# memory usage after indexing Lexical itself
# using :erlang.memory()

# OTP 26
[
  total: 200081864,
  processes: 97217168,
  processes_used: 97187736,
  system: 102864696,
  atom: 1466705,
  atom_used: 1452046,
  binary: 2958496,
  code: 35956838,
  ets: 14065992 # 14MB
]

# OTP 27
[
  total: 481878632,
  processes: 79210832,
  processes_used: 79184048,
  system: 402667800,
  atom: 1360193,
  atom_used: 1358458,
  binary: 50157968,
  code: 26346438,
  ets: 279298656 # 279MB
]

We may want to add user-level configuration to disable indexing for those who have to run OTP 27 on very large projects, but I think that should go in another PR.


One final note: I haven't reported this upstream yet because I was trying to create a more minimal reproduction, but I've unfortunately not been able to. I'm going to spend a bit more time this morning trying to reproduce, but I'll likely submit a bug report later regardless; perhaps someone more familiar with ETS will have suggestions for how to minimally recreate the environment that's causing the crash.

Copy link
Collaborator

@scohen scohen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm embarrassed to say that I started down this path this morning and plumb forgot about VM.Versions.

@scohen scohen merged commit 6572a6a into main Jul 23, 2024
12 checks passed
@zachallaun zachallaun deleted the za-ets-compressed branch July 23, 2024 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Crash with OTP 27.0
2 participants