Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lift 32-bit limit of ETS hash table size limit #8589

Merged
merged 3 commits into from
Jun 24, 2024

Conversation

sverker
Copy link
Contributor

@sverker sverker commented Jun 17, 2024

Fix #8198

The PR also includes a commit that introduces C99 type bool with values true and false in the ETS code base.

@sverker sverker added team:VM Assigned to OTP team VM fix labels Jun 17, 2024
@sverker sverker self-assigned this Jun 17, 2024
Copy link
Contributor

github-actions bot commented Jun 17, 2024

CT Test Results

    3 files    143 suites   49m 39s ⏱️
1 591 tests 1 541 ✅ 49 💤 1 ❌
2 330 runs  2 255 ✅ 74 💤 1 ❌

For more details on these failures, see this check.

Results for commit 7755458.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

@sverker sverker added the testing currently being tested, tag is used by OTP internal CI label Jun 17, 2024
@sverker sverker force-pushed the sverker/erts/ets-table-size-limit branch from 7755458 to f26d379 Compare June 24, 2024 15:25
@sverker sverker merged commit 30754c3 into erlang:master Jun 24, 2024
14 of 15 checks passed
@sverker
Copy link
Contributor Author

sverker commented Jun 24, 2024

This PR has now been merged to master for OTP 28.0.

@UlrikVendelbo @mikpe

What has not been done is testing the API on a table with more than 2^31 keys. That would not only take a machine with enough memory but also some suitable tests to get some code coverage without spending too much time and memory.

I tried to create a big table on my laptop and got

  • 250 million smallest possible objects {IntegerKey}
  • 17 Gbyte of memory consumed
  • ~5 minutes to grow the table
  • ~1 minute for ets:match to traverse the table and visit every object

That is only 1/8 on the way to a 2 billion keys table.

@sverker
Copy link
Contributor Author

sverker commented Jun 24, 2024

To be clear: We do not have easy access to such capable machines. So, I'm fishing for some test assistance.

@mikpe
Copy link
Contributor

mikpe commented Jun 26, 2024

I managed this far before killing a 128GB test node:

size: 1546649600, memory 13055561606

That's with arity-1 tuples and fixnums 0... as keys.
I'll see if I can find a beefier one, but no promises. Since we migrated from Mnesia to Aurora we've downscaled all of our production Erlang nodes, long gone are the days of 2TB RAM :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix team:VM Assigned to OTP team VM testing currently being tested, tag is used by OTP internal CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Raise the hash bucket size limit for Erlang ETS tables to a higher value than 32-bit signed integer
2 participants