Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in binary SDP deserialization #159

Closed
vasdommes opened this issue Dec 16, 2023 · 0 comments
Closed

Memory leak in binary SDP deserialization #159

vasdommes opened this issue Dec 16, 2023 · 0 comments

Comments

@vasdommes
Copy link
Collaborator

When loading a matrix from binary SDP, we create new buffer, read data into it and then call El::Matrix::Control() to acquire ownership of the buffer.

Ring *buffer = new Ring[size];
auto data = boost::serialization::make_array(buffer, size);
ar & data;
matrix.Control(height, width, buffer, leadingDimension);
}

However, due to a bug in El::Matrix implementation, buffer is never deleted in destructor, which causes a memory leak:
https://gitlab.com/bootstrapcollaboration/elemental/-/issues/3

Amount of leaked memory can be several times higher than sdp.zip size:

  1. sdp.zip is compressing zeros (see Fix #148 Binary SDP format is larger than json format if blocks contain many zeros #149), but after deserialization each zero takes the same amount of RAM as any other BigFloat number.
  2. If SDPB performs a timing run, then it reads sdp.zip twice: for the timing run and for the actual run.
  3. If some block is shared among several processes, then each process reads the whole block_data_XXX.bin file and allocates the corresponding matrices. Thus, the block is 'leaking' several times.

In practice, I've observed ~5GB leak for sdp.zip ~0.9GB, and ~40GB leak for sdp.zip ~8GB.

@vasdommes vasdommes added this to the 2.6.1 milestone Dec 16, 2023
@vasdommes vasdommes self-assigned this Dec 16, 2023
vasdommes added a commit that referenced this issue Dec 16, 2023
El::Matrix::Control() does not take ownership correctly, which caused theleak
https://gitlab.com/bootstrapcollaboration/elemental/-/issues/3
vasdommes added a commit that referenced this issue Dec 16, 2023
Fix #159 Memory leak in binary SDP deserialization
bharathr98 pushed a commit to bharathr98/sdpb that referenced this issue Mar 1, 2024
El::Matrix::Control() does not take ownership correctly, which caused theleak
https://gitlab.com/bootstrapcollaboration/elemental/-/issues/3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant