Memory profiling #152
Comments
|
Already working on it. I ran memory-profiler and need to collect the results. I'll try memray too. Any particular pack configurations or parameters you want me to run? |
|
Not really, just see how it scales with number of processors |
|
Below are some memory profiling results. I used the Memray and memory_profiler tools to log the memory usage during various pack simulations. Information about the system I ran the tests on is here: Small pack tests, 16p2s, 1 cycleMemory tests for the
Memory tests for the
Small pack tests, 16p2s, 10 cyclesMemory tests for the
Small pack tests, 16p2s, 10 cycles, SEI degradationMemory tests for the
Medium pack tests, 32p20s, 1 cycleMemory tests for the
Results from the memory_profiler tool.
Megapack tests, 1 cycleMemory tests for the
|
|
Memray does not give reliable results for the ray simulations. It seems to be stuck at 16-17 GB of memory usage. Not sure why. I checked htop during the simulations and it was inline with the memory_profiler results. So I should redo the Memray tests with the memory_profiler tool or figure out how to properly use Memray with ray. |
|
Thanks @wigging, can you plot memory over time for a medium pack for 1000 cycles? |
|
Memray produces a huge log file and will probably use up all my storage space for 1000 cycles. Might have a similar problem with memory_profiler. If I can adjust the logging frequency of these tools then it will be feasible for a 1000 cycle simulation. Otherwise, I'll have to use something like psutil to log memory usage during a simulation. |
|
Well probably we would see a trend with fewer cycles. Going from 1 to 10 increased by 20mb it would seem for the small pack. Would be interesting to see the increase on the medium pack |
|
Did more tests with the memory_profiler tool for the small pack. I also have some medium pack results which I'll share in the next comment. Small pack tests, 16p2s, 1 cycleMemory_profiler results for
Small pack tests, 16p2s, 10 cyclesMemory_profiler results for
|
|
Here are the medium pack results. Medium pack test, 32p20s, 100 cyclesMemory_profiler results for
Medium pack test, 32p20s, 1000 cyclesMemory_profiler results for
|
|
Thanks @wigging does it break down what variables are contributing at all? |
|
The memory_profiler tool just gives total memory over time and it works with the Casadi and Ray liionpack solvers. Memray shows how the memory is allocated but I can't get it to work properly with the Ray solver. So my next test is to use Memray with the Casadi solver and see if I can tell what is increasing the memory. |
|
Here are more results using the memory_profiler tool. Compare these to the Memray results in the next comment. Medium pack test, 32p20s, 10 cyclesMemory_profiler results for
|
|
Here are results from the Memray profiler. Medium pack test, 32p20s, 10 cyclesMemray results for
Memray stats results. Memray tree results. Here is the HTML file for the flame graph generated from Memray. |
|
The flame graph represents allocations contributing to peak memory usage. According to the flame graph for the 32p20s 10 cycles test using the Casadi solver, most of the memory usage is associated with the following liionpack functions, methods, and properties:
|
|
Would it be easy to run the same script for twice as long and compare the output to see which objects grew? We could also just try throwing in a bunch of del statements to objects that could be the issue to see what works. I've never really understood how garbage collection works - especially for parallel processes but potentially as the processes are getting continuously used rather than destroyed that might be doing something gc is not expecting. Maybe @martinjrobins has some insights? |
|
If I run it longer, more than 10 cycles, I run out of storage space and Memray crashes. I could run 5 cycles and compare that to the memory usage for 10 cycles. |
|
The latest Memray update (v1.1.0) fixed the large bin file size. The table below compares the storage size of the bin file generated from Memray v1.0.3 (Run 1) and v1.1.0 (Run 2). The bin size from v1.1.0 is much smaller at 1.7 GB compared to the previous version which generated a 24 GB bin file.
|
|
@wigging great so are you able to compare which objects grow most over different numbers of cycles. If you share the scripts you are running for memray I can help fix the issues |
|
Here are more results from Memray using their latest version which reduces the size of the log file (bin file). Results are based on the flamegraph generated for each pack simulation. More details are given in the flamegraph html files but the tables below summarize the objects allocating the most memory. Medium pack tests, 32p20s, flamegraph comparisonMemray results for
|
|
@wigging I just ran 16p4s charge and discharge cycling for 10 cycles and saw no increase in memory over time just watching my memory usage in task monitor. I even saw it go up a little and then down a little. This was on windows 10. I wonder if the problem is a linux issue. It doesn't make sense that output should grow in memory over time as the arrays are created on the first step with fixed size and filled with zeros. Do you have any intuition on the differences between memory management or multiprocessing in windows and linux? Here's the code |
|
I also saw no increase in memory for the 10 cycle tests. Memory usage seemed to increase when using 30 cycles or more. Run your example for 100 cycles, not 10 cycles. Use the memory_profiler or Memray package to log your memory use. Not the Window's task manager. |
|
Here is the file I'm running for the memory_profiler and Memray tests. """
Medium size pack with 640 cells in a 32p20s configuration.
Run with memory_profiler and log at intervals of 20 s
$ mprof run --interval 20 pack_32p20s.py
Plot memory_profiler results to a PDF file
$ mprof plot --output file.pdf file.dat
Run Memray profiler
$ memray run pack_32p20s.py
"""
import liionpack as lp
import pybamm
import numpy as np
# Define parameters
Np = 32
Ns = 20
cycles = 80
nproc = 20
manager = 'casadi'
# Generate the netlist
netlist = lp.setup_circuit(Np=Np, Ns=Ns)
# Define additional output variables
output_variables = ['Volume-averaged cell temperature [K]']
# Define a cycling experiment using PyBaMM
experiment = pybamm.Experiment([
'Charge at 20 A for 30 minutes',
'Rest for 15 minutes',
'Discharge at 20 A for 30 minutes',
'Rest for 30 minutes'] * cycles,
period='30 seconds')
# Define the PyBaMM parameters
chemistry = pybamm.parameter_sets.Chen2020
parameter_values = pybamm.ParameterValues(chemistry=chemistry)
inputs = {"Total heat transfer coefficient [W.m-2.K-1]": np.ones(Np * Ns) * 10}
# Solve the pack
output = lp.solve(netlist=netlist,
sim_func=lp.thermal_simulation,
parameter_values=parameter_values,
experiment=experiment,
output_variables=output_variables,
initial_soc=0.5,
inputs=inputs,
nproc=nproc,
manager=manager) |
|
This is what I got for 32p4s 40 cycles using your script: |
|
What are the specs of the machine you're running on? Can you run a test for 32p20s for 100 cycles using casadi and ray solvers? And run the profiler using |
|
|
|
I think something that might fix it would be periodically saving the state of the system and restarting the processes. I need to implement saving and loading states anyway so might be a good work around. It seems like it does do periodic recovery but not enough which is quite strange. Have you tried decorating any functions for the profiler / know if that would reveal anything extra? |
|
I put the results from Memray in the above tables. See here. We can go through the Memray results if you think that would help. The |
|
The size of the output doesn't make sense. It should be much larger to start with. This is about 460 mb |
|
I think this might not be a memory leak at all but a slow filling up of the arrays with real numbers instead of pointers. There's something different happening when you do sys.getsizeof and that gives you the full size. I profiled the code and added a new array filled with random numbers that should get fully populated with real numbers, I'm guessing the zeros array all point to a single zero number in memory and then each step in the algorithm fills a new column of data hence it looks like a leak but the actuall arrays should be pretty huge for a 32p20s system. Here is the difference in profiled memory |
|
Looks like the main solver function changed in the latest liionpack release. All of the memory profiling I did was with version 0.3. The latest version is 0.3.1 which has the old solver code commented out. So I don't know if any of the results I've shown here are still valid. |
|
It's more or less the same code but easier to step and solve. All the memory allocation for the above arrays is the same |
|
Here's a comparison from using a 30 second period and a 60 second period for a 32p20s simulation. Peak memory results are from the memory_profiler tool. The plot below is from using the 60 second period. As expected, the simulation with a 60 second period has lower memory usage then with a 30 second period. Probably because there are fewer time steps thus fewer results stored in the arrays. So for these large pack simulations with many cycles, the memory usage could be flattened by not storing results at each time step. It looks like the liionpack code only relies on the previous time step so there's no need to store all the previous time steps in memory. Results from previous times steps could be written to file or deleted depending on how the results should be viewed/analyzed. I need to look at the liionpack code in more detail but I think this might be a good way to reduce the memory usage for large/long running simulations.
|
|
I ran some NumPy examples that build a large array to investigate what causes the memory usage to increase. I won't go into all the detail, but the examples and summary of the results can be read here. Based on the examples, the following should be considered for liionpack:
A key finding from the examples I ran is shown in the plot below. It shows the memory usage of building a 500x2000x2000 array. The array is created on disk using an hdf5 file. Notice the memory used is about 100 MiB. If this array were built completely in memory it would use 8 GB of RAM. However, since this approach writes the array to disk, it requires 8 GB of storage space for the hdf5 file.
|
|
Great, are you able to work out a prototype for saving the output while it solves? |
|
@TomTranter Just want to check with you and see if you want this to be a part of liionpack. Writing the output to hdf5 is only needed for large long-running simulations. Small to medium pack simulations don't need this feature. Also, this would introduce more package dependences to the project. |
|
I think it would be a good feature and happy to have hdf5 deps. It's a very widely used package |
|
If needed, we could make it optional - but hdf5 is recommended if we ever wanted to do a parallel IO. |
|
Maybe Dask arrays would also work but then we'd need to get Dask actors working |
|
I think saving the output should be optional but I don't see why the format should be optional too if it's simpler just to use hdf5 |
|
@wigging did you get anywhere with hdf5? Shall we close this issue and open a new one for that? |
|
Changing the NumPy array type from float64 to float32 was the first step to saving results to an HDF5 file. This will reduce the file size by half when results are saved. But I haven't done anything beyond that. I will close this issue since the memory profiling is done and we plan to move forward with writing the results to file to avoid memory issues for long running simulations. |
@wigging could give this a try https://github.com/bloomberg/memray
The text was updated successfully, but these errors were encountered: