-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TdmsFile.read() leaks memory #199
Comments
There are some circular references between Until then calling
80 MB sounds about right. As you say, calling close doesn't have any effect when using
Yes, accessing the data causes it to be cached internally, you'd also need to set |
Fixes #199. This circular reference meant that when a TdmsFile went out of scope the memory it used wasn't immediately freed but was only freed after a GC collection.
@adamreeve Thanks for the fix. Memory does appear to be freed in a timely fashion as expected. Though with these new changes, the following code now throws the following exception: tdms_file = TdmsFile.read(path)
tdms_file.close() As previously mentioned, the The problem is in the following code: class TdmsReader(object):
# ...
def close(self):
if self._file_path is not None:
# File path was provided so we opened the file and
# should close it.
self._file.close()
# Otherwise always remove reference to the file
self._file = None With the snippet I provided, |
Ah sorry about that, yes it wasn't intentional to change that behaviour, I've pushed a fix for this now. |
Something seems fishy with memory usage of the
TdmsFile.read()
method. It appears to do as it states - load the entire TDMS file in memory - but the memory doesn't appear to be released (at least in any sort of timely fashion) afterward.Take this simple example:
After the loop has complete, memory usage of the Python script is 460 MB.
Now, manually clearing the
channel._raw_data.data
value will net the [more] expected memory usage of the script:Memory usage at end of script = 17 MB
Though, simply accessing the channel data increases memory usage back up to 460 MB (the data is kept in cached properties?).
Manually calling
gc.collect()
immediately after thewhile
loop does free some memory, but the script memory usage is still ~80 MB afterwards (one TDMS file data, I suppose).How are we supposed to be freeing memory here using
TdmsFile.read()
? In all these examples, thetdms_file
variable is re-purposed inside thewhile
loop, yet the oldtdms_file
memory is not being freed once exited or between loop iterations. My Python scripts are consistently running out of memory whenever I try to useTdmsFile.read()
while looping over a number of TDMS files of decent size.The text was updated successfully, but these errors were encountered: