-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Huge memory usage #72
Comments
Hi, Thanks for reporting this issue; this is indeed a little problem. |
I've created a large file using this script: with open("large_file.bin", "wb") as f:
for _ in range(1024 * 1024):
f.write(b"X" * 20)
f.write(b"\xff\xff\xff\xff") This will create a file with size around 20MB which contains a lot of strings. import time
import rust_strings
from memory_profiler import profile
@profile()
def main():
time.sleep(1)
x = rust_strings.strings("large_file.bin")
time.sleep(1)
if __name__ == "__main__":
main() The memory huge consumption reproduce:
I've tried to debug it but I don't think there is a problem. |
Indeed, the issue doesn't show up when providing a file path to strings, but it does when using the bytes input option: import time
import rust_strings
from memory_profiler import profile
@profile()
def main():
with open("large_file.bin", 'rb') as f:
data = f.read()
time.sleep(1)
x = rust_strings.strings(bytes=data)
time.sleep(1)
if __name__ == "__main__":
main() |
Hi,
While using the lib, I witnessed a huge memory usage (peak of ~ 230Mo to extract strings from a 22Mo sample) from the python lib but not from the binary. I suspect there is a lot of overhead while allocating strings, but the memory usage drops when the strings are returned from the lib.
To monitor the memory usage, I used memory-profiler and a python script that loads the data in memory, waits for a second, extracts the strings using rust-strings, waits for a second and exits.
Do you have an idea of what can cause such a memory usage?
I'll continue to investigate on my side.
The text was updated successfully, but these errors were encountered: