Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limit buffer for mongo backend #349

Merged
merged 1 commit into from Nov 13, 2020
Merged

Conversation

JoranAngevaare
Copy link
Member

@JoranAngevaare JoranAngevaare commented Nov 12, 2020

What is the problem / what does the code in this PR do
Address #346

Can you briefly describe how it works?
Keep track of how many runs (datakinds) are stored in the buffer. If to large, dump some.

MWE

for r in range(10165, 10165+10):
    if st.is_stored(f'{r:06}', 'online_peak_monitor'):
        st.get_array(f'{r:06}', 'online_peak_monitor')
        print(st.storage[-1].backends[0]._buffered_backend_keys)
        print(sys.getsizeof(st.storage[-1].backends[0].chunks_registry)/1e6)

Copy link
Contributor

@darrylmasson darrylmasson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nesting a for-loop inside a while-loop might be really inefficient, but probably won't be a real issue unless someone decides to change the defaults.

to_clean = self._buffered_backend_keys[0]
for registry_key in list(self.chunks_registry.keys()):
if to_clean in registry_key:
del self.chunks_registry[registry_key]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python's complained at me if I modified a dictionary while iterating over it. List-casting the keys probably helps with this, but I got around it by storing the keys to delete in a second list.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you really have a good memory for errors, got to give you that. This is true.

However if you look carefully we are making a copy of the keys by iterating them when converting to a list. Therefore you are not iterating over the keys but over the list of strings that are extracted

@JoranAngevaare JoranAngevaare merged commit 152801b into master Nov 13, 2020
@JoranAngevaare JoranAngevaare deleted the max_buffer_mongo_backend branch November 13, 2020 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants