Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numpy caching of data in online monitor storage #698

Merged
merged 7 commits into from Aug 10, 2022

Conversation

JoranAngevaare
Copy link
Member

What is the problem / what does the code in this PR do
In #346, we uttered the prophetic words
It's just the backend-buffer that is sitting otherwise idle in the background eating your RAM that needs clearing.
We now learned that this is true when we have longer than usual runs and data with large volumes.

We also should have known that the sys.getsizeof wasn't taking the total size of the document but only of the dictionary.

This PR fixes both with using numpy for the buffers such that the max size of buffers is respected by the backend.

@coveralls
Copy link

coveralls commented Aug 9, 2022

Coverage Status

Coverage decreased (-0.1%) to 91.678% when pulling 47e5894 on numpy_mongo_buffers into 1562542 on master.

Comment on lines 126 to 127
for key in np.dtype(dtype).names:
result[key] = [dd['data'][key] for dd in doc]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For bookkeeping, this is about 10x faster:

n=int(1e4)
bla = list(range(n))
n_bla = np.zeros(len(bla), dtype=[('a', np.int64), ('b', np.int64), ('c', np.int64)])
bla_d = [{'a': i, 'b': i, 'c':1} for i in bla]


%%timeit
for i in range(len(bla)):
    for key in ['a', 'b', 'c']:
        n_bla[i][key] = bla_d[i][key]
22.4 ms ± 97.1 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)


%%timeit
for key in ['a', 'b', 'c']:
    n_bla[key] = [dd[key] for dd in bla_d]
1.98 ms ± 50.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

@JoranAngevaare JoranAngevaare merged commit ef39e3f into master Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants