Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorganize caching in TOD class #14

Closed
tskisner opened this issue Apr 12, 2016 · 2 comments
Closed

Reorganize caching in TOD class #14

tskisner opened this issue Apr 12, 2016 · 2 comments
Assignees
Labels

Comments

@tskisner
Copy link
Member

Currently the read/write methods of the TOD base class are being abused to implement timestream caching. This prevents descendant classes from using this cache. This should be moved into an explicit, separate object. The TOD base class can still contain an instance of this "cache" object, but this will then be available to all descendents. It also opens the possibility for us to explicitly manage the memory of this cache outside of python, which may be needed to address things like issue #13.

@tskisner tskisner added the bug label Apr 12, 2016
@tskisner tskisner self-assigned this Apr 12, 2016
@tskisner
Copy link
Member Author

I now have a proof of principle code in a separate branch that implements aligned memory allocation in pure C. This is the start of the core C library that we need in any case for lots of things. I can use cython to create Numpy arrays from scratch with these memory buffers, and also disable the destruction of that memory when the numpy array is destroyed. These memory buffers are allocated outside of the python memory pool, so can be returned to the OS. Based on memory_profiler output, this technique seems to work. However, it does require care to ensure that all numpy arrays with references to that buffer are out of use before deallocation.

@tskisner
Copy link
Member Author

This is fully implemented as of 3434030.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant