Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RANSAC consumes too much RAM #17

Closed
yjmantilla opened this issue Apr 6, 2020 · 3 comments · Fixed by #32
Closed

RANSAC consumes too much RAM #17

yjmantilla opened this issue Apr 6, 2020 · 3 comments · Fixed by #32

Comments

@yjmantilla
Copy link
Collaborator

yjmantilla commented Apr 6, 2020

I have been testing pyprep against our datasets (GRUNECO, University of Antioquia, Colombia). For example I have 58 channels @ 1000Hz for about 5 minutes. This will need like 10 gb of RAM awarded from the OS to python. This is quite a lot for personal computers.

I have two possible proposals to solve this:

  • Run RANSAC in a channel wise manner.
  • Automatically identify the resample needed for the data to fit memory

I like more the first proposal and @sappelhoff agrees with this approach. This was discussed in #9 .

Once #9 detrend_correction is merged into master I will upload this code so it may be reviewed and tested. It considerably changes find_bad_by_ransac() and a tiny change in run_ransac()

@sappelhoff
Copy link
Owner

looking forward to a PR, I hope this would not slow down RANSAC too much 👍

@yjmantilla
Copy link
Collaborator Author

Im kind of academically compromised right now. I could do a pull request with the code as is, the problem is that is has no explanation. Or maybe by the time @adam2392 is available I will be more available too.

@yjmantilla
Copy link
Collaborator Author

yjmantilla commented Apr 8, 2020

Regarding #8 and #9 this is a comment to remind me that I forgot to include another do_detrend = False in another call to noisyChannels in the robust reference function. I will fix this in the channel-wise ransac PR.

Also include the seed for the ransac as the matlab one according to #21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants