Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialize objects early to save memory #879

Merged
merged 4 commits into from Feb 11, 2021
Merged

Serialize objects early to save memory #879

merged 4 commits into from Feb 11, 2021

Conversation

adelavega
Copy link
Collaborator

@adelavega adelavega commented Feb 6, 2021

To address memory problems with Clarifai Extractor, serializing objects early should save memory

@cypress
Copy link

cypress bot commented Feb 6, 2021



Test summary

7 0 0 0


Run details

Project Neuroscout
Status Passed
Commit 6c1d82b
Started Feb 10, 2021 10:16 PM
Ended Feb 10, 2021 10:17 PM
Duration 00:47 💡
OS Linux Ubuntu - 18.04
Browser Electron 87

View run in Cypress Dashboard ➡️


This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@cypress
Copy link

cypress bot commented Feb 6, 2021



Test summary

7 0 0 0


Run details

Project Neuroscout
Status Passed
Commit f5b6f97 ℹ️
Started Feb 10, 2021 10:15 PM
Ended Feb 10, 2021 10:16 PM
Duration 00:43 💡
OS Linux Ubuntu - 18.04
Browser Electron 87

View run in Cypress Dashboard ➡️


This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@codecov-io
Copy link

Codecov Report

Merging #879 (3b61487) into master (d196891) will decrease coverage by 0.49%.
The diff coverage is 50.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #879      +/-   ##
==========================================
- Coverage   84.70%   84.21%   -0.50%     
==========================================
  Files          57       57              
  Lines        2858     2864       +6     
==========================================
- Hits         2421     2412       -9     
- Misses        437      452      +15     
Impacted Files Coverage Δ
neuroscout/populate/extract.py 65.38% <50.00%> (-1.95%) ⬇️
neuroscout/populate/utils.py 59.09% <0.00%> (-22.73%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d196891...3b61487. Read the comment docs.

@adelavega adelavega changed the title Add regular loop for n_jobs=1 Serialize objects early to save memory Feb 10, 2021
@adelavega
Copy link
Collaborator Author

I was able to extract Clarifai labels for the dataset after these optimizations.

@adelavega adelavega merged commit ed95556 into master Feb 11, 2021
@adelavega adelavega deleted the ext/nnd branch February 11, 2021 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants