DL: Remove fit final multiple and improve serialization of image_count+weights #485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kaknikhil
changed the title
kaknikhil
force-pushed
the
dl/remove_fit_final_serializer
branch
from
2dcab61 to
1d5528e
Compare
orhankislal
reviewed
Mar 4, 2020
| )( | ||
| STYPE=BYTEA, | ||
| SFUNC=MADLIB_SCHEMA.fit_transition_multiple_model, | ||
| FINALFUNC=MADLIB_SCHEMA.fit_final_multiple_model |
Contributor
There was a problem hiding this comment.
It seems the sql and python functions for fit_final_multiple_model are still in code even though we don't call them anymore. Is there a particular reason for keeping them?
Contributor
Author
There was a problem hiding this comment.
No reason, I will remove them.
JIRA: MADLIB-1416 Based on our experiments, we noticed that at the end of each hop, fit multiple would start using a lot of memory (from 80 GB to ~ 140 GB on the master host of our gcp cluster) and we attributed that memory spike to the final function. Removing the final function brought down the memory to around 30-40 GB and also made the places10 query 3 times faster places10 with 20 msts on a gpdb6 cluster with 20 segments ``` With fit multiple final function ~ 2 hours per iteration with memory hovering around 70-80gb and peaking at around 130-140GB at the end of each hop. Without fit multiple final function ~ 35-40 mins per iteration with memory hovering around 30-40 gb and peaking at around 40-45 GB. ``` Co-authored-by: Ekta Khanna <ekhanna@pivotal.io>
JIRA: MADLIB-1416 We need to add the image count to the model weights to get the final state for fit_transition. Previously we would use np.concatenate to join the image count and model weights together. A cleaner and faster way is to append image count to the weights list and get rid of the np.concatenate call. Ran unit test `test_serialize_image_nd_weights_valid_output` with large model_weights `[np.array([1]*100000000), np.array([1]*100000000)]` to confirm the speed improvements. test_serialize_image_nd_weights_valid_output with old code and large weights: 17.8 s test_serialize_image_nd_weights_valid_output with new code and large weights: 16.4 s
kaknikhil
force-pushed
the
dl/remove_fit_final_serializer
branch
from
23f8342 to
5dc38e3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on our experiments, we noticed that at the end of each hop, fit
multiple would start using a lot of memory (from 80 GB to ~ 140 GB on
the master host of our gcp cluster) and we attributed that memory spike
to the final function.
Removing the final function brought down the memory to around 30-40 GB
and also made the places10 query 3 times faster
Improvements to "serializing weights with image count"