[Embedding] Fix coredump when destruct FAE object and enable auto selection of ValuePtr. #114
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
candyzone
reviewed
Mar 14, 2022
| @@ -281,7 +281,7 @@ class EmbeddingVar : public ResourceBase { | |||
| EmbeddingFilter<K, V, EmbeddingVar<K, V>>* filter_; | |||
|
|
|||
| ~EmbeddingVar() override { | |||
| if (emb_config_.is_primary()) { | |||
| if (emb_config_.is_primary() && emb_config_.primary_emb_index == 0) { | |||
| switch (sc_.type) { | ||
| case StorageType::DRAM: | ||
| VLOG(1) << "StorageManager::DRAM: " << name_; | ||
| kvs_.push_back(std::make_pair(new LocklessHashMap<K, V>(), ev_allocator())); | ||
| kvs_.push_back(std::make_pair(new LocklessHashMap<K, V>(), cpu_allocator())); |
There was a problem hiding this comment.
改回 ev allocator ，有问题的ev_allocator 已经revert了
| StorageConfig(StorageType t, | ||
| const std::string& p, | ||
| int64 s) : type(t), path(p), size(s) {} | ||
| int64 s, | ||
| const std::string layout) : type(t), path(p), size(s) { |
There was a problem hiding this comment.
在这个类是Embedding整体的配置
以后是要对每一级的layout 配置，存在多级存储下面有不同的layout的情况
| case LayoutType::NORMAL_FIX: | ||
| new_value_ptr_fn_ = [] (Allocator* alloc, size_t size) { return new NormalContiguousValuePtr<V>(alloc, size); }; | ||
| break; | ||
| } |
| bool IsMultiLevel() { | ||
| return is_multi_level_; | ||
| } | ||
|
|
||
| void DebugString() { |
There was a problem hiding this comment.
加了新的配置，修改DebugString 方法
改成TF的标准样式，
std::string DebugString() const {
return strings::StrCat(....................)
| if (max_element_size != 0 && false_positive_probability != -1.0){ | ||
| kHashFunc = calc_num_hash_func(false_positive_probability); | ||
| num_counter = calc_num_counter(max_element_size, false_positive_probability); | ||
| } else { | ||
| kHashFunc = 0; | ||
| num_counter = 0; | ||
| } | ||
| if (layout_type == LayoutType::NORMAL_FIX) { | ||
| if (layout == "normal_fix") { |
lixy9474
force-pushed
the
fix_fae_destruct
branch
2 times, most recently
from
48c7a22
to
879c92f
Compare
liutongxuan
approved these changes
Mar 15, 2022
candyzone
reviewed
Mar 17, 2022
| self._layout = "normal" | ||
| self._layout = "normal_fix" | ||
| self._layout = "light" | ||
|
|
|
rerun test @bot |
lixy9474
force-pushed
the
fix_fae_destruct
branch
from
879c92f
to
20203bd
Compare
candyzone
approved these changes
Mar 17, 2022
lixy9474
force-pushed
the
fix_fae_destruct
branch
from
20203bd
to
516dee8
Compare
…ection of ValuePtr.
lixy9474
force-pushed
the
fix_fae_destruct
branch
from
516dee8
to
072f2db
Compare
candyzone
approved these changes
Mar 18, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.