recommender: clear_row don't free acquired resources

[rimms]

I tried to measure memory size on before and after call clear_row.

The result is following. The value is RSS size which got by using get_status.

method	after start-up	after update_row 100K ID	after clear_row 10K ID
lsh	4312	521888	522048
minhash	4308	216708	216840
euclid_lsh	4336	236324	236672
inverted_index	4292	434044	434192

We cannot use jubarecommender long time without some solution (e.g. stop service and add memory ...).

[suma]

Current implementation doesn't provide its solution without clear.

Because their remove_row only free id value(std::string), not execute compaction the storage and container(like unordered_map).

https://github.com/jubatus/jubatus/blob/master/src/recommender/lsh.cpp#L92
https://github.com/jubatus/jubatus/blob/master/src/recommender/lsh.hpp#L76
https://github.com/jubatus/jubatus/blob/master/src/recommender/recommender_base.hpp#L81
typedef pfi::data::unordered_map<std::string, row_t> tbl_t; in storage/storage_type.hpp
https://github.com/jubatus/jubatus/blob/master/src/storage/sparse_matrix_storage.hpp#L65
https://github.com/jubatus/jubatus/blob/master/src/storage/sparse_matrix_storage.cpp#L147

[beam2d]

As @suma said, current implementation doesn't run any compaction on tables in recommenders.
We can add compaction codes on master tables, which include orig_ and master of algorithm-specific tables. Diff tables are cleared at the end of MIX.

Note that we cannot save memory before MIX, because we must retain markers representing removals in diff and share them with other servers on MIX. For example, bit_index_storage::remove_row only overwrites the row in diff by an empty bit vector.
https://github.com/jubatus/jubatus/blob/master/src/storage/bit_index_storage.cpp#L68

[rimms]

@suma, @beam2d, Thank you.

I understood that it's specification of Jubatus.

[kmaehashi]

I'm currently facing this problem in my project using standalone-mode... and hoping this to be solved in Jubatus layer.
How about defining inherited version of each storage class for standalone-mode and removing entry in there (instead of overwriting zero for distributed-mode)?

[beam2d]

We can erase an entry in the diff table that is not contained in the master table, because we do not need to MIX such entry with other servers. If we fix the implementation to behave as such, the standalone-mode server just erases items on removal, while distributed-mode servers correctly MIXes removals.

[kmaehashi]

In today's meeting, we agreed to fix this using the method proposed by @beam2d.
I've reopened, changed milestone to Near Future and assigned to me.

[kmaehashi]

After applying #659, #685, and #694, I observed some mitigation.

Some things to note:

• In the current architecture, we cannot "completely" remove rows, as records like column2id_ cannot be cleared.
• As far as I tested, RSS value does not shrink by just erasing elements. I suppose it depends on how unordered_map adjusts its bucket size, and memory fragmentation may also be involved.

Here are RSS values tested on Recommender (inverted_index):

0.5.2

1. Startup : 5776 KB
2. update_row (10k) : 13564 KB
3. clear_row (10k) : 13544 KB
4. update_row (10k) : 18896 KB

0.5.2 + Patch #659, #685 and #694

1. Startup : 5784 KB
2. update_row (10k) : 13456 KB
3. clear_row (10k) : 13472 KB // does not shrink!
4. update_row (10k) : 15824 KB // about 3 MB better than 0.5.2

[kumagi]

We should try rehash of unordered_map.

[kmaehashi]

From discussion in meeting in 2014-03-03, we asked @gintenlabo to try rehash and other possible ideas.

[gintenlabo]

I tried table.rehash(0); to shrink buffers, but memory usage is not changed after clear_row.
c.f. https://github.com/gintenlabo/jubatus_core/tree/fix-issue-294

[kmaehashi]

As we think this behavior is by-design, we close this issue.