[Enhancement] reduce memory consumption by tablet meta in be

[dataroaring]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Tablet meta consumes memory in be, when there are many tablets, e.g. 40w in a be, then tablets meta consumes a lot memory. e.g. a tablet in a table with 200 columns consumes 70KB memory. Doris store tablet meta in rocksdb while it loads them into memory when doris starts. We can use memory as a cache to reduce memory consumed by tablet meta.

Solution

1. add a lru cache to TabletMgr
2. maintain cache in tablet mgr when create delete and update tablet meta
3. get tablet through cache and rocksdb

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[IoCing]

I want to do this. Please assign to me.

[byrantwithyou]

I'm interested in this task. Would you mind assigning it to me?

[byrantwithyou]

Here are my understandings of the task and how I will implement it. Any suggestions are welcome.

1. Instead of loading all the tablet meta at once during start-up, we'll bring them to memory when the tablet is accessed.
2. Add a configuration parameter to specify how many tablet meta the memory can hold at most.
3. Maintain a lru cache for the tablet meta in the tablet manager. When the tablet is accessed, we update the lru cache and evict/replace it if necessary.
4. We adopt a write-through cache. When the cache updates, also write to the rocksdb to avoid crash issues because the meta update is lightweight.

[dataroaring]

The task is available.

[Arthur-Sun]

I'm new to here and could I have a try?

[nanfeng1999]

I'm new to here and could I have a try?

hello，we can do this issue together，you can add my weChat yzy20200701

[liangjiawei1110]

I'd like to try it.

[nanfeng1999]

I'd like to try it.

ok