Add: big key & top key #1691

chejinge · 2023-07-07T11:54:29Z

Pika big key分析工具-redis有丰富的工具可以在线或者离线进行数据的分析来发现big key pika目前这块还欠缺，业务需求也较

chejinge · 2023-07-15T11:26:44Z

新浪有这个工具
两个工具 big_key top_key 需要确认一下

chejinge · 2023-07-18T08:27:36Z

参考：

#!/usr/bin/env python36
# -*- coding:utf-8 -*-
# Author:david
# 兼容redis cluster 加密和非加密的端口获取big key 的情况
import requests
import pymysql
import argparse
import time
import string
import subprocess
import redis
import json
import subprocess
import sys

def get_params():
    parser = argparse.ArgumentParser(description='This script is used to find  redis big key')
    parser.add_argument('-p', '--port', type=int, required=True, nargs=1,help='Specify redis instance port')

    args = parser.parse_args()
    return args.port[0]

def Conn(mysql_host, mysql_user, mysql_passwd, mysql_port, mysql_db, exec_sql):
    try:
        con = pymysql.connect(host=mysql_host, user=mysql_user, passwd=mysql_passwd, port=mysql_port, db=mysql_db,
                              charset="utf8", connect_timeout=30)
    except:
        print
        "Can't connect to MySQL Server!"
        return -1

    cursor = con.cursor()
    try:
        cursor.execute(exec_sql)
        rows = cursor.fetchall()
        cursor.close()
        con.close()
        return rows
    except:
        print
        "SQL '", exec_sql, "' run error!"
        return -1

def check_env():
    redis_cli="/usr/local/redis32/bin/redis-cli"
    global datadir
    datadir='/data1'
    retcode,output = subprocess.getstatusoutput('ls /usr/local/redis32/bin/redis-cli ')
    if retcode != 0:
        retcode,output = subprocess.getstatusoutput('yum install addops-redis32.x86_64 -y ')
        get_error(retcode,output)


def RedisConn(redisServer, redisPort, redisPwd):
    try:
        r = redis.StrictRedis(host=redisServer, port=redisPort, password=redisPwd)
        r.info("server")
    except redis.exceptions.AuthenticationError as err:
        return RedisConn(redisServer,redisPort,'')
    except Exception  as e:
        print
        "Can't connect to Redis Server!"
        print(e)
        return -1
    return r




def get_big_key(port):
    #print('♥️'*50)
    #print("分析中......请等待.....内存越大时间越久....")
    if 16000 < port < 16999:
        redisPwdSQL = "select old_password('redis%s')" % port
        pwd = Conn('10.208.255.227', 'zabbix', '68ffc1ca109d5019', 3802, 'zabbix',redisPwdSQL)[0][0]
    else:
        r = requests.post("http://api.dba.hulk.qihoo.net/mysql/getPassword?type=redis&port=%s"%(port))
        pwd = json.loads(r.text)['data'][0]['pwd']

    url = "http://api.hulk.qihoo.net:8360/?router=/db/redis/api/ins/get-info-by-port&hulk_uid=600259256&port=%s" %(port)
    header_info ={'hulksrc':'dba'}
    h = requests.post(url,headers=header_info)
    all_redis = json.loads(h.text)
    redis_num = len(all_redis['data'])
    num = 0
    for i in range(redis_num):
        if all_redis['data'][i]['type'] == 'slave':
            sip=all_redis['data'][i]['ip']
            num = num + 1
    if num == 0:
        print('%s do not have slave node,maybe a redis cluster node'%(port))
        for i in range(redis_num):
            if all_redis['data'][i]['type'] == 'master':
                sip=all_redis['data'][i]['ip']
    if 16000 < port < 16999:
        pass
    else:
        sql = '/usr/local/redis32/bin/redis-cli -p {0} -h {1} -a {2} info  server |grep redis_version|wc -l'.format(port,sip,pwd)
        retcode,output = subprocess.getstatusoutput(sql)
        if  int(retcode) != 0 or int(output) != 1:
            print("\033[1;31m此端口不是redis 端口,或许是pika端口\033[0m")
            sys.exit(1)
    sql = '/usr/local/redis32/bin/redis-cli -p {0} -h {1} -a {2} bgsave'.format(port,sip,pwd)
    retcode,output = subprocess.getstatusoutput(sql)
    if  retcode != 0:
        print("\033[1;31mbgsave 失败\033[0m")
        sys.exit(1)
    for i in range(1200):
        time.sleep(1)
        r = RedisConn(sip, int(port), pwd)
        bgsave_sec = r.info(section='Persistence')['rdb_current_bgsave_time_sec']
        if int(bgsave_sec) != -1:
            continue
        else:
            break


    retcode,output = subprocess.getstatusoutput('sudo -u sync360  scp -q -o ConnectTimeout=3 -o ConnectionAttempts=5 -o PasswordAuthentication=no -o StrictHostKeyChecking=no /home/sync360/rdr  %s:/tmp'%(sip))
    if  retcode != 0:
        if output.endswith('busy'):
            pass
        else:
            print(" copy 文件rdr至远端机器失败: %s "%(output))
            sys.exit(1)
    retcode,output = subprocess.getstatusoutput('sudo -u sync360 ssh -q -o StrictHostKeyChecking=no -i /home/sync360/.ssh/id_rsa sync360@%s "df -kh|grep data02"'%(sip))
    if output == '':
        datadir = 'data1'
    else:
        datadir = 'data02'
    retcode,output = subprocess.getstatusoutput('sudo -u sync360 ssh -q -o StrictHostKeyChecking=no -i /home/sync360/.ssh/id_rsa sync360@%s " timeout 3600 /tmp/rdr  show  /%s/redis%s/dump%s.rdb>/dev/null & "'%(sip,datadir,port,port))
    #print(retcode)
    if  retcode != 0:
        print("\033[1;31m分析rdb失败: %s \033[0m"%(output))
        sys.exit(1)
    else:
        #print('*' * 50)
        for i in range(3600):
            time.sleep(1)
            sql = "curl -I -m 10 -o /dev/null -s -w  %{http_code}" + ' http://{0}:8080'.format(sip)
            retcode,output = subprocess.getstatusoutput(sql)
            if  output != "000":
                #print("结果请访问网址  %s:8080 "%(sip))
                #print("\033[5;31m网址将于1小时后自动关闭!!!\033[0m")
                #print('♥️' * 50)
                print("%s:8080/instance/dump%s.rdb"%(sip,port))
                sys.exit(0)
def main():
    port = get_params()
    if int(port) < 5000 or int(port) > 30000:
        print("\033[1;31m!!!请输入准确的端口，不在合理范围内!!!\033[0m")
        sys.exit(1)
    check_env()
    get_big_key(port)



if __name__ == "__main__":
    main()

yaoyinnan · 2023-07-22T11:29:29Z

chejinge在跟进

chejinge · 2023-07-23T03:26:14Z

#include "blackwidow/blackwidow.h"
#include
//#include
#include "src/redis_strings.h"
#include "src/redis_hashes.h"
#include "src/redis_zsets.h"
#include "src/redis_sets.h"
#include "src/redis_lists.h"
#include "src/strings_value_format.h"

using namespace std;

#define PN 10
#define MINLEN 1
#define MAXKEYLEN 10
#define MAXENCODELEN 12

struct maxKey
{
char name[100];//position
unsigned long size;//acceleration
unsigned int type; //top key type; string, list and so on
} max_key[MAXKEYLEN],max_tmp;

struct encodeSize
{
char name[100];//position
long count;
unsigned long size;//acceleration
} encode_size[MAXENCODELEN], tmp_encode;

struct keyPrefix
{
char name[100];//position
long count;//velocity
unsigned long sum;//acceleration
} key_prefix[PN], tmp,all;

int perfix_key_init(){
for(int i = 0;i<PN;i++){
key_prefix[i].count = 0;
key_prefix[i].sum = 0;
}
for(int i = 0;i<MAXKEYLEN;i++){
max_key[i].size = 0;
}
for(int i = 0;i<MAXENCODELEN;i++){
encode_size[i].size = 0;
}
strcpy(all.name, "all");
all.count = 0;
all.sum = 0;
return 0;
}

char *key_type[] = {"string", "hash", "zset", "set", "list", "unknow"};
unsigned long type_size[] = {0,0,0,0,0,0,0,0};
unsigned long type_count[] = {0,0,0,0,0,0,0,0};
#define KRED "\x1B[31m"
#define GRN "\x1B[32m"
#define BLU "\x1B[34m"
#define YEL "\x1B[33m"
#define RESET "\x1B[0m"

int perfix_key_print(){
printf("==========top 10 keys==========\n");
printf(YEL);
for(int i = 0;i<MAXKEYLEN;i++){
if (max_key[i].size > 0){
printf("key名: %s ,类型: %s, 大小(M): %lu, 大小(Byte): %lu\n",max_key[i].name, key_type[max_key[i].type], max_key[i].size/1024/1024, max_key[i].size);
}
}
printf(RESET);

printf("==========all==========\n");
printf(GRN);
printf("key总数: %ld ,总大小(M): %lu, 总大小(Byte): %lu\n",all.count,all.sum/1024/1024,all.sum);
printf(RESET);
printf("==========prefix==========\n");
printf(KRED);
for(int i = 0;i<PN;i++){
    if (key_prefix[i].count > 0){
        printf("前缀: %s* , 个数: %ld , 大小(M): %luM, 大小(Byte): %lu 占%lu %%\n", key_prefix[i].name, key_prefix[i].count, key_prefix[i].sum/1024/1024, key_prefix[i].sum, key_prefix[i].sum * 100 / all.sum );
    }
}
printf(RESET);
printf("\n");
return 0;

}

int perfix_max_len(char *str1, char str2){
int len, max_len = 0;
len = strlen(str1);
if (strlen(str1) > strlen(str2)) len = strlen(str2);
//int i;
for(int i=0; i< len; i++){
if((str1+i) == *(str2+i)){
max_len++;
}else{
break;
}
}
return max_len;
}

int perfix_name_sort(){
//printf("sort\n");
for(int i = 0;i<PN;i++){
for(int j=0;j<PN-1-i;j++){
if (strcmp(key_prefix[j].name, key_prefix[j+1].name) > 0) {
strcpy (tmp.name, key_prefix[i].name);
tmp.count = key_prefix[i].count;
tmp.sum = key_prefix[i].sum;

            strcpy (key_prefix[i].name,key_prefix[j+1].name);
            key_prefix[i].count = key_prefix[j+1].count;
            key_prefix[i].sum = key_prefix[j+1].sum;
            
            strcpy (key_prefix[j+1].name,tmp.name);
            key_prefix[j+1].count = tmp.count;
            key_prefix[j+1].sum = tmp.sum;
        }
    }
}
    return 0;

}

int perfix_add_perfix(int len){
long new_count = 0;
unsigned long new_sum = 0;
int max_len = 0;
int min_len = 0;
perfix_name_sort();
for(int i = 0;i<PN-1;i++){
max_len = perfix_max_len(key_prefix[i].name,key_prefix[PN-1].name);
if (max_len >= len){
new_count = new_count + key_prefix[i].count;
//key_prefix[i].count = 0;
new_sum = new_sum + key_prefix[i].sum;
//key_prefix[i].sum = 0;
min_len = strlen(key_prefix[i].name);
if (min_len == perfix_max_len(key_prefix[i].name,key_prefix[i+1].name)) break;

    }
}
new_count = new_count + key_prefix[PN-1].count;
new_sum = new_sum + key_prefix[PN-1].sum;
strncpy(key_prefix[PN-1].name, key_prefix[PN-1].name, len);
key_prefix[PN-1].name[len] = '\0';
key_prefix[PN-1].count = new_count;
key_prefix[PN-1].sum = new_sum;
return 0;

}

int max_key_sort(){
//printf("sort\n");
for(int i = 0;i+1<MAXKEYLEN;i++){
for(int j=i;j+1<MAXKEYLEN;j++){
if (max_key[j+1].size != 0 && max_key[i].size < max_key[j+1].size){
strcpy (max_tmp.name, max_key[i].name);
max_tmp.size = max_key[i].size;
max_tmp.type = max_key[i].type;

            strcpy (max_key[i].name,max_key[j+1].name);
            max_key[i].size = max_key[j+1].size;
            max_key[i].type = max_key[j+1].type;
            
            strcpy (max_key[j+1].name,max_tmp.name);
            max_key[j+1].size = max_tmp.size;
            max_key[j+1].type = max_tmp.type;
        }
    }
}
    return 0;

}

int perfix_sort(){
//printf("sort\n");
for(int i = 0;i+1<PN;i++){
for(int j=i;j+1<PN;j++){
if (key_prefix[j+1].count != 0 && key_prefix[i].sum < key_prefix[j+1].sum){
strcpy (tmp.name, key_prefix[i].name);
tmp.count = key_prefix[i].count;
tmp.sum = key_prefix[i].sum;

            strcpy (key_prefix[i].name,key_prefix[j+1].name);
            key_prefix[i].count = key_prefix[j+1].count;
            key_prefix[i].sum = key_prefix[j+1].sum;
            
            strcpy (key_prefix[j+1].name,tmp.name);
            key_prefix[j+1].count = tmp.count;
            key_prefix[j+1].sum = tmp.sum;
        }
    }
}
    return 0;

}

int perfix_merge(){
//printf("merge\n");
int max_perfix_len = 0, len, str_len = 0, has_len = 0;
if (key_prefix[PN-1].count == 0) return 0;
for(int i = 0;i+1<PN;i++){
str_len = strlen(key_prefix[i].name);
len = perfix_max_len(key_prefix[i].name,key_prefix[PN-1].name);
if (len >= max_perfix_len){
max_perfix_len = len;
}
if (str_len == len){
if (len > has_len) has_len=len;
}
}
if (max_perfix_len >= MINLEN && max_perfix_len > has_len){
perfix_add_perfix(max_perfix_len);
perfix_sort();
}

key_prefix[PN-1].count = 0;
key_prefix[PN-1].sum = 0;

return 0;

}

int perfix_add_key(char *name, unsigned long size, unsigned int type){
//perfix_key_print();
//printf("name: %s ,size: %lu \n",name,size);
int str_len,len,has_max_len=0;
all.count = all.count + 1;
all.sum = all.sum + size;
if (size > max_key[MAXKEYLEN-1].size){
strcpy(max_key[MAXKEYLEN-1].name,name);
max_key[MAXKEYLEN-1].size = size;
max_key[MAXKEYLEN-1].type = type;
max_key_sort();
}

for(int i = 0;i<PN;i++){
    if (key_prefix[i].count == 0) break;
    str_len = strlen(key_prefix[i].name);
    len = perfix_max_len(key_prefix[i].name,name);
    if (len > has_max_len) has_max_len = len;
    if (str_len == len){
        key_prefix[i].count = key_prefix[i].count + 1;
        key_prefix[i].sum = key_prefix[i].sum + size;
    }
}

if (has_max_len > 0 || size > key_prefix[PN-2].sum) {
    strcpy (key_prefix[PN-1].name, name);
    key_prefix[PN-1].count = 1;
    key_prefix[PN-1].sum = size;
} 

perfix_sort();
//perfix_key_print();
perfix_merge();
//perfix_key_print();
return 0;

}

static void Usage()
{
fprintf(stderr,
"example: pika_keys dump/20230330\n"
"example: pika_keys dump/20230330 --list\n"
);
}

int main(int argc, char *argv[]){
if (argc < 2) {
Usage();
exit(-1);
}

std::string db_path = "";
int display = 0;
if (argc >= 2) {
  db_path = argv[1];
}

if (argc >= 3) {
   display = 1;
}
  rocksdb::Status s;

char key_name[100];
key_name[100] = '\0';
unsigned long key_size;

perfix_key_init();

//cout<< db_path <<endl;
// Init db
rocksdb::Options options;
//options.create_if_missing = true;
options.keep_log_file_num = 10;
options.max_manifest_file_size = 64 * 1024 * 1024;
options.max_log_file_size = 512 * 1024 * 1024;
options.write_buffer_size = 512 * 1024 * 1024; // 512M
options.target_file_size_base = 40 * 1024 * 1024; // 40M

blackwidow::BlackwidowOptions bwOptions;
bwOptions.options = options;
blackwidow::BlackWidow bw;
blackwidow::RedisStrings stringsDB(&bw, blackwidow::kStrings);

//strings
std::string path = db_path + "/./strings";
s = stringsDB.Open(bwOptions, path);
rocksdb::DB* rocksDB = stringsDB.GetDB();
int64_t curtime;
rocksDB->GetEnv()->GetCurrentTime(&curtime).ok();
rocksdb::ReadOptions iterator_options;
auto iter = rocksDB->NewIterator(iterator_options);
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
    blackwidow::ParsedStringsValue parsed_strings_value(iter->value());
    int32_t ttl = -1;
    int64_t ts = (int64_t)(parsed_strings_value.timestamp());
    if (ts != 0) {
        int64_t diff = ts - curtime;
        ttl = diff ;
    }
    strncpy(key_name, iter->key().ToString().c_str(), 99);
    key_size = iter->key().size() + iter->value().size();
    if (ttl < -1) continue;
    if (display == 1) {
        //cout <<  iter->value().size() << endl;
        blackwidow::ParsedStringsValue parsed_strings_value(iter->value());
        cout << "string " << iter->key().ToString().c_str() << " " << parsed_strings_value.value().ToString().c_str() << " " << iter->key().size() + iter->value().size() << " " << ttl << endl;
 printf("[key : %-30s] [value : %-30s] [timestamp : %-10d] [version : %d] [survival_time : %d]\n",
   iter->key().ToString().c_str(),
   parsed_strings_value.value().ToString().c_str(),
   parsed_strings_value.timestamp(),
   parsed_strings_value.version(),
   ttl);
    }else{
        perfix_add_key(key_name, key_size, 0);
    }
}

blackwidow::RedisHashes hashesDB(&bw, blackwidow::kHashes);
path = db_path + "/./hashes";
s = hashesDB.Open(bwOptions, path);

std::string start_key;
std::string next_key;
std::string pattern("*");
int64_t batch_count = 1000;
bool fin = false;
while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = hashesDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

  //cout<< keys.size() <<endl;
  for (auto k : keys) {
      //cout<<"k:" << k <<endl;
      int64_t sum = 0;
      sum = sum + k.size() + 12;
      std::vector<blackwidow::FieldValue> fvs;
      blackwidow::Status s = hashesDB.HGetall(k, &fvs);
      for (auto it = fvs.begin(); it != fvs.end(); it ++) {
          //cout<<"filed:" << it->field.size() <<endl;
          //cout<<"value:" << it->value.size() <<endl;
          sum = sum + 4 + k.size() + 4 + it->field.size();
          sum = sum + it->value.size();
      }
      int64_t ttl = -1;
      hashesDB.TTL(k, &ttl);
      //cout<<"ttl:" << ttl <<endl;
    strncpy(key_name, k.c_str(), 99);
    key_size = sum;
    if (display == 1) {
        cout<< "hash " << k << " " << sum << " " << ttl <<endl;
    }else{
        perfix_add_key(key_name, key_size, 1);
    }
  }
  
  //return 0;
}

blackwidow::RedisZSets zsetsDB(&bw, blackwidow::kZSets);
path = db_path + "/./zsets";
s = zsetsDB.Open(bwOptions, path);

start_key.clear();
next_key.clear();
fin = false;

while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = zsetsDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

    for (auto k : keys) {
    //cout<<"k:" << k <<endl;
    int64_t sum = 0;
    sum = sum + k.size() + 12;
    std::vector<blackwidow::ScoreMember> score_members;
    blackwidow::Status s = zsetsDB.ZRange(k, 0, -1, &score_members);
    auto it = score_members.begin();
    for (auto it = score_members.begin(); it != score_members.end(); it ++) {
          sum = sum + (4 + k.size() + 4 + it->member.size() + 8) * 2;
        //cout<<"score:" << std::to_string(it->score) <<endl;
        //cout<<"member:" << it->member <<endl;
    }

    int64_t ttl = -1;
    s = zsetsDB.TTL(k, &ttl);

    strncpy(key_name, k.c_str(), 99);
    key_size = sum;
    if (display == 1) {
        cout<< "zset " << k << " " << sum << " " << ttl <<endl;
    }else{
        perfix_add_key(key_name, key_size, 2);
    }
  }
}

blackwidow::RedisSets setsDB(&bw, blackwidow::kSets);
path = db_path + "/./sets";
s = setsDB.Open(bwOptions, path);

start_key.clear();
next_key.clear();
fin = false;

while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = setsDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

    for (auto k : keys) {
        //cout<<"k:" << k <<endl;
        int64_t sum = 0;
        sum = sum + k.size() + 12;
        std::vector<std::string> members;
        blackwidow::Status s = setsDB.SMembers(k, &members);
        for (auto it = members.begin(); it != members.end(); it ++) {
            //cout<<"member:" << *it <<endl;
            sum = sum + 4 + k.size() + 4 + (*it).size();
        }

        int64_t ttl = -1;
        s = setsDB.TTL(k, &ttl);

        strncpy(key_name, k.c_str(), 99);
        key_size = sum;
        if (display == 1) {
            cout<< "set " << k << " " << sum << " " << ttl <<endl;
        }else{
            perfix_add_key(key_name, key_size, 3);
        }
   }

}

blackwidow::RedisLists listsDB(&bw, blackwidow::kLists);
path = db_path + "/./lists";
s = listsDB.Open(bwOptions, path);

start_key.clear();
next_key.clear();
fin = false;

while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = listsDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

  for (auto k : keys) {
     //cout<<"k:" << k <<endl;
     int64_t sum = 0;
     sum = sum + k.size() + 12 + 16;
     int64_t pos = 0;
     std::vector<std::string> list;
     blackwidow::Status s = listsDB.LRange(k, pos, pos + batch_count - 1, &list);
     while (s.ok() && !list.empty()) {

       for (auto e : list) {
           //cout<<"e:" << e <<endl;
           sum = sum + 4 + k.size() + 4 + 8 + e.size();
       }

       pos += batch_count;
       list.clear();
       s = listsDB.LRange(k, pos, pos + batch_count - 1, &list);
     }

     int64_t ttl = -1;
     s = listsDB.TTL(k, &ttl);

     strncpy(key_name, k.c_str(), 99);
     key_size = sum;
     if (display == 1) {
         cout<< "list " << k << " " << sum << " " << ttl <<endl;
     }else{
         perfix_add_key(key_name, key_size, 4);
     }
  }
}

if (display == 0) {
    perfix_key_print();
}

}

AlexStocks · 2023-09-30T07:42:55Z

1 big key/hot key monitor;
2 compress big kv in proxy;
3 topK;

sjcsjc123 · 2023-12-01T13:12:39Z

Could i have a try?

AlexStocks · 2023-12-01T13:17:26Z

Could i have a try?

I have assign this task to you.

sjcsjc123 · 2023-12-05T15:34:26Z

#include "blackwidow/blackwidow.h" #include //#include #include "src/redis_strings.h" #include "src/redis_hashes.h" #include "src/redis_zsets.h" #include "src/redis_sets.h" #include "src/redis_lists.h" #include "src/strings_value_format.h"

using namespace std;

#define PN 10 #define MINLEN 1 #define MAXKEYLEN 10 #define MAXENCODELEN 12

struct maxKey { char name[100];//position unsigned long size;//acceleration unsigned int type; //top key type; string, list and so on } max_key[MAXKEYLEN],max_tmp;

struct encodeSize { char name[100];//position long count; unsigned long size;//acceleration } encode_size[MAXENCODELEN], tmp_encode;

struct keyPrefix { char name[100];//position long count;//velocity unsigned long sum;//acceleration } key_prefix[PN], tmp,all;

int perfix_key_init(){ for(int i = 0;i<PN;i++){ key_prefix[i].count = 0; key_prefix[i].sum = 0; } for(int i = 0;i<MAXKEYLEN;i++){ max_key[i].size = 0; } for(int i = 0;i<MAXENCODELEN;i++){ encode_size[i].size = 0; } strcpy(all.name, "all"); all.count = 0; all.sum = 0; return 0; }

char *key_type[] = {"string", "hash", "zset", "set", "list", "unknow"}; unsigned long type_size[] = {0,0,0,0,0,0,0,0}; unsigned long type_count[] = {0,0,0,0,0,0,0,0}; #define KRED "\x1B[31m" #define GRN "\x1B[32m" #define BLU "\x1B[34m" #define YEL "\x1B[33m" #define RESET "\x1B[0m"

int perfix_key_print(){ printf("==========top 10 keys==========\n"); printf(YEL); for(int i = 0;i<MAXKEYLEN;i++){ if (max_key[i].size > 0){ printf("key名: %s ,类型: %s, 大小(M): %lu, 大小(Byte): %lu\n",max_key[i].name, key_type[max_key[i].type], max_key[i].size/1024/1024, max_key[i].size); } } printf(RESET);
printf("==========all==========\n");
printf(GRN);
printf("key总数: %ld ,总大小(M): %lu, 总大小(Byte): %lu\n",all.count,all.sum/1024/1024,all.sum);
printf(RESET);
printf("==========prefix==========\n");
printf(KRED);
for(int i = 0;i<PN;i++){
    if (key_prefix[i].count > 0){
        printf("前缀: %s* , 个数: %ld , 大小(M): %luM, 大小(Byte): %lu 占%lu %%\n", key_prefix[i].name, key_prefix[i].count, key_prefix[i].sum/1024/1024, key_prefix[i].sum, key_prefix[i].sum * 100 / all.sum );
    }
}
printf(RESET);
printf("\n");
return 0;
}

int perfix_max_len(char *str1, char str2){ int len, max_len = 0; len = strlen(str1); if (strlen(str1) > strlen(str2)) len = strlen(str2); //int i; for(int i=0; i< len; i++){ if((str1+i) == *(str2+i)){ max_len++; }else{ break; } } return max_len; }

int perfix_name_sort(){ //printf("sort\n"); for(int i = 0;i<PN;i++){ for(int j=0;j<PN-1-i;j++){ if (strcmp(key_prefix[j].name, key_prefix[j+1].name) > 0) { strcpy (tmp.name, key_prefix[i].name); tmp.count = key_prefix[i].count; tmp.sum = key_prefix[i].sum;
            strcpy (key_prefix[i].name,key_prefix[j+1].name);
            key_prefix[i].count = key_prefix[j+1].count;
            key_prefix[i].sum = key_prefix[j+1].sum;
            
            strcpy (key_prefix[j+1].name,tmp.name);
            key_prefix[j+1].count = tmp.count;
            key_prefix[j+1].sum = tmp.sum;
        }
    }
}
    return 0;
}

int perfix_add_perfix(int len){ long new_count = 0; unsigned long new_sum = 0; int max_len = 0; int min_len = 0; perfix_name_sort(); for(int i = 0;i<PN-1;i++){ max_len = perfix_max_len(key_prefix[i].name,key_prefix[PN-1].name); if (max_len >= len){ new_count = new_count + key_prefix[i].count; //key_prefix[i].count = 0; new_sum = new_sum + key_prefix[i].sum; //key_prefix[i].sum = 0; min_len = strlen(key_prefix[i].name); if (min_len == perfix_max_len(key_prefix[i].name,key_prefix[i+1].name)) break;
    }
}
new_count = new_count + key_prefix[PN-1].count;
new_sum = new_sum + key_prefix[PN-1].sum;
strncpy(key_prefix[PN-1].name, key_prefix[PN-1].name, len);
key_prefix[PN-1].name[len] = '\0';
key_prefix[PN-1].count = new_count;
key_prefix[PN-1].sum = new_sum;
return 0;
}

int max_key_sort(){ //printf("sort\n"); for(int i = 0;i+1<MAXKEYLEN;i++){ for(int j=i;j+1<MAXKEYLEN;j++){ if (max_key[j+1].size != 0 && max_key[i].size < max_key[j+1].size){ strcpy (max_tmp.name, max_key[i].name); max_tmp.size = max_key[i].size; max_tmp.type = max_key[i].type;
            strcpy (max_key[i].name,max_key[j+1].name);
            max_key[i].size = max_key[j+1].size;
            max_key[i].type = max_key[j+1].type;
            
            strcpy (max_key[j+1].name,max_tmp.name);
            max_key[j+1].size = max_tmp.size;
            max_key[j+1].type = max_tmp.type;
        }
    }
}
    return 0;
}

int perfix_sort(){ //printf("sort\n"); for(int i = 0;i+1<PN;i++){ for(int j=i;j+1<PN;j++){ if (key_prefix[j+1].count != 0 && key_prefix[i].sum < key_prefix[j+1].sum){ strcpy (tmp.name, key_prefix[i].name); tmp.count = key_prefix[i].count; tmp.sum = key_prefix[i].sum;
            strcpy (key_prefix[i].name,key_prefix[j+1].name);
            key_prefix[i].count = key_prefix[j+1].count;
            key_prefix[i].sum = key_prefix[j+1].sum;
            
            strcpy (key_prefix[j+1].name,tmp.name);
            key_prefix[j+1].count = tmp.count;
            key_prefix[j+1].sum = tmp.sum;
        }
    }
}
    return 0;
}

int perfix_merge(){ //printf("merge\n"); int max_perfix_len = 0, len, str_len = 0, has_len = 0; if (key_prefix[PN-1].count == 0) return 0; for(int i = 0;i+1<PN;i++){ str_len = strlen(key_prefix[i].name); len = perfix_max_len(key_prefix[i].name,key_prefix[PN-1].name); if (len >= max_perfix_len){ max_perfix_len = len; } if (str_len == len){ if (len > has_len) has_len=len; } } if (max_perfix_len >= MINLEN && max_perfix_len > has_len){ perfix_add_perfix(max_perfix_len); perfix_sort(); }
key_prefix[PN-1].count = 0;
key_prefix[PN-1].sum = 0;

return 0;
}

int perfix_add_key(char *name, unsigned long size, unsigned int type){ //perfix_key_print(); //printf("name: %s ,size: %lu \n",name,size); int str_len,len,has_max_len=0; all.count = all.count + 1; all.sum = all.sum + size; if (size > max_key[MAXKEYLEN-1].size){ strcpy(max_key[MAXKEYLEN-1].name,name); max_key[MAXKEYLEN-1].size = size; max_key[MAXKEYLEN-1].type = type; max_key_sort(); }
for(int i = 0;i<PN;i++){
    if (key_prefix[i].count == 0) break;
    str_len = strlen(key_prefix[i].name);
    len = perfix_max_len(key_prefix[i].name,name);
    if (len > has_max_len) has_max_len = len;
    if (str_len == len){
        key_prefix[i].count = key_prefix[i].count + 1;
        key_prefix[i].sum = key_prefix[i].sum + size;
    }
}

if (has_max_len > 0 || size > key_prefix[PN-2].sum) {
    strcpy (key_prefix[PN-1].name, name);
    key_prefix[PN-1].count = 1;
    key_prefix[PN-1].sum = size;
} 

perfix_sort();
//perfix_key_print();
perfix_merge();
//perfix_key_print();
return 0;
}

static void Usage() { fprintf(stderr, "example: pika_keys dump/20230330\n" "example: pika_keys dump/20230330 --list\n" ); }

int main(int argc, char *argv[]){ if (argc < 2) { Usage(); exit(-1); }
std::string db_path = "";
int display = 0;
if (argc >= 2) {
  db_path = argv[1];
}

if (argc >= 3) {
   display = 1;
}
  rocksdb::Status s;

char key_name[100];
key_name[100] = '\0';
unsigned long key_size;

perfix_key_init();

//cout<< db_path <<endl;
// Init db
rocksdb::Options options;
//options.create_if_missing = true;
options.keep_log_file_num = 10;
options.max_manifest_file_size = 64 * 1024 * 1024;
options.max_log_file_size = 512 * 1024 * 1024;
options.write_buffer_size = 512 * 1024 * 1024; // 512M
options.target_file_size_base = 40 * 1024 * 1024; // 40M

blackwidow::BlackwidowOptions bwOptions;
bwOptions.options = options;
blackwidow::BlackWidow bw;
blackwidow::RedisStrings stringsDB(&bw, blackwidow::kStrings);

//strings
std::string path = db_path + "/./strings";
s = stringsDB.Open(bwOptions, path);
rocksdb::DB* rocksDB = stringsDB.GetDB();
int64_t curtime;
rocksDB->GetEnv()->GetCurrentTime(&curtime).ok();
rocksdb::ReadOptions iterator_options;
auto iter = rocksDB->NewIterator(iterator_options);
for (iter->SeekToFirst(); iter->Valid(); iter->Next()) {
    blackwidow::ParsedStringsValue parsed_strings_value(iter->value());
    int32_t ttl = -1;
    int64_t ts = (int64_t)(parsed_strings_value.timestamp());
    if (ts != 0) {
        int64_t diff = ts - curtime;
        ttl = diff ;
    }
    strncpy(key_name, iter->key().ToString().c_str(), 99);
    key_size = iter->key().size() + iter->value().size();
    if (ttl < -1) continue;
    if (display == 1) {
        //cout <<  iter->value().size() << endl;
        blackwidow::ParsedStringsValue parsed_strings_value(iter->value());
        cout << "string " << iter->key().ToString().c_str() << " " << parsed_strings_value.value().ToString().c_str() << " " << iter->key().size() + iter->value().size() << " " << ttl << endl;
 printf("[key : %-30s] [value : %-30s] [timestamp : %-10d] [version : %d] [survival_time : %d]\n",
   iter->key().ToString().c_str(),
   parsed_strings_value.value().ToString().c_str(),
   parsed_strings_value.timestamp(),
   parsed_strings_value.version(),
   ttl);
    }else{
        perfix_add_key(key_name, key_size, 0);
    }
}

blackwidow::RedisHashes hashesDB(&bw, blackwidow::kHashes);
path = db_path + "/./hashes";
s = hashesDB.Open(bwOptions, path);

std::string start_key;
std::string next_key;
std::string pattern("*");
int64_t batch_count = 1000;
bool fin = false;
while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = hashesDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

  //cout<< keys.size() <<endl;
  for (auto k : keys) {
      //cout<<"k:" << k <<endl;
      int64_t sum = 0;
      sum = sum + k.size() + 12;
      std::vector<blackwidow::FieldValue> fvs;
      blackwidow::Status s = hashesDB.HGetall(k, &fvs);
      for (auto it = fvs.begin(); it != fvs.end(); it ++) {
          //cout<<"filed:" << it->field.size() <<endl;
          //cout<<"value:" << it->value.size() <<endl;
          sum = sum + 4 + k.size() + 4 + it->field.size();
          sum = sum + it->value.size();
      }
      int64_t ttl = -1;
      hashesDB.TTL(k, &ttl);
      //cout<<"ttl:" << ttl <<endl;
    strncpy(key_name, k.c_str(), 99);
    key_size = sum;
    if (display == 1) {
        cout<< "hash " << k << " " << sum << " " << ttl <<endl;
    }else{
        perfix_add_key(key_name, key_size, 1);
    }
  }
  
  //return 0;
}

blackwidow::RedisZSets zsetsDB(&bw, blackwidow::kZSets);
path = db_path + "/./zsets";
s = zsetsDB.Open(bwOptions, path);

start_key.clear();
next_key.clear();
fin = false;

while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = zsetsDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

    for (auto k : keys) {
    //cout<<"k:" << k <<endl;
    int64_t sum = 0;
    sum = sum + k.size() + 12;
    std::vector<blackwidow::ScoreMember> score_members;
    blackwidow::Status s = zsetsDB.ZRange(k, 0, -1, &score_members);
    auto it = score_members.begin();
    for (auto it = score_members.begin(); it != score_members.end(); it ++) {
          sum = sum + (4 + k.size() + 4 + it->member.size() + 8) * 2;
        //cout<<"score:" << std::to_string(it->score) <<endl;
        //cout<<"member:" << it->member <<endl;
    }

    int64_t ttl = -1;
    s = zsetsDB.TTL(k, &ttl);

    strncpy(key_name, k.c_str(), 99);
    key_size = sum;
    if (display == 1) {
        cout<< "zset " << k << " " << sum << " " << ttl <<endl;
    }else{
        perfix_add_key(key_name, key_size, 2);
    }
  }
}

blackwidow::RedisSets setsDB(&bw, blackwidow::kSets);
path = db_path + "/./sets";
s = setsDB.Open(bwOptions, path);

start_key.clear();
next_key.clear();
fin = false;

while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = setsDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

    for (auto k : keys) {
        //cout<<"k:" << k <<endl;
        int64_t sum = 0;
        sum = sum + k.size() + 12;
        std::vector<std::string> members;
        blackwidow::Status s = setsDB.SMembers(k, &members);
        for (auto it = members.begin(); it != members.end(); it ++) {
            //cout<<"member:" << *it <<endl;
            sum = sum + 4 + k.size() + 4 + (*it).size();
        }

        int64_t ttl = -1;
        s = setsDB.TTL(k, &ttl);

        strncpy(key_name, k.c_str(), 99);
        key_size = sum;
        if (display == 1) {
            cout<< "set " << k << " " << sum << " " << ttl <<endl;
        }else{
            perfix_add_key(key_name, key_size, 3);
        }
   }
}

blackwidow::RedisLists listsDB(&bw, blackwidow::kLists); path = db_path + "/./lists"; s = listsDB.Open(bwOptions, path);
start_key.clear();
next_key.clear();
fin = false;

while (!fin) {
  int64_t count = batch_count;
  std::vector<std::string> keys;
  fin = listsDB.Scan(start_key, pattern, &keys, &count, &next_key);
  start_key = next_key;

  for (auto k : keys) {
     //cout<<"k:" << k <<endl;
     int64_t sum = 0;
     sum = sum + k.size() + 12 + 16;
     int64_t pos = 0;
     std::vector<std::string> list;
     blackwidow::Status s = listsDB.LRange(k, pos, pos + batch_count - 1, &list);
     while (s.ok() && !list.empty()) {

       for (auto e : list) {
           //cout<<"e:" << e <<endl;
           sum = sum + 4 + k.size() + 4 + 8 + e.size();
       }

       pos += batch_count;
       list.clear();
       s = listsDB.LRange(k, pos, pos + batch_count - 1, &list);
     }

     int64_t ttl = -1;
     s = listsDB.TTL(k, &ttl);

     strncpy(key_name, k.c_str(), 99);
     key_size = sum;
     if (display == 1) {
         cout<< "list " << k << " " << sum << " " << ttl <<endl;
     }else{
         perfix_add_key(key_name, key_size, 4);
     }
  }
}

if (display == 0) {
    perfix_key_print();
}
}

Can you provide an origin? This code looks a bit messy.

AlexStocks · 2023-12-16T09:55:56Z

下一步工作：

sjcsjc123 · 2023-12-21T14:27:11Z

这个熔断tool好像做不了吧

chejinge changed the title ~~完善 big key 分析功能~~ 完善 big key 及top key分析功能 Jul 17, 2023

yaoyinnan added tool enhancement labels Jul 27, 2023

OpenAtomFoundation deleted a comment from Mixficsol Nov 1, 2023

AlexStocks changed the title ~~完善 big key 及top key分析功能~~ Add: big key & top key Dec 1, 2023

AlexStocks added the 3.5.3 label Dec 1, 2023

AlexStocks assigned sjcsjc123 Dec 1, 2023

AlexStocks mentioned this issue Dec 29, 2023

pika引擎支持 hotKey，bigKey查询么？ #2228

Closed

AlexStocks closed this as completed Feb 8, 2024

This was referenced May 8, 2024

feat:Add codis-proxy bigKey/bigRequest fuse protection #2641

Closed

feat:codis-proxy bigKey/Request fuse protection #2646

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add: big key & top key #1691

Add: big key & top key #1691

chejinge commented Jul 7, 2023

chejinge commented Jul 15, 2023

chejinge commented Jul 18, 2023 •

edited

yaoyinnan commented Jul 22, 2023

chejinge commented Jul 23, 2023

AlexStocks commented Sep 30, 2023

sjcsjc123 commented Dec 1, 2023

AlexStocks commented Dec 1, 2023

sjcsjc123 commented Dec 5, 2023

AlexStocks commented Dec 16, 2023

sjcsjc123 commented Dec 21, 2023

Add: big key & top key #1691

Add: big key & top key #1691

Comments

chejinge commented Jul 7, 2023

chejinge commented Jul 15, 2023

chejinge commented Jul 18, 2023 • edited

yaoyinnan commented Jul 22, 2023

chejinge commented Jul 23, 2023

AlexStocks commented Sep 30, 2023

sjcsjc123 commented Dec 1, 2023

AlexStocks commented Dec 1, 2023

sjcsjc123 commented Dec 5, 2023

AlexStocks commented Dec 16, 2023

sjcsjc123 commented Dec 21, 2023

chejinge commented Jul 18, 2023 •

edited