From b959e3068e8211228b3a39ef9edbb9df2930d341 Mon Sep 17 00:00:00 2001 From: cosven Date: Tue, 14 Jul 2020 13:14:16 +0800 Subject: [PATCH 1/2] cherry pick #3084 to release-4.0 Signed-off-by: ti-srebot --- TOC.md | 1 + media/tikv-rocksdb.png | Bin 0 -> 27160 bytes storage-engine/rocksdb-overview.md | 61 +++++++++++++++++++++++++++++ 3 files changed, 62 insertions(+) create mode 100644 media/tikv-rocksdb.png create mode 100644 storage-engine/rocksdb-overview.md diff --git a/TOC.md b/TOC.md index 4ff1fccb03f52..3c06cdb969db5 100644 --- a/TOC.md +++ b/TOC.md @@ -417,6 +417,7 @@ + Storage Engines + TiKV + [TiKV Overview](/tikv-overview.md) + + [RocksDB Overview](/storage-engine/rocksdb-overview.md) + TiFlash + [Overview](/tiflash/tiflash-overview.md) + [Use TiFlash](/tiflash/use-tiflash.md) diff --git a/media/tikv-rocksdb.png b/media/tikv-rocksdb.png new file mode 100644 index 0000000000000000000000000000000000000000..639e38b1d391c95bf584bd8ec556849b6dd18678 GIT binary patch literal 27160 zcmeFYWmH{3(=K>$0>Le~2Zs3;LCg}p?MQ37k(@B_^-b&7{`gDu<6{bGhd8+4vw7O!dMeV6)yWE2U)-|8>frk37tMsRz#TCEgCR!YA>+oE3lU_j$x_=kAZ>l zeH^-5yb-!i7p{gUskVUz2J~sAm*s}=3Yau{_A0=Lp#if{(4`wz^H7m@# zW?D&;Q_LE3ZW-WJjC?iuSarEr_3>~eh1ufYqS#nX0srZSUpx#@B)fP#WW ziZj!22_pMBU;gVO+3>1?3x8AY8KZvnKSd9Af>r04AsY$2#~-%?3*0^tX;kPQ?C*on zF)=ZLmBz;}RxYNJva+|2jffy{f+8Qbuyt=aMNvXTp9P+bk$b3}HciJRB|rZeXJzxd zeI2*1TBtUi1qV{}kT0vG|9|u(@vbb$fw&)yFOX5Ee9k=E*X`lzfFSxk6&00$fZt+` z+2!tNX;Bd^3(M>3Z97-pVb^Y)HRDZfJuyo84JskofDu8j9AKzNqGI)AAbI@xbYjv> zE3HqWvgzb*6~F35U;GW&5Asy4s8x~3-^(WQa8N0izWe!KYfKt}ko}1$W@u<=d3hPI z+J}b+!RG@7QlCG<^71hvIpI-JL>I=T_6so?_A*h*l^;U-L+E&qFPe{tqt(OeyXi_2 zKCZhA;mj}TEM?*y^4Q5wgjit*f$*z%w;NFR^lC*p=yP&<{kB`xayrXv#e^sA@w@*S zIt``wVxE5G{DI4xH*Z{BU9){}QWtI74+_%ba!4Ljs@UOCRY$GoR}q!b~FW_nB%N*rG3LNj9bpX{e|=k$~tClZf?3z zp;Wa5&`3l?L}kaz!{gekpNh&fO}3A+p5Dsp>gvplX}xx~_h~boN&#Y!!}G)SU&SRe z>mi|`B~w{W=8C`nVY$`Ue+0pfm&^L`bJB`0nGsQu0jzgUwzWNbv+!XjgD&lT>7{b8 zt$G;`qu9P{ZoZb&oTd$%2|}}I$Se%tOcb8@oi|!Mh3OE7vNX_}yzof8kE?E zo7jjP3G`8M?eMO0oRI*Bo!%%--gj74Y4Oz(`9Cfx5p5?JL<=ii93Vnog!6f(c$fO? zex7RE?hiB1y|GMSuMk8_VD|!YCrVQf*K}{h0WQcaTEcl+j*CgRX=1~{^vjo*Q}>SH z!NKD4a(@4pCt~+yR^SpBhu)_DIw6Wi%;%w)l!+v~;WC6;czcN=uf>_R5E$1Eom-6DrvNiD>@+y28m!S1 zRO_$|%*t{576WaC#%ijPaYyF8M)^+46aU{>V9C_o|(PT zDTsKZM4w1QkKg3et*!;hA}S;i`7q@F*yv938_8*l_mu^o*pk^_zZNabIhrPox*iPbpCu{#e1qtG%@I zWiUq>@v<wGE>p1x6_)W-neEeyKf9HwzbFb@ zolryiF=fEV+*5Kp|$v)3jlf6i(~2SrN{q)=B#MeYlXZ6P{+BC22|UvK8< zcD?;VHpc1qrd z7%;Jalfm#QyelL5E8-@Bfihl63PFAvNnRF7c~*%^GrUrYjmIq2Ur$(}b*wScfT)Cl zRWK7uuZHyY1MYQC-+x{us5GVnL!Ek<+FKkwTACl~0O zgfiSn@e5xoB|Mtn%*}fkt}PCreIN0egPs@4`WYR*NDOf>)glyVgD3OTf)#aSJ{-u! zk!Q{1Bm1IemWKPz^vLjVixlYDTSFgrQ5T=O@z*+?+w@=4a22F% z{wk;1pZ6G+uDu@_#D~^a7wR*@=U27P?q3S2gL`RDM%@%z&tu=O32INOWT}Z>EQ| zwv)#pB}q#5OG;WhmmUWztTc~aDqJsU7vy=nzf8U>dguKk=E=`|(o&FK5^#U+(C)jx zJA$3IdI^f^D$ah0kGE!NOpMF7Je@@2r(diH{yJ$gd^}$htnGf@V!0(iSPh*`KE0c` z)>9vIYO4&?j@j-&0Udx8pZfaM+Q&XoEpb=8M&pu}AcTKAedXmP`BlQjX-37osHjXD zu>KG?mS}@rCqoLcx}nwv9^VK(+m+d5dKr_P9AVr@AW|~q?`F;o`epCQP1F3y8&B=% zN3kv-AU`MhSQ9~#-#w(q@yL0k*14j^y^w|mx)<+vv(Z^TqkFRz=Dngv6uZZGe%~WL z8+PAYQ4ez<KQ3m$p3YqQG4Y0fn~`#OO!|Bq6e}gFfO307Us5#CV+#15$zpV zLgI4C);KbLd>+4|d0Ij8(J5=y=1 zhxH?moRy!oU1n!CTjw8vH-E>r8K7!Cv_D>iA2)LD`=0tAWH6X}etEbkW1L6p7-6cu zN}bWKzXV%z*=d8XuQ9hK+wiS~33FykLe&Jlx)-<|+Z@jrbl6~pC#>o*w9D1er?__g zkFPKKOpAL}uP&p5thu$zf8x%)uy?fXg_SOKIM2FsRg5lay2uB$S?v`><2!RUt1gtCZq=tZBd{KH7tx3e4lzXmy-kO1M5<8;-%_v|Fxo z+ZvC|*H^lpy5&0c+=+hPk@V|OQv?eh&&@~jRzCe<0W^UG>p8PP&*uxhh@;MOrV|Z! z+PGG-e~qM$+q6sT%}~g2RfyciB#~s+{AG=EJPO<SI2j?w4@*K(&L`G0^x_&Hz|J3zVz%sEMM@zq8oc_`# z2Kn*gLC0e5Cb*4r>F1=j!s*S;oSk)Lk(ccxl;2jHt+Opw$M5$PxgdNQUo;rRu!Y)r z^5InGDr*i*gU#ySY(9PSmJ$(ceeJGW?-?x znHPINoq&v%MqDw%q0Gl*z70=0>;BbXGXs+4wiOk7B1p@<-WiwX7s19-hqXYxaaZ;C zI}BNeek|6FrE+SM+#O5xdDcT^dfskl%tJDGP@PUL=0gPI#p!h!LB9Eahj|9VMOr_j z-X)D8Fz$|Mow07;slmxo1M|xB=8!F%{R#ckD1EhtYP%y#K4flqwN|+Dg6*&JGbR~S zzfRx&xjTjNZg4^K9&RUKiQL~^cRux1%#I1HfO#O691)PTXL86!5!U}^&I}11OL6mA zbXLkjOV5HQdo-+9OER-|71zpFOWFy}l1DCEcFchk6w#e8+Z_tboUY!k>wnIsbLQdt z(o1p+f66md>0;lR(T3Lor?GXAdf?ltuQIjRKQYU&Ird2ciP)YJ&r>vDU zD#_>ltglnBN8_ZS=O=d5rua_#dcxiHB~(sKERdaKOqb~Bf%EZzDQm^`KB6c2A&X_7 zYHZvkn_RG=z6o;erIuwxQsY-ozPNWpJ-p_geKhc(7V9lHO?uOF`4ZD!_*s{{vJKp? zs5;er@iLyi?{sKobvRNZr!NS8v`jrd27Buf(o2k=`|KTsIg}PNy88aOZj&Vjc{lnC zU7X11o#7HQCx3l+@>z4y+tu%{R8SmQBw=xv>WioS(@bk}QTjDFpt$twGnt{RGy6Jt z-<#Ep7!Md#SXTA$EkKxZ1Job)^e)OUqC9ek+S9gOQ7BGeuMpzh1WKrpd zM#EV(#IXf16HKd44&Gf6ctcPL9aWq8_Nk3bw*rR2MrQIn7sG*@=E$X4@(SuudwejQ zWy_h|EAwhH{YVCfPa6v|gj}B4Wp~BUTWC{{Q9+5{7uGmY$5?AjpG>|hV9jZIkKu(I zJ|t!poN1N4<@jp{0glT2WsC#fbw^bKm(9OzS$;L$>^G+pCe08JTzMUZ*h!skX+ClW zBu6Oh*vVtXK27>+`$DEmea@^{79V@ov7kD~a=V+`WKvxRTV5I*9pH$02#^UTn#>u1S{+{Ji4lUWQZO zZr%1p>sQg;0zHq%Hz0*pjTUaE`afFpo?kOi+#k<1%eedwUK*;V()e|b`LA-M>0c(N z)H)au;vS_pl6JWHZ}fc;z*i&ss`{alC~fm@WfuJ;Cb{H`Ra6YlenL0F-@d+`r-gWy z)+TwJG#vOP^~72r;MdgtYW?!ux_w%y>bv;*A=9^o=LtDIJRI7RPnY8j%cG&=>O({$ zPZ*Jhare=x{ndQFS+jk~(``JZq>LLhXsU7VozE(ZhOg#+dyUb~DdT*7U`f_1*%v?i zTJJ+NE?hLXRlS*`l{VUrku&c=gM(MC$0-))?1j3V4%UXS0mz@(WV={uMuQ zVTI8-PZGZp*v8tykKgVltzPQLa`^c``~P*K0R%L&!CYVaFZn5z^X`hHxY`fF3Q($p zgnm2pAN93cdiZTF=faZjS}|oNhb!V?bV$x_O~XF$APgScFX9SVZoh_}(Ceh6m1uph`>oTAHn`bz3 z%v;0Sbw-AD^d;2yISt)zoIQE{)b{=mtsoTbZ)+@wiD7^0Dkd3% zFC(epxuvE-C?#nToB||)woeqE|M2~fQQXies@?7J9SIAO>6D%8NRlsy9$N1{%|%46 zozZQLlScNQ?%QUsrmFnC8h(2%JQm#zyb!czlxy(eOy~XyD?baWmt8OdCH!MR|G1ynW z2^u%fV1~o(42V|-*Ez@1!6J@0f0pr1gvlTq!qCn3_0q{+tIvehH8xk4hq$km`nbkd ztiKB$PB!dRWs)-U%S%PB4WAw(ms6<{55{>u0653Fh3C@VEHdcG77#>gseTLK^lH)P zT33S+5VsDal8PrbZdx#b=4HHo?;NRB6ZviWhwHOONc$fKz%LPK{fn0d1h?+8fZ&k= z3F%*$^*@m2|Nrt|nEU_7nqXIr?=}7o>SWeB%->)fRJQ+qOo@dZEiQ|b`zCA_2Ic$t zZR+GC941-jSZvJvZuW(9r6K*`^Q8Wp096uV(zYwEz5p>P6Oc-%#ikM*FNmPAN9>m+ zRQM$)5u>#1Or+na-u3xdGOFzW3|WCj0`^=F4LsFn0^}5WF)Y;5u)$F~Zl@wyyx92I z;XM0c?T+ODudP)|TLgPiE)eos};(%Z|NK_|smK2ttm; z-07uuNb!O(+EPA)U4H@ZrWTpFy_rn2+HCG_DOFR<1ZXde&j_pyF@eR zE*{W;mzNAD=x3)zA_KDC@bbkn?!1#*cAYi@^0Z#0EJWR?VgP!|U+w*{(_&=}l%d&J zlIsCzDw}tKa-i<`m>o2*sDrVM!gc>aExd?V9qmWjtR{%;QNyeTz}a_+K_o0yEvu!x zDX2D!8k7)vCs5h48s%^eaC> znX^UD8LFvZ$B#`)P*=n6U>rlx?N7s`0%1-p)-ug%Kpix{9t+H{mkkfAX&+LyaXR)0tBj$o?Kt&TctkB z%(}k(!h}jeZTp^b4-5KbwpTtrAfnXaTv{h&T$#Y$km3z{pp6Xo=0D>pqG>TS9(zpr z<}7}9zZ{*U0AbJy=q9p_gFpo(6lBv!{I?}yL<}Dq$`sHL0w^m@8J{o0I&3|mEqSfl zmvc3s+_sj?ox!7qC$-_mTjF%SSx3+fCUPIpj@?ZX z_IX8~E51yb57%n2D2KuHk8oV$Mh>RDRu*T5{V;BV>gU`&V@ak0p+h;k?Q$9hZ+wea zc|UzX+038;rBysF?y>lMwURtRYYZ%46-(<+Lnl2mb%!ZuqtsE|CVns*SHlV253}2A zYrQEaRCOIQzyN+kC&H;~x#|6Gc>DzfEY6;RdnMkcwk+%r)5}4-O1?%0d*Z5!PWoXd zy<%%QaC=MBsIzZ|u115cStE{`Hh_s+qq;D6^m|!iv;H+&;UqJEWYn~EDybTn-2HZz zH9Or>`QEIE-;yZGkyKyN1ZHAqD1@majK?*2@D?&xh8Nk2#e=#0^oLf9JV-6XQJv# zZ7BfBYPtEsO{5y?*UA*DwzQY1($CNT^n0KCEy`c-hrG8pc-AVh#qzA6yL&Er#jX)@ zk{Rt(IHs<%|He1#X;sVglUEJogdVM4yL{3@*MK)P-&UR+a z`II@9l$10#r#`Qd+|a-|uTeU?G&fhETwJGJUYc{f%m7YPl2fRdo11-sKm9PKmgcgS z>~d<@a^}k8E?%(`Kl)|bsy=5W#52U37MqWi>o-7OMM_$8*^&T))x(^!bcu+F@F4e$ z!KxIiLOqx@OaEvuo=w%O*eqbH*9DOOwbsKdZGa3!pTvx&EFFIg~dbVEXF!OD*M>a?_BaVsF2gmLGhs)8fxC#I_0N~CTB;QxW#KoIYjfw>`-^?v6 zETkHaj*bGv+Q!BPfMi$bS+2n%qoAO;9aH{nYinZzx8O--CjJn~*&kt;p9iSNd5tXQ z9I2q$?v(dZ@ryQ`>7z+@fDGN;-FoF}G#Wa%09A4v*B=M4W@G#YKfx*x5Cc(w|xx_b=^%ovX+&N zQZGeu8xW(6n(cl!wXmRRV6e9CxYZYZb90lD@@~FeZXN|KGfXsTr>{?9-fpUX-sMDK z*2-1ZUmA|{zj6U&Wo@`7Cnvj5-Kez6=KxyslpC1)J1neeuKxED2x^M#92~9%>gHy^ za`LD!F#l`x^z;kgF;S39!+ut4m(N!kR31s>P4@KmHZLUuiUUx2U{@H1aeu3;t80qj zoPVirZ`Y6i)zs8<_pbp|pls?#GoR}b1Aui%G^wbJr|RwRcOS2QqfQd7RWajKTv=IJ zT3QP^S-G?tN}y{^3hKjCN6V9pr&8gPlam8T%}iTKAD<_xV{X!|B?yGqy{-OV0~QOo z>D2}>rZn&0$JCP?0CNINV2lXBmiMzv_*RK|brH~*D`p~MW6wrPdwRt5FJ(-KO2ax^ z0Nts@wzRYqs}#QX>gnr)=jFM803@O8dlfmk(R%B7D=RBw;Gis@$z!`zXE|Na+uIxM z$0uKSIh4qd$qi@-xV4n}#j4)xN;EeIpy*t8W`S-51Ox@1ve^Q@KGzYdRrEas+&QKySl7@x`4HeZ$8i)1nNUD^SR4Y|zAEG)M zWkE~J13+owG3z!tZb>*fIVmeom1)&3bbatl8QqluxY$;Y>XH(AfGy+Z=BB2mrlYg` zm1JMLIF>DVHpE)5eKJ)2>lalPOZK0V(o#K7vyZ9D|C(LP!TYSR+;!vY54yl8JkGna zW@h;kd8usXV{%+8>?VCth6v)qZh+j*0K3EMUXh<~HkQE+{39nNg};=WA-{JpT{xb_ z?|pN;h!1E1=m~t{$+RIBax?6 ztV;0qtLNozb$Z}KjV5EELN+HsG_Y}t|24oI1jcmq^z@XJ&PTb{sNTOPcncrg)p_1r ztO7_oX;D^G)O5d_uA-u{H=2%j9B$!cbiLjfiN^w9S!FdfP5?=UjLWz_`*u4LFpeKI z3M7JlK7CR6mX?;a=Hn!W#(9&P`uf1;ot~P)WzJBjZ?(LVvF_#LIy&De4kjC*ZwZ412Yv0Z8gV6*K zQBd@1zsOZ>S@qKVToH6Vd!NM&K>jLQL3FGQAJuv4&-~{Yda>tW+ zJQcwBk&yVEwcWz=5~Yz14G(ub9G3b&6#8$58)1_Xy>&(ZQrq0xnjsm2O?JEH{|YF9 zMZa}mDo{ev%gYOR6=-Q`Rl(%Eu@1WBuWM+qTf52o)stgTv$`;~y0FB`X0-3FYEW){ z|0wxF&BSCoAoGs59avv?zuVob9z#@4hl8Q$Y=PIvYPNM@SJ&%`OwaDY>LDv+*pXC$ zD+QP80?W(ThEc#ufu3;r`T70=e_k;+gSbC+UuipCyy$d+w;o-^hzZo>D*DF7X zX6m*ASIjp0?UA-~kyN7F+uA-aB9D)xk&G}1R-_ShJUWc^SlqXz-QC{wSmuX`_zPa$ zhw*nlbyY=eFdH+nQt7|)b|(wJU2jVg7uE%9)KVV3_&fmRJ>B7LSUk>tSmg zoD-pZNa`YPa0i!N9OdLiH;y<4rq#o%Mbh?)Hw8~m=*Ji<~~_mMW` zZZ=q(S}aU6_+5ay8QSKOWJXaSGh<5j=RDk?CbvovRVN2$MllWnFZ6oynDzIu(%e~N zvn7ebc3_d&t)`QyBJ3_XPTPUIyHcW>^Q%UHjTbY4`bUFbp#5(zVc+C|y{KFj{PSDT zfmmQ45h0BxzpO*wnz@7|-XHt#60p)k1;BlY}Ob|DxyzI?kmj`z7*_9uw zHwu!v@UcGtSZP+Hf!wx@54crFdmljji1T*b7ed3{h}fp(J_e3{yZgTGAfl^l~AjI6CW<8XOJRFjO1E{Q6GTvd=V6-rB6 zVb-2;z`nOT{J2c?NatcewgvSIt>dd9j~tokFvBpuSO*eFGg8?Vv{d0`a^9cc$AGJp zoZ{#{7w_5P=R9c4D4=ShiSE1I9ZxUSXu4A}a{pk596xl6N6F9N+wwU5PLR3e+xUnq zAp=@dj+Fnd2nr}vF!&y4rZp(-H99Dfb}5XD8HN4F0rb?oh#!^?=J#6!O_(~I6Sd~U z{d{ZAg!IuqG_DHm_?8V@O0vX`8Y^)xX{%jSwh_Yl%Fi>qX~zL=U(9bxqPXc8=8`+L zTQ=HNPZT1-EM-<}C2Ea(>dlgN94DGp#wSWLhMNUhUsFveIQ^iNxQ{=i$YYJfT@2a4 z2c&Tswb6iV=bI_W1zcwzatWN}&I?dvP$usSf4w+KwJ5jHdy5ZcM4^O@&arS+XvU?0 z*Jf`+D>Vpfk55Z`xH5`Ap!(P^JIjy&Fg-{#=lp) z7w$1GVt+3{e{$T0MN)u@rfo)4p^kvc!;qAPsWANf%UMz%8g*s>?Ek$HtjVkymMfSf z4f+jbN?gV;?Clq(>kIO*ynK`>ibFKqsUT=}fUqE0-`I9{R4#e99330&-XaA&ZK~jc zrViqh1Ad1kNBK<5#6mfZctG8URb5gZ^Fu|}y2m8?z=3)Vx)(l{fdCD}liK1Lwg$A( z1mM$h2M_N_w4Zz=p#wF{4j{Na8TUi!h@&dH{3=<~RknmOZPTUiNKU@Z@>y@bN}k&$ z0F%h(jMEKuLGE`d1_XUz7HVGX-1wr%%SgP~)N&Cud2m1_@?;O^VH5m=$_=(E3qgi^q>7sS7vIA#=Z4jtYl>U9W4+hz?&pZlU?!l2kgNE)*ibQ4g7 zuQ2wy;-juM9x#*kLdq_~)s=!*?CG}*#8_5cD4KTN&thoWjCO8!C=&@lYR zBB%cJTUs8q_}Om9m5rUi@}466Kt9v!F*6SvDFkWe^NJIrr>8tm1TtzWNM_PR{D8&& zbJnr>Qa|nsC&;jy1w_V*#1Tzf3IBzR7Y-EjgmcHa?Fx!>M+@0gQV+|J{K=8G?nb2f z_S5+)&AY=BCEz5Vz_pnBP zK=?*~3;A0p3t1s;q}h|jMV5*DwttZAE3zj%Xn_xp4s29+JyPA#Z9Sg1DfjNH)*)Nw zdy3bkik+sP;zKlX&QLTj+8=-0CLc@?eoIvV32D2nK14BMH|w7XCf8u`ds3EowZZKt zZ41QIcd(4gNir+mIXr~JNmB;S!hlRyLQQdi`Si}}`8$E;1ivw|4UKnt5S#U7T`nxY z%78MUz6~|`n`2C&%_W!}LK&X9H~x0L!jIW>AI_IVp#XWyEru0o(Lv=RVv|-3rJ934 zDry*Ui*lpjy@Wc)eH(ixh31PJ0277D(|EjxOLe{^a}t(tgV+NNdgws|mmz^@{koK%lO>qV-gZ@4#?(tQX8T6cKucp?3cVlyI| zh*f1|PChj+^ujU3X(XfF!6uDrZ;z60bDG8YG!e)2Nl2PR zK9wAs@fY#nBr+8GCM@s;@S#x|0o5|P<-1sC0GTv z`jf>x#prvixF0wH-UZ`NJDbFn>^B~k=QG`qE95-*l)-!PoCD>retH+CCC1>r1rbmc zr?8detoW}N_QFYjEc?=&)fGKGynZIgVrdhNb52VJMBXYZQm@Af%> zs@eR!hD=53w4k2u=(2Wo2{TS6nFAXATHf#QoBN*sx%}Y^UcttGJy^p4%--1*owt=Z zFRQ)-tMEI}i4iuG;e5)@B3tCJbm1Bu^zO(@o*xdpOZkd6Ku`S5u+NmovJl3(fW*)J z<=3dilYU{>`R<@3;&t zQILlQ$%b8UeK9NZB?V^>5e(iXKnLu-`mQdz?`3r6_itVal-Mf2f3HT3f6Nd84*WKA~1z^i#SJgkj{#ohh5{|nmdA2cBAu<)|_0?$q! z!<-Jv@KpBvg7JDJBa-(8_x9l<$~xbr+~rDM^Edtm4|pjYILs+|v|p;hVR?+Y2yp8p2L3?(petI}5us)rzTt9GVtcmwPkF{Yym*qM#6|tqLGQF7W}0t3s~;bgtiTV0)U1}4t6Mtp2o~Df zviGDE_V!w1Pr@;Gjdm+7ri1alk$4K}oTL3ezl@?|70`hU3+^5a#FqUZzOQK$WKTk^ zIu%!Vno#4abNaCp->y*n-niSe*g7BPl*&uvWwBXg(vQskW?!R$7Odh?eEulUmi89S z4zjUhrgw7*Mo6>unuF51u@-j&Kc6(N#Y9E50&&gXJjj`unXWxYP*$ejOh(aOV75km zN4xCxWQEDmYWMHiH@8AVzO@9AKdcPqd>v2ohn(wN# zaqxoEK>eqBDKauL8mT}=T%5x=|Lw1Wf|A+bs3W_RRhkkMUv?T7g?81)x4$~{j?!h6 z`rd2R>A&lvT9%lE&#GNLw`;Mnpr`DR6vWI6H3VjcOv!b2#i)t(#PZyojK1Vuh z1et~{kH@v+@L6OZKPtxqd|?FuI6d7t3ZuXDzuu9)lnJNQ6WtYvIu-f9k~FlptlvXs zKqLwsbq?Tr8_AJFQmS&#%FR{xk0j~e*!-K}_s9KsIG}{&kADm^++-hmFnWG`kwX<0 zO~0TKyqEGngJ%ZAf6f8GF8~h(S}+2d-+3<{kP*zw3yBpEDy=-BKO2}|B~?AdNh zzk2B2WkFqG*EsO;PkytOIejMv8?Z+j74CJnSzS67`s3Ht+h1;B7qc1$#D+mi)=lts zWfYW@P`Vv{Kpy>(`FIw}n>PS3^6hdD-%x3~PRQCe>~bW#b$iNWxZp%z_i;61=JpIP zf0)Y+bmkxDXz(_mcvzni;?bn0&5sWz)#QiOEgE{Me0`|A+(WpsYt7LFMh2awjqVz} zTqIYV70RaO&L82i=$Fk}vFKg<%SoT#W*5xrCkZ^4TyIhzY;D2|?eWmYgFa;h7@t1V z)koTzcUZpU7Ho|i^znBRyw0;(9@NFyLK^N@ZF!X+WO4wQ>h)p11OBISF#x3ju$Lpm zwvjti9z0)+FsE?QWRL3rlJ0FuW;-qYA)mmN-AqCJT4J=H9p;6k#PhOa@L~qQF1py1!U4I6+8LO1VY_rrAERfZH6{$f*` z;|~ky4SHorL)7Gt`ugXOLpBTXI|cx3*3r?CmL8hQm%(Gw-dtVZ}^4uBM1bj#tYB%HJ=OMI)Co~%Z?W)YD;-7 zOv}h=d-#c^rH52O4Q_p&6_9Lt{yBNF9lPLz(2f8I z(f1?GLwoNhR*`JuWut4*CnL*XW}iQAXc7S&mh|(102L#hv zx7)cfvyF5OA#fm%!{Wj(qiHypoBb%G+H&*Xn2HbkT5cWI&?s}^N2pc8usTya^wz>A zo=Hb^aA81a9UeGy-|w-vP)iX$zZZLbYX*)J7;xqM5tE%CjA`Lfx!`1uy~LYgK2-VY z<4>4G!_T8Yr;iKLB(#ZFb3CZqiAiP*s2~ndX2n~05bn-=#YSOLJSeht39ovl>o8vH z;c+(Nis^~{Yid^YXqjfJ^F3Kg1-!yw8i(LS+sl{hl}8+N|74f7oJe;; zLuiLJO13Nm{pZWdrT1g?pREifs!ZM;5oC|JqW!6I@ceyBvpJt{9RL+iW7BVJ<9qVN zIs$nP@>{!!G*##8MU4z&CVoBUamPID8V|@z2L*7x*8;<;(rht(M18VHfHN; zCvm_$md@{JqWOPtKH8inbe4ziG=7dDTAA}1&bO^tqyb=h7*@5d0F3vYA0W@$JRpb7 z0mlUWU_1CEiM$dNE$3kT?RTuVFIv&JP`(F)o9 z<~iQ1zOqZy@Z>0kOM}jJBqIfc-o2TUvWna<7A$fAx7swq7AG+&%?fZ*Vj-gxH*#2r(p8n?S9%==rx*Oq_atg z)(aDWiEJ?D(RUc4Ol}2QKldk!uIeXKM|-9h42cp1T~<@WS|fjN$23C&b&eSrT65HN zM3-jQmm|p+ znTgS6Hz9^p7#*-Azju_`!6^;%V<+E28}LI69fl&?llgVpx0*k|Ze-1Yq~G&U+zGSr z0P7|nq;x_{_Ato0VxWYodK@Gi-3rd}#h`eV9dx$=Tc}nAT*zBN1uW;ElA<(H5Egyt zT80NLR=@~zIFUfX6Gdeg2|n+n^oCr887jgA6iIeP3Gs`BzNc!?W9QN2zY?81DGjS8 z7K$BS>lySdWtTS-s@!0dwCWRc_-(vgd8uYN=^5q<=Kfy(1RZCD&*X)@PR&9vy z8;F%agWX_+N>Jg6rvjs1BZJX8U7!iu;q6e2XW0EzP3)Ts1FA$bg39opw-=r}S*mVe zwpSuH*T_l#?A@cYA5@*>0?7Y*L;YULKHv}V`g zfs|CEZ4?MI5`{WiFuFHNrd{_KT6HZPf}Qv*9pCvIx2I2^0w+TNMz0K0{{cl-ZXiP9 zz@Q3?wvGDp_eqbwhDhiL4Y>zPBrCrNCR@$xY+BB|_5OwUk9V?S?K1L`i?>Ds!S!Gz z4!rzV$jXTo$DX;OyV zvL*<-Rp{%{zNPLbs+pK5$ikds9RblF&ECFs#>m?gbIr)-w;QPCswbBXae5cV=gL(k zn(a9=t9cx{IRUAFFJ0Q`N%#O+|8cYRgkrlJXO7xn?RtU`gVvUBbtVKR{@(r0YUhcA zeDMge7Ay2$S{Py(KqkF4zX529)77>|pmL9BOk@`zlQI_j6O8P}yTi~FvIY3LxjjE2 z<6A?V2Oz)l^74v`_H)8W2>jlf*rt1MjBMX8j{DtolfEbxPNSHUZyAd=zjb!jYxlVa zcrpK5cYm&o(5Sr8S!A6Tq4YanwC<;8y2K#_5)VUTnm~iE&2;j>BLFt*O#^?}lVh2> zwU}5BEQA2&-3tvsWPOW`CH-#L*KTCTTsXbByv*lWQ;OH{6xeq}5*`F(!G{L<2Vll}V*kdh`$Is6ggj)eXN4-UH&A@`@z+DAClTEH}##j)9s zb#8_MnlPC7x5w|}+yLWiA#8R&o?o3tZc0L|jp0FGCBv{kwVnn-N6f+~O;q_7b{g4A zL-ig8(poGu$_94-?PG@lhZ2iQ=a8a@tq7#V{in}=iSuVa4mwO@R#6D3=0zfCIK%!_ zCP()Z$cBMKc})6GpAn|qpBt0p{}Jl%WB&D76o=|vra&UlzZLq=WB!VEtW)q$w10!$ zFsZD60Mq_V(Ge5w&J#r<`mY)O_h5f#*jf<{9rkyy|BKB3=fVDap8ut={~pXalGegQ zkMMs=R%M_1zZmR)uYeZL|0&sj>;9iM|7){_15Z0Xl*S?|*2lo(?YIpH08Z11RLg>}*m@Oyg?JT%h*N z*f`5`H`Ne1ce1*I5}ZqVJ)rHqY;0{#g%h^0un3QiwzaeKyBMU|8%f1F{@GVinC-{I&OT$q2~-0&*3~_2(+En* z%D#0|+=AEfBdz{;qOtp-6z;!r0YpTCx;L79?wtTC6nMUbghUl3!5S{Jt_P4*0MO_F zA-!;N1Dvv9d29?(aT7CH%5-1#JebNa0AjfkdJs@>i}Lr| z&ECk;k`hMTNNZ<+Gl%2NJr7~(hy~t23<3fIbaaE)zNV(zKuAk6DN&G-oqb91@VJ0l*Y!TOz7?RdF%#S=9*OH}$8+(QFU?H19*n03Do~rw zqgZV2eR~d#$cB@ckkQZ(KG?6e4h#$!hOxC<)-M5lpwPw7v%F8=xP20SkNEp${ z?ri;5x^f)w|#f`Dc9SCcBfxHVzNfym2vW4m>7(f~z&Be&b2tJtn z@Bi*AuE{{*cyzR!ygaHK;})8ZpFx$^yU!LDMO}u9^ocfA5+T5c(piAXS5i{yilM&y zGM)uw?-VFxk4;ate|YznAbK~3_jFy4zCA_I-%nF>;h(1%XzEE?!z(dF_@E1{B9yJ% zm$3{j0|S81x8c+YP&caA?jI9(28vRFDx^T27l3pB5g!luv~~5#)zymW0|`?lprjAG z7egyS1E8VP1bkblqylxsfkL_>*ES$F?4*vB$om!@J$rNwwW2GX?=(|^iMWsOEhYvP z1tmH<8f9M_C_y z4-;mV%ig+u=vIX+yu%|*cexz3#K#8;`jMsI0x>R=pt{>bMs-dOIfa*xLHv8jPIy%j z+*k3&Rj;E7JJr&%GCp;Ezw?n)fSqr0^qZ^gK6o1&k;dl#0xWIG{1Hrb1Yn26f{w?( zdoMZWW#6c50beOm@?zi=r`u~fJf1Wn6q&s~991eLCL5Gjs7H@CH1dMUdU|^CahY4U zVtApwW+o4SfZM?DdM9;gdXz~n$k^vrWTl8rK`JFaBO(5K607^^vSHX(qoWo-5kSvR zqY3K%QXii?px{UY3=f)zC`M*f z(!7I`Sx=6uc5n3u@7;@z+p|gmPp5W~@7VZs@CQv+rW~#=8MT@*5Sz7@*Oz>m+Y4Ud_kO!{hJ7@jW)2XOHVJ^^1HONrjaIXI0gT#YjfDi4wcJi_g?V)q4A*#GkCtwd4Cr&S-9G+lS9g z4#yT&7ToUJhiQ+bM8($H)KU*85c{>6nFL(HcCdMC%kuKm-qO;w;GOU~^VHL3B#U&+ zliAo*DpsD|ey_*X(@1#wtKZ2Q0x!B?R4vRX;I`sn5ZZ~c*vMrNK*O6`IxKoXxH+DpiwMYxTraO6o zSPb9t)T`6UT0a7Y>6>~tJT+DKhQC5SM56X{`HiFX+yA4rw+f2miQm0vafbvA8UiE) zclTsNa0o7oySvKEm)^A&rkNjD@&Oha0}&K83B@H|hraJ>8UJV+hW~SA7)|6O)Vu zA`wNJH32}ej9QK$-8i3$r-!FV5L|L1^7I@hlg(Pw;aVse#g+y%M`Ka8to zQa?Gf>9?g&QdOkxPn64kDPz}bA_!y167_g%euwhBG@}r{U)2~ZH6s3{UzKm9Mo(9j zkZ@3Vzdwwi^$`)pzUq%4=2EGKVdM70PHDywjbR+Ip#P?6_e zT~#ut=)+^Y8c2ptfR;?({6E&}r_Yj~5&6m&^7>4Kx__~@o> zA7?@i8C68Towv4ED0~V1bo(6>V&iJ2IblkDvyyGY+jwOONl4`|c*?%px3sUf>hAU> z`%|9RJZp}w2BlJ#q#`doNTPO|YURgdh zt{#ZStd{aBJ+G<%d$?Q45WPEWTn)cJ9)qum{H4iB<{f_?{~a){LvZrALW@Eua`h|I z*UZ5Tu1WI{`;HraZquZ6uc_2g!+-T#(_@t8IQGv7IjQK~dPwxYdRaL9j%M`x!c(Va zmVZmjb=h;nZ9xEpZ137R*SR#S&7_Lww5DI3+2zS)j7kOTzuNL$bq2m(2TC2skB3y^ z_apI*6Bjn(rotCm<2I_959{+DxEj8Tc%n)FbS8{w;fojB&)$cX90l{9Aq4VWe@g?Z z7~t`%zY5`xUgM2LH-tGALOyc(-Qv$nt9?PDDXUB5S@+LNt@F^rN>_*VXDK9%8R#Si zG6*$XD~QK=id?xwtAc|O9K$$F9$Ydw8OGa*YJh$IVzLBn)>0sVUm|_Jn@@826&6x&`f32^D;2eyx&d~j& zw_H1WpQU=+YO#1=l{_xeITa7$Y2-;<9>i06K_5l%G5VM$sBp^e>&q*b z<1WuenpL~kwKlhQt+BgXTZj^Oafg`oqL^~C`*AOngKjX-6Ld2iQ zSGtXuVA-u{PQA%}GzqQlB&(Qy+3f0FMV;fTUxjA$q_@}MIbIi5A3(lc$!&0;eo+}e z_Jnl6UMc;$z1S4C&GxZTYz$BNONP!g!71*#!%suh2y6S&k>MA;Cy`~`4?^yvhOj@g zS>?F(-*lB;X)PcVK??|7I6hp@@Z({nWVnBFE-9By6CE_``w8X}+&_LnDIRL! zV5MdhN3W!@AaCK>+2h6;z}8vmqwId*pRvMN8To#@W;1=MMw)14)juEp`DjiWZDpu` zQ4ukNtTD%+fm77>0=Rq8M z6hnr!wWz@@7j)?3)%WLz4p|B=jks}>-4VuSDcV8b4FX?#;T8VA6+sIS&Ve(Tls^IK`B|bd{T!qVzC7 z>P#t}xUH|M`eelOcQI?kNG!|Zvx5v4F^_?!MaU|RGzy03-WTX(7yqg zCivY6J>3a1K0zq}hig@0Ml)Xv%R+pA-!3J4Nlu%>eJ{2#a+ix~YNu5oKd8%;lAY_x zTEDJI>b4_~7AU<~Yeoe-lW|Ea_*x`nR3cVFSBFND#7fn0op$9@1dfA;Efx?{1o-9QebB3q$0|qI{JPT+{UCaI{%~U{F z%mF+$YBP-(D_{U&T%SYo*C3HMS()PaU7e zWTuT;;Bi!v!(tJDchtOYt+M^ues;Rz;gfby=|mh6saPd9x$VIvU#O^-M>7s9i_X?S zsvSF7X;{RvoBhVgf;;r3A(wW`f%(rQY*D9=8hVxKi<(7Sb|@+CSLSc;I6y%B(8yk7 z2Px^pomQIY4AI&s@uNO6a?-u9$h<}T{+EJh5>nzJ^^o1lj?5U1i4}ophCc=c^%{gC zRAVZtzO}EPudNGyE00^oI`LJgW4H>xqhR&}3-R+sl8{$mO8i#m$n;p>sY z%kkhtlPn^WC{49;zEk)TuHw|h6YbGeNr6YT>#CK=v< zhe1Op4P^5 zY!Tf);!!~cc+`)5lwLrdP9B|BLnpl!v>ME3g(lsy?cCY(u}-K4jNjwoei;j;?OiLp)`yx+g!3 z51FaD9}ktNCVYMw#8>`uxzxcUnN3d#*x_v{h@G(pEJ2Lmaabr|5e3frOtqMkuxGJ{ zbW9rpkWarMIVHu!rsgLpz!1-B*f(@%1=Oo8lRQQXFEXP3+Eh?j$YXxNuCK z{p~pX8|W1S0zb1aVUmY736RUWt7CP|nq@1Di*+4RM}btf#kT+)GY;xm87wovrZB6= zaE4S#=H-2Gx#)Ti5(hiHXE(KD^Jsb^f$5f+CscDITJ}* zu;w2vq7Pg^NU+sP;?gy+gz6H?t(@f{c#9QyjTkN1qTCQ+@WUfJPEXjM)Xd#MpUOXH z38#ZlgwK%iAmjgxL1Z(&+;lDzF~-lHPdu5gcoIqRL6JSA0jAiVwZTKgxa>cjhh{Dy z>(@SGWH#RBqx_(d}Gi=!2mn=)Ov-gOOY&^ncF_40GB! zIk-i=v~Mcob6R^+U6cxgT<$13THl{ihc?SP)(x(qLUZ064WD_*Y|ducMv$9?CxzZm z0H7)K2TJHG{UM)xEaFN1t?!(4M1%4C#@nySayj0|{G2f_@ksE=!emo@O{X&`4s`iq z)V_spQUSD>nY$8WozpMbjIEq=#RpJQ=oV`Xr~lBbC9|*)U~oEZ#I?iIkR=IKKOnw= zTznAJ$z-*p$G~nFSMKkt%^tPHa6BXUH-#)G>`$Qzsx#EB$g z^6b^t@l(pj12@PBb8z_RJy{P<*sX;cVy!wS%j;^&92?*O)ijRBfH=Nc>9dk1lOT_~ zc*0Y1&ff;8vN?<;@%{@1rR@3$81M9BOUe)?H0w?nTm7L9(B$sDc^9oybNMWH)qxA_ zkk-;Aowz*++@&#Af{NQ{{84rtZbLgU*SSwl#YzqtRI$mm6wv(>%Q(n6B+s4<$DjD* z9*df5dIWcw%fBFc*Qz0u!?3&s^(}w*o&xz_c7a zYm&MuA_CUe9xB;#F29g-Pa($8T4AUO2H=`}Ch|E(7)&IM8l5)m^;|r#o=hb{fn|6& zwO?&6L0Ir(Z9G~e@xb*5UDnRB+}nup*JKDFVY9s0g-JRHrm?RxK++7O@Du^=qQ_}2r< zzJaFb<}b|i*AD~MEZ+f9=%RckR$U83OUT{OcUL?t`C!$?7_iDb^14YfUXSdsH61GsHGxDGyADp~P zho}IyU2Jgx54li18|Bh3AJAH}k_$we{u-0|D-r7O*GkF8ax%Z1xh;u0jqb0=?$NA? z5aG}EN>G<;WhZ`o3%kPK_h?nL!b3e+0Qt!ph@|6}&G*88Pm!@S-)iokZb@JK*tbGR zm9q#u&*vqOKOAE5u?Oio z{H2emvX@O)-E2nvxnM#a{Db;3)b-sO$EbpuQl8xel&&dOAHOk>vhypwz=t}jYRSe- z(s2a=Wj6eV;UBLLYpYutf>pRp*Ty(z0#fiOWftS{nnvf{oOw)Zq`r=wsd%k&2lh9y zK>94mk*LPr>SD8GOyJjWYDmqhX$HIzJsXT+6knmeRH2rf4k7>2Cw;v~j2sXtM%9Cw z&6JGXBPB*z2npp~32o%6Xseq$3n#XO&C0-X zwoIEcd?>I!aK&CC=`<_~2@n!#+xlM)@je% zv;#|o=9d*n>?!WWP{|9RBf`NhoC77@piM7YwjPV@ZjJ_S(Zb#=3hLTXpbRlK`oxI} z-0@}UFWvT~O3)YCLt4aUShss;7q4R1t3{9<&C)>1AQt}0JZ7JSBTG$#)gtKzPhhWd z1-_tqg4fBJnX~1E{n#x#7XnI2PQM@M{IR;rt3)FKsSrzmQftIl`uF zeyl1|YANDD$`o_)D@5&lwzA$No&HL*shigaa0KL#=F3lEC_6Sh(k)&bOw4U4>>+Ii zy*sfCFuMgGE9t!8qLNwqmW9sp^F@kAo}@5O^p3^GehG|N^0!x0D56!&OK!{iLvb&< zdG&k)WaduL?<+t_qJZZd_HTiG2b)J|aZ}c?o03gc}oN zfGy_Mj|~;I3-ZYM>`Udm=*CNVjeUS_8B2!J*7!HilIul$YYHo7 zCI-4*c|d8}gM$&x&S#a7^Sf)tCgauaT4P>g6*vXeIMW=8nT7G$jj74D5UhJH)h&tT z*q@x$!l>1cex?cBV?1w!*LDYKI|Be!-ye%cC51jw=#k|I3Z#69tzWu$+_`k{eroPJ z?}~+1rxNZC+y|9Hko8IWO-`@LQ#vkKrpToe6NEKP;g-eqDUH2Ph4kensIn~ng-@Qf)hR2+}X)SK=~w9m6m(w<>uTV7%H5G>BmiE@S~oj9~{ofo%@l9 z95lfpS?9>0StqtDC5W2MBU%CE^`J}@G06A?e`V~@ek@SmG*K zL;KxHFAb16eLVZW6F)D=zvdPx8~uJjiTcssjb%@=y)D=FpCWD7E2~cyu~*JtO=Ztf zt%9Vy<#_18yZnWhtUm%H;mZXQ+b!~5*u#ATd7Jo!BZo(+M{?8&A2i8cA#W1J3COW1 zPK*#!#{64M#%u&nFjdpGX1*aQitIbsT>jSMBw1v7W9S^(^Z7!-^&MNlS7JqmXW!a| z;UMfV{=QvJHGa*NgyWc&Z)f8;$cy`vC5(dcz7L9iAA&B_G^{cIj^n`QdEZaIwZ}oL zd~n>y%kKF3z1eRo(dT@(uyirv<+}FxcZlchCM+SCo$9O zQB<-;_ljw_3)Lyd#?856@Af>)arApw>M+AzUIf#ygShA>h>%Om6SlbVOiAV#4-ZT8 zYxp#`7H8>e-op+?vmTLwnzlBW9UHb5UoWpgx!*xO&$b1(OtxL@OA;DvfOyMJL zHY|uT<~Mc^l<)x=%xkECc=_oxGUL_xG>F3H*k$`ZPSE!AJu!zS+DFQ?dUsau`uEQt z$!#21D$T!LVw!yHgn;tx9-rPOocjp}3%~5+RU)@JS!1_rU8_yzs_D%wU&byP_oHYs zwBai3o0#`6>jntS!p>`)!P?EFKc;V1Dda!_@T-wKO$B6;tmP)p30+)-D?DjU;`%D% zt4%oPLf`q_-9uYe;CNTEgT%eE^|)JGa45nHQG%GY+&0dly<(xHIv;}vPb%v^#D{*x&6`l9xfHlZIS z?XH91yE3Z;$k>zb5>=*>jZE&hE$Iy!B;BHdSxeC5AT0cV$9xPx^j2+&z%$f;_%)>7;v}kI*_o#eDDte6Q?~*~Y@ZW=z zFFh;od0l2^R#wHqg}p^S$B@S{9m+85#n(sx3A@>ro`(s5_im0YA}U+e^Be#$%m3RW zK<;8#5Uq>fa0pMyW{1^gofg>FuGFVUBJgnh$E)9Fva8?Z$d3ww+b#(@62+1GOOJsR%vy z(?qAnD92Zde&!$q()<5yWzCwciQVnk!IT=rwx0X19ch*db2x>44BU?qd9bfb1aNawzwZXCRlEr5d&eu_DU+`^-WR!0K8aGm;d2y-%NsEAe{_Q z<$bpJnt6+oB-;+x#Cc+3PeG97^VtoXJ1ke)DO}Y!1{tui9)7nxNATk#YzkLkcazV} z<8(Va&aOYGcvC*>Pw?!^pVm4L;4&T{P(f9%duz zxn_T40|;v3*efnpXH`K*7%(!wVjna8*cs*(a4xXYxGVoPDnm-;Xv$L_A4m5Pz9_A8 zc4>sa?8k;p-F8t1PUEK^7odwnpAhLJo5*G4p|_62$A(72BakC{Phr)ZK-WS$UzCDo zEy&K_a3G_4iIv-VK&ErWXpeLvrA8?w6A55jH5%}&*hxEfysBzE#Dlf|{sKnKo=vu+oPD;I%lkVMw2Byy)Wj_}{LZA$YFm@2)tSfi)_ z&OOij5VPXep!hk5dDz5)%UMJ4bYZP7w>jZ!A7=5!OUEW!n_ZMKD&-7npzRC+SeW<2 z$+$$M3I_hKIDI`m_;S_0Oa{?D_FbzN{0O9O-}-IzUUmVgN1xj1?l@0+Yj@PBcA>o` z8DT_!3=M$QyB^H}mTK|jcRY94OOwU#i47%=9OSnzkT=+)-mj7!g+owNQ#2UEDhq&)2z ziC4U^)vO``E}z(l!Myqn z3uESw_N;lxv9TjpS0jUigA1M3#evkLl17%0KbBNDEC|Vt%0KemG0)l3)U+{HR^c{s zDsnEq*n`}4qsf2uXuhXu*zh0eH^SrRTbgvzqG~WvRl~)7G%8ZHf-ppIjm}j+{RSrbb*L$#yjr(Z z4&~+L%a37lHU?a}TM{a=Dz>D_Ck_Z+`=?*Q8k&M#*cbhwGoRh@aaiH&>&hCCifsKA z*Q#IOpI;DLaQvmvbvKVKWpW6xo&@Ec`{SB$^enf<;$*E>`zMKddU}2)jzGsnZWy)p zy1PjIA{aB^!`r$Qy=OIyUSEuEn`o#SUAj@(zp~QK&FrI?;+T;;!-{;Bs8gv-G&AdV zZtXhg)7=}9n@YW)wsxVVv7)DdAndyZs3MeOZ6&HMSt18dg?B76>cnHirOBVPuP#o@ z-#Zk?;cuhlo@?~tGG&vk5ETNM?VGzj?d04Q-ooDjMwqII@{W?x6hOA%)M3n zYs(ia+#1P&g|^W z(m7`j4-0KP=4KX~DV#3fqw4aztDSq$Q*FzdG=eC5y{d{(-j$Y8FX$Nj({*$6SDn7- ze5y}wa4azKxKV5ih2u-3!kN5-DWUYS z5(ia&o0(yfc5;)%hiL658u!bP(V?Mc->0^bG>4jtOWezgsFoG*`sJ76*=Qm+(xwV! z`7GX%f_JH`i^GG11Z!>mRs1zoeA7iomsj5`+7j2MWsQ;qoRGESg}J$Q;b0FHm8s4@O1R{*+>M;PoD})L z);i^wWJ@$uQ-|+))y;5Mx8B~~+_at09j5yK6a^4aDg^C^z1^+s{5P_KIQ&>c1tR)m ztii*XoZB3yR=D=m?Sq-w4uYAJZ`OwZ$wblmnI!FrAK~sV=+vfXd~#~mqod0cuMS2) z`_~XKniw_j`zyQ1-;ej!BD2#J)+Sp5&f5rR2|DiK^0FDk6}~DO{_%_Vp8J#oc&=e3 zHUFZXJDWm_`NsC*DWFcSK;md{e^C6&^fNhNfcz`5L-O~GGN7z Jl~RU*{|l^nIQ0Mk literal 0 HcmV?d00001 diff --git a/storage-engine/rocksdb-overview.md b/storage-engine/rocksdb-overview.md new file mode 100644 index 0000000000000..91d8c8da17d10 --- /dev/null +++ b/storage-engine/rocksdb-overview.md @@ -0,0 +1,61 @@ +--- +title: RocksDB Overview +summary: Learn the basic concepts and working principles of RocksDB. +category: reference +--- + +# RocksDB Overview + +[RocksDB](https://github.com/facebook/rocksdb) is an LSM-tree storage engine that provides key-value store and read-write functions. It is developed by Facebook and based on LevelDB. Key-value pairs written by the user are firstly inserted into Write Ahead Log (WAL) and then written to the SkipList in memory (a data structure called MemTable). LSM-tree engines convert the random modification (insertion) to sequential writes to the WAL file, so they provide better write throughput than B-tree engines. + +Once the data in memory reaches a certain size, RocksDB flushes the content into a Sorted String Table (SST) file in the disk. SST files are organized in multiple levels (the default is up to 6 levels). When the total size of a level reaches the threshold, RocksDB chooses part of the SST files and merges them into the next level. Each subsequent level is 10 times larger than the previous one, so 90% of the data is stored in the last layer. + +RocksDB allows users to create multiple Column Families (CFs). CFs have their own SkipList and SST files, and they share the same WAL file. In this way, different CFs can have different settings according to the application characteristics. It does not increase the number of writes to WAL at the same time. + +## TiKV architecture + +The architecture of TiKV is illustrated as follows: + +![TiKV RocksDB](/media/tikv-rocksdb.png) + +As the storage engine of TiKV, RocksDB is used to store Raft logs and user data. All data in a TiKV node shares two RocksDB instances. One is for Raft log (often called raftdb), and the other is for user data and MVCC metadata (often called kvdb). There are four CFs in kvdb: raft, lock, default, and write: + +* raft CF: Store metadata of each Region. It occupies only a very small amount of space, and users do not need to care. +* lock CF: Store the pessimistic lock of pessimistic transactions and the Prewrite lock for distributed transactions. After the transaction is committed, the corresponding data in lock CF is deleted quickly. Therefore, the size of data in lock CF is usually very small (less than 1 GB). If the data in lock CF increases a lot, it means that a large number of transactions are waiting to be committed, and that the system meets a bug or failure. +* write CF: Store the user's real written data and MVCC metadata (the start timestamp and commit timestamp of the transaction to which the data belongs). When the user writes a row of data, it is stored in the write CF if the data length is less than 255 bytes. Otherwise, it is stored in the default CF. In TiDB, the secondary index only occupies the space of write CF, since the value stored in the non-unique index is empty and the value stored in the unique index is the primary key index. +* default CF: Store data longer than 255 bytes. + +## RocksDB memory usage + +To improve the reading performance and reduce the reading operations to the disk, RocksDB divides the files stored on the disk into blocks based on a certain size (the default is 64 KB). When reading a block, it first checks if the data already exists in BlockCache in memory. If true, it can read the data directly from memory without accessing the disk. + +BlockCache discards the least recently used data according to the LRU algorithm. By default, TiKV devotes 45% of the system memory to BlockCache. Users can also modify the `storage.block-cache.capacity` configuration to an appropriate value by themselves. However, it is not recommended to exceed 60% of the total system memory. + +The data written to RocksDB is written to MemTable firstly. When the size of a MemTable exceeds 128 MB, it switches to a new MemTable. There are 2 RocksDB instances in TiKV, a total of 4 CFs. The size limit of a single MemTable for each CF is 128 MB. A maximum of 5 MemTables can exist at the same time; otherwise, the foreground writes is blocked. The memory occupied by this part is at most 2.5 GB (4 x 5 x 128 MB). It is not recommended to change this limit since it costs less memory. + +## RocksDB space usage + +* Multi-version: As RocksDB is a key-value storage engine with LSM-tree structure, the data in MemTable is flushed to L0 first. Because the file is arranged in the order of which they are generated, there might be overlap between the ranges of SSTs at the L0. As a result, the same key might have multiple versions in L0. When a file is merged from L0 to L1, it is cut into multiple files in a certain size (the default is 8 MB). The key range of each file on the same level does not overlap with each other, so there is only one version for each key on L1 and subsequent levels. +* Space amplification: The total size of files on each level is x (the default is 10) times that of the previous level, so 90% of the data is stored in the last level. It also means that the space amplification of RocksDB does not exceed 1.11 (L0 has fewer data and can be ignored). +* Space amplification of TiKV: TiKV has it's own MVCC strategy. When a user writes a key, the real data written to RocksDB is key + commit_ts, that is to say, the update and deletion also write a new key to RocksDB. TiKV deletes the old version of the data (through the Delete interface of RocksDB) at intervals, so it can be considered that the actual space of the data stored by the user on TiKV is enlarged to 1.11 plus the data written in the last 10 minutes (assuming that TiKV cleans up the old data promptly). + +## RocksDB background threads and compaction + +In RocksDB, operations such as converting the MemTable into SST files and merging SST files at various levels are performed in the background thread pool. The default size of the background thread pool is 8. When the number of CPUs in the machine is less than or equal to 8, the default size of the background thread pool is the number of CPUs minus one. + +Generally speaking, users do not need to change this configuration. If the user deploys multiple TiKV instances on a machine, or the machine has a relatively high read load and a low write load, you can adjust the `rocksdb/max-background-jobs` to 3 or 4 as appropriate. + +## WriteStall + +The L0 of RocksDB is different from other levels. The SSTs of L0 are arranged in the order of generation. The key ranges between the SSTs can overlap. Therefore, each SST in L0 must be queried in turn when a query is performed. In order not to affect query performance, WriteStall is triggered to block writing when there are too many files in L0. + +When encountering a sudden sharp increase in write delay, you can first check the **WriteStall Reason** metric on the Grafana RocksDB KV panel. If it is a WriteStall caused by too many L0 files, you can adjust the following configurations to 64. + +``` +rocksdb.defaultcf.level0-slowdown-writes-trigger +rocksdb.writecf.level0-slowdown-writes-trigger +rocksdb.lockcf.level0-slowdown-writes-trigger +rocksdb.defaultcf.level0-stop-writes-trigger +rocksdb.writecf.level0-stop-writes-trigger +rocksdb.lockcf.level0-stop-writes-trigger +``` From 8c9e72c846eee9ac3d527bb9189d715e42e29b45 Mon Sep 17 00:00:00 2001 From: Ran Date: Tue, 14 Jul 2020 13:17:13 +0800 Subject: [PATCH 2/2] rm category --- storage-engine/rocksdb-overview.md | 1 - 1 file changed, 1 deletion(-) diff --git a/storage-engine/rocksdb-overview.md b/storage-engine/rocksdb-overview.md index 91d8c8da17d10..37c63d4084026 100644 --- a/storage-engine/rocksdb-overview.md +++ b/storage-engine/rocksdb-overview.md @@ -1,7 +1,6 @@ --- title: RocksDB Overview summary: Learn the basic concepts and working principles of RocksDB. -category: reference --- # RocksDB Overview