From 00e9ded1149b1acaea13723299b3c6e308b47376 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:25:53 +0200 Subject: [PATCH 01/13] Remove demo binary --- demo/cpp-shared-lib/cpp-logger | Bin 35792 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100755 demo/cpp-shared-lib/cpp-logger diff --git a/demo/cpp-shared-lib/cpp-logger b/demo/cpp-shared-lib/cpp-logger deleted file mode 100755 index db2a8eb70a30b53cb6efe3dfb537aaa518767818..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 35792 zcmeHw3v^r6neLGv#Cg2M?bJ{L5}}Y9Xwp#d{r>%s zj}+_9G4R z3doujxAFmes>(IWMS#l~j@LKy47EI+D9O(&1fCQUU$xL&m|rF6WKK3wqMZ1OrRur7 zM&?Qthl#J8iYk2|sq8bChv#!x=4L*ouT#($Bdm7tD)}3kiRArj3oPO z`rQpR4R!WJ)b3*SWS5FR>b{k$*DL6jQNjsvO35IHr97&;MAQ@gbMLmlY%RBzT{FDw zw$H_%d3fWyKV`? z$SFP^f#45Lf&US3Q{e}S(EE$ff4vC(iX!yCLL;Zr>n$SZyG8I17r_q|!CwwXPNg?d zg#P9t_&1BlxxPrdzF))-hlQL@&foDQ9sJ{2AbOc;86-KyoGXhdJ2$ifyosO>3gf{@PrGle;dJ@} z-nhqzdxJ)z-Pg9(i;uP@&nk~M7SkjCHpmD=h(FZXOF4u!_@F2ft@V7;<#Z(mU}@On z3q^gKYa`LVwq8!;tc`mk{%9D@=?(gH64;{~z21-{a0jCa53JGa+9N@u)yeQ{jXEDJ zpHf64M05uuK_c|@^!IyWdOQ)0ctb&Bz|-ph4_edU@kG%$-5d4pbfmp0>3Op3Ym=)_OukEr0<=+nQT0z^84@!ce09e5X%kkZ&E#sfRh#g^@nt z`+To0=U%%2W2irTaFe08tqBX+e`}C99T4IVCSuWqp6k76(%?POlncmc4A{o{de~2kSWcd2M6eb9ZyKvEXaOb9Qr|T>14Msqi z7zis7ZzP%s>3U2szz_XFLxFfB8lntW5lExB0+1oN6S@-C!@k%67@#Q<^&sHAMlc!y z0^)s}J-)zZPq#N1QlK(n_^ zDl0qMmjML2&fa9I0AQ|iy3L@uY7$CzD!vm;2?ubmGyurSzz;9LO;Qm1cM(?&_bHblK$Oa{u-uLGn9yc zA7SuC%7B2AR|IF0&xVCOIj@)V>0L`ns-o-@6Ih8KyouvS3-FzSzG9HqOL_?(yPM-_ zu6G?TvdnS55fSCj7M~{8c=6SkRmDtr^L(e@guhq{K|W~0)Atyec9`(Y{$3OQG82A}314l( zKWM^VZo=<1;jb{^_nGilned|~{D(~VqbB?{Cj4}9+5^)bnD)T52c|tR?SW|zOnYG3 z1JfRu_Q13UrakcA$pdd$KXgtTdc8s$DgU(x%Msf~jgpCD+R#%K2U#;G>L34%qKr>0 z_-AC+YZRgnP|bMu#fgcDT@0oLobl}O983#0qgv`{mieIN(Z0?l~# z%Q=`9X2!Eyb1*H)jAuWagK5EQJnPNDv@kQC{X`C?1)1?|OAe-mnDMML2h#$~c=p;H zObajL*||BG7F@=&mK;nAE#uigy)WBK3oK}V4yJ__v_A*af(qK7gJ~fJ?a#rqfP(ht zU|Kjq`*ScYn4tYRm=;RV{v1pTBxrvQriBr-KL^u-2-=^6X`v47moR$s`Hj?@tqSW; zz$fp>hksW9KVJYpT>u{{fPY#5KUx4kTmXN!0RCnH{M7
x(@fd8%lP8Pu70$4AA zHx|I_3*c1+a9aVqv;eLzfNKii1qJZc1@OEA_yZEwMz+}8nO=mdHnQJ#FFHCC6u=$= z?f}q{I+Iz)kXvlA%*~<{$TW-6rc8qOf^03-rryll4dB#c-FE9W!(h&hPc5Hn`DDxbmUZjbYFl5YRefz}j!h{jKZrrX zuZ@%~SO`WfHEg^0BZ}f2O}(q7-syn0V;IziCM?!325|Li>0uid{n6^)ijLGdEp>!A zn8m-r#iym~e*x~HcT0>iZ6sMmH1-)q3%41Apu$kIjE%$W`4;rcT3E$?JNo0&(WU~ zd=KYQwWprSe1l`QO!A#3zG03|0Ij8$UV0lcn6-%Df{(+-cN{zT?8^3;>3 ztdk+5jjVh6td=^a9msr0a~{)zqk3va$Z`~Vo<`gpuxMNV5<}M{U(iPGu(`=31q~w< z#p%X(>GJxEmio8HXOWV0`R(ulMJqk9v6^fwFGZa&zCw6NSbG*11vSDU8X>HH zgV1P%usS0^VRcr3!s?R(6jr}1KwywUnDB^KfWofL0u*+I1SmZ+Mw&0? znje~s5t;>=`r=s;S=HQivuUK5jEz?PJKc<@Y;gSpB*_q#Za{stvtn$(+D^c-4sQDRRRm(ZWsw_^8=xtENU!&8lrLTZdo7bh{(< zYUW4p(0yf?-Fe$nhqR%yR_hmrXTn26XD!xkT`09!AG^f1V>cHxvd&gb);L7y2OyBy zbhxYp-L~6C@0^+KYBt$5DD2umcI^^?%6e7Wb%n6&@~Q0lZO*PAS4`T)1i>zq?CKPD z?I^TsLxEinR!#bAk1%Bu*|k>y3&}33VgBkxgpt3tzCDG%-i9^YUnM5Hm>}4-knD;H zyB;jGtG~dmDwAEK!ju5nbxZ&bvWse%U0<8S?aID2gRyGf?W=>YnQO= zXrW#A7TDE~%T31Dm@p+qcAXYL71>2K%&zg-+^$bdWtTr^*SB%6k#;dbu&avf+AHkJ z6xy|q>`FgkQ!xuM$Jq%4j`2HfvR*U;Hn*6o`i|8e}$fHgLsntvvq&PCg9@wAK(v!HjA#{yo2yuzMs=ww4YSfsgm%5-nuj#CE=HP5_OdVW<4Y|fjm-%DE3OTWxb)Jy zsqSW~OC5O}!a!f&ausk5grl!2ghZlikIICrAY6^pa;nmLO|z1D_m2}3L+-L9E`nF+ zt}+|#89F;*yh6c1dODiU@K9iZNujb&G?}>tp;X1|*`P`F_lkPvM$|iaJsa$)zF*Yu zA-AG_A+O&{Ak}XW^@E~b<@LK{y<60867|)*J|^p{MSZiVZ|3!#vi>ZZJW^+CGHhF{ zJO7CSeB=_FHgs;Lb^Fb5KDwu$&)9~}`mEcRfMDsX_^@vOIMF_F%cr-yUpN%5Tgqlj zH)flF&CTD2&VgkHFk8=Yp4qHSq}J_L;>UN*)bW;~r^>W5uV8{U8(%QLwcVohVZu{p zvt-}JYy<18G_Txt55fUoP;C|28RrO#^*V;%B=A+)JN{zEe@x&V*@w=W@mC4_!t7r$ zX`GaE4hyj)U(I&p;ZF#Bb@u*0OL`1@oz7x?TV7Dq1s$o^5K5Wr zUZ;>1RR~?8&aQ$uxQf~Vc>ykOk&rh^{gW^6*rdE)p30X;)Y(HYds5yeF7MZ*8TRct zU*6tHc|!&Ah&ubbJo`3qc_|@p@O*i@Cgs%@$Rq0PJLt?w`(j*PlaM!dzP!_u^8WBz zetU^J8_umtnQ2{Vjv;`Rc+ z0r>mbJu+Kr#&-gbDLB#JfiTO_Yryl_LiXvLoh+yk?Uw9(OuxE!Nw_5Y!92J$TSC(_ z+`biZOS*m^7O^MS&-P->%EahynXdO@6wcnq!V4=vsGu-AjU3M{FlIf7Qk(PT%vZAH z!2Pxi;LJRlU;r2+ph5ub8}_>-6g9`F<_!THCE!H?j1usi0QM0ugLzXdS)OBh`1+w= zyMZq#TDM(B&Di?XgZOIOni{)(W6K>a8(Tip;yH*GJi*osmyWH3tXqg3-*(W3vyhE8 z@Kv>1hNo4vD87kjY@#)TpgF(2e7MSdD?={i!*-^NanOqI22l!RR*TZ6Osgnu$Se`1 z&WwYV#ES2A0J8hp+U(y|vk8>*&zX+_iOKAi>Rd84 zz4Uhoj!YjADDQyu%;$J{3zef(#=ujZ{S?;isHA3>ip9c(**^%JBbySXs%%u$R%HJd zmC{S=NY3?K=0Yl8$;&$Fop^=OR1v-uc+uGFFnELYFGTV>NHSkQ9eePWI+)#t_Q>)( zrt(Tt*={P|Vk&PmmESYTTw}toHI>(y%Ii(#Pnyb~GL`QtQa0Ofm}nN7%3m_slQxyV zg>v`g`YSbky_T*y*@j5KVlw$h8(F$>Ie~9%)Y9|1P|Th@Degl=@BCwX>iB20)JxjX z%cncnwK+$fKZk!U&^|G*6y$GD$_L&_veM-rp{(hwF%OSAUs;AS9(B%~7_(kOmy~i) zFCzr}u=Q~HA8{wNYy@vkKk;Y#P$}%8=NRp&x7TZ__cB#* z0&cD3rgtC%Yb2TXkfR~VnH{MN8)9g9S&Zk8XvbD+2WYaOVQfb-|3>8L`bX)xL+VK8 zUkPMO@R_H0O)0UW=23t@qlsc>Hk9r%`E$|uK3r)N^n|57^*kB(R~YxsTCM5H-rLxB zGL)>t-vMuX>V@_LZ>Sxqap#|DYB>r%j9S0)WP8)0 z_;GF9v&moFu330{T0R5s(0n0%nUwS4pRL39L7GO7kIujynU`N8*OuADLRQ~OT9ipTt&go&cC5wM>g5$J zhs(+^ltV`Du91X-r>gnA;vkPJiqhk!Y&mB+-ee;@^=c!a~23{%T7fqxZu zd-P}KA4xeD5<5n&JGDD=IZ>vU-i7B_nJ{Y1Eo9GsJ4R-#f*QniAA}s~Sadl1Jm^^a z51Ogb59AMRA8z}U?&hAcJx`vb5G4myXh(2=g)fe*tPnZYkq`|{PxX1R`eEDzL)+mZ zFLd0TzDm_nZMf6F)Iou`5w1CsdG$qV+r|od4MBKA_LuWFlj51!0y;GG0Ae^}Km^%J z{=xM!57ts>JUmtV+f#S~bhhOuzu#MNpO*evHO%iwJt_P@GD{nYS7=*bL=y(LVsx6g zTbubOE%l6c{0*)4#{n#Q4ZUbFN;{m-XepOA^d!F7|0;R3J-v4ms(wZ?>3fzovWw(u zBRf>ew!Xx~!AKJKzwwPgJD$~QD~4WzD9wonN;??&&?v+mP9A6<8SF>@wU1P33m7Nb zL7bQ^JgFUjt-W>^X?xIORP1A|a6Z|d+Qm4I3=V>k5-y}>-hU%sAlzDt)}>Lze_uNm zge-^{Tv>llRkxE^d^brwt2v*fZ!8DKW-J)|j%9wp zrVK+Q<<$2@yiib}D-Uuzraz}WFztb94@`St+5^)bnD)T52c|tR?SW|zOncz}g9j{F zZf6^$sI-xZ?pM2`aTWVvI9-i)2mU#i)YUp0#l|RtW{IkV?)3CxFCN+q1x2LUqS=GZT1~P}J*(E!a5)(4D%W3T3LY3>y^b zab-n(KJ>H|Vga;H&kHa)&b+8MYD#ewx zzId{`dq7$44f>Y)R=1c_UEK)*p*Qkg{6uQ|a&1(fDR%T|ie?MM1R68&9a~BSB_} zF`%r8c0~AswM;G6JtAm=<3kY3NFvDUa!ffi0cX zyP{FNR2w4G22`In5`nor@Tq@53Fz@?(5Lw0UdTs*ZGbbNVE?Bmfzg-|3}fFZ6a!EK z>yr_v1efkteAqTB5>@s7m^Z>YTH3FMyvc|!Fsak88_{H3Se1;$gJA4Y<9a}c)4bSR zhnXIWN5jwx^x==0Z!B^y(C0@O~WEmTCmN5kyVDCxKm$7A}%Q2fsd4a&|C*OUSGGLf%AT zAQycq?UaNT`_(Wu!U_$jUHC5p(oPo^m~kC|0D++PgreAVsYmTf8U~`eOZRz`30)27 ziJ-x{PVLr1h8peeuIZxfzwE*gc5xyG;9dkY*%8M+QNf6==0X+JY7d&7Fa|=wM1cA* znozNam!W%ea_9n_yu^Lp5RFkrAec}g4xH)Td}*82!O5 zk|ToZ(?dEeC7WbpBDkFV18TqvRLC1fO&}On2XtcPg9n_(^}_sMf?c9&k_IAG@9qwg zIs+cU=2}S~1+?{;1v%-y-t^M z6uu&7cq3?9PL3GG;508TQVQfXJ zf(YyLCgA5Fu1;L}UW6IYxaeRwhH)1Bc4aMMjCv{>r%n9)Y9g74At0j31OlFh?=D^E zO%Sn*?g?Uu?iKvhj@%Ui&jxWbGZ4sqpz?*1elU}G3M(k-R^?ExMiC#B3ryBb`L zF1M?x)=}%Mb=B6^*45V6Hqzf)J4bBEvLv2G{Lw!R-Lt}%xp{dc)=xlT~);88P);BgZHa5B&o7@h! z)9rHCy6fEa?gn?G+wE>@f{9H~-vrT3U~2-3T4cGh^rQXAEAeL>nV5J8dF#(7CZ0pS z4*BcIFZq{=33^q+_KS&$VdT%@;YV31_Qu5nt=*`98+jQXYz^bF7QLHoU`5{ncpV-~ zh6s;*55aht_%?ESPF?{zJTYU*a@QKg(qCn{a?Xs3U8pBK-ndc%(DPxUXq~|xOW07X z!{6xPi3xf;Myae?QCWSnb#`CHpz`r6mVBhvHXm5VkKb}Bfuj=>jFFH?B-g;-Cg4Yj zU!;Y|cj7MtecK4Ptg4(5d!-#Ct^5m!oI zbfS*h{af%HL|&GF;hj01?WD7n>vWW?F3{ON`_l!O7J7k|?AQV^8_-YGzm)KJT_NAs zp|2u@=%}n3D!F}DWyQd3^5sKB5C6PKbvIX5eUbc=E=8MnlK$;zvt#I@l3Sqz7E!7fY8_s_C+2l?!*4w^TZ| zU({0R9-6VTa-Z5!xv#mSa%_ER&n2@e-Ovx;gEps|wN-^EKV&aPpO+#p`w%|BG&E!T zMLWyWWnV1)QVF$vC*tww*u=yQC@XE1RohE$t7MPb08UI;X5HKc4?JNhXV3{t425r4 z${(#!Ua{=4ykp1yeC1iYk}fMhWLLg(Q8_UCXO#cMt~^#z{;*wnW@h;VcID`-@~_&J zU(6{_fqqWP@?MQ{-=*aTYLu^DR{nI2@}27PKh-Gd%gcXfSN2_DMS)=^; zy7FJwDEHXPzh9#~Yb(F+M&;WJ%F{P0rx%oOy;1qi^|0ly*Ox%r-bEh+(_cPvJ=Fi? zhQ+`PFD9A078CjXi%HDOi%HDs#U$phi%I*A8WQuZ8d8O(mcKx)oKDjonD)T52c|tR z?SW|zOnYG31JfRu_Q13Uradt2foTu?_ws<;UtjL0FZa*KN(oDJwg4pwpDciB?|({j zc%~>SFW@)zENxoGbK0YalHA`PI~K7d*XZP)<8nWLxi%)(VcvOvf<5@zEA+`_iWqx0 zTAtm6^)*pHOUU0<#tXFeMTypXDB-6sc}cJ-C&YgFSQ27sk0{XkAtj+s!BQ1V{j_+B zB=OCn;nHrma2)-Bm6Fsi_e_^`_=OB>AIKRBZ~Zd3Q#cl{Co@>uiC+&hSUMg*C}Xf_ zE?4%S_$kjzD-`KaIi^|Uog&{P@|eg6MZQbqdqlohsbvki6 ziKE7uWT5l&E|=X=XRlo(D)Kq=Aq&7VE{Dqg4N(X zOF7{-LBCSq9UtR}7X^ME6chdEbsRzKsg$k}csYKsHB{hhfw$m9i_hM}$a3SyGT^7O zpU&eV`C~V6dii;)z*h-7CB94G<#;9W{lJsnSb^SQhQCaay$4e3f#AQavaTBC7#@D*4Nc;B6SyRLD^Z z_C#^*Y|**&rx}w2a3plj?u#p&754?Z?}-0Rmw3N$I~+gN@t76nGXXV z0LR#;INi^ArrZYn+>$Gl!t1&Rc=DU0nG+rq^bY`kE$FKTIYN%}KLvg&zr8~ElDUe! zUkWkG>qY2i;s!XC9QtYcRCqh^WasG?E}1>&LGg(q^uH>CFURk8r;>kZ5&S|locu4( zKV{GKpm$Lb`fw5a-9_-bi{O7q_!8VaRo-6t`6%$zkA?AkpopBSabusVy@!kVCsB+Z z_^HmX`z`R)FY3p+h3vT^iu2(&y6%)gX*thtWcVwU{(JKAbkjeZ~t0E&Ra$B7h@tcl|OrdpUMyWfmhFOmU5zqoHvW$mtjIUm7Gr$!EXkB zs&fdp0zZ}g^ctMuv`2B4nYSzG!K*av^(efwgqMr#cvFgw!^0VDD?4so*3#ketoGoY zfp9WmcyO3nS2Wbt*_%6R&{dbjdkc8?1_$sNbUYl68pQiTc&(y6+~y16^gCBAUJAj> zQ4!r^^s$2uJvg%vZv(Z5JHoZDwvKRvs}7&FO^x*F#D}ZC0WTNSHaP16I0;kd!V3Wn zjjkrb5FNc<(ByJA)H#TRuyqa>-Y;-8){{u5n|(O(wn8o5PH004uUY4zll&YWr-R<% z2z!iynBHa&CgNV4%SOk=*?rNFU&*yQk?iuc^K<0rw80Ly#}kawiGT|HmoV@qp&jqn z<<4*P80=ugh@OCB{ZS8IpX&04Jbt{mmGF3z{R-ZxiG_5$eFSw=Qm_LLJ>GcSI{+UU zcqgwLcIqB~GK_Z_JsuMTcL$9;QjcfFnwC{c&KfK@gs+($kB77E=oGvE z$RU-SK^(P*Q}P%OBK(u=wK{e#WjjI|r(v#jdfM7rU8Gd^dkyd8*@5RP{aa5^EI4K} z&nTSa=V`wcXA9!3DG%PVLm1|rZJ4K>pIDfO`uAES58Nzza;VuF_=kc4@s8I!@oKOW zB_oO>5AjYXt{zViaom1qTU&3Z?@pXsNp!&;94HE#{fVdtFG}HnM$s|+%|)|!m%G~N z=uQgYAUn4)j$<1W26O$ITh_WfO6#ZWalPAK{M^qXr%mR!fld#FMRWwC=}5?&*Kio3 zIP1}j;>SMbFgQH%ztDBzW5=noxZp=v=IqQ}2n7eRPM)I5N#!Nr<}ZHd_?a`*+=tUl3oz_-+2+ZUFmd2z;jx-IGtdES$n|(|KCT?x{xi>hHQPppT*v{s z=N(v@CzP)33zC|5;_CS#>G)i{t7)#dFh$Fq?KlOujSm0C1z1aW4;uUURfB;r;uk*x z;Hoh2B3}52Q~catvsreWVTwTur>*8B1Pgf!#?PFtA-uc3GQm$D) z8g%{E&Wz4GB)3_fID0{g^G^&uuPsv? zWL@;IYIer6Sps)O{@KQKB6J>&V#j0?Z%89Y;IKR(OPFfv@t9)Ai}Sj@Cz7<|m214B zi=SppqPmj7kiQ1+)N>Th+^wOBE+Y?k69L8UABfQ5z$`c7yoUY6126qEG68r{7st;{ zh)|SbAw#jVVabl79dEKLtdP)siXDdw17vp}dpyb}5q3QwCT{^hq)7;Ba8$6~@6%%j zrr~t%H<+vaet#e!Ohh-O*XfG8?GQWz0 zxlf`sr|SUVD`Jk{E%;^bFcYJUuU&bm0g#!WHuj_>a~lV9pIC0lOYMNn{BnI!=4u}G zo-e<_IrV)wR_1g*Y{<ZZCCMlApP@Xofm4Zb6G;*?E^2_yRnQtkeFVz2=0{(@c<_t1_Pzn}u3d~VX7VsYv>*F$~cjC>d zpgWW^sH8`1;xf;z$L|esu0nrH{W7QL6jl?zT>l>x{F07Kn~Z;nw}PKyU+S0Z@<*k9 z(SEvqDM>zQ*AIZBHc5W@eDQ=>fL&P7V977@YSf$gl|cxj6yuq^&N(!xU&8dtJIRvz zn+JJKwcvLb^j|Zt&V61d^gD7GUXc7X1$eTRk~uF>!Rj@kf3I*HT}GGTFV}8H#h+_Q k+XaecXZtwxU988MlhiA5b5WRz|C{%4ip2#C1)%bO05!tuq5uE@ From 89d922d101872c943d77768c64c14b72e047becb Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:26:04 +0200 Subject: [PATCH 02/13] Fix naming export --- liblogjet/include/liblogjet.h | 1 - 1 file changed, 1 deletion(-) diff --git a/liblogjet/include/liblogjet.h b/liblogjet/include/liblogjet.h index fd95cca..0afee08 100644 --- a/liblogjet/include/liblogjet.h +++ b/liblogjet/include/liblogjet.h @@ -4,7 +4,6 @@ #include #include #include -#include "liblogjet_export.h" #ifdef __cplusplus extern "C" { From 9e382a7204c0678d0c2f72cc318ee11609c13a5b Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:31:42 +0200 Subject: [PATCH 03/13] Update file-replay demo --- demo/file-replay/README.md | 29 ++++++++--------------------- demo/file-replay/run-demo.sh | 24 ++++++++++++++++++------ 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/demo/file-replay/README.md b/demo/file-replay/README.md index 6e633f6..6f6a082 100644 --- a/demo/file-replay/README.md +++ b/demo/file-replay/README.md @@ -1,7 +1,8 @@ # File Replay Demo This demo replays recorded `.logjet` files into the OTLP collector mockup -as fast as possible. This demo depends on the first demo. +as fast as possible. On first run it generates its own BOFH-flavored +`.logjet` input file locally. ## Build @@ -15,24 +16,7 @@ That gives you: - `target/debug/ljd` - `target/debug/otlp-demo-collector` - - -## Setup - -Run the first demo in [`../logjet-file`](../logjet-file) so it creates a -`logs/` directory with files such as: - -- `bofh.logjet` -- `bofh-1.logjet` -- `bofh-2.logjet` - -Terminate messages recording manually (Ctrl+C). Then move or copy that `logs/` -directory into this current directory: - - -```bash -mv ../logjet-file/logs ./logs -``` +- `target/debug/otlp-bofh-logjet-generator` ## Run @@ -45,8 +29,9 @@ From this directory: The script: 1. starts the OTLP collector mockup -2. runs `ljd --config ./logjetd.conf replay --path ./logs --name bofh.logjet` -3. blasts all recorded OTLP batches to the collector +2. creates `./logs/bofh.logjet` with generated OTLP log batches if it does not already exist +3. runs `ljd --config ./logjetd.conf replay --path ./logs --name bofh.logjet` +4. blasts all recorded OTLP batches to the collector Expected result: @@ -54,6 +39,8 @@ Expected result: ## Notes +- first run seeds `./logs/bofh.logjet`; later runs reuse it +- set `BOFH_RECORD_COUNT` to change how many records are generated on first run - replay order is `bofh.logjet`, then `bofh-1.logjet`, then `bofh-2.logjet`, and so on - replay is immediate, no delay - the collector mockup accepts OTLP/HTTP `POST /v1/logs` diff --git a/demo/file-replay/run-demo.sh b/demo/file-replay/run-demo.sh index 4a97e10..72c79db 100755 --- a/demo/file-replay/run-demo.sh +++ b/demo/file-replay/run-demo.sh @@ -5,7 +5,12 @@ SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) TARGET_DIR="$SCRIPT_DIR/../../target/debug" COLLECTOR="$TARGET_DIR/otlp-demo-collector" LJD="$TARGET_DIR/ljd" +GENERATOR="$TARGET_DIR/otlp-bofh-logjet-generator" CONFIG="$SCRIPT_DIR/logjetd.conf" +LOG_DIR="$SCRIPT_DIR/logs" +LOG_NAME="bofh.logjet" +LOG_PATH="$LOG_DIR/$LOG_NAME" +RECORD_COUNT="${BOFH_RECORD_COUNT:-128}" if [ ! -x "$COLLECTOR" ]; then echo "missing $COLLECTOR" @@ -19,15 +24,22 @@ if [ ! -x "$LJD" ]; then exit 1 fi +if [ ! -x "$GENERATOR" ]; then + echo "missing $GENERATOR" + echo "build it first with: make demo" + exit 1 +fi + if [ ! -f "$CONFIG" ]; then echo "missing $CONFIG" exit 1 fi -if [ ! -d "$SCRIPT_DIR/logs" ]; then - echo "missing $SCRIPT_DIR/logs" - echo "copy or move the logs directory from ../logjet-file first" - exit 1 +mkdir -p "$LOG_DIR" + +if [ ! -f "$LOG_PATH" ]; then + echo "generating $LOG_PATH with $RECORD_COUNT BOFH OTLP log records" + "$GENERATOR" "$LOG_PATH" "$RECORD_COUNT" fi echo "starting OTLP collector on 127.0.0.1:4318" @@ -42,5 +54,5 @@ trap cleanup EXIT INT TERM sleep 1 -echo "replaying bofh.logjet from ./logs using collector.url from $CONFIG" -"$LJD" --config "$CONFIG" replay --path "./logs" --name "bofh.logjet" +echo "replaying $LOG_NAME from ./logs using collector.url from $CONFIG" +"$LJD" --config "$CONFIG" replay --path "$LOG_DIR" --name "$LOG_NAME" From b9e3684198081e881f3ff7639904be0bfe5da2e6 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:37:22 +0200 Subject: [PATCH 04/13] First swipe of updating dependencies --- Cargo.lock | 215 +++++++++++++++++------------------------------------ 1 file changed, 70 insertions(+), 145 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 1ec53c6..b1babf2 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -204,9 +204,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "axum" -version = "0.8.8" +version = "0.8.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b52af3cb4058c895d37317bb27508dccc8e5f2d39454016b297bf4a400597b8" +checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" dependencies = [ "axum-core", "bytes", @@ -253,9 +253,9 @@ checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "bitflags" -version = "2.11.0" +version = "2.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" +checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" [[package]] name = "bumpalo" @@ -292,9 +292,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.57" +version = "1.2.60" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423" +checksum = "43c5703da9466b66a946814e1adf53ea2c90f10063b86290cc9eb67ce3478a20" dependencies = [ "find-msvc-tools", "jobserver", @@ -327,9 +327,9 @@ checksum = "6e4de3bc4ea267985becf712dc6d9eed8b04c953b3fcfb339ebc87acd9804901" [[package]] name = "clap" -version = "4.6.0" +version = "4.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b193af5b67834b676abd72466a96c1024e6a6ad978a1f484bd90b85c94041351" +checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" dependencies = [ "clap_builder", "clap_derive", @@ -350,9 +350,9 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.6.0" +version = "4.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1110bd8a634a1ab8cb04345d8d878267d57c3cf1b38d91b71af6686408bbca6a" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" dependencies = [ "heck", "proc-macro2", @@ -700,6 +700,12 @@ version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +[[package]] +name = "hashbrown" +version = "0.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4f467dd6dccf739c208452f8014c75c18bb8301b050ad1cfb27153803edb0f51" + [[package]] name = "heck" version = "0.5.0" @@ -753,9 +759,9 @@ checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" [[package]] name = "hyper" -version = "1.8.1" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11" +checksum = "6299f016b246a94207e63da54dbe807655bf9e00044f73ded42c3ac5305fbcca" dependencies = [ "atomic-waker", "bytes", @@ -768,7 +774,6 @@ dependencies = [ "httpdate", "itoa", "pin-project-lite", - "pin-utils", "smallvec", "tokio", "want", @@ -839,12 +844,12 @@ checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" [[package]] name = "indexmap" -version = "2.13.0" +version = "2.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" dependencies = [ "equivalent", - "hashbrown 0.16.1", + "hashbrown 0.17.0", ] [[package]] @@ -901,9 +906,9 @@ dependencies = [ [[package]] name = "itoa" -version = "1.0.17" +version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "jobserver" @@ -917,9 +922,9 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.91" +version = "0.3.95" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b49715b7073f385ba4bc528e5747d02e66cb39c6146efb66b781f131f0fb399c" +checksum = "2964e92d1d9dc3364cae4d718d93f227e3abb088e747d92e0395bfdedf1c12ca" dependencies = [ "once_cell", "wasm-bindgen", @@ -927,9 +932,9 @@ dependencies = [ [[package]] name = "libc" -version = "0.2.183" +version = "0.2.185" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" +checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f" [[package]] name = "libloading" @@ -1096,9 +1101,9 @@ checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" [[package]] name = "mio" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" +checksum = "50b7e5b27aa02a74bac8c3f23f448f8d87ff11f92d3aac1a6ed369ee08cc56c1" dependencies = [ "libc", "log", @@ -1315,12 +1320,6 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" -[[package]] -name = "pin-utils" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" - [[package]] name = "pkg-config" version = "0.3.33" @@ -1546,9 +1545,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.37" +version = "0.23.38" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4" +checksum = "69f9466fb2c14ea04357e91413efb882e2a6d4a406e625449bc0a5d360d53a21" dependencies = [ "log", "once_cell", @@ -1579,9 +1578,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.9" +version = "0.103.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53" +checksum = "8279bb85272c9f10811ae6a6c547ff594d6a7f3c6c6b02ee9726d1d0dcfcdd06" dependencies = [ "ring", "rustls-pki-types", @@ -1608,9 +1607,9 @@ checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" [[package]] name = "semver" -version = "1.0.27" +version = "1.0.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" [[package]] name = "seq-macro" @@ -1792,12 +1791,12 @@ checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" [[package]] name = "terminal_size" -version = "0.4.3" +version = "0.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60b8cb979cb11c32ce1603f8137b22262a9d131aaa5c37b5678025f22b8becd0" +checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874" dependencies = [ "rustix 1.1.4", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -1854,9 +1853,9 @@ dependencies = [ [[package]] name = "tokio" -version = "1.50.0" +version = "1.52.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27ad5e34374e03cfffefc301becb44e9dc3c17584f414349ebe29ed26661822d" +checksum = "b67dee974fe86fd92cc45b7a95fdd2f99a36a6d7b0d431a231178d3d670bbcc6" dependencies = [ "bytes", "libc", @@ -1869,9 +1868,9 @@ dependencies = [ [[package]] name = "tokio-macros" -version = "2.6.1" +version = "2.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c55a2eff8b69ce66c84f85e1da1c233edc36ceb85a2058d11b0d6a3c7e7569c" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" dependencies = [ "proc-macro2", "quote", @@ -2041,9 +2040,9 @@ checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" [[package]] name = "unicode-segmentation" -version = "1.12.0" +version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493" +checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" [[package]] name = "unicode-truncate" @@ -2109,18 +2108,18 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" -version = "1.0.2+wasi-0.2.9" +version = "1.0.3+wasi-0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" dependencies = [ "wit-bindgen", ] [[package]] name = "wasm-bindgen" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e" +checksum = "0bf938a0bacb0469e83c1e148908bd7d5a6010354cf4fb73279b7447422e3a89" dependencies = [ "cfg-if", "once_cell", @@ -2131,9 +2130,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "18a2d50fcf105fb33bb15f00e7a77b772945a2ee45dcf454961fd843e74c18e6" +checksum = "eeff24f84126c0ec2db7a449f0c2ec963c6a49efe0698c4242929da037ca28ed" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -2141,9 +2140,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03ce4caeaac547cdf713d280eda22a730824dd11e6b8c3ca9e42247b25c631e3" +checksum = "9d08065faf983b2b80a79fd87d8254c409281cf7de75fc4b773019824196c904" dependencies = [ "bumpalo", "proc-macro2", @@ -2154,9 +2153,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-shared" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75a326b8c223ee17883a4251907455a2431acc2791c98c26279376490c378c16" +checksum = "5fd04d9e306f1907bd13c6361b5c6bfc7b3b3c095ed3f8a9246390f8dbdee129" dependencies = [ "unicode-ident", ] @@ -2248,7 +2247,7 @@ version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" dependencies = [ - "windows-targets 0.52.6", + "windows-targets", ] [[package]] @@ -2257,16 +2256,7 @@ version = "0.59.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" dependencies = [ - "windows-targets 0.52.6", -] - -[[package]] -name = "windows-sys" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" -dependencies = [ - "windows-targets 0.53.5", + "windows-targets", ] [[package]] @@ -2284,31 +2274,14 @@ version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" dependencies = [ - "windows_aarch64_gnullvm 0.52.6", - "windows_aarch64_msvc 0.52.6", - "windows_i686_gnu 0.52.6", - "windows_i686_gnullvm 0.52.6", - "windows_i686_msvc 0.52.6", - "windows_x86_64_gnu 0.52.6", - "windows_x86_64_gnullvm 0.52.6", - "windows_x86_64_msvc 0.52.6", -] - -[[package]] -name = "windows-targets" -version = "0.53.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" -dependencies = [ - "windows-link", - "windows_aarch64_gnullvm 0.53.1", - "windows_aarch64_msvc 0.53.1", - "windows_i686_gnu 0.53.1", - "windows_i686_gnullvm 0.53.1", - "windows_i686_msvc 0.53.1", - "windows_x86_64_gnu 0.53.1", - "windows_x86_64_gnullvm 0.53.1", - "windows_x86_64_msvc 0.53.1", + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_gnullvm", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", ] [[package]] @@ -2317,101 +2290,53 @@ version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" - [[package]] name = "windows_aarch64_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" -[[package]] -name = "windows_aarch64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" - [[package]] name = "windows_i686_gnu" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" -[[package]] -name = "windows_i686_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" - [[package]] name = "windows_i686_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" -[[package]] -name = "windows_i686_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" - [[package]] name = "windows_i686_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" -[[package]] -name = "windows_i686_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" - [[package]] name = "windows_x86_64_gnu" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" -[[package]] -name = "windows_x86_64_gnu" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" - [[package]] name = "windows_x86_64_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" - [[package]] name = "windows_x86_64_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" -[[package]] -name = "windows_x86_64_msvc" -version = "0.53.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" - [[package]] name = "wit-bindgen" -version = "0.51.0" +version = "0.57.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" [[package]] name = "xxhash-rust" @@ -2421,18 +2346,18 @@ checksum = "fdd20c5420375476fbd4394763288da7eb0cc0b8c11deed431a91562af7335d3" [[package]] name = "zerocopy" -version = "0.8.42" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2578b716f8a7a858b7f02d5bd870c14bf4ddbbcf3a4c05414ba6503640505e3" +checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.42" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e6cc098ea4d3bd6246687de65af3f920c430e236bee1e3bf2e441463f08a02f" +checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" dependencies = [ "proc-macro2", "quote", From c604d6853ae7fd859afa3906ecd819aedd852421 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:37:43 +0200 Subject: [PATCH 05/13] Lintfixes --- liblogjet/src/lib.rs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/liblogjet/src/lib.rs b/liblogjet/src/lib.rs index 2265d8d..465a307 100644 --- a/liblogjet/src/lib.rs +++ b/liblogjet/src/lib.rs @@ -115,6 +115,11 @@ pub extern "C" fn lj_error_message() -> *const c_char { /// Creates an OTLP/HTTP logger. /// /// `endpoint` may be either `host:port` or `http://host:port[/path]`. +/// +/// # Safety +/// +/// `endpoint` and `service_name` must be valid pointers to NUL-terminated C +/// strings for the duration of the call. #[unsafe(no_mangle)] pub unsafe extern "C" fn lj_logger_new_http(endpoint: *const c_char, service_name: *const c_char, timeout_ms: u64) -> *mut lj_logger { unsafe { new_logger(endpoint, service_name, timeout_ms, BackendKind::Http) } @@ -123,6 +128,11 @@ pub unsafe extern "C" fn lj_logger_new_http(endpoint: *const c_char, service_nam /// Creates an OTLP/gRPC logger. /// /// `endpoint` may be either `host:port` or a tonic-compatible URI. +/// +/// # Safety +/// +/// `endpoint` and `service_name` must be valid pointers to NUL-terminated C +/// strings for the duration of the call. #[unsafe(no_mangle)] pub unsafe extern "C" fn lj_logger_new_grpc(endpoint: *const c_char, service_name: *const c_char, timeout_ms: u64) -> *mut lj_logger { unsafe { new_logger(endpoint, service_name, timeout_ms, BackendKind::Grpc) } From 7f4ccc9a63cdb0e6c402d01833c2f078c573c8a8 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:46:15 +0200 Subject: [PATCH 06/13] Fix broken wire forwarder mock --- Cargo.lock | 1 + demo/Cargo.toml | 6 +++- demo/ingest-overload/run-consumer.sh | 25 ++++++++++++++-- demo/src/bin/otlp-wire-forwarder.rs | 43 ++++++++++++++++++++++++---- 4 files changed, 67 insertions(+), 8 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index b1babf2..0901af8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1222,6 +1222,7 @@ version = "0.4.0" dependencies = [ "colored", "logjet", + "lz4_flex", "opentelemetry-proto", "prost", "rustls", diff --git a/demo/Cargo.toml b/demo/Cargo.toml index ea6ff2b..1b3a0fc 100644 --- a/demo/Cargo.toml +++ b/demo/Cargo.toml @@ -7,9 +7,13 @@ license = "Apache-2.0" [dependencies] colored = "3" logjet = { path = ".." } +lz4_flex = { version = "0.11", default-features = false, features = ["std"] } opentelemetry-proto = { version = "0.31", features = ["gen-tonic", "logs"] } prost = "0.14" -rustls = { version = "0.23", default-features = false, features = ["ring", "std"] } +rustls = { version = "0.23", default-features = false, features = [ + "ring", + "std", +] } rustls-pemfile = "2" tiny_http = "0.12" tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] } diff --git a/demo/ingest-overload/run-consumer.sh b/demo/ingest-overload/run-consumer.sh index b2963f9..6789611 100755 --- a/demo/ingest-overload/run-consumer.sh +++ b/demo/ingest-overload/run-consumer.sh @@ -6,13 +6,34 @@ TARGET_DIR="$SCRIPT_DIR/../../target/debug" COLLECTOR="$TARGET_DIR/otlp-demo-collector" FORWARDER="$TARGET_DIR/otlp-wire-forwarder" -for bin in "$COLLECTOR" "$FORWARDER"; do +require_fresh_bin() { + bin=$1 + shift + if [ ! -x "$bin" ]; then echo "missing $bin" echo "build everything first with: make demo" exit 1 fi -done + + for src in "$@"; do + if [ "$src" -nt "$bin" ]; then + echo "stale $bin" + echo "rebuild it first with: make demo" + exit 1 + fi + done +} + +require_fresh_bin "$COLLECTOR" \ + "$SCRIPT_DIR/../src/bin/otlp-demo-collector.rs" \ + "$SCRIPT_DIR/../src/lib.rs" \ + "$SCRIPT_DIR/../Cargo.toml" + +require_fresh_bin "$FORWARDER" \ + "$SCRIPT_DIR/../src/bin/otlp-wire-forwarder.rs" \ + "$SCRIPT_DIR/../src/lib.rs" \ + "$SCRIPT_DIR/../Cargo.toml" cd "$SCRIPT_DIR" diff --git a/demo/src/bin/otlp-wire-forwarder.rs b/demo/src/bin/otlp-wire-forwarder.rs index 2909fb8..139b0e0 100644 --- a/demo/src/bin/otlp-wire-forwarder.rs +++ b/demo/src/bin/otlp-wire-forwarder.rs @@ -5,6 +5,9 @@ use std::net::TcpStream; use logjet::RecordType; use otlp_demo::DemoConnection; +const WIRE_MAGIC: [u8; 8] = *b"LJNETV01"; +const WIRE_VERSION: u8 = 1; + fn main() -> Result<(), Box> { let mut args = env::args().skip(1); let source = args.next().unwrap_or_else(|| "127.0.0.1:7002".to_string()); @@ -76,19 +79,49 @@ fn read_wire_record(reader: &mut R) -> io::Result> { Err(err) => return Err(err), } - if magic != *b"LJNETV01" { + if magic != WIRE_MAGIC { return Err(io::Error::new(ErrorKind::InvalidData, "invalid wire protocol magic")); } let mut header = [0u8; 24]; reader.read_exact(&mut header)?; + if header[0] != WIRE_VERSION { + return Err(io::Error::new(ErrorKind::InvalidData, format!("unsupported wire protocol version: {}", header[0]))); + } + let record_type = RecordType::from_u8(header[1]).map_err(|err| io::Error::new(ErrorKind::InvalidData, err.to_string()))?; + let codec = header[2]; let payload_len = u32::from_le_bytes([header[20], header[21], header[22], header[23]]) as usize; - let mut payload = vec![0u8; payload_len]; - reader.read_exact(&mut payload)?; + let mut wire_payload = vec![0u8; payload_len]; + reader.read_exact(&mut wire_payload)?; + + let mut crc = [0u8; 4]; + reader.read_exact(&mut crc)?; + + let mut crc_input = Vec::with_capacity(header.len() + wire_payload.len()); + crc_input.extend_from_slice(&header); + crc_input.extend_from_slice(&wire_payload); + let actual_crc = logjet::crc::crc32c(&crc_input); + let expected_crc = u32::from_le_bytes(crc); + if actual_crc != expected_crc { + return Err(io::Error::new( + ErrorKind::InvalidData, + format!("wire record CRC32C mismatch: expected {expected_crc:#010x}, got {actual_crc:#010x}"), + )); + } - let mut _crc = [0u8; 4]; - reader.read_exact(&mut _crc)?; + let payload = match codec { + 0 => wire_payload, + 1 => { + if wire_payload.len() < 4 { + return Err(io::Error::new(ErrorKind::InvalidData, "LZ4 wire payload too short for uncompressed length")); + } + let uncompressed_len = u32::from_le_bytes([wire_payload[0], wire_payload[1], wire_payload[2], wire_payload[3]]) as usize; + lz4_flex::block::decompress(&wire_payload[4..], uncompressed_len) + .map_err(|err| io::Error::new(ErrorKind::InvalidData, format!("LZ4 decompress failed: {err}")))? + } + other => return Err(io::Error::new(ErrorKind::InvalidData, format!("unknown wire codec: {other}"))), + }; Ok(Some(WireRecord { record_type, From 32374a05578f0ff8bd6520f58e34fecf99e0cca1 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:48:57 +0200 Subject: [PATCH 07/13] Build missing exporter plugin --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index 8993f71..1cf6be2 100644 --- a/Makefile +++ b/Makefile @@ -32,6 +32,7 @@ test: setup fi test-unit: setup + cargo build -p ljx-parquet-exporter @if command -v cargo-nextest >/dev/null 2>&1; then \ cargo nextest run -p logjet --lib -p ljd --bins -p ljx --bin ljx; \ else \ From 0632b26eb56ff6dd39ab1c9419fed7197055497b Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:53:05 +0200 Subject: [PATCH 08/13] Update outdated headers --- plugins/logcat-ingest/src/lib.rs | 22 +++++++++++++++++++--- plugins/stress-ingest/src/lib.rs | 27 ++++++++++++++++++++++----- plugins/syslog-ingest/src/lib.rs | 22 +++++++++++++++++++--- 3 files changed, 60 insertions(+), 11 deletions(-) diff --git a/plugins/logcat-ingest/src/lib.rs b/plugins/logcat-ingest/src/lib.rs index a456d4f..bcd55c4 100644 --- a/plugins/logcat-ingest/src/lib.rs +++ b/plugins/logcat-ingest/src/lib.rs @@ -19,6 +19,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; pub struct LjAttribute { key: *const c_char, value: *const c_char, + value_type: i32, } #[repr(C)] @@ -29,6 +30,13 @@ pub struct LjLogRecord { body: *const c_char, attributes: *const LjAttribute, attributes_len: usize, + event_name: *const c_char, + service_name: *const c_char, + scope_name: *const c_char, + resource_attrs: *const LjAttribute, + resource_attrs_len: usize, + scope_attrs: *const LjAttribute, + scope_attrs_len: usize, } pub type RecordCallback = unsafe extern "C" fn(*mut c_void, *const LjLogRecord); @@ -41,6 +49,7 @@ const LJ_SEVERITY_INFO: i32 = 9; const LJ_SEVERITY_WARN: i32 = 13; const LJ_SEVERITY_ERROR: i32 = 17; const LJ_SEVERITY_FATAL: i32 = 21; +const LJ_ATTR_STRING: i32 = 0; // ── Plugin context ────────────────────────────────────────────────────────── @@ -242,9 +251,9 @@ fn emit_record(ctx: &LogcatPlugin, line: &str) { let tid_val = cstring_lossy(parsed.tid); let attrs = [ - LjAttribute { key: tag_key.as_ptr(), value: tag_val.as_ptr() }, - LjAttribute { key: pid_key.as_ptr(), value: pid_val.as_ptr() }, - LjAttribute { key: tid_key.as_ptr(), value: tid_val.as_ptr() }, + LjAttribute { key: tag_key.as_ptr(), value: tag_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: pid_key.as_ptr(), value: pid_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: tid_key.as_ptr(), value: tid_val.as_ptr(), value_type: LJ_ATTR_STRING }, ]; let ts = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_nanos() as u64; @@ -256,6 +265,13 @@ fn emit_record(ctx: &LogcatPlugin, line: &str) { body: body_c.as_ptr(), attributes: attrs.as_ptr(), attributes_len: attrs.len(), + event_name: std::ptr::null(), + service_name: std::ptr::null(), + scope_name: std::ptr::null(), + resource_attrs: std::ptr::null(), + resource_attrs_len: 0, + scope_attrs: std::ptr::null(), + scope_attrs_len: 0, }; unsafe { cb(ctx.user, &record) }; diff --git a/plugins/stress-ingest/src/lib.rs b/plugins/stress-ingest/src/lib.rs index af8f4e4..5fc40cd 100644 --- a/plugins/stress-ingest/src/lib.rs +++ b/plugins/stress-ingest/src/lib.rs @@ -12,6 +12,7 @@ use std::ffi::{CString, c_char, c_int, c_void}; pub struct LjAttribute { key: *const c_char, value: *const c_char, + value_type: i32, } #[repr(C)] @@ -22,8 +23,17 @@ pub struct LjLogRecord { body: *const c_char, attributes: *const LjAttribute, attributes_len: usize, + event_name: *const c_char, + service_name: *const c_char, + scope_name: *const c_char, + resource_attrs: *const LjAttribute, + resource_attrs_len: usize, + scope_attrs: *const LjAttribute, + scope_attrs_len: usize, } +const LJ_ATTR_STRING: i32 = 0; + type RecordCallback = unsafe extern "C" fn(*mut c_void, *const LjLogRecord); // ── Plugin context ────────────────────────────────────────────────────────── @@ -138,11 +148,11 @@ fn emit_record(ctx: &StressPlugin, cb: RecordCallback, seq: u64) { let ts_val = cstring_lossy(&format!("{}", 959727165693860u64 + seq)); let attrs = [ - LjAttribute { key: svc_key.as_ptr(), value: svc_val.as_ptr() }, - LjAttribute { key: scope_key.as_ptr(), value: scope_val.as_ptr() }, - LjAttribute { key: mt_key.as_ptr(), value: mt_val.as_ptr() }, - LjAttribute { key: pnr_key.as_ptr(), value: pnr_val.as_ptr() }, - LjAttribute { key: ts_key.as_ptr(), value: ts_val.as_ptr() }, + LjAttribute { key: svc_key.as_ptr(), value: svc_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: scope_key.as_ptr(), value: scope_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: mt_key.as_ptr(), value: mt_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: pnr_key.as_ptr(), value: pnr_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: ts_key.as_ptr(), value: ts_val.as_ptr(), value_type: LJ_ATTR_STRING }, ]; let record = LjLogRecord { @@ -152,6 +162,13 @@ fn emit_record(ctx: &StressPlugin, cb: RecordCallback, seq: u64) { body: body_c.as_ptr(), attributes: attrs.as_ptr(), attributes_len: attrs.len(), + event_name: std::ptr::null(), + service_name: std::ptr::null(), + scope_name: std::ptr::null(), + resource_attrs: std::ptr::null(), + resource_attrs_len: 0, + scope_attrs: std::ptr::null(), + scope_attrs_len: 0, }; unsafe { cb(ctx.user, &record) }; diff --git a/plugins/syslog-ingest/src/lib.rs b/plugins/syslog-ingest/src/lib.rs index ebc2baa..603c78b 100644 --- a/plugins/syslog-ingest/src/lib.rs +++ b/plugins/syslog-ingest/src/lib.rs @@ -13,6 +13,7 @@ use std::time::{SystemTime, UNIX_EPOCH}; pub struct LjAttribute { key: *const c_char, value: *const c_char, + value_type: i32, } #[repr(C)] @@ -23,6 +24,13 @@ pub struct LjLogRecord { body: *const c_char, attributes: *const LjAttribute, attributes_len: usize, + event_name: *const c_char, + service_name: *const c_char, + scope_name: *const c_char, + resource_attrs: *const LjAttribute, + resource_attrs_len: usize, + scope_attrs: *const LjAttribute, + scope_attrs_len: usize, } pub type RecordCallback = unsafe extern "C" fn(*mut c_void, *const LjLogRecord); @@ -35,6 +43,7 @@ const LJ_SEVERITY_INFO: i32 = 9; const LJ_SEVERITY_WARN: i32 = 13; const LJ_SEVERITY_ERROR: i32 = 17; const LJ_SEVERITY_FATAL: i32 = 21; +const LJ_ATTR_STRING: i32 = 0; // ── Plugin context ────────────────────────────────────────────────────────── @@ -230,9 +239,9 @@ fn emit_record(ctx: &SyslogPlugin, line: &[u8]) { let appname_val = cstring_lossy(parsed.app_name); let attrs = [ - LjAttribute { key: facility_key.as_ptr(), value: facility_val.as_ptr() }, - LjAttribute { key: hostname_key.as_ptr(), value: hostname_val.as_ptr() }, - LjAttribute { key: appname_key.as_ptr(), value: appname_val.as_ptr() }, + LjAttribute { key: facility_key.as_ptr(), value: facility_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: hostname_key.as_ptr(), value: hostname_val.as_ptr(), value_type: LJ_ATTR_STRING }, + LjAttribute { key: appname_key.as_ptr(), value: appname_val.as_ptr(), value_type: LJ_ATTR_STRING }, ]; let ts = SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default().as_nanos() as u64; @@ -244,6 +253,13 @@ fn emit_record(ctx: &SyslogPlugin, line: &[u8]) { body: body_c.as_ptr(), attributes: attrs.as_ptr(), attributes_len: attrs.len(), + event_name: std::ptr::null(), + service_name: std::ptr::null(), + scope_name: std::ptr::null(), + resource_attrs: std::ptr::null(), + resource_attrs_len: 0, + scope_attrs: std::ptr::null(), + scope_attrs_len: 0, }; unsafe { cb(ctx.user, &record) }; From faa934b0644fd2617aa89f1a83e14c9445a9b368 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 00:58:11 +0200 Subject: [PATCH 09/13] Fix memshow demo --- demo/memory-buffer/run-demo-memshow.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/demo/memory-buffer/run-demo-memshow.sh b/demo/memory-buffer/run-demo-memshow.sh index 3516a35..505a2ef 100755 --- a/demo/memory-buffer/run-demo-memshow.sh +++ b/demo/memory-buffer/run-demo-memshow.sh @@ -7,7 +7,7 @@ LJD="$TARGET_DIR/ljd" EMITTER="$TARGET_DIR/otlp-bofh-emitter" COLLECTOR="$TARGET_DIR/otlp-demo-collector" FORWARDER="$TARGET_DIR/otlp-wire-forwarder" -CONFIG="$SCRIPT_DIR/ljd-memshow.conf" +CONFIG="$SCRIPT_DIR/logjetd-memshow.conf" for bin in "$LJD" "$EMITTER" "$COLLECTOR" "$FORWARDER"; do if [ ! -x "$bin" ]; then @@ -39,8 +39,8 @@ for i in 1 2 3 4 5 6 7 8 9 10; do "$EMITTER" 127.0.0.1:4318 --once --message "this message #$i must be kept" done -echo "sending 7 BOFH flood messages" -"$EMITTER" 127.0.0.1:4318 --count 7 --interval-ms 0 +echo "sending 5 BOFH flood messages" +"$EMITTER" 127.0.0.1:4318 --count 5 --interval-ms 0 echo "starting collector on 127.0.0.1:4320" "$COLLECTOR" 127.0.0.1:4320 & From fc40182015e60c4b630d1cf7feedbd8bf28151f4 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 01:04:08 +0200 Subject: [PATCH 10/13] Add remark that TLS is over http :-) which is correct in this demo --- demo/remote-drain-tls/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/demo/remote-drain-tls/README.md b/demo/remote-drain-tls/README.md index 7930804..5b16722 100644 --- a/demo/remote-drain-tls/README.md +++ b/demo/remote-drain-tls/README.md @@ -118,3 +118,4 @@ These are demo credentials only. Do not use them anywhere real. - appliance replay listener uses the internal wire protocol inside TLS - remote `bridge` forwards OTLP logs to `collector.url` - OTLP ingest and OTLP collector export are still plain transport in this demo +- this is intentional: `remote-drain-tls` secures only the daemon-to-daemon replay link, so the remote collector still uses `http://`; if you want HTTPS on the collector leg too, use the `secure-pipeline` demo instead From 903560f60331c00026e3d579cccbc0323d144a04 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 01:06:55 +0200 Subject: [PATCH 11/13] Remove demo noise --- demo/upstream-reset-resume/bridge.state | 2 -- 1 file changed, 2 deletions(-) delete mode 100644 demo/upstream-reset-resume/bridge.state diff --git a/demo/upstream-reset-resume/bridge.state b/demo/upstream-reset-resume/bridge.state deleted file mode 100644 index bebe233..0000000 --- a/demo/upstream-reset-resume/bridge.state +++ /dev/null @@ -1,2 +0,0 @@ -stream_id=1772129127930933566 -last_seq=7 From 4e470348d6988056117b47e3be04c030f74b279c Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 01:07:20 +0200 Subject: [PATCH 12/13] Ignore demo noise --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 69cd3c5..1ead1be 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ TODO* .* bsd* *.logjet +bridge.state From 59a132560bdb97e4839d2a8196a4f7365e19f4b8 Mon Sep 17 00:00:00 2001 From: Bo Maryniuk Date: Tue, 21 Apr 2026 01:20:51 +0200 Subject: [PATCH 13/13] Update main readme --- README.md | 111 +++++++++++++++--------------------------------------- 1 file changed, 31 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index e54a4c1..95bab8f 100644 --- a/README.md +++ b/README.md @@ -1,93 +1,38 @@ # logjet -`logjet` is a compact append-only binary log format and Rust library for storing -and replaying raw OTLP protobuf batches on unreliable storage. +[![It is alive!](https://github.com/tinythings/logjet/actions/workflows/it-is-alive.yml/badge.svg)](https://github.com/tinythings/logjet/actions/workflows/it-is-alive.yml) +[![Unit Tests](https://github.com/tinythings/logjet/actions/workflows/unit-tests.yml/badge.svg)](https://github.com/tinythings/logjet/actions/workflows/unit-tests.yml) +[![Integration Tests](https://github.com/tinythings/logjet/actions/workflows/integration-tests.yml/badge.svg)](https://github.com/tinythings/logjet/actions/workflows/integration-tests.yml) +[![Insanity Check](https://github.com/tinythings/logjet/actions/workflows/insanity-check.yml/badge.svg)](https://github.com/tinythings/logjet/actions/workflows/insanity-check.yml) +[![Licence: Apache-2.0](https://img.shields.io/badge/licence-Apache--2.0-blue.svg)](./LICENSE) -It is designed for telemetry relay and local persistence on weak hardware where: +`logjet` is a block-oriented binary format and Rust library for storing raw OTLP protobuf batches as an append-only stream. The project is aimed at telemetry relay, local backlog retention, and later replay on systems where sequential I/O is far easier to afford than elaborate indexing, background compaction, or large in-memory state. It is also intended for the less well-behaved parts of real deployments: links that are intermittent, slow, lossy, or simply unavailable for long enough that local backlog ceases to be optional. The emphasis throughout is on predictable writes, bounded reader memory, and partial recovery after corruption. -- files may grow large -- writes must stay simple and fast -- reads must be sequential and streamable -- corruption may happen in the middle of a file -- later valid data must still be recoverable +The repository contains more than the format crate itself. The `logjet` crate provides the storage format and the reader and writer APIs. -## What It Is +- `ljd` is a daemon for ingest, retention, replay, bridge mode, and file replay. +- `ljx` is the offline CLI for inspection, viewing, filtering, and export. +- `liblogjet` exposes the relevant pieces through a C ABI for C and C++ callers. The demos are not merely decorative; they are small executable scenarios for retention, replay, transport, plugin loading, TLS, and exporter integration. -`logjet` is: +The storage model is intentionally transport-neutral. A stored record carries a record type, a sequence number, a Unix timestamp in nanoseconds, and the raw OTLP payload bytes. The format does not interpret those bytes beyond preserving them faithfully. That boundary is deliberate. `logjet` is meant to be a reliable persistence and replay layer, not a query engine or a general telemetry warehouse. -- a block-based on-disk container format -- a Rust library for appending telemetry records and replaying them later -- a corruption-tolerant sequential reader with forward resynchronisation -- a transport-neutral storage layer for opaque OTLP protobuf payload bytes +## Format Overview -Each record stores: +A `.logjet` file is an append-only sequence of independently verifiable blocks. Each block begins with a sync marker, followed by a fixed header, a small extension containing the block base sequence and base timestamp, a payload region containing multiple records, and a trailing CRC32C checksum. Fixed-width integers are little-endian. Compression is applied per block rather than per file. LZ4 is the default codec, and `none` is supported when compression is undesirable. -- a record type: logs, metrics, or traces -- a sequence number -- a Unix timestamp in nanoseconds -- the raw OTLP protobuf bytes +Within a block, records are encoded compactly through deltas relative to the block base sequence and timestamp. The essential fields are the record type, the sequence delta, the timestamp delta in nanoseconds, the payload length, and the raw payload itself. Sequence and timestamp deltas are stored as unsigned varints. This keeps the common case compact while preserving a reader that can operate in a strictly sequential manner. -## How It Works +The format is organised around recovery rather than random access. If a block is damaged, the reader validates headers, lengths, codec information, and checksum, rejects the invalid block, and resumes scanning for the next sync marker. Once another valid block is found, replay can continue from that point. This is the principal reason the design is block-based. Whole-file compression would make later valid data difficult to recover, and per-record framing would push overhead in the wrong direction for the intended workload. -The file is an append-only sequence of independently verifiable blocks. +The result is a format suited to persistent telemetry staging on unreliable media or modest hardware. It does not attempt to provide indexing, ad hoc search, or analytical execution. Those concerns belong elsewhere in the stack. -Each block contains: +## Using The Rust Library -1. an 8-byte sync marker -2. a fixed header -3. a small header extension with block base sequence and base timestamp -4. a payload containing multiple concatenated records -5. a trailing CRC32C checksum - -Record payloads inside the block are stored as: - -- `record_type: u8` -- `seq_delta: unsigned varint` -- `ts_delta_ns: unsigned varint` -- `payload_len: unsigned varint` -- `payload: raw OTLP protobuf bytes` - -Important format properties: - -- fixed-width integers are little-endian -- blocks are compressed independently -- default compression is LZ4 -- `none` is also supported -- CRC32C is computed per block -- recovery works by scanning for the next sync marker after a bad block -- a reader never needs to load the whole file into memory - -If a block is corrupted: - -1. the reader detects the failure while validating the header, lengths, codec, or CRC -2. it treats that candidate block as bad -3. it resumes scanning from the next byte after the rejected sync marker -4. once another valid block is found, replay continues - -That recovery model is the main reason the format is block-oriented instead of -using whole-file compression or per-record checksums. - -## What It Is Not - -`logjet` is not: - -- a general-purpose database -- an indexed random-access storage engine -- a query layer for telemetry -- an OTLP decoder or validator -- a replacement for object storage, Kafka, or long-term analytics systems -- a whole-file archival compressor optimised for maximum compression ratio - -It stores opaque OTLP protobuf bytes and focuses on durable append and reliable -sequential replay, not schema inspection or analytical querying. - -## Library Usage - -Add it to your project: +If you want to depend on the crate straight from the repository, point Cargo at GitHub: ```toml [dependencies] -logjet = { path = "../logjet" } +logjet = { git = "https://github.com/tinythings/logjet.git" } ``` Write telemetry batches: @@ -151,10 +96,16 @@ fn replay_batches() -> Result<(), Box> { } ``` -## Notes +## The Rest Of The Toolkit + +The format crate is only one part of the repository. `ljd` handles live ingest and retained replay. It accepts OTLP/HTTP and OTLP/gRPC log traffic, supports plugin-based ingest, keeps backlog either in memory or in rotating `.logjet` segments, exposes a replay listener for downstream consumers, and can operate as a bridge to another collector. `ljx` covers the offline path: inspection, viewing, filtering, and export. `liblogjet` exists for environments in which the surrounding software is written in C or C++ and wants a stable library boundary rather than a Rust crate dependency. + +The demos are useful as small reference systems. They show file-backed retention, memory retention, late consumer replay, replay handoff, bridge resume, TLS on replay links, plugin-based ingest, Parquet export through the external exporter ABI, and shared-library use from C++. In practice they also serve as executable documentation for behavior that is easier to understand by observing a working scenario than by reading a static paragraph. + +## Build And Explore + +From the project root, `make` builds the main release binaries. `make demo` builds the demo artefacts. `make test` runs the full test path, and `make check` runs clippy through the Makefile. Readers who prefer orientation before compilation should begin with [doc/README.md](./doc/README.md), then continue to [doc/overview.md](./doc/overview.md) for the system shape, [doc/configuration.md](./doc/configuration.md) for the YAML configuration surface, [doc/features.md](./doc/features.md) for the daemon feature set, and [doc/c-cpp-integration.md](./doc/c-cpp-integration.md) for the shared-library boundary. Standalone Rust examples remain in [examples](./examples), and the scenario-driven material remains in [demo](./demo). + +## Final Word -- Examples for standalone usage live in [examples](./examples). -- C and C++ shared-library usage lives in [doc/c-cpp-integration.md](./doc/c-cpp-integration.md). -- The reader is sequential by design. -- Compression is per block, not per file. -- The payload bytes are opaque to `logjet`. +The guiding preference in `logjet` is restraint. The format is compact, the reader is sequential, the recovery strategy is explicit, and the surrounding tools stay close to the operational problems they are meant to solve. That makes the project less ornate than many telemetry systems, but substantially easier to reason about when persistence, replay, and failure handling matter more than decorative complexity.