From 595fbc0c8d5867fd605c3b475efd7209521cbb4d Mon Sep 17 00:00:00 2001 From: Tom Baeyens Date: Fri, 5 Jul 2024 17:46:27 +0200 Subject: [PATCH] Contracts v4 soda-core 3.3.9 (#817) * Contracts v4 soda-core 3.3.9 * update image for experimental support --------- Co-authored-by: Janet Revell --- assets/images/experimental.png | Bin 11722 -> 13652 bytes soda/data-contracts-checks.md | 5 +++-- soda/data-contracts-verify.md | 35 +++++++++++++++++---------------- soda/data-contracts-write.md | 17 ++++++++-------- soda/data-contracts.md | 15 +++++++++++--- soda/new-documentation.md | 1 + 6 files changed, 43 insertions(+), 30 deletions(-) diff --git a/assets/images/experimental.png b/assets/images/experimental.png index 1d9b4cda2ebf218f5aff8376015d1af786571677..a87736534b9876947beebb4e4afac95025470f85 100644 GIT binary patch literal 13652 zcmbW8Wmp_d_opEcf@^Sh4Ga(qIGkkikGDL4|{Z!;q5&s=~p&5Qm-XAtS*)-M*`Q zgeCB|Lx835f3AIc z@t6Vw>@$|adj{F)4W$*`cgL|``y>;SILP;D6+^jJadWD z;@(L#@*L4rsK0QAaK+&9nS%qPUqEoc3!%x#p%l6Y;JbC7b!fqPmvGKUK2Oi{vr6c@ z{i~&JdyBJgj>}nUd+BXValH<8wyNIYl}t+;X>C21ZT4>?4M<;v2T~g&6a7h)5s}$k z@g1UB%f0%mIeOPHdW?ND_mi)`+~Z6bSbshay?Cf!WmVj0#gpvA1|Xut|C2;h2DyMD z9o%=(IGjRcstVtwDPW^3r^HW5j=_O}ugYybljyez@c-y4!pz8FU2 z=@DqgCH}92#axdh9d$w#p7x}R0IF|AQ2AUG#EF~O#FX47?ymxpf*-e z-mX+*P5SFiIQgu)7;o8SH%5NVUa%UTgqIk$&&+O!LWNXBZb0 zN%7o`9u(|J6jCy1bt?uB2z^Lw@MGTGfbaD)fXkyFMj=HH*7b>K5{)`%aZ2vXqi86&se#TRCj@*o6vk zxnRQ@9ADW-%LSb=9@8}!KL48+cXO9fq|NR2dTy@fih9rk75%ppj_(T&(~h{C0v_RC ze@_daH!bBglqFMCOUTPyj<1+-r-ZF2Q`7loH*BTvO z=;+kPzgH4$X2UztJu?5k?R{uk41SV67SaRLJ$dHu9IG+UMqgq)jHNS7g^%CQNDSi~ zOr$lg7G2Ii=l0gUvCf~x+6C*SXgc`xm#Q2LE_|ThKx=yc?3D*L=WcU)H(OVy?xfG|aX<-FcbA+9t!Sx3;IJWB@ieWjcLt zJ$K3M;P-t7DCyuEn{KM}peyPx%;?S|%7f`hR(bis;F@+l6{FK?y~zl8zVB*~=R%+7 z>Q*&5Uqc)M<+a^!oLU?bB9zhNJ>Plcu|21(uGT&`)a87`u08DYkyD(|ZFw?p8B`e? z6^13?|1k&ZJmT0nxZ<$qIQ#LHQrAdzvI0vU1eDlWJmUZCviUgga4dIsf2Dt}LuW#dg@<|NHUF!1&HQS( zqwU%u)$>W=lDGtKC189_wM8p!1*-7m*zscUx)L$E9q#xGzO6n{M8DN|l;`IxmblxH zTfGh(1JB!xA0vHfulZ;Ugzhx%KEFJFxCgj!-0+Vzvsc?m1DqupL=OtW0#x*~1DqT6 zNI@o7C`zk!_T3z@d+{k0i&g-k)@|Oi$1|TtaE81fe1aV!?}9sJcD}+u-R$=_uKN=d zrxbd(?vJ;Q9hlBCE#yCHCG{cjJ3CvGg6|+<>;=C|Njc9I11P?y_2N9l*lvdn&6vI4 z+4=QhrNg%4G(Z2~hFB)EaNAO7pE%;De~l+@`eU2s@(?mJ<-;Di52Jo!AGqn^GJUZJ zZ;w_;X*por+$A)q+v)80s%#dU>x{X9hYQ=-r&)0CM*V$?TPk_jC^A%TqxQV4;oW25 z+(kjjsn~CQp>>1$vuc)2>nlCJbJbl#feUoT<yBLW;pIfwFqP}1x)?1v6*+_oi@*cm?vpPPe$j*oCV&G?cc=bQXu`a`A$=81>lUR zgr%|mfyCQiKEw-yE{R?Sx3udSXo3Sw*1glmzVSRNHDK5HmI?jS?2o%y>k<5!8V3i& zN}a#i*8tBhwTjlUoMCWB52!rD! zqOUeYznZD%xK*Vp5>u^*AR4o7t{(>*vuQjA0{84dZIVaqyOcBT8P{(zvA_aZ2g+Gu zXD97O+3?wKAcA0@N;L%^$D}=>>0_~0%SuHqdX?fkH0$lGEvheGsz`W55@~Kg?oP#+ znzKCrktYG&EGjoG`bZ)Jih8vzo+u$t+#P-um0(Y!)rWDSo!;-#rJWVH*Yu_DT`gTwo&< z$*N`M;k)tuUVM=cZiYNxfGR8q!u58+&?!hQ*RVGA*2S0O?`^IYy(bhW?ul5Q&ikip z!tX_{XhulQ#59xmt=Hb%Z67$k7{EXqd604P>->runr5NLZ}~%{deHbT;R|UX(zr%uA^C=kqqBZ+>ssZH9fifxCTy0v5a1 zZbW0kLJ6a9QNAAzg`~{B%;_WuI5ITgr)|@_d3`gFW!a;=yC>>%vYn}1Xf)}wrfAT~ z_C!|Ss+z0*^EK#$PXBk_Lhnvp&0KWN=`1yU7=@FNj_rod&IBp+rUuJ-cVzsn)3YT?d zV-)qy`OtM181@dVfps@NDd;`VhJL!?b8>2!52%i2ksZhFN%4uSoH}W17A#W~rQF#M z#(>mR(myFjyEv^ppWS=t-2={Lq40Vo@9@c{P!D@aBI<*>$Z5{aFJ9)c`rSR?tF5Yy zGdv!*^(wFS_y9G!wJIwrZZ`)IIn$X>^G&j?xWRbrm~R!_AG1qU@J1w-1yZn{0_(9Q z9>)f@CNvuLbyB`VkRG8TUqhpHFC>VMlfXkNg*Qz~IU+Smv+_5Qr9PQfBmz;{IEoUu zW18_Es=RdHejq$4(0wlBEJ&kNUwA^k(NBHddEi&WJz?t%8ua1OV23Dh?T4OEx=;Rc zjg9R#FjrYoQa8=`YI594V;>4Q7Rv^^ZRY;wr+a=-?N`9Yd;bpE%=<#}eXpAg2l+uF z3822{MTbX!=NYsJGq^5D+$1%sj=#@e2IUMJNvedbs&6av6BKBWuDDxt7$&ohH@s<( zgEno6%ngr`u3=8?rVpK{1c*mO%?0(pFqy7;efu!J*bCN{p-nln55DtAIj>c%0j!es z^XVo8u&(GGRdCJCjx&7JkRyt?a*U$TwD0Ldd16->zycd`;7ivl3;SIQFV*~JIz6CM zk(?$rW~)8w$Z>ox38ca2vJ%aDlCSEHDOFVQ#IX=N`~V^=Cq(W{d=IAOT9hc0vdKoZ z-j>@AM619p;kN8Pc*ZaLXpCx@EC<~ck%{S+&!V*RXFXCS>SvPLkP8SZ&l38fz7YN(O|_3^)qPd4g=kEB|ekJU9qaDG{UJdobH_slHXEf5!K!e#YbEqf3w;^ z?^>(^I54LaKadiznJ_gjZ^z;ZMb5T19hoV1NkFrE8X4llTHNP4#sUEp7zLNI%dg>c zMzDZ+k0pyXBCE%pA~_kTrWmwjjDByL@;})3YLEmk4DG#^PUGAU^)S(`I~crfv}4p6 z-_&eFGovxT@J#|ZPuCB04I<9Vh!#0YF-IF*f9zS{(P3ZDGn*V`OaQcIeDho5{xMYi ze3t4Fb_xi7ZBSMG!bgmtKTLU7L0|Tp&=vrt&@iwL--~HprA@?hlC?g3P-q-zqy5Dd zsbU#CT}HgU03NDP(qbGzMkrH7+M=BI_@orx|1Kw$;jnXSKv9kub+FAkA|6fapj#Q; zZ>1>V*N%j)u+(K(TRS_l#=MeLT!owTNPoUud%0cPdF<+l2V3fZ_2N(DK|IXwd}$b0 zDed9?pNMlX{kwagGuCV7O{wjL*eBst3DMoKcaiDe>cXA43}`@6n9JitrabhOdswQu zfMeJ5+46(ucFspf#FYcC<|iuQxoJCnsb);H#0Ak@%aX|r68pU>iTthl zu;XF@fzIdUQ$Db#V9=E8NkC~6L!7>gGlL$Bhk`RCp+SfV%51x$y9osnM7%_neiSO# zDoMFgj-qECaWDGVcN@JSGq5Ti*GxHY)^1}v`H-==*i9XeifyAg_bA(<(Gm9}*x!%| zpAAJo4ETdw)lhsUW0rWJKQs*ol0U-BKlDmU9|N@G_B;7kdn@gVM$MQULTB8&4i!7N zB5C_jJ!XXG-}LGWb}R(@BWMW4sR{k~c-OS!J(sF%VwAjb>nOU*l@u*x;g*hkwEjufuxpN86!6>!ohOB2lwf=_0)^ zOvFk_jWo81F2s0C>6Iu(^v;k@x?b)8-a5j@DQbpEbeX_C9vs@CHq}jcsKKX@%uC@X zqfa&`A+N+ykecpvC@gHEkJ>1qlX%f__h#rFsFjjon;$q44xM*l2C`3hp;qCza-2p|G-*)(*n_Ps*c!;Si?s z5zA};RrenIy!qR}J#vVgaF$fb*T&N!X{@mk5rCF3`G$Xto1p1r;8GDq(g1Zo*wD2+ zBstx*bTR$M0j{d#p$yMFN0S|oPR(+*tMO_8%|LQ1lze~0YcFfs0!#8(-@SG>PLSXm z;-{8r1`*=Ei0P`SJVmdvQMsB<}IYMG+G=q#>! zm&$S!PAv=0WD66qjaqbUXMfs`ZRO;QH{=5zj>}`mO`uXL^}=711f7Jjb@fsG>&)s| z7Vo(a8zq*#An!J*_%wu8xp3@r`zaBetP&O2%g7`IB%PG8z_|U9Qo-z2%25>jPlo`n z=1H`}-Ymn&9b*|Se$B$YF==NH@eXo?Z4$KBdExI+ky!a}+~UkWRLipVavnTW!lB~3 zTI6G`)w8eElP63V^?Eg};T}lo9IhXfTf9-}r(`{;?cf4(RBtYYGP*59b|d#X3$Kf_ zPY6@%8OAkoTrpe9VtaM{$@edoyCqJrD$nXW*^d(c_ia3w~_Cr;LnY@aPy| z+fV@)yh@oiFa53tu&%lGlWyIblb$(~8bwiJuwotN z4egrV@%0_lWsKps8i_~HT7C~!VUxk{-do4E(A))fE; zoyq?+YVqhF$L}zHGs<+Fu=PSOh`1O4H1q8Lj2~B(hS&s*2-~%7p@03Do{DM_?0_%X zRQ44x`~HHduaQb#Vfr`D;GrEl3{^C7vxW5N0EBT`LW3G9eASD5En=^mK#TnHaq(27(lTvN^stl z@K|u5RkJWI!m#sM0LU*fjInLSyB@L}b1f6qQO(K_>Dk~p!!bmp#`q%Eq!A-Bd+H!b zxR4!Z(80f|4J%y$D)-qQLgPQ@Kom5KB#EGxMjAC*ywW>OBi$^@{dPCuSjVAHgEE_n`CTpjJ zm&SgCcdMy7dU`M?ae%LqG=OMNjf+brk}4g2r(-|6VFtqCI}>RiEwpAffC$mq9~X=n zzCSq+UrwP^{c%W4v_8Fegcp^8WnRHJcPtMm+|)#8>!}kRKqExq2|v%WDq5_hXMS~P zFlX)KDA4B{h}!f&9ps04%C73Ft@q@N8xGQ^N64d$0OD_D1C~sbvHdR#9vif$I%301 zd7Ad}PNQqyTiwJ`RaJZ20)8UrN+h?M#KsaDBehbHx7LknzMn_na^fx=2x+T*k<*A2 zutRw|KR~KXj)Jw9w&W-2!hlCrn0i{r&0Y@&U@?ip7?Vr@E8q3a(h0}F&v{`>^9FgO zuPqIjAzPZ`S?fUIMJZz&y%= zeBsfHgz$GP;5f!OZo|uRw<$J9mtz)^a6S%cpsUYh`_O+z1Bo7t&G zpn;~Eg7WoRm|=j7`hFoQThwVE6Uf3im@1Qso~JGXWndB$H4l^I@mXxAb-HDeeePuB zWxW=bseGz&Y!V#nj7+j>skeklvFl6gdP3sfcP5)@aVRD#sbI1$CB9Vio(|Q_Ps7Uz zHVyi^j6A~oRvl!7pAl(t_qm7hMDXkr4J=SqCZJCHmZc1anggF>V%X1Gb>TGeJ1QM{ zj5M}qMmeA-O!XSRJ)sBsS_K_1-)#vRsO;w$Qg7Dk{QU6fN^rERDl0IAq5%TDi!2>9 zvIrTEBR&iIIf@s0$AmUr$q=sEPfeRG#vbvWMB z=3}phf2O4er9pu>@m?gJRbCZVSvuNcv6j9xp9|Fr?VmbcZbPIxmBDn6QP|y4EXN6J zx3Vj!H1z>qIt?1$E#3m3+&7vqUMB}SRn53oR}PvdMZYoO9n-+fWcSF{Qnk!9Ig$Ys zvn@Z&>^G{|r&TiS=h{8gMp?dmAM6w`CnwBKd$Jm-_v_T0bk2Bz-FM2dLCTAW`ldm) zh#3teGl_P!KLwE3kT!I&OU1JGu4kgWZMR~cpVJmJt zYIv{}7%qJmvn> zQgsVBlAMpYj-H~O7Afu4Z@cGk(Ft-4) z5n4J{_QsSHMPwR2_FVHk?I7f2fzYGGp>LNET^Z|r`jO;cz1KW%W%~tiCNMtu9J5A> zM~oOQH{Z2nm|Q*?l~VrOhAg(FK8YAo!4@Za`WoLFx5z$3(u($-_imG$dNUfTusWbD zp3)vNLoy^+XisiISghXT8L4u_%9U=6_0=M3?a3nQsVgp(;IfFnc37V5W4@AbSWJ!4 z`=_vES-*u!zux^jbXafBw5)xh7 z23W$2I1Ly$_dj(nvuO3WerVy3SIo-dwvB;}>6zFM<}@U%fHYK-b$s=O@Hhez#=c5c`!q*UUED|1uujTFZUBJS{7B;z-1www&i+f=cS_8 zF-QhKI{j)C0W=7YvQor^03=!;tRT9u4ACY-p2 zwj%WZNHZb>@eQ*O4jerFS~6WVps^1uW7Hf}F%dJ?av0O11p15w{>*mhe7FRSl#m*P zKY*}>J%+jKZ%D#p@~<|f&4UkU3+LEKcn{r61sMHHpL3WoA*1J0zU&_75b1CMakJP;J+b(X)-nOKDLJv)66qHzUb6>^*DXqaLwAS$RYW(k$e zkk+CM9rvy2Hsabvy+}F1v#vf{OlB9Ma<3y_BidH9gg( z=aX(`o)#yPKS@?SF8xUDmy($C*D3KIo(c0o)F8sws+@}m*8XVr$BRv;?0$%(S=0FW zZlLR>X@`TMYW5DK5#kHe?UQei`eShJl?x zt#7=P65*v1PD}Dxq^3S;N%dsKJCX!pk8O5GmGWbw=l3@+U@v1XQQ&al6=l-;>G7Q> z?N<7+-}btgb8WKKOXqATJUjltbqfrk!4>K==2&Zyxy11_CH~}a9tI|6Rf5U?a2u;1 z$e${9Rqv;h-ID9oc`;D&QYl z70EIi{V#qtiUXrn{F(%i|5yz`B*FPZtBS^&ivDG%%O6_x|J}R|OxA`mI~~XnhV7al z?*i-CLWj}q*WR85vA&_*gtMQnPk}$X4p`EnXJIKf80oYnoB!ufdXkO0h z9HQ&yA=q?q)?&Y;;bgG8uE#Ud&Dz_}Ny!-deFANqu`?QDl&7unyy@WR)?sbL{(Duo zo=W|sqCx97;ZD0`sQ2j>QpN(9xCpanzANQ;8Jw2W;7|^AnV@NP9jB-%o)f&@iZ0hV z7CIBQm(OVz*1fq*u&CABTsIQtW7jTj*Rl#Iw}tDmzENGc_g7oDR8UTuce^XCUUTin zn#w*#;fl%=gyP>-DGa!aZ@?CJ_A_0@0rN1))KOVs5o_7;u;y4~&OJqrQ-0PABIRFv;rA`I5b`>fy?SS+5hamtU`jWR)*_bYi>wFmG7+MBm%0Z6)~$L~We>DO`i{Aw-jX~sU9{udmaSyeFPH%0(0UDF2(6eaptMYIb>@d zpIYm!SUqrD&D@`o7u92RIMoIIsZFO+G$BnTGN{|8hFy;#-VygIXQuJwt^>h1WG^%HrvCNvWrzCWA zh;7%#v1w*a?Uq-KzIs7x^7?$!(GkSeiL#AS*r1L=0U0i2K3rQ+B}4$faolLg!5X1_ z5Y$}$HUgNlnQsH7z4+C5aP6gVECIwcp$i@1#*rzbq}e*Uc?mWqL?YqLR& zgF|6cd%Y^}*yr78CyY&Ps{j-zt0{t4AalhWpV?-g*K149kwur(*f(V}I;qo)8#sD` zbw=4ojz4fqDao;<<$WfAOGor-WSze!fUsd1N%_2tD*S8cpqb$sJZ^YoRJG)lyqvTa zNX$5x$PLDXR`V8nT!PVJCi;Ajs**I2oESba^77Ate2E~A9q2gHR@s7fk12+@m$eko z@PswW&eP4>I~){*&6YHj9~X+4&VD)e#;J+mU%GZSHHu-~@^s9U)fC6uPsJna>t#1_ zFy>6~UZA5l{gnRTT>$}1upbxQ%~SCqKw5Sk!aSmk5xqS$PK<@WhEu_i_#>#|l4eRD zu&Tg=8u3cbU)}$5?(eilVnUB_v(E)2C^-BW4D^A?v;$P`-uX&UP_mYQ_!#PB0!Pjd z&1leDx65xy&!0i>D5g7VJKOErdfnG^XU>W~taSQ|`_&xlo=s*yC0io-Q9Zn0UY7@Z zsK`l_?e_3oY&_=*$k0awafc2a;ae+BACe<=vVt+hEhoH3pA422WXs&|4McuV#42DX zi!9&Oeb>+^1Ybosbaskzcug6+y_*Lt#Z6lG%D$i`f-;sW$woXBE*7oPW!4j+99C;I z`faG(nBG_MPV(U}q`X8|?X^?43YdA0ZiMq=73k?JBV+Z@0r1PS*(j@xlqWNeZyo$N zR01C>&L)szadC<-9Z6jo%SejhFwNr$e#d%&ewl)lyES|({C$GRsiQ!ItMZp_K1)4^@Te#wZ(>9glvre6)ls9qDhY@QV#fRR;v!POAl6JN^CDNPI*9-s zLQew^O7jaT%gpRbJ=3|&>W%n}bIX|E;Nk=uI+q?RyE1-`!C7}5B*_tkXfjCBGx#vD z6H%%A%DzhcX$%eWjvfV&g;Hu?hDo~<`TBPpVyzM8b{e7{t%^E!>ZCcnI%_)G;Kn25^M*p=2~Y98ptD`c#Bi)T#>B=yo? zhO_^P0@_K@BO#kwF5eM18Y-+v_peeNeggtPV{pRsr#n9nl}d=^ zhyagR=-BxlfUgQ$)7VKv@n^1Xx{5i>PuXn&)*0lmZ4jS3rod|(Gp4<{a(gRA1BgkN zkh{d4U-mO2m=;JyHOd^U<%9617!>I+W-&ody}0&f8wuFk`|gXj4Ous=V-D#l-F#$>}dNzH&n6zT`0 z3rKB?lvFU>$h-B@4I8c?#K*=>i81^6kSA-dtWp;os?6DMrT*a^-YQ8_TN9i8)<%Ct zy3v$j_3Sz*AQ``@0{6pdU7FTN5@x=@$ar)jKI69;G_FGvwUj7jrBD7ZVt!ZWZeICV zjPzsPpt@xPEIj3Us5EcXkaG7J^Ts<)cFypyc-F5-lX%@>BJO)k<4;sXKz~cm)nz4J z^hLt)Xi5~4JA6Ul<>t}64CZqw80n+KrnHe!sv{I$4eWJyBLsLe(T{}eL*1{ zaB%acg|v`A``Fs{E!jer<>^qw{}lFqnW)*24q{|YK0 z&2*zIMEY&4z=3SHx@(tu7$4O>4-}j=%4WTBTF0kflL~YW^n{ZNP+i zvI^?!LRm16hDBK&REUA}AUzRbCW}!v*dRoc(zA>MVl8RID`hL`7;Eho z?r<9s zJ-Y3FxL9PMT(HF3qs?D(OLgZQ4KxDMNJi@T0feJKk=GxB?Y!=y>|Wfb6VWA?E`JO? z%AP`)$3GY63C7`q9$2;UaDC!zGxZrnEtuF>PJ7oHVkwsS4pi4r%_7&SWXu;V?k1aJ zs=2u7inMbeA_LpyvK0c5E$SHuR*@uGFoE3S7V{{1rCzlz%{_r`s4d^ zN7E@Gb#<}mdsR;T=pn8pOs*w&$6MM~&>&~YyZp}j=R%{SNC00nAh z1n}KJlmW4fy-cVaWIYkvxKTn-GCFkbL+hJ1UAmM``E}znkVn#MW?_P|zH37W?M);Y z@o+|yW>gKuxaMAAS-y$>Bt^mpu$%;uJAK9P?+jdBW_Ua}H)D;6rxBJMsD#w$pDQjK zDOa*8;*DCpeY#Msl`rd0O?;QXa3R|lq(DqnJ-{A64EQU5!6}eoAAOY#Lz2Lv##G-u^`n=8i%*ji-z(KY@rPE#P z(5@!bF?=h?TWu)$H1KIMG?orgBFw4TF0#xmWHHJ3JV=0ZHH0}A{jHq=`IWtBVLkND zLTKB*M;2k8P;?GAG|26SBAs5B(V0hC)_OrY@~T(mAg`*qD*4sE7rM$x=BTY zZerJZ;|h;G#df<)M2bgKZPyb7X?nAzsVaBrveEfo5ueHi_L)ZSS0nkSFwYNfo=0?Y=z#<_ep^y^_;FK@cMT7QmN%mnBzAb$kMfQ+B)4efu6FTa9TT#P>bWFqwYDQl`E1y!T% zFsC1-VrZ~b_|SjDUaP=JsMdin>d;?}@}^MY@D(3Bo?vcb&zmQVPU<-+j3@$mf7C+` zT_kYnb+1$5V7|J%QAx|8m-TfWc+BV&IdBczjmrUjWaZ1R&UtTLgN(tjomg10i?g!x z6fFF--+o8IJE&vy`5kUQDWDo%tAwg`=pN(NvKleG=Z}^%YW~Y5>O7AX2yzQsF>4(h0-fft73n{@>M2 z*Oo>qn|bu%x6=zn4$>9y^*_YOrH#b$BKmJ4o|LXn)qAD(5O@Q7C7>}Dg8!$$GSN9ea}paO^CRF?Ax#nL`< z@8z&I-`houSsDDON?u^phu4mUuZFZq%HdSGPR#D5qUw%!?6a6sE?5;v+U8yv&>jhc zeeJmWhGP2#dztDbM#3xp@g-)RuL4+nKRT!vmxbkk);fXc_?VG1k=JZFS7qBmaKMQZ z-t;citf;^)K1!IJSLP9OYp~0VW~+d2KdtE~@eQRDmJFBI;W=c~wIDQq8#8!EIgP|f*d}anJ zx{I7`q2oTwwkS1i@g99Wau(2t1CX2M13%YC@vG|>FVSHS+`d3UP8*pT3QYD>g{^ln zaXSg^EVaLYXkCM4y^I5$+AWm@9dB7lR!KpoY~;;^)NrI|4x$hK=MBEKfDd-D@7#_C z&w7IdXw}t9@FtvGF2513Dur3AXvA@em9598K1mNenJh|q^Wn5xKDr^FJ-w3WcTT~I zk|R?Qu6&yQsXvIF&`vDoX|wPlxHJC{Fbz1;E06SkMzkS5lwFjdKC z)-#laoi4H?0Q)p)Ei!Gu$(y5q79F1jpx!=5?XyJ2js)@Q$RoE?!Xfb*DVZ`Hd^cej(V#iOc#TJbiRT9W%q?eKzezv$tGz&De)CEWM7X>!!}39}3?5Op5Pg<7_L} z6NJ!f`pK6q{nQUyEO}>>Q?%JYP7HLG8O9cf*Fo8C$iU9o08zfxV-UaX9Bsi;CL-)= z!B4}fTpJB0J|7VKRnUO4`dkei?cv9QVl{?4M{km?RESKl*z?!z20xgljA!S6y0Uh3 zk0Nd1x0^;z8HjMdxnknUdIga-K;sUjZmo#0y(Di=(u6#7dH%@q7vB|D=zWZsLVcpJ z6HN(|D;<65GoVOMHi_pI9h{IvtxO4!0Ni#*5s!dXGtzIPVFhiHQf--uEn_x4RS3o=z9 zC}VjeW+);5MR=ucT9+lfoNzd;uB(l85bB&|^XB~f$Iz{+xd&)!}7t7TWj!6fwg z@M+xt2B`dFNaiP#DxUGizjC=cDNK>8z$m2u-@z|`k}r|4vj4Zi!-yKB67T;V@`3}O z`N{lCrt`lmBziF~W~@_Q>7-VX6ez{>Z+q^sv7bNBrKxu*`|`Jv{dE$|B^ZdIznR^S z#~m~1#wwFy|Ic!n*h~?yUqd*EI9TfSVwm^Swf;T{03xb1T#w9sf&6*hWf`LxGep0D-}DsP*w&}Vd+Rks(&|4WPJ{DUxwD~eDy0&YF84R@SA-y!sr|n5zB}YJ zyVoCoxrZ2}K)Cg#MxQU6YK0s-Rfsx4f1gFT*r1GY)?M%j3OcEkLg0(XfDicDO+F&; R7ym>w$w_?x)<_r!|1WE=&~5+# literal 11722 zcmZ{KWn5cNuyzX+cc%n*4^W)q65J_LBxrFfP>L0TdvJGmDNww)y9X;!+)8mMaKrz1 z@B8ijaB_C`?9Sxu*`0aj*$r1$mB&JVi~j1>D=Y<&jOMFXuR-wg7gS{UZycE&EByA_ zO;cX#RrLhm04~T_>M2;MsJwc4fcolnsLd;+mnQJ*E&O`*3NiQfD@6GH^-EnY!v8&e z4a!CQzw)auFAdkFKw+<5alBKIk<|Y3`Xmb_lTh}0Fp^=9;iRVrA4vlV?+rIb{x@di zsjkg--bm{hYr*HeO|Ou)3(e7rExW8xyglU(a65gg<%BE3zvCj3zA_f9Rxc3DtA5)fxLUBSd_oRp-TJ#<%jSVKMM4 zI}rLxR{}7P0}jHLorbq^(*q#cMYOr7%m~A{OFR>V)uhM?Q3?^Dm=QoiUeOT|+?X5< zbzuHZI4EvmSMR}F$X}4n zO8g%Fq6A)$To^m5wyuJk!NXNYCVNfqkPk2O5xlmHd6Uv*R;x|Xo%?n~U;Tp%sK7_7O*cwSZTZKl^2Ya(LfGW7O2v4h z3ftH&hpZ#&!&XNONHX`!=v zl?(E_)1}XAEb#(MlfuT+ImpAWE2Qx-6;Guto$|^cJ=xXxg1?5FK!BaMrY6mJEBi>R zD|)zUcE|%~YinRtCp)eEmbdqD((TpV;z*s>BFNvr~bS3%0v{v2$cNl1Y3T2 zqv)~C;m)(kX--VsUe9(y?@zX+siXZfsnf^%q!z1T>%2kdw&No+p$Y1&IeR8IwNuG{ zT`JKBSzgBzK-&aWmtH~Kn_A-*%YyR<_GEI4y@OaL((0q@6)Ta9>8hLh$X_>&q5ut- zf?1+SasFq$>*y1l1gaGAnRg=*YjVKbC*Hv&ho>&;iL2sOOa3>;&IjL1YW$sinYi-& zc(b{KUpIOCc{p9vWFnqSRqTkpj@3ruZ{7u%!sZCR~6f-s%t+aNL zO&3-eUAri}i=+DR{n(C_q}B0&#OgAh>)7epV3{buAAIwL(Ra0$-snfEER~yE#-p+? z9%CR8b3>B}fPNLr- zZ*&bu8$ds2W@ebj9HR__NucelkS_;r^5;%9E z9uR^tmpstpr#IwVT$k9iZ^RNjONDB|4;IpiNB7F7=N7(kg(gSNg}(Yh-16A%o{tMw zQY8fVnBrLplKEY&I^u59N4_%$x|T1c5e!Ua}(>eJZ(Ul63~W2eDku6W^L6Vg>xUp zNSc<$KepT^)K)7j?u5T>@b4An^;=t&A)oLzeT!p2SV;i!7N5Yir!*2;_Gx#_Zf?3& z)85(`r!utPfjHpsi`q|7rTExdIkAH5_jQ%(F$wqi15bSeZ%%o0I6aL0 z+GGQr{s=#bc*cFK(5nzYx6%_y+6dhcnYV-1azhB3SV@PRc3O=&xS+YM-oA*J9Wgl^{Q_>?}Vi&^;AsPKyA$Q#R1!?{%%xZ2L*jDw(=Ek~pml#@0!*%O>NhvT z7YMx>%UAEbaV`Rv^-nSEBg7gBE>g_nmM$PS85;|a6Tk1UuJ25^y=gLqko`VPNL(~6 zbtQ0HHak7HN$J=((>bbZ6Qi#5HF$-#sObNKyG8cm{3IKHX4;KQl@vEt zTOl<#GTN=c#2#ypuT|5cUREdqvr=r>QpHHU-=S?u5de#OeBOwlI*uN66@jLPzGKAm z?I$=q3pPu4`E2E(hKSh4>uI`G1gV0htQ#3VlUwaqY}BP)mQM zRwRe3eXyfaWZuEHwVz?6bTF&CwR3wSjg;LOSA+J}e$ey@iBAxLThs`ikRIcgy>)#i()QxwB_&zJxcbV%xnmH%V zM<|6<8!>JGA2#|aa7cG@QjUz&zq4${Nss#}ty5SfZSHFrl2}^8TH@_apN+4rQmD0ZhYkj2eyT z7lB-RnGdo8xjm)H)(#=}`t-b#n2UIV9~G_BPdS$mn4Ud-P?D*4km*u;+DuElz$R~43CxJfPR@Y{bPVa8}RtWp-jcE%b=W1jqHs2*58blJmJEmU<+=kI$ zo@6tqAiQ0>$xdwkjUu_0ajIq=0c5VuA z&7LV|Z&--PRGnFywWH6f3ny|zWPD}Xu?uQZ5X*+GV=miTcd~cixfB=D-XlFsQ|{hc zx0JO2!N0f016Sf$UiZ$kc^)x@XnU&$RH;yWa}*kICq4dWLgX~g zg;PN7RWACry|6av(78>is$*2G(?(|~Iz|;9+TTXL*<-OUuQRE+khIT6sI`@(s;Y0= zOM|Lb8$}z<2gLbLOEfSB(nN|T6=>gMr;wyMxrN*rvx9lEWL4PIMSRey1(__7aF7tUu4*en%GF4sD|=b#obijt`9uPE|VFX ziUvq-ZThNa3WnVbqlPrmI*v*=Z99CMMW_+z-XAUv?Q1W3C}x@)^6C_sb1=iY2&(#% z+s*|R>zi(=EwXsHTD+KwhF)xUOIebw|A>?f3=-Va9)s~mJS(Btby0OqLYV`pgmWy4 z7W@^Ws0Ou&w?uu4CoEnC#^hPi@(FLjbcrDrIek{fEbDqq)26weO-!E38lB-@9g3OC zKCp;#uOA;9E(|XD+Xg4`bSOg!Z*rZEJvFVJsT$uG|8Bln`Ojx=s}b)T)Ub&rZWyMyzV__Pd`%aGdLkr2%l+j%pAXkC@;cb2M3gyUPhi=v&0Eu)XCR}}X#kUF z`5XAiX?S_#_o9uhKuf8wbuVpY?&2j=Lb*BUH4^v(@){d{?4;2!}{3wgdXo#M5v=+BG@$h2{SQNZ9|)5?SE z1pKOV?jX@4Su96dSA3aVA8p#8y^mNFY?QhokWC+5{gTzE3ZCGkx&Mhrqx}gAL!OVIcG26&?V}TQ~c8< zVv;%7St35-V#O|rPqIsSTT5ZYl+T~{9_P2CPL-4XQ#9Cs6e+CB%<7by!iYK zFRdpmf?Tdtqi;$!FXug%*J$OZ3nj?*R@w8$kI+O1B)>L&>h31+s7CDocXCu*wR&S3 zu^zEhHQ2v&lZcw7s%b6l8Vi~wf;7vCgB!AS{N6v zHMz97TEu@`gZ_<$xbo)woDt&slWXoYNHecbIlv`M>68^B8YD`-7p@&~g1^P^7vjU3 zF%`xcZ#}NU)Q=Xtjl2f~q`j&w5j3c4=@F%&-}X-Sr@8}rEl(1Oh>W2k^p+MEXp1O# zj2|Q%-?{rbQ*YWIZU6-W#BW%vQa-boBMshQ1V)?><Ep&SC@(1wy<5TxYvf4otk5 zgbML$q~H>dVI}9EM&F&=y|fDn!;UU*_#aaCk+m$F;+;BKrb`h%ICq(_;$aCWEs-Sf z)sjDiTdY=tMF+>dc`sXA`N13XvCf$B)_o~Wy)b$Gl;iYSf#r*gN*sf6r`@hBXiG_t zwtc(3YNK3?h~4HxKfaE0K8Q~>ehb_q_hEj&>nMf(xBBi|k#Op)?LPIVjUSiaa63#@9guIWz=_tkV(hSNG>Qp(X zyyU>6Hy8c)Z=P%)G9DrRlh^RQWD-%I`Kj zBJ}K^IZ#pDy4n<1>GObnfEli#s6gMQejfgotp4ovCJ(yNA6eq2h5#jb3q53WFK;w=q*3! zm`xQDwCNlU1Qng-&@F>G^=_dYE^qtd4A$>Qq3Ncb@5^QVR_^-LcR}Ksh8ica`R3Jr+($_-1OFL`HA(V1`@AsEK%eW5 zQh0C2Y#tMu8Ltz<_V}*pnK%V>&h|kB;!1c@EN0rO=0EoR@hU(D!N9iYBh1wf%0A~( zF549eH&A9`u5tPJLKIj=J2fjH2ILmc*&KXw*wHC=S>IjC zs=_4=6~Fa-SD)QT;V#XUJ2|)uwzK!Ogh_yWy%l{iC(hv@)Irx;oa4Q__v4moEGQ3X zjYF?-q@yI($nc45bUtYFhKS>Vv0PSD$s*UH$(XF>K`Gsc51`_T<);SXl%(-a8^rJ*}u8U(^cy5GZucau3B0I zoUD*>l`<=eshQMc(H|C`+15gREV>S5N~McR?J@uEIW|oxG1R51*&Xv$7$`5_4l;!S zeWhs=!k(`2( z(@><#+o-}!>fXN5f+_ud^&MqPwhx9ql zY$hFo!Qk?&C?fiDpm3{np`taS#q&+&ZQhbgi^tOqa~W)3$rk93>i!i9oXk77T=1n~ z6FhhOOvy0j4E08=S!)8BnYU;y@d*11hMPY{CMZP4B57@tPL7{C2)yu2>b;~=4*t$NGTh|KJUQ;doO$&) zxHW=J%fp&Zj>@BE%Xc1QI8kZmwSwi6mQ>n1iPOLsdkRo|j| z=b9UP$PS68n#QG4hTSXsOC_Kpy3Q%mi!jY^mRXTm<0H9s%4bS+t^Oa(zjkphET z0F3tvr$JV0D%~S0{B!#>&^ftg*yw(~&-1^7 zQtr!jA1iIX{g641ncTr@xR$jVIc5*IFd6o-5&cuz2lObmrgb+nqb?>ub1np>;YM8~ z2wn_e_{Rv2Ttl}Z!+y+MK}`Jz7imj^XghZmW4Y%=f5gKe)(+FVQ@#F$C#H5NSdK_F zS`3wn3V5aAsMj3qB*PqwZ=>MvKr(D zJl_cK-N8K{^Jj_&xI`fJO9Li%dTQguz`x*>0(gB~9ERHoDH!q8y?f|y=N35yY)7f4 zKgTFb0vIL#wCJ`zy@$T}i-zwYFuEQOrP4ZR?p7$moROFDU!+HBY89xM(l>{QNCmc` z#f5}*1qtOF7_(!OYyl|Ff8h{_kT(gpFE9O8B^<`J#sW)&%t+Y(P4P8j(*Oh{Gib0- z%h%0!XZaJ66{2$gAXu-!;PmPgj9(E4*0@CBseZ+ zPq?R94IUJof?+KM%6u}mtPT-?c#{(0aT%Q|F3r8Kh!;OcDdhSDtgPpoHHo4UmlR3T zA=-+f1$0|P8Xxksb|Vv64&7RVQd$bWjJS*har5|#nlYrk4fit5`*Kp^@G*^NV)_ij z*tc&t;Knl1=qXmXcx;mQW5ICXhYe|{L<~Kyd5mw0gkPu@Mt%W z3*SbZ|C4{n!U6}$chu?(9ZNfIw!#trA#%$X(7gO-cKv1LMdQI?cQl-KOn-shSnz(9 zy=vYsyN`TwxiiIYQrN=vJh=${!da7@ER#MCeNjbNEVGDM~0K z;k~s*5r`(J%eXFN+VU6<>z!?7HH*I;_?9RPI25DlzG*eUS5p6i0aMW|Y>C>|*<89L zeZiuxc%|w@8G=2FFsd}u7-fGt?;oQbAXXsHd=%6Iio*&5DsP-~OgaPG&a6Ar58A?v@H9S-j*&X_;r+{+p z)6KQpS6nw!?X8f`LrTBg^{_CR7k>9?4Uj?N`om&?F%9XCu*{u;zu=8dw%6=hk+=eL z>RUm{zwGrU;!0j~1`F4(+Wn8aOhmyPWWL5wOmQ9I(NNFhdo8$89_bhS#|miVyav=@ zkA-XELGP!>EIqEQhoX81OY-7#ud7!BJcwA@Cnnd&7lu6?K(}8<0_#HhT4QR-#AY0w z>wecdl8;^fX{lkftxMD$gmwcVl(3*DOL~_Ae=lBvd=TeziMD)UHiirqX}y=UH{U0_u19}xW98$?KS!>E zZsU>ns|yH&qhcQ?c6uA#0>?k=zZhnLqnaVp{6%K>Iu^gi*~>x9D-$`;ux5l-OOK(c zBC<362#uSu;+%nto_M>fYoJhlj4+#jax3?oiBKuHa%i5nqJ&f|rdHOYW<`p}deANlWkr-FD(gJABE_eCj2ktMlMP}11&vq~yVtqcin5Ny;EdTiLHA3mR8A7_ zH6d@x{>BrxrI^NP^V}f&k~WhRg{*!#OQ&Td`hpPZsV&?Xt$LAtAR_o{etT z3c`*@;KYkePE-{f#+y1Hd4IN+!=t=DAJdX#6zhq z!io^_(*q)*K+y!idE6}`!0g@%MdJ!> z=@BSVG+zitR`pwB)a|I|=ql4!8Q}$Rh-Ia$iEt!;MMS0mg7}=1_yq^Q_&dAhIaVf z{Xjz@LFJPcwUJ?BU0VEs)d3LaBSxC6q{<320ue+am>1;A>Zs;kSMXCAdvtIGtB%3W z=)tI194xH3aM>&Zu>7#r^iM>jf5sv9K8s6vF}j;@1giNg_u})Lun>sK%|0x)E~O|% zlOK9XE8AFP-&v)VRlVeX^B4i>Nz@f~M)vso!Z6R>hJ0hL?~xXGnlCiPA>1_2(BINX zBYNacy7Xhy3y>z_UI$-;^oi?SPw{0>b-Yu3Ug=M9`;WF_aexFriY)B>&Qp@f9>|=0 z2SA?w%0a6QYUHD@ZKzo#>81JO=-Ze0RSsH?g?RBCyi@I(V3m08fr?0Tk&u4{^nKm+ zROlf@BVHQBDXnW8N*!m+p9_Ee`V01J18d~#k@+etaF=RyR-HF|>yX}4%7XXGRSl6ciPc*FlgBe9 zfg%07-My;d-c|maHRVi!#AH-9wsV1D=b3MdEwo#?O@kNYsF5*!5^kRM)ad!c>*Oxc5hpO z2L*+v0(r7~!PlhNH^!b4FkmTIcplCOHojEUXK>nRcV8X&f( zN28I@l^Ag1c2_G*nF6G)Vme%Y|vevm%UN z{sZS-p|%rWotxb^&}_ zUas&qS7ew2L6Q2hBX4$nxbF=Vsjnc#2fS#G;Ec`CCiVP+`Ye!U%;GxV$G$JrB6*m5 zbC&)Ay*Ob=ufW&hR71m%5l z21y{TW9xKK zLQLsN!OKEDpo>e!s;@sX6WQHY9rfQoLPwNV@i^R`q>)^HYR&GUW?7U#=UbrDr(jpb zvM;aJ+*DEO;(0)Iqt_!JIHFg+ey#`JD3=ux61Qm|`kAw>$(Hl9nsYjcc5w=*k{;*H z4Z6&+O0MfN#J%>Mx;%DWyFygx=Qa)$nhLI7B{cpM*HrRu_X!Cn3`vjv0O%P#=H2_{r^!JV(7CetsX^W+1WS`MN1>+_ zjxOt1KbJo_Ri=SM!dSk!)RuHIj80f=9@k%%VBdaWF(sA*Gr0G{4us(zmpKNG2I{{o z&ZLw^`bvmahO{oh_qIP)YT3BgMe|Y&dz45WXrhHqSBr?r{?c$Gq=s5K*>Qs4sp`6fTr~do@%0Y(Ave2 zP)!;+?AScEIXrU@(E+E9U;kbOlb)~>e(bD2WJQNw`8aVDeUZel1_K@dUbqhXxap8p zSb3#;U2_cK6Gr&oa2@fL=sWlH^bFXs-AR1uxL{%oeiHG$Tqa!tzx}wXnB;L%AGOrP z@dlSs%J9pBexu@Tgr{Q>>;wlaK0Hh?rJ-%1y1BJMwvX!Jp{P(USBhb%exYC0cfYRc zgW^AGR_=)iHiK(ElE!`Wj`luEsUwD85A5L?mlc2>JriPm#Gm!CjQ)Uqz$-07)qV@9EPHk21y2KiUPr!Qh!rmOufO4QAs3Z2JnF^8zlNJa zm4zpg%!XdWwsTn ze1`l7Ncivr0@=e^vUTXxjMNNX)C6dwqB%=V;X9emRa}Y=%oQU@)q+H -![experimental](/assets/images/experimental.png){:height="300px" width="300px"}
+![experimental](/assets/images/experimental.png){:height="400px" width="400px"}
*Last modified on {% last_modified_at %}* Soda data contracts is a Python library that verifies data quality standards as early and often as possible in a data pipeline so as to prevent negative downstream impact. Learn more [About Soda data contracts]({% link soda/data-contracts.md %}#about-data-contracts). ✖️    Requires Soda Core Scientific
✔️    Experimentally supported in Soda Core 3.3.3 or greater for PostgreSQL, Spark, and Snowflake
+✖️    Supported in Soda Core CLI
✖️    Supported in Soda Library + Soda Cloud
✖️    Supported in Soda Cloud Agreements + Soda Agent
✖️    Supported by SodaGPT
@@ -338,7 +339,7 @@ For a list of the available formats to use with the `valid_formats` column confi Also known as a referential integrity or foreign key check, Soda executes a validity check with a `valid_values_reference_data` column configuration key as a separate query, relative to other validity queries. The query counts all values that exist in the named column which also *do not* exist in the column in the referenced dataset. -The referential dataset must exist in the same warehouse as the dataset identified by the contract. +The referential dataset must exist in the same data source as the dataset identified by the contract. {% include code-header.html %} ```yaml diff --git a/soda/data-contracts-verify.md b/soda/data-contracts-verify.md index 3ae6a1a9..ac75ee2a 100644 --- a/soda/data-contracts-verify.md +++ b/soda/data-contracts-verify.md @@ -6,15 +6,16 @@ parent: Create a data contract --- # Verify a data contract
-![experimental](/assets/images/experimental.png){:height="300px" width="300px"}
+![experimental](/assets/images/experimental.png){:height="400px" width="400px"}
*Last modified on {% last_modified_at %}* -To verify a **Soda data contract** is to scan the data in a warehouse to execute the data contract checks you defined in a contracts YAML file. Available as a Python library, you run the scan programmatically, invoking Soda data contracts in a CI/CD workflow when you create a new pull request, or in a data pipeline after importing or transforming new data. +To verify a **Soda data contract** is to scan the data in a data source to execute the data contract checks you defined in a contracts YAML file. Available as a Python library, you run the scan programmatically, invoking Soda data contracts in a CI/CD workflow when you create a new pull request, or in a data pipeline after importing or transforming new data. When deciding when to verify a data contract, consider that contract verification works best on new data as soon as it is produced so as to limit its exposure to other systems or users who might access it. The earlier in a pipeline or workflow, the better! Further, best practice suggests that you store batches of new data in a temporary table, verify a contract on the batches, then append the data to a larger table. ✖️    Requires Soda Core Scientific
✔️    Experimentally supported in Soda Core 3.3.3 or greater for PostgreSQL, Spark, and Snowflake
+✖️    Supported in Soda Core CLI
✖️    Supported in Soda Library + Soda Cloud
✖️    Supported in Soda Cloud Agreements + Soda Agent
✖️    Supported by SodaGPT
@@ -24,7 +25,7 @@ When deciding when to verify a data contract, consider that contract verificatio [Prerequisites](#prerequisites)
[Verify a data contract via API](#verify-a-data-contract-via-api)
[Review contract verification results](#review-contract-verification-results)
-[About warehouse configurations](#about-warehouse-configurations)
+[About data source configurations](#about-data-source-configurations)
[Verify data contracts with Spark](#verify-data-contracts-with-spark)
[Validate data contracts](#validate-data-contracts)
[Add a check identity](#add-a-check-identity)
@@ -35,13 +36,13 @@ When deciding when to verify a data contract, consider that contract verificatio ## Prerequisites * Python 3.8 or greater * a code or text editor -* your warehouse connection credentials and details -* a `soda-core-contracts` package and a `soda-core[package]` [installed]({% link soda/data-contracts.md %}) in a virtual environment. Refer to the list of warehouse-specific Soda Core packages available to use. +* your data source connection credentials and details +* a `soda-core-contracts` package and a `soda-core[package]` [installed]({% link soda/data-contracts.md %}) in a virtual environment. Refer to the list of data source-specific Soda Core packages available to use. * a Soda data contracts YAML file; see [Write a data contract]({% link soda/data-contracts-write.md %}) ## Verify a data contract via API -1. In your code or text editor, create a new file name `warehouse.yml` accessible from within your working directory in your virtual environment. -2. To that file, add a warehouse configuration for Soda to connect to your warehouse and access the data within it to verify the contract. The example that follows is for a PostgreSQL warehouse; see [warehouse configuration](#about-warehouse-configurations) for further details .
Best practice dictates that you store sensitive credential values as environment variables using uppercase and underscores for the variables. +1. In your code or text editor, create a new file name `data_source.yml` accessible from within your working directory in your virtual environment. +2. To that file, add a data source configuration for Soda to connect to your data source and access the data within it to verify the contract. The example that follows is for a PostgreSQL data source; see [data source configuration](#about-data-source-configurations) for further details .
Best practice dictates that you store sensitive credential values as environment variables using uppercase and underscores for the variables. ```yaml name: local_postgres type: postgres @@ -51,21 +52,21 @@ When deciding when to verify a data contract, consider that contract verificatio username: ${POSTGRES_USERNAME} password: ${POSTGRES_PASSWORD} ``` - Alternatively, you can use a YAML string or dict to define connection details; use one of the `with_warehouse_...(...)` methods. -3. Add the following block to your Python working environment. Replace the values of the file paths with your own warehouse YAML file and contract YAML file respectively. + Alternatively, you can use a YAML string or dict to define connection details; use one of the `with_data_source_...(...)` methods. +3. Add the following block to your Python working environment. Replace the values of the file paths with your own data source YAML file and contract YAML file respectively. ```python from soda.contracts.contract_verification import ContractVerification, ContractVerificationResult contract_verification_result: ContractVerificationResult = ( ContractVerification.builder() .with_contract_yaml_file('soda/local_postgres/public/customers.yml') - .with_warehouse_yaml_file('soda/local_postgres/warehouse.yml') + .with_data_source_yaml_file('soda/local_postgres/data_source.yml') .execute() ) print(str(contract_verification_result)) ``` -4. At runtime, Soda connects with your warehouse and verifies the contract by executing the data contract checks in your file. Use `${SCHEMA}` syntax to provide any environment variable values in a contract YAML file. Soda returns results of the verification as pass or fail check results, or indicate errors if any exist; see below. +4. At runtime, Soda connects with your data source and verifies the contract by executing the data contract checks in your file. Use `${SCHEMA}` syntax to provide any environment variable values in a contract YAML file. Soda returns results of the verification as pass or fail check results, or indicate errors if any exist; see below. ## Review contract verification results @@ -80,11 +81,11 @@ When Soda surfaces a failed check or an execution error, you may wish to stop th * Append `.assert_ok()` at the end of the contract verification result which produces a SodaException when a check fails or when or execution errors occur. The exception message includes a full report. * Test for the result using `if not contract_verification_result.is_ok():` Use `str(contract_verification_result)` to get a report. -## About warehouse configurations +## About data source configurations -Soda data contracts connects to a warehouse to perform queries, and verify schemas and data quality checks on data stored in a warehouse. Notably, it does not extract or ingest data, it only scans your data to complete contract verification. If you are using the Contract API, you only need to provide one warehouse configuration in the contract verification which Soda uses to verify contracts. +Soda data contracts connects to a data source to perform queries, and verify schemas and data quality checks on data stored in a data source. Notably, it does not extract or ingest data, it only scans your data to complete contract verification. If you are using the Contract API, you only need to provide one data source configuration in the contract verification which Soda uses to verify contracts. -Best practice dictates that you store sensitive credential values as environment variables that use uppercase and underscores, such as `password: ${WAREHOUSE_PASSWORD}`. Soda data contracts uses environment variables by default; you can pass extra variables via the API using `.with_variables({"WAREHOUSE_PASSWORD": "***"})`. +Best practice dictates that you store sensitive credential values as environment variables that use uppercase and underscores, such as `password: ${DATA_SOURCE_PASSWORD}`. Soda data contracts uses environment variables by default; you can pass extra variables via the API using `.with_variables({"DATA_SOURCE_PASSWORD": "***"})`. ## Verify data contracts with Spark @@ -92,7 +93,7 @@ Best practice dictates that you store sensitive credential values as environment Where you have a Spark session that potentially includes data frames that live in-memory, you can pass a Spark session into the contract verification API to verify a data contract in data frames without persisting and reloading. -Use `with_warehouse_spark_session` to pass your Spark session into the contract verification, as in the example below. +Use `with_data_source_spark_session` to pass your Spark session into the contract verification, as in the example below. ```python spark_session: SparkSession = ... @@ -100,7 +101,7 @@ spark_session: SparkSession = ... contract_verification: ContractVerification = ( ContractVerification.builder() .with_contract_yaml_str(contract_yaml_str) - .with_warehouse_spark_session(spark_session=spark_session, warehouse_name="spark_ds") + .with_data_source_spark_session(spark_session=spark_session, data_source_name="spark_ds") .execute() ) ``` @@ -138,7 +139,7 @@ During a contract verification, you can arrange skip checks using `check.skip` a ```python contract_verification: ContractVerification = ( ContractVerification.builder() - .with_warehouse_yaml_file('soda/local_postgres/warehouse.yml') + .with_data_source_yaml_file('soda/local_postgres/data_source.yml') .with_contract_yaml_file('soda/local_postgres/public/customers.yml') .build() ) diff --git a/soda/data-contracts-write.md b/soda/data-contracts-write.md index 0d3d854e..ac83d118 100644 --- a/soda/data-contracts-write.md +++ b/soda/data-contracts-write.md @@ -6,7 +6,7 @@ parent: Create a data contract --- # Write a data contract
-![experimental](/assets/images/experimental.png){:height="300px" width="300px"}
+![experimental](/assets/images/experimental.png){:height="400px" width="400px"}
*Last modified on {% last_modified_at %}* **Soda data contracts** is a Python library that uses checks to verify data. Contracts enforce data quality standards in a data pipeline so as to prevent negative downstream impact. To verify the data quality standards for a dataset, you prepare a data **contract YAML file**, which is a formal description of the data. In the data contract, you use checks to define your expectations for good-quality data. Using the Python API, you can add data contract verification ideally right after new data has been produced. @@ -54,6 +54,7 @@ checks: ✖️    Requires Soda Core Scientific
✔️    Experimentally supported in Soda Core 3.3.3 or greater for PostgreSQL, Spark, and Snowflake
+✖️    Supported in Soda Core CLI
✖️    Supported in Soda Library + Soda Cloud
✖️    Supported in Soda Cloud Agreements + Soda Agent
✖️    Supported by SodaGPT
@@ -72,12 +73,12 @@ checks: 1. After completing the Soda data contracts [install requirements]({% link soda/data-contracts.md %}), use a code or text editor to create a new YAML file name `dim_customer.contract.yml`. 2. In the `dim_customer.contract.yml` file, define the schema, or list of columns, that a data contract must verify, and any data contract checks you wish to enforce for your dataset. At a minimum, you must include the following required parameters; refer to [List of configuration keys](#list-of-configuration-keys) below. ```yaml - # an identifier for the table or view in the SQL warehouse + # an identifier for the table or view in the SQL data source dataset: dim_customer # a list of columns that represents the dataset's schema, # each of which is identified by the name of a column - # in the SQL warehouse + # in the SQL data source columns: - name: first_name - name: last_name @@ -114,20 +115,20 @@ checks: ### Organize your data contracts -Best practice dictates that you structure your data contracts files in a way that resembles the structure of your warehouse. +Best practice dictates that you structure your data contracts files in a way that resembles the structure of your data source. 1. In your root git repository folder, create a `soda` folder. -2. In the `soda` folder, create one folder per warehouse, then add a `warehouse.yml` file in each. -3. In each warehouse folder, create folders in each schema, then add the contract files in the schema folders. +2. In the `soda` folder, create one folder per data source, then add a `data source.yml` file in each. +3. In each data source folder, create folders in each schema, then add the contract files in the schema folders. ```shell + soda | + postgres_local -| | + warehouse.yml +| | + data_source.yml | | + public | | | + customers.yml | | | + suppliers.yml | + snowflake_sales -| | warehouse.yml +| | data_source.yml | | + RAW | | | + opportunities.yml | | | + contacts.yml diff --git a/soda/data-contracts.md b/soda/data-contracts.md index 571a413a..8bfbb985 100644 --- a/soda/data-contracts.md +++ b/soda/data-contracts.md @@ -8,7 +8,7 @@ redirect_from: --- # Set up data contracts
-![experimental](/assets/images/experimental.png){:height="300px" width="300px"}
+![experimental](/assets/images/experimental.png){:height="400px" width="400px"}
*Last modified on {% last_modified_at %}* @@ -43,6 +43,7 @@ checks: ``` ✖️    Requires Soda Core Scientific
✔️    Experimentally supported in Soda Core 3.3.3 or greater for PostgreSQL, Spark, and Snowflake
+✖️    Supported in Soda Core CLI
✖️    Supported in Soda Library + Soda Cloud
✖️    Supported in Soda Cloud Agreements + Soda Agent
✖️    Supported by SodaGPT
@@ -70,7 +71,7 @@ Soda Core 3.3.0 supports the newest, experimental version of `soda-contracts`. T * Python 3.8 or greater * Pip 21.0 or greater * a code or text editor -* your PostgreSQL, Spark, or Snowflake warehouse connection credentials and details +* your PostgreSQL, Spark, or Snowflake data source connection credentials and details * (optional) a local development environment in which to test data contract execution * (optional) a git repository to store and control the versions of your data contract YAML files @@ -81,7 +82,7 @@ Data contracts are only available for use in programmatic scans using Soda Core. Soda Core CLI *does not* support data contracts. 1. Best practice dictates that you install data contracts in a virtual environment. In your command-line interface tool, create and activate a Python virtual environment. -2. Execute the following command, replacing the package name with the install package that matches the type of warehouse you use to store data; see the complete list of packages. +2. Execute the following command, replacing the package name with the install package that matches the type of data source you use to store data; see the complete list of packages. ```shell pip install soda-core-postgres ``` @@ -97,6 +98,14 @@ soda --help To exit the virtual environment, use the command `deactivate`. +## Upgrade data contracts + +In the virtual environment in which you originally installed `soda-core-contracts`, use the following command to ugrade to the latest version of the package. + +```shell +pip install soda-core-contracts -U +``` + ## Go further diff --git a/soda/new-documentation.md b/soda/new-documentation.md index 49350ccd..a883956c 100644 --- a/soda/new-documentation.md +++ b/soda/new-documentation.md @@ -14,6 +14,7 @@ parent: Learning resources #### June 28, 2024 * Added [release notes]({% link release-notes/all.md %}) documentation for Soda Agent 1.1.15 & 1.1.16, Soda Library 1.5.13, and Soda Core 3.3.7, 3.3.8 & 3.3.9. +* Published documentation to accompany data contracts version 4 release. #### June 27, 2024 * Added [release notes]({% link release-notes/all.md %}) documentation for Soda Agent 1.1.14 and Soda Library 1.5.12.