From 636d4cd685f83da81a035f0b3de2dc6015bcb8ce Mon Sep 17 00:00:00 2001 From: bcumming Date: Tue, 24 Jun 2025 07:37:10 +0200 Subject: [PATCH 01/13] document affinity program --- docs/alps/hardware.md | 17 ++++--- docs/running/slurm.md | 116 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 124 insertions(+), 9 deletions(-) diff --git a/docs/alps/hardware.md b/docs/alps/hardware.md index 83eac33c..676990ff 100644 --- a/docs/alps/hardware.md +++ b/docs/alps/hardware.md @@ -40,13 +40,13 @@ Alps was installed in phases, starting with the installation of 1024 AMD Rome du There are currently five node types in Alps: -| type | abbreviation | blades | nodes | CPU sockets | GPU devices | -| ---- | ------- | ------:| -----:| -----------:| -----------:| -| NVIDIA GH200 | gh200 | 1344 | 2688 | 10,752 | 10,752 | -| AMD Rome | zen2 | 256 | 1024 | 2,048 | -- | -| NVIDIA A100 | a100 | 72 | 144 | 144 | 576 | -| AMD MI250x | mi200 | 12 | 24 | 24 | 96 | -| AMD MI300A | mi300 | 64 | 128 | 512 | 512 | +| type | abbreviation | blades | nodes | CPU sockets | GPU devices | +| ---- | ------- | ------:| -----:| -----------:| -----------:| +| [NVIDIA GH200][ref-alps-gh200-node] | gh200 | 1344 | 2688 | 10,752 | 10,752 | +| [AMD Rome][ref-alps-zen2-node] | zen2 | 256 | 1024 | 2,048 | -- | +| [NVIDIA A100][ref-alps-a100-node] | a100 | 72 | 144 | 144 | 576 | +| [AMD MI250x][ref-alps-mi200-node] | mi200 | 12 | 24 | 24 | 96 | +| [AMD MI300A][ref-alps-mi300-node] | mi300 | 64 | 128 | 512 | 512 | [](){#ref-alps-gh200-node} ### NVIDIA GH200 GPU Nodes @@ -81,6 +81,9 @@ Each node contains four Grace-Hopper modules and four corresponding network inte ### AMD Rome CPU Nodes !!! todo + [confluence link 1](https://confluence.cscs.ch/spaces/KB/pages/850199545/Compute+node+configuration) + + [confluence link 2](https://confluence.cscs.ch/spaces/KB/pages/850199543/CPU+configuration) EX425 diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 07841c81..6cc3181d 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -9,6 +9,8 @@ SLURM is an open-source, highly scalable job scheduler that allocates computing !!! todo document `--account`, `--constraint` and other generic flags. +[Confluence link](https://confluence.cscs.ch/spaces/KB/pages/794296413/How+to+run+jobs+on+Eiger) + [](){#ref-slurm-partitions} ## Partitions @@ -27,7 +29,6 @@ Each type of node has different resource constraints and capabilities, which SLU ``` The last column shows the number of nodes that have been allocated in currently running jobs (`A`) and the number of jobs that are idle (`I`). - [](){#ref-slurm-partition-debug} ### Debug partition The SLURM `debug` partition is useful for quick turnaround workflows. The partition has a short maximum time (timelimit can be seen with `sinfo -p debug`), and a low number of maximum nodes (the `MaxNodes` can be seen with `scontrol show partition=debug`). @@ -38,6 +39,116 @@ This is the default partition, and will be used when you do not explicitly set a The following sections will provide detailed guidance on how to use SLURM to request and manage CPU cores, memory, and GPUs in jobs. These instructions will help users optimize their workload execution and ensure efficient use of CSCS computing resources. +## Affinity + +The following sections will document how to use Slurm on different compute nodes available on Alps. +To demonstrate the effects different Slurm parameters, we will use a little command line tool [affinity](https://github.com/bcumming/affinity) that prints the CPU cores and GPUs that are assinged to each MPI rank in a job, and which node they are run on. + +We strongly recommend using a tool like affinity to understand and test the Slurm configuration for jobs, because the behavior of Slurm is highly dependent on the system configuration. +Parameters that worked on a different cluster -- or with a different Slurm version or configuration on the same cluster -- are not guaranteed to give the same results. + +It is straightforward to build the affinity tool to experiment with Slurm configurations. + +```console title="Compiling affinity" +$ uenv start prgenv-gnu/24.11:v2 --view=default #(1) +$ git clone https://github.com/bcumming/affinity.git +$ cd affinity; mkdir build; cd build; +$ CC=gcc CXX=g++ cmake .. #(2) +$ CC=gcc CXX=g++ cmake .. -DAFFINITY_GPU=cuda #(3) +$ CC=gcc CXX=g++ cmake .. -DAFFINITY_GPU=rocm #(4) +``` + +1. Affinity can be built using [`prgenv-gnu`][ref-uenv-prgenv-gnu] on all clusters. + +2. By default affinity will build with MPI support and no GPU support: configure with no additional arguments on a CPU-only system like [Eiger][ref-cluster-eiger]. + +3. Enable CUDA support on systems that provide NVIDIA GPUs. + +4. Enable ROCM support on systems that provide AMD GPUs. + +The build generates the following executables: + +* `affinity.omp`: tests thread affinity with no MPI (always built). +* `affinity.mpi`: tests thread affinity with MPI (built by default). +* `affinity.cuda`: tests thread and GPU affinity with MPI (built with `-DAFFINITY_GPU=cuda`). +* `affinity.rocm`: tests thread and GPU affinity with MPI (built with `-DAFFINITY_GPU=rocm`). + +??? example "Testing CPU affinity" + Test CPU affinity (this can be used on both CPU and GPU enabled nodes). + ```console + $ uenv start prgenv-gnu/24.11:v2 --view=default + $ srun -n8 -N2 -c72 ./affinity.mpi + affinity test for 8 MPI ranks + rank 0 @ nid006363: threads [ 0:71] -> cores [ 0: 71] + rank 1 @ nid006363: threads [ 0:71] -> cores [ 72:143] + rank 2 @ nid006363: threads [ 0:71] -> cores [144:215] + rank 3 @ nid006363: threads [ 0:71] -> cores [216:287] + rank 4 @ nid006375: threads [ 0:71] -> cores [ 0: 71] + rank 5 @ nid006375: threads [ 0:71] -> cores [ 72:143] + rank 6 @ nid006375: threads [ 0:71] -> cores [144:215] + rank 7 @ nid006375: threads [ 0:71] -> cores [216:287] + ``` + + In this example there are 8 MPI ranks: + + * ranks `0:3` are on node `nid006363`; + * ranks `4:7` are on node `nid006375`; + * each rank has 72 threads numbered `0:71`; + * all threads on each rank have affinity with the same 72 cores; + * each rank gets 72 cores, e.g. rank 1 gets cores `72:143` on node `nid006363`. + + + +??? example "Testing GPU affinity" + Use `affinity.cuda` or `affinity.rocm` to test on GPU-enabled systems. + + ```console + $ srun -n4 -N1 ./affinity.cuda #(1) + GPU affinity test for 4 MPI ranks + rank 0 @ nid005555 + cores : [0:7] + gpu 0 : GPU-2ae325c4-b542-26c2-d10f-c4d84847f461 + gpu 1 : GPU-5923dec6-288f-4418-f485-666b93f5f244 + gpu 2 : GPU-170b8198-a3e1-de6a-ff82-d440f71c05da + gpu 3 : GPU-0e184efb-1d1f-f278-b96d-15bc8e5f17be + rank 1 @ nid005555 + cores : [72:79] + gpu 0 : GPU-2ae325c4-b542-26c2-d10f-c4d84847f461 + gpu 1 : GPU-5923dec6-288f-4418-f485-666b93f5f244 + gpu 2 : GPU-170b8198-a3e1-de6a-ff82-d440f71c05da + gpu 3 : GPU-0e184efb-1d1f-f278-b96d-15bc8e5f17be + rank 2 @ nid005555 + cores : [144:151] + gpu 0 : GPU-2ae325c4-b542-26c2-d10f-c4d84847f461 + gpu 1 : GPU-5923dec6-288f-4418-f485-666b93f5f244 + gpu 2 : GPU-170b8198-a3e1-de6a-ff82-d440f71c05da + gpu 3 : GPU-0e184efb-1d1f-f278-b96d-15bc8e5f17be + rank 3 @ nid005555 + cores : [216:223] + gpu 0 : GPU-2ae325c4-b542-26c2-d10f-c4d84847f461 + gpu 1 : GPU-5923dec6-288f-4418-f485-666b93f5f244 + gpu 2 : GPU-170b8198-a3e1-de6a-ff82-d440f71c05da + gpu 3 : GPU-0e184efb-1d1f-f278-b96d-15bc8e5f17be + $ srun -n4 -N1 --gpus-per-task=1 ./affinity.cuda #(2) + GPU affinity test for 4 MPI ranks + rank 0 @ nid005675 + cores : [0:7] + gpu 0 : GPU-a16a8dac-7661-a44b-c6f8-f783f6e812d3 + rank 1 @ nid005675 + cores : [72:79] + gpu 0 : GPU-ca5160ac-2c1e-ff6c-9cec-e7ce5c9b2d09 + rank 2 @ nid005675 + cores : [144:151] + gpu 0 : GPU-496a2216-8b3c-878e-e317-36e69af11161 + rank 3 @ nid005675 + cores : [216:223] + gpu 0 : GPU-766e3b8b-fa19-1480-b02f-0dfd3f2c87ff + ``` + + 1. Test GPU affinity: note how all 4 ranks see the same 4 GPUs. + + 2. Test GPU affinity: note how the `--gpus-per-task=1` parameter assings a unique GPU to each rank. + [](){#ref-slurm-gh200} ## NVIDIA GH200 GPU Nodes @@ -144,7 +255,8 @@ The configuration that is optimal for your application may be different. [NVIDIA's Multi-Process Service (MPS)]: https://docs.nvidia.com/deploy/mps/index.html [](){#ref-slurm-amdcpu} -## AMD CPU +## AMD CPU Nodes +Alps has nodes with two AMD Epyc Rome CPU sockets per node for CPU-only workloads, most notably in the [Eiger][ref-cluster-eiger] cluster provided by the [HPC Platform][ref-platform-hpcp]. !!! todo document how slurm is configured on AMD CPU nodes (e.g. eiger) From 8cd17b2898f372145a334883342221f2cd601eab Mon Sep 17 00:00:00 2001 From: bcumming Date: Tue, 24 Jun 2025 17:38:46 +0200 Subject: [PATCH 02/13] wip --- docs/alps/hardware.md | 38 +++++++++-- docs/images/slurm/eiger-topo.png | Bin 0 -> 53384 bytes docs/running/slurm.md | 104 +++++++++++++++++++++++++++---- 3 files changed, 125 insertions(+), 17 deletions(-) create mode 100644 docs/images/slurm/eiger-topo.png diff --git a/docs/alps/hardware.md b/docs/alps/hardware.md index 676990ff..cfd43d80 100644 --- a/docs/alps/hardware.md +++ b/docs/alps/hardware.md @@ -80,19 +80,45 @@ Each node contains four Grace-Hopper modules and four corresponding network inte [](){#ref-alps-zen2-node} ### AMD Rome CPU Nodes -!!! todo - [confluence link 1](https://confluence.cscs.ch/spaces/KB/pages/850199545/Compute+node+configuration) +These nodes have two [AMD Epyc 7742](https://en.wikichip.org/wiki/amd/epyc/7742) 64-core CPU sockets, and are used primarily for the [Eiger][ref-cluster-eiger] system. They come in two memory configurations: + +* *Standard-memory*: 256 GB in 16x16 GB DDR4 Dimms. +* *Large-memory*: 512 GB in 16x32 GB DDR4 Dimms. + +!!! note "Not all memory is available" + The total memory available to jobs on the nodes is roughly 245 GB and 497 GB on the standard and large memory nodes respectively. + + The amount of memory available to your job also depends on the number of MPI ranks per node -- each MPI rank has a memory overhead. + +A schematic of a *standard memory node* below illustrates the CPU cores and [NUMA nodes](https://www.kernel.org/doc/html/v4.18/vm/numa.html).(1) +{.annotate} + +1. Obtained with the command `lstopo --no-caches --no-io --no-legend eiger-topo.png` on Eiger. - [confluence link 2](https://confluence.cscs.ch/spaces/KB/pages/850199543/CPU+configuration) +![Screenshot](../images/slurm/eiger-topo.png) -EX425 +* The two sockets are labelled Package L#0 and Package L#1. +* Each socket has 4 NUMA nodes, with 16 cores each, for a total of 64 cores per socket. + +Each core supports [simultaneous multi threading (SMT)](https://www.amd.com/en/blogs/2025/simultaneous-multithreading-driving-performance-a.html), whereby each core can execute two threads concurrently, which are presented as two PU per physical core. + +* The first PU on each core are numbered 0:63 on socket 0, and 64:127 on socket 1; +* The second PU on each core are numbered 128:191 on socket 0, and 192:256 on socket 1. +* Hence core `n` SMT + + +Each node has two Slingshot 11 network interface cards (NICs), which are not illustrated on the diagram. [](){#ref-alps-a100-node} ### NVIDIA A100 GPU Nodes -!!! todo +The Grizzly Peak blades contain two nodes, where each node has: -Grizzly Peak +* One 64-core Zen3 CPU socket +* 512 GB DDR4 Memory +* 4 NVIDIA A100 GPUs with 80 GB HBM3 memory each + * The MCH system is the same, except the A100 have 96 GB of memory. +* 4 NICs -- one per GPU. [](){#ref-alps-mi200-node} ### AMD MI250x GPU Nodes diff --git a/docs/images/slurm/eiger-topo.png b/docs/images/slurm/eiger-topo.png new file mode 100644 index 0000000000000000000000000000000000000000..31661fa8546b508b3df584adcd49548ba0708274 GIT binary patch literal 53384 zcmeFZWmF#BvM$=V1-Ibt?m=H9xVyVUfZ#4cLvVKwZUKUOg1ftf;O=hc&G)T+#$J1` zvCbXm*S&Y_;s;~Q?yjEQb9U8JRZqR)N(z$5i1>&A03b_Cy;lJMNN)gu!hwebkHE^# zpZ)n~A}jeGc>VL2+g_Xi0HlEQdl5B{%%fFLVL4j@;B0q9&Rb08;h~b^xfO>_OVP) z@5WsL=j61MZ*_H>PR#*mqVQyK2nYxpADi`9SXdnAp~-^LY=+v}+O&RGL4glaqO)$2 z2cwC)bVWvhPo9L}UwB-fYxUR)Gb{2+QE6yPvYxpKbc$Q)k4Jo{cys!=b?4q%(Fl)iIX4S<>W(d z@%*g$W{_Ul@0=|vSXko<5^fF^TXjbsuwYTaY2Lwr)Nk4wS}H4&KhYWh++PB#1DvKmcemHw+foj}@(&YMm0`*B(#HWuRUp z_knMPpF|lFXrsYBt*<{Ff(&>X7-iN-Zg3MEePrppPJ;}n)$F&psyUji%ebET$ubPU z|DHB?PkGUhkQjT7prZpJ>0udsbPQZ95s&cC5h_v~7Xc4L1k`>_s3SUtd-LsZ)_ttm z9ebR#bfxRrSWq91mf>do;q(PPoiWyawL@LnKnO-nM2N0hSgoQ&F9&&-2i4~1eND|- z^7{7Juutgz03^^VK3@1q5#^0~=@qeov3*>+{o^Yl`8uIt$7A{(-#(XJ0So-qgR+_JUQ*WDaai+E0TED`ORCA1bpfKLEX_#ge65M1 zqG7o_CTS}{er`=mJ80xO55?o~esZ51J7BfA5zZ1~h#on8*cRp1Km6R7(MXYUJmZE4Rtw>?8J~9f}3q*&@KV7}l z_AZ_?kYH*0@|?~l;AS;(4?%z9^tw?~_$>QeZK9t3y1x9iUbjv~)qiyz{Fs#LOn50u z-$^Vm+$`qz{rOR993%kXbvwJ+kA9d%52@*!a~m@OK>LMK`b*i}1X@HdIRJdliFj;y z?(ZyYd3HUqmjcV)rln<3LzPwn3ID}9k$V=Q(V_bVs+cwa@czu>2uq`{gBLf&rKy}p z*r$AoTwVI6I#(c>LkC3PfAN+cl>St=oS3NYRTXY!&)#FeM^9dOunYl|I56eYej$A+ zcbVOx02@D7_2tH(&lbL+X=|%q&Cd=f+w)d~nd6@6UPRcx4lm1j3{o!cu_V`5_Ux=Y zh+#Z=RhcICKfVaYUW!xs?acUIvO!)1|E&<@~o!bh+#l6qivWIyD`gYW4T8?X8SdAv05Dwuk7V@Y2~! zB;@vUmRc;^@glw%DX*$!hYNI1($9~2M^-|9eTh?CvZ%dQ~%NM^iqIAhFz`zf`Go+KPtn11KvY#7nrKiYr`=jC980)VgH-s?+; z>?@UziPEbv%ED#5_5xcaK%1n{m}*S>s7J=w4*qGK0k)4(qL<=nL`K_g%wbDWthmxe z%d=(93Y!}G9bg`JNGRv#9Pqn_GwbDsWTj11a_^cw!eFYu;J$L!Hf23~zh{+NoqBq> ze_aFr^C=6AhV_f`_Jwy zIaQCm^|5`rw!1<6U4;ntW)dbx4^4zrQba2bcq6}!PTT$C43$)a@NmICF(1l854>kSqs*@K zBZFNl0zzIp^`GOI-P=yhdMAnm<>O7Cc*d^29mq)*{&7V`+r|q>K#6J}OaQS)mSt<} z=N5&(F8F>gTAy()^lv=wzjntZl6~7C5Ga3|1gaAE*`j@7(IRVnR?d%TO&!0<_U=Hl z4Oa7;3{hWXbCQV0lGJvn-TF(k;Sg9upfect953=Zff^3iXgN78>E*%nBd{S zJ$iVb`zXiTA1>G);oaKd&n0KFd-5{%=QG2izz&>@+kJ|>%F)oNfdC?D!A8Anp_am* z?XHdM^V`l4e~^R;vK~oFLFKwS)6u3S;%A<)9pcus(V`9~#+324$NHO=jp~}(kGA9_SLDVimSGm|>)(cAd^hR*=IKEy1{&eDWp5yWiX9vJdu?qGeLS+@*t^+N z!jxA^BaK&n!d%*BjcB0+?;*&NvH^j29JhqBp*d;Svkjlgoqr28aN0A#K`+FtnV}Yb zQo`^b-7IQ5wma)B*_T)K`2sDWI$G? zE2@+}p_|Jwa(&ShH=%?tj3V($>{j!EfRcFT1h zlbw_v9xPTK+2g-z@fB67<{RLuD4m^#FCH2uLgsBu%==EGr9JmUs@|*L&QB>z`#p3( zmmKl4)tCcaQ*5@ELR+5PFlq)r;+RlX2h(4Lim$uUs5Lf=tCm>5(aamquA9OhI4)_s z554h=7ZH{IQ82I*x_TJf1h$rB%C^4}RM%^eLl-YzSKbMh6y`-d$4<$16t_AM^ZE>B zJy$`E#j)E_KO9KYtUv(Cp}jvq#4Vl@&koT{{sx85uf*&F7DL_A6tk{Oy7&X=&}gGs z5+exL8JEjMbi|X^HzUbOAnY?@!%Q235m=yM$YD7}?kV%SMrJ-i8 zo1eD5G)j3gr66=f(zX?Cu{dq@-v{h!ENxZpN2}^BB1_zn{Omp*mp@_^kYQQ-mA)0@-GZ)swQ^dH+TY>ivQ;mKyO*hn|& z@kaO>ERvhbKY@)>I99guzrg)rEo=S0Z*R!?jzXl$1JTN|n$>r~HjBZ&yX(EwQ5md- z{6V^7!B2yp_no!B@sX4mf|Y)Tg!DjkDSnFj6dvoLPLRq~+Fv)Mp=at|eU4m#B`U?5 z(H@P9jitF74L;%RELQ65kH>!%T>NWO_m8kd1Rk&PN`3D@#9wD$;dF7V{`K=T>J(k~ zFf{w_SbWkL4ic`;Oo+7F8TovkNyJ91(EaUHS9d_PkQHsOk=%h|VNcA(3C&aU*flAVqh zQikONv4?1pm|N*_G`fne8;Q(ssg%IU{QEWLgTi88*#0KH^@(s*4+o3t`8wieeEp?` z)6eEs#nYKxeZyab-X=6nnTXo%8=^UhNtVQFx29Uq5i;W+ct=mG zeR0DWPC7gqV@Jklm6cPdfPlh^yay@@C>BF9HPpX#A1o^?~A%42t0c{OcGJUO_wLfIBjFc$U z6#t>FF)7dOreK~I3y^ep5qu4kHb_`bJf{T!I=s>gU)nGE9nUGG^ag?C(igkAg;E&s zfrUkM8A&=)5CDjuN=&+r%Ai@ag!s>5k&19seIV&4J00^Ne%5*q0k8-%(jU}C3Ox27 zepo%RTVL@^mV`)3yQA#PGG+*@Bvs8!8i%9xec9ov_Tiyt5D+STQ?k-{&Z|T8{auT1 z%U8JfMR7_PxWr{+@BuLwrXyd4ucg2J$tbpo9i zCx#R#A!cktRW{x8>cz&_TX6Ub6FcRu_1gGpCdL=uyw$b*cn-mUKbR5=mReLUC`b8^ zwc>-{Ak_Rg^%Hj;kNHG~G?@M;_=9XZBm5^slIr;PsV*cujFilbTczL10?{72zu21e za@j96aLXR~mQdH4s>*^q?z>DtD4Qi0zfrAr_j^;$pJpaky){4*H($E}U}{?$S9!KOCcjyZi;dk>9{ z2d?{<9E1V|vt)_N6Sm7ADGse?nC>la1kO6HLL3}?ACqD{)_Fj>)1A)MOyeG4U#t+2 z5xtT46Hkq)SEYWYfYYOBb4kj8$s0Hgseb-yT547prDorCuzD^)|2U)Zs_6r;UnY+y zc8KX6g^jPFt4gW+wbs{&^MrNapxo#`M(bi+ebOAq00M@Z>$+Tht(NM7kR+D*__g1> zZ9hABym0zrke#+~w4;)jMl~o>UW9yL-_Lbu*YCRQ3yP>vit+=iS%n+Bi5e>?dGbEg z!(AIJt@Mu#wzm1Ctsn5DS=a(?F-ir%Qe!}%gOd!D%6u z7YUEJGP!#OwXvu!OZ;?c2%hOqT-93>m>=HHuzdBqXNH{=S4*{dTN`Wj4zR>6AITQ_ zIXyPRDg6qw3R}jI+dLT#ZdObDVcV}4z1nj-sIUYVYGObesLzs;)%Chw&pJw}>j5dt z(Skb=SgSW{u~t@#C!HenUvva*yulm`@*wUDkZR%0$rlfrWA-yg>e zwV9SN$KPOrEwFab@LMJ=xhh--XZ8&caQo6WUnX`+R~xWy-X}pqW9Jq%OXuh0bqLbT zU$fd_YL9!HHKnm*FBw)GQ~xkJX#q9az1HC2{C@2!n*RN)yG;FoaVdW6Q}dp5uEVNq z?!^hQml?NhT0EcSCNBW^DdYHH%+W9DEE_Sy$`E)n}#9K*`E4Cb;xb ze($Zb*&xhIx1;;UugOaTZba=?YMO_hhZ>%!2Scv9g6jr?yUxl;>M)RiRvA7$@4oh| ze>{ty_o-h;wzBNKS5lYv3#&nOnBzT7PVGm(JiI}`;(YsW*I2^LEa+H%ZuWEv%ad`pH1bc}opv#}=@#we` znW8U}=&98Z62!w*JQrc8c0*7r?v0Z65sBThvA&@xe@C)RvVOJm1&8xjlv5UMst zXhMmdz8By(^74F{8^8d&H+gw*+RZLXJieF?)`IH*FBJqf#`&aWytHo1k?A2mlB)Ww*OcW3!pnLwZ4Yoh`lgC-EAH z81qdczRmnhlT^gSs-ihCGEg8KMarQf)M!<$v2TjlKXW{ihoDmdo1$bu$9||iV2jN! zYu8X4N0G9Lou!c8__BYArmI{1yqn;-s7B-3Q1h6E=I-~q2)(0C22LSAO~jC(GzoQj zAbWOwO|--sy6ABry54-058C4Rk5kydJYJ~#9>!7=ebb()0Qr&cra=z%9btwfM551| z3smL$Mzt3lYebD8zOd7l#ii0laGoc2a6YKE?)=l!ltql5D3#DGo|QF5*OVlR(47LZs7>cC8+U@*MTReoTj|Lx2`q>cle zO-)H7N9lyYOXpLgr*;aBVt)mDOiDy@!G0JZ&3@XS=iQ?w4ltlywIEo_?7xq_EFITSH|X%r<=buGd%rkfNSlKg^9~`y=0x^_t?1t(2(qnhj-t zAXzv?g98?r`akeq^zU-INg``lOGQ^0AVC7cEsv+!S*!i1LJC9mIZaEFyP7#FLk6=2 z&~ydFSv(S;P2SWP1GZt)w9Noy%2Ko4j#&|64@YlFrk1;t%lDV2)1Th7S))o28<>MMfgm;h=J3Fw(_<0XqpW{^|nq4a)DQ|&)jsgKX@0uYi zppnlO@VwaC_Yd1*b1>==6u-|o`H;rpT6TB^<4v)j(P(1Fb-(|m_;AZ`>ho%UF?}m{!LU8}}Q>#UaY6L0`u3f85y{DROjR2a2LwVl2r>>4h zqLJ^W!D@c0q$gy#Upj_vwA)M#+(QlcF}z>R)|b2ysqOi#lzp5JCneKxo;l&ko{x1t zH_ujR1hu20z=A_J-i9L&*@zv^?`g(XM3D467b;ri+RK71Qia)u-;bOEM^8fqiU^s+ zoy=%GosP3_ngknkz%Enyl*_wLsJfZ+)c0(zG6m%L)nmf(>Nn^E=iTl6cr#N$mgbW0 z+WTfC{;DDsGxSB{t$bcLCfl)8s_;**dd&nG`3B!OuH=1M8eGg;i4n$Uy%g0fAts9~ z=P_QJp5iK#dRUA1?OY$(aP%!bbid|g(cJL%5Soxc9vo__NJK{$aerQTEPi)ltizLX zGK!Gz`5J<>Z^h1?4)1S&VXr>_858-^wQo3$nj)7v$29j!>wU3ce@2=+8|s*pxU z$~u(u0;CLLQ&Y7PVnQ=X+eVGc_lFO_tvkKt(sWvi)@{t=LITY2oW5fX zo?$}AZYhp#`ukj0)u+{vL*!01*O{Cxk0&C0!{WpLzI|p$-6j{l(#V+&)5Y?;+`>0c zxP3?JYsc4EnIgi2OPVrqMQ)(;x2^D}rfA2j2AwxUBpOfFvW7%rUN;p;HK-7}jqAbZ zIxA~L3l3uyS2r;$jRW!0mTkXJNWP4-JjX7Y)ZA)BRg^QN5PLN_1q)y__B=G;=cQFs9lddToWEKZ-6I6e&Nmn20tQ(@ zQ6UFKx4emkgh69Yy{n+$co`)5XJ#heE}O+l+lW>YCt4)=xZ9W}UHy*T!?U-e#F!zk z_efB6HWI8XzIS<^ikrz57Fi!!9UIyQ@>@RprO(Dmu~&UBty>k!(eIXsfcu`nB-`5P ze0;uhIp;Pd9qy$w(b;Hn5`k;DjH*fT%k}=`h#QtjC`4e))lG@n%j;wXw^f>(OGvSM zZlCAJZ&TgjQYhImlE}K8oeCMCx?K5Y@EhaqCm9D~0gb(b899M0S?6+V3*wWY=KtUV z^m0{gYM#HZu_OxjhA&an)Hr?NbDS+fIf|sI;jQ#Hof9{&xka5E>5;Q{_P9tzIlkQ7 zbk^mayx7oMse}b`oiGd?`L1xgVNl~Ru%X{x! z%VfIO;?>ukWm`n_B2%-qw4`v8Y4a6jXfx-7{=>W{iek%fq<9hGFW==1D~n^k?eiUT ze}j!PZ6XO4mGTu(+a1Mav7QkWAY1u5B~Q_5ROPWwI>phmjU(h(a?q z=kO55(c@Qa$6M6K*{(Eh0-k4~O6YXgtl!4W53)m%R`u*6eq$rnYchl}DXdVHV!~F) zE(8sp!+P`WK4sds*2FlqZ}@aDva_=d3=FV&JUu6jw;#A<7Cw1dY|49IltbN3(VUlMnbFBo<*OK z@k$@*r)KwHd;qHM9POns$OJ5=rXdE5$a3=X{yG@7CzScKjM|>fGN7fDd!bg(4a}JC zFhf7(Z`vmfABB;A!l=q+#uzpndmV>P4v`X-IylS_vbXlzTTc^R^6%uFEg~>$ih0o& zt!B^NjzH~#Iy%2fQi|6~jy1D32r@i;tJ7R?34<6toJI?O3ke}*@hbJcoQ+jn4C@o! zQzQWaK}Kfg$hK36L9J9sa4;Yo^r%U$D}U};C{3dF;_er8Q+%kJ6&)gwzAU-+Ykb>X z9~MR@>tYMKf!J1+C zM&$&5Aj1AANv<39i4K|^Q7-NY5;kS{dEZ-VWEYH3Gcy^b5aQ=;^2KG7xCP(n$U?PN zme%jIe|=LHgMwwgg*iG9sc0x2TYgg{9+Klm@%gnQRH0+B?zU;@D&i5@%rKjPyN(Py zzKyJeWblY&(3T6f2<m>pkm1G3(#6{ey*+B&WD! zGiYy9M?&y$XR%Kw7ul$ti;g@9>b**R8|@&{uw>o4WKwu8M^}#?^)+vaO=%3V4tyKV zZYLY~aFB|2@4fnW{QX%ZtLu!fKokNdU&9pBS$wSt=S@q7+fjrH7z_Q|u!{K`Sp1j2 zphGogPP-85d)cm%`6yuI8UF10@cwPxXLJ(*h1NALe@3~abf=rB~m7xyNW-AWy;bpUJJjIr}?reJ|MqA(RbZqQ8 z^_ffG&tAgB+{)D++gMC34$K{25fHQb!p;SEyckdE6;{t~xB51-wH7>Vl~Y}PuTn;v zMFUv&x8%y1^&{nuSE~!-x;}@~>^SSu4XeNQ(NI#h$ux0OTMrH~cp}pUbSG`@UmW>rA z-}9uuT&e;Z3&en+3=in`jUBbq%t8! z>G#c(45QxXUBzmJ(N+(J5VM{h)1k(ouS-i;bMMZn>pDpIJ0u<}SG*6_4(FOLnR~_C zrjwfI7SD~FZeo+bq?0}V+xr#)PD^7NuHW-i;X!9Qc3W%yhn+Vv_>2K_$GSY9|``bS=AdQy7#3PZ&Y+j?%v|HAsgkPgXsLv|y81jw8+bI;m? z*bf?`IIb6-D*8B}Y{`dSAaGQ7%cOLX+fuv*O+%~j^iO|Ng`b5AGh9B?6#mGt4|>#E zZt;LEj{j>_yLZLy_SQ#7Mn-y^bl&HzuA$66_1sh|X;O7OT1;(i;65 z3>nbT&-?`%yndO0DrY26`bvqWJ}$fL@I_u4JPh5F5{%<)ot*ycN z^1UHl?(*V;U)b9=pN-&YiL2k82Nm_-^OH#ul8hQ+>#23*q|#)kR?|B(3P1*}d}rA5 z6%|pgVr9vJHpc;Cn|bfQz>eAoSe$2*Xn+5(?rZSg%}3`7lRUKgX&zryzBh744^ao> zX$BSVpoIzVHckb)(g4)K(Z+wm$J;e`x|5R%{1EGWw*=-+w*>3FVt6rjbv zjnxzuDa05YfBm#Nmlw0MpMv)l6N;L6Ga4py3}udv2KD{63P3fudkbxSwOKZ|FiH3m z;gSeWZoSSwrB&>8@$uGg6TGQ-@mqWq2n}a*z z98+>d%U%M(Ztsp?FUfd^_GYSGvIsFFbCc>RHjy0PsLA(8a4vVt&U9C6uV*+|eiKW) ze=qRv-J;dRIxgGloWd`%UnL74u}=-cSzy@($pz}fXpkl31a_iY+Mbuh3U;5&P4AkU zCcT2INg zuZxc_4!fFAv1if0&pxGCqJpGG(v0!IP;hnmac*sc3IF_4wou@WK>734>oc!uQT-&j zQ0bKWv9cu0FU~$ZEzFHuMbp%B@>J{+KWGJL##Ouwy&^d-o)<)>qxP7q6w96-Ov}FQ zR4++5%~d&C-4VeXBYSy=@aNz9p36E8K*0K3#glS?#H*{w^E^UL(GSXWjXHjBfstD@ z8}ST6sS8pm1?4Ey?G|Q}4+`XI-AO7^_Z0nGZEeQp%T3$T4TYfm9 z$fbHjJ#6xTVbYA%zs_R(BY(omwyDvY2+i5R6?jlyRSEmQsh{8b*33e!K8{}!d%Z1d znB-yoBWZ3s&X`1~5Ed3my}I}@p$+Ld-`$(yM4E37WS~$jxF*42yDL%PL9(^Ukw|PtQ*h)yB`N zuII^pg(8&}hcA9F+M<7CC2fB-vyYE!PRKY-u@`}oRC~KZwVoVMUB!sOgnJM3uMY^aHZOigb z$G?61+qEuEpUW=w;R-d)!2CQ}q734EAY%DcQa;Q4N2+Tae)Cc&P{ugj^DBr_Y~0aP znc%^FnFWTD%7NA5mKmelA{q$={ELr^pT)8?U_gAodS*yes*-_oEqu={oiKz( z&h##2>52nB=&73w6IFryxz?u28q~|0L>${f+un(wyKz}*&apr53*|4X0(C0S=>=Yo zKLkEx2%u}BS9+L>EQ_#YVUNbR9)AYJSZ{;lU@iANede*%tA{X&_pLVE?W>-u@&OV=z%hrKQypT#OC6Qs=JmBMv%JiA?s&Pw63jBgS54z^C2 zrhYj((ZS56*;L0Q(uHQ08_$kU6G40xk8VP5GCfgW4XSJ=v-;sU!TDTuHwfj0>zXm zow_20fb2#?*Xu$0I8CJy=)IgATnFp-(BYEN_YbM%`Bg^Kn)C0KT+^swNhr@v-~MyR z^Dnp}a2ufV%;aR8zgW`bSMlQXyu6EDUAf&3DgP~_7$?<1^tGVi=eCo$a_t_x5ZT|^ zNG1M1Q1}B^!7sEY(x=Gs($dm`g3ax1DzLVp;b8_|-sXR$4gLS32K{@asZC2!m1;B^ z7A#&ZS;xEC-osF|tvrhUdvPn^BJ}3G_L7}6IYV2oPuB@ntqQN&O}IIMiHF)woVqUN z9RP%kKheA`xmr)@%T4;+iK$+WM0}cNwmduXwJxr5d7npsFJne(V)0Tg8|)6N{Zqlt z+m`DzFD%c7z!&O|Bv@Mt@#LqGnUKpuw|!0EsBCdzW3PAbOqPJ8pw3PmcPwPod?v_= zjXk5^SW5j$#ZbM)^%c#)hy1PE?=yoosyEAZ5dWKqj?~)lgFH|}HErGvX3rim$`jwc zcRS?vq=emAvqd_$bQcc{tLsq+(9M|?FRv5<;ZPL zBBDbJ@`zPYkHAh*$RXTR4m?b_W5Kt45kZdc;$E#FRGlrRm*^O4={GqgTx2At z%CqI^dSl%EeuGY6W6OU5T;^k=Y!*MWtFvey!tMpB12x2^I^x8*`>~0ASbcuB-*KE+QwPP5*G4p!wVG*W)58jU>t#23{I&UWSc>NdO`Y|&nOcJ%+{31yr zxo3>fvYR*O2Uoq85>Vh+2IP$0=S-5I&4=;I(1Cx+O6J>iC_`AX(Rkc{eRS@Vvn>$! zQ7JrKDu_n>Z7sP;5t!|e2u6lKPDyaW?w!@4D=kpn4~5KCZcVXbZK4TK*=lAZ$v=`C z{jt&P=4H+2npVcAzF#{9B1QwkX~rUl*{-8es#?;N8dANFx>wTIFdlnbFB>tFwpXS{`s|8#;|<$NK@apR^}HX zk6=DNq|)BQ?&2PZ;pj&|c$g+Pn7>EH_itg=XAl@>eT1Gd7bjiIn69gAee1%;>ZIkv zlms1+pDKuXRB=Rf9LK!S5DLF$a;5(kq!Rle zXmu9c4=uYpgz3G_I%=m7U9pXyQIM)EVpm{VAJTRNomk&R-$*7COLvoesH_VGg8~36 z#NqC-CLWdm2tc=u{~Q?mP+Y|HhN?ni(?kS$?i*Rj)P|e@H~S$jYX=LV@>}1&%M=Dk zx~uhv>h^e8q+HiW4XV6*tbr2Sfh`MMB~o=Yt)imRzRxk3ozz<8_jJ-EE|uP;Xu>Y8 zN9XnqX6JIao@@<8%4PFtz<&-#0bqwYl!7tvTJc&pRk7>O%>&El(bV=oQWF!Txqi6o z_!umzuK{f`SZ+-#4S@w(_Wt#A!VyP>)b_L{QOkI(K85Z!_Pk8{VdV84G7#91zB=}8 zv;43-=T5DAI(8s2GLpsPx@_%$3%bj@d7i^!FMToz=Nhn8-XKZ2#PU?0xR@hd8@K?PL-D5X6n}4?N zaazF-+)8S$)oS~Sggt-#ZDIW=UqeT%e5k)Nw-9!(B{Mw@IfK;>BSLo+)h&|%OaC5A7Sf{oTc6Sx=liNnmX?q3jF`Y7>H1kDyx%}KY28OZ5s3)#iM>HK9LYDDk zO)jj>LXC3O!GT;$BtJL1)0*=io8?mq`gk2#y{r)$t<57}-V*)0bB)8cI@)1^Fu@^Z zXR&hGmZrZ4TfR1K*!PC9@p!M#iK0;tN*B3sBU>ywBWHo_+tx)*zaY&WcX%DcdC4%^ zHywGwOA~qkINe~+5Y)g;(fYXdU-4qbTGWaY@*#;VUdfaxr=}h$awc}gS6>R;t!lnw z`;q&PJe2wl+{`#LMy`C^@zReHuQMBOz`NY#Af@lsNj$f`@4p3#2|gC1>APLqQc1Ou zn#a!eALc*eFC3hhN*-le*|kB>WbqKY+cU3$Jr6jLDMNygp7KN3%_st?J4*Wwy7NgFW8pt)+#+{)^D{G8Kzu^NpJed<-d^H-oAG~x zl4{^iAX|$+LVaa|r@;P~qWXl9m$<)iskjHpzoJe4!=RGxC8)Bov5A(#ae#~HSF>yt zZv4+ldGhj=7FdLVhRLKAL#$}9)TSF~GYYwpw!K^sKmfZ4qH%2pU+DVs-9O;$i~=~9 z{P&$ZDUJHL8o+F66;V^lw7;}0NrQ9Tf0gimrJVh{{*OFjMFlXw+U|yrxuuLW*ajU3 z!_uNk^Vsc;TyRfeQCEeAK1Wrvx+OqMPit=k?luWzY=$@Kcq1aQx!-LHR?wU&L-PH3@M+53jeHd%e9 zS%!|0@-6lqa7Xv;53lsEdtKkbykV%SX`R1=)j%YpOdU1oFMnQR0#h7J)KrZ$kWT-f zx0?P38=TB+WlA%z9PZh=kiXrMfBuC+0VGun&4M|Q$uFzHr_1=(sNP<2OVc>ao2a`8DzN_<<7BZi@+84L4SuPdzy1C{hb%BKmqfz7CGclAx}#d2j7Qw#4^{l{;o zOYST2xu|Vk$zV?TSQbpxq1bdbksj;P+~S}~*ItO`dP<7w)de^i!;ykG$|#MZ!)2%B z&R?bNz+bg|@~CYVVsljoRwMMW!NG-Ce0yDYx8+Ph!R|STQce|VB;Q7Kr;~n`iq)1G zf|WH6{_`GTO04Zqq$Rg~%=<$@?C+%H?~1Shz~h|fM}(UBkYS#aJ+#pDgdAd1+Dm(C z3vL3;e9eDw0T|HM4tXX@1;bt+i1Ox>kzpDe@3$0MoGvBy>@!M6B{%Pq+>#IYc+Pez zOP1XWUhuBwEDx^(ACi@>jQWLMfqaRVyb>E`u1%iO7b2 zm}`foKRwBlw5Us_D|P11h64U{q+yj?mtg?4?E<$SL3dGz&mGQ_LDTbmEvrNmIABe!e96b$e?tNqbaSZ(h`2C~kLevF@IKIwzTw9@*;p8u#6+_E(*N znW(Pg&lvO^ALsG1u;&#XPQPj1sX+p2_GigZqdQrGPIm|0;hy|APr53yc4n>72ApN( z*KMxlZE&vJEtT00)q`=W$_{-Hw4RPnGyN}}uGJf=GUvQAbnMr12w{w`bTHINWx;Z;HsCIS|9c8}|8?0=@5Q=+|LSCI7zPBR zI>+UX^f{jSpfm}8s!JSHEYH3LB{J|dUo*b8w!~^Ove4c6NOk6{>TJ_~CvM(Jru%(2 zcK?)i)D0y1ZEJQv#5w=LA!k{=;o@W)-F)+C#(fZTnah81aIQKyKJpEn{HOz+R#}R> z(w!=(oS(s6B1K)53nxNcdHXneSpeKQv#Pu9v=#zx*@!M(?k)e6=T4qY#M&<4Z)=7C z=D-(GGf-^euEtQxB_NKZixpLU(iQ{LN2IXcyQ+h$IatT5eMipAB*#6Fc%)lf<`Ia9 zQc5Nez%^C1l1QV#XM|pZXXIDhJ({eAyc}H)b~Ioh@etSaQgt4=(HKHK@lk>6?l=+e zxQqb@+$o2xQB{$ zI+&(?wlVd4v5~RCA;*gJ@{<2uk-a5~{DcWtQ}BgMoc6e=LHUBVOTFw`?<2UyiEDiaOK`c3SsKN#@u>eAmB%spR@bgcwRut3niast8au& zsKlJj(YM6*Vdf6Hd?1(c`8;cYQ2TxecUlG{A?{;eSTF<$80heo<|&8CPz+8{LMEuE z#9Dv+d)d9BkpTA9qF4KmH`R=R`hN0i&~w$J8+_^UAw+A_9ZN(;_4O+zb_K zl9R?JKX2X=vppS-eFs-ge6G)++og;}6!i!ywM?8AkQTSo2pNjvl1X?>5|+rD2j77t zbzl7%mzIMAIshj4A|Pb=M{+ctqv)4@_RMDGa;Jer!9}5O>travH{Iw(r>&TA>!WD= zim6ya?RFe!;4A@hT(a$+-F<-x+{oY@V`}TKT6HYuxmfO%NNPS)aM`zKDFAml$zb8h z=uG(S6J^rTN+aI@6pQ_7i8$`M?@MYov90S5pYu56pE{|YN4hPJ*FMX~;*5duww9YM zKW%db&LDjmlhG2hT+V~1`^U*^JFHVP5zwvUx`3SG9Is*-vFk2-vFjFOCG^g=r z&%peco}L~aF8@u*tB@UjLmbh=p$mFJhy7=(;EfuAa!<{b;MKX##+5hcj_Kn zpEnplWmfTQ+?Drcy*Q)KFeC~`o4)%WCiwsFnc#O1^hQ71(m09~9kX`5d6H0prTCUQ*Z|z{fvM`1 VH8@lfA)mW%oyBzq<3c#22pi9i#%Rv3!W%)GxZ5=Li!Iy!<;~V%EPc}) z;=L<2^taKk+Bd<+FMj;HHmmPGQbahC{Cpp?MwVOaa4ZOe*+_I{c&hdGfi}TpqDGlg zGf6gEueDgvKBh966ZkmeJ-SvgZp%FCP15FixUEP?N9;n_euqkc{PKM5Y(H$Zo8*?m zqkQrsy1W!i`@P%SW~*rZ;dBsuA|>pL^S1qmFJ9ea6FlepOH*;j{r}A9jy>GiSymTG z1C-`fm?9YR&A&vn&IUUr`NeX&po)yY*14}k$gdazUF%fF1s|Yk#Ha?;(9t18pka;_ zBfMt>57xXtVV3Gt>NsjtdTWTI(YSv4{3REjK9GC}Ly4QRd7UHx27}d159LDW^}7Lc zfBUId^HzNEusokPNec$EtT^LVMGl+cCJ`sFvH=E^2Ce^?Enk)Cv}w{lM_W2 zpNIHt4CabQk~MG+pwW(xN7

=?_mU-i!ab5^=IuH&ixOV*SuRjdHO9{ zQ*4HZ2INsu$seAKMO#v+j8}wyq)CR8p+51M?Bu$ArZMsci}j<&IZYwM(2Hu)sXlKX{w1f!Su%^EmIILF%o0RYV~;BT>Zo8Wu``q+*BbxR3ip zXHmNd3#7fXbMU#}K+5t{VVzlniJbv$tAa@b@8l^7bSHAcbjjN1g{wlbrQLAbu zUPuCW>gX*pxW#v<)ow3%y0H#Ew1M+_v$gs8_n1)#%`o#9k!y$;uK1&qz8!e1J$;z1 z;=W7}*OqfyRK$H z1c4AV0)Kj^c-DHiO4zBEzamBm-)tZ7$%wiA?mDPSD)6)X$PRpQtigmo%-8Jr8R8lz z@61NVZUYZ5P^(o7&Wy_D5XF8!qwJuwBY9Tm*RA0tsc8u+bgD5Vr3@*COf0KP#XA4Y z{F*9xGBF8b_97c^+&E150_nX7Jr6=aGxHvj%^QwX;(>$}v|;mOL6na=!D@AxD&8OS?+(1r;H zh+E@5klP3M` z6dM>boC11G5|MS(4pmtd9><$DncqSAW*s(heEB_VC81-TEpU($9R&{vO|nKWGa9VY z1j&PBxeod4lG@lH zICZD(0Pmb)+ocT`0saqJh?-&o_FF;a%DfWt+~@OfI*)*#N9=xAPGyh7{Wp1zP9 z=}XZRe*FxoXeOoSC!O<9hJKD`5WVUOp8p6OEz00oYiB1zI~%|H3MGl;m=wX`1kyx& z6a`S_+m4flAJDz_Jd-=;bwl)TTBYUfoh?bUbHo2C{#)xH`=U~eLv#x zsvD9S<|RUj);;xXFm^zLY4ucOi{bSQsLK{^He; zc4MPBW>+SScT*K-kn!O5gN~NUM{n%i`Ml6M9;bpFV&p`3_o!XeZv+%G%R2rD{R9vT zB{|<-aL>kTxU7uSOG5?O*ps{j)+H59`S@HtI zz^SEtD_%>jxzd+s$YY;|83<&~`&r$*?kNa&qy>|K1ZwpOit`U$K9DaXw8Y&w7YPJn zY!{R=-2FqB$Ev~jY56z1^ZGv~A>h_g_|NRlH)J;E-8(`e6u)aLu9W4UQ%ly@O~$D2 z@0NK@VIp?akAH=&u8mn*Vpc4u9^F4DXT^rvDCa`$}sU7AzUK>Y>EH^$Nahf#q- zJeLiU*z*Im^7-MF1>~w1K0yUjH;pb-`wKsrg_B7Cpq~*jBV$=vbHvh3_EyL89_y*U zgg*c-udThry{lJC2I*o?h?0X}We*Slzk4u2r|fW{zyc z#FsV1h~(?6Mf5zHgNg+1W2^RqnKYesyzhCpC!yT~{1aN3S*HWlJ$5tYkPgMR$4|l? zsRRw1n?|E<5~Xv*T~XTzv1bi_*b96&Q^IG=C0Q7FlB6GT91o5cXWSf~gF`%Ri?I!P z+ukwejWVJUf6HPwU40&S8c2U>Cx;|#lXx&o-w*YWPZuH`BbvWb3)yP+g+#@I52nOI zK-mr*HYa{0adKzsdHbDM>1vC8G38=v#mXLs`wZLL*pZn^Y9GrjGeb!C)G2>b+u3!j zq>Om9t?s)C0}T5uLZ5S24`f+a=*D4=mGA z*3(r(A`%gWCeRa*_tDlVzW0x6wSnEM=bMqR(I)}=CW2pbGu*qS%VfIgK{ zzdjXqX(@T{Cp=A7vuf|8hwYjXBPC{w3+i8wrOGFK!{^fiDe!`6(+Cutx3qJgmfx+z z3r!r&#SrUK#&n95hA=&}spVnF=THSOlhwY0EAnzIY!41!`?w@^o@$&o19}&?nKW^*j+&Fjnw% zzhtNQUczfD)`1HzvFu)}3Tvu!vjSgWP~anj7ACJia^e`zC>J)`cza>`G$?yya>vYv zqipug3&JG8k|W_!L~Wt6_bn`EDD0JO8yi08haauuAO|?zF2oc@(N(QBf<`sLR4g{S zunF-dNBeT2cobv~nM!k`IA9-DKBmog)J}g{hB|=KW&-J%A1L&X@;!e`1@_ z!KI)!<{{@?%6B66!b9{*P}k|e$~?Yjuj%PtVWGzMM$~20;^Eqc#AIZ`-dVlFfw~>m zl^_a3i-s6QJ3{z^{@Bjgc*{@Bi0R?QIqG{!`c}@2*|~L=EZ$_yoL(ou+1lj1kYVnZ z?QlG@a3S^Bj=v;z{T1W+K!TZ)fOGsV?ady)O>3Gut9i>E>6I0>ZJrGIfy6v28nw|chV1)Mj|;mK_&8K+*xYd;pmEU=$>W}Kol`{)OL|(bO^{2bJ=&&CF!3S5%jc=I zP%E1e)Mhkh@AS&9`N~j{s`)&&L=2CO0|!B%AA(P|@F>yC2(co)CiAzFfoHZf91t^7 zk9RHQ1p+Sz1`dnvJ|N$hGMy{05ba$pvm7%hQjLZ%jAe0QgZ|8S2h9s2E7B!M*z;Hs z!s&E-^N#qgzj7+Q%sIr1LbQjS7pi;;g1vspe~;BppLW{6n#<#K*1G2L(JG_tw}Rc^ zCjwpr=s5rIb5{F@tD5pZlk9N+Fnh)^kF|q~j|M$Ypu;KNgxc*}Sz6odvprq%qarY( zjGG6MaK7MDVSaAOG~D#T`PwKg;>S3T6MDtriE0v4jU@U*+&ZIIk4f^f@UW0NfKXn%(Ks1AHm3LHP3Cwz~*nhz-!`LhF;61Z{_A;$U2WA}g>xmlpiz3_P!9R-|0qsml3 zuFSr8HXAK!Ski(7s~!e(UmkWj(3n~%pRQ+RZCyr8BXRvaxvG~}%_08qV*)*2U32S) z&DYUM73LO$jCSVgQbSb}i=d|xVl`vudkRcJFcidBoLn4FO#_zDR?YH0UXYK_kj>rK zJEpx8r^u^ab1>M|+eKpH2MR`BHL?gS#RqwFwAUYJ@3+2aMpXGY3;_$CqxLZoF?n-l zHTHD%MMGD_wFjGv(?<-@n2UApY|B8MtTX~0O4UjY52JG5U=guu$zi6@Yj#qv4PN{x zPFXPyHi{3!tiw&cv#6KuPg6mUjYvB~j94Erg0LK@mkLjcX`U&YDC$P<^ut=U!wYk= zHKsFZp0wo#{67-i9TwOfnYVIHFl^5BR^tmctXi^X)Ww9jlr{@{c8QC9OK3GZ+)5aF z<P;V{s?=}zU&pfFrY6aEG{dX zAGI)o5-^_R#@*$x6dOJL!K48ZirRVI-8f-(8vPz;6cv!l-UksIxOIsSTHVP|eGrAC z%eQU7I&?F>h-2clARnB`y!N)aSg{$iV{U+gn~h@7_?uAKtc1Inu5b0Nb{;u?Z;W>a zr`wO1Z3|Fn#Dx%SlQ>!;RHQubQ;eU6;B6ygu> zw7HwWyuU(!(CEv`%-pv4R%jFYN&lT6nxEbxny|EIzX8`zDvHZj)JHV=0yp~ow=&Y) zH*pFpJT-`Z8wL8Hjgees?5N_5xNSzS?%$bK3d}T#f6X-IXZ_Sae#X`RgnQ8caH{<` zEJ$|e5eEd^yZ(Nty^V`xq@_RC%u@*K3FYMD9DTd*H+%SPHdXsgPbp2Uo!A&y%MT9d z!zg7VZec|K{V%6!TkPduzkZF1iUNRtcLbw4u$g@O_U(UnHe&DaKei(OzU=(3Ym#xK z9q;JuoH>52Sua?!QX5Om)s+-+F>b2jNnEKRG zZl*~5(d%A!LZQHF6GE;usS+d^eUdkapFUeyPkU{(<59n_0YFXhLZH4_Y_&4^rx4GW zg=mBT7SwQ4OSfbBiZ>ts{{gM+VXs*02as&5p%?xbOFtlSqOCg0|($j zv8ckRFbx<7ETn7IiFsFM7>tB2o}E?g3Q%esg|Gj%hn3mZBnc=On`tU|4tCK( zA@=GSf29GJ*8EWKw36zxL9v=bAMjCu1zC2VeNZg$(_ZL&11Tu8mMo*7SfXoNp;%dy zVp7qzakExnD)Rf;r0;)^X0s0e0cJIEKzlsmc?4Imo6QSQ(vH%)OBFZy7y2g+yn&jK z+?^(f0XK>l{CmsI_`I8o4ui1)L)d{^hoz0#*<;VTN{jiiNu{ckT02Lyh#nDs=)kMf zkL?~s4WDNM6$t~l^IxVG`+HpE=gdgnYl(O}H+HzV5S7@#Jlf<=?sDd-<&<{@FS_`? z5uVl-`Bku@CRL8KK&Vse z2E%t#Q)=K`rSHY%?cbr__lIOY6~Ra@Xom45g}$HH-L%itzq3RPVI@i)wtH9w4Jz>e+@&OJ#iOIc4G|uc$mFfxOnOi^zS@QXIv2fgLT1XZ6O! zM!@wj?k7m;jX;)8eiThswqgY^PQeSWK<1p{Z+~cuDDnU1-mevSu_XAMB<8$-4GU-#lZ;xWmI$TCLgYalmo<5D^ zSH=_Zl#PP2T8{5PY+{8RroKB4^hM>w-^}qHccyIRsx&zm9(2lTwdGaqZKs z+##4Wbr1}U(2=5-!saqz0jnJWvk(^QZJB(SS<>Y3c8SF}$#Tf_J{m6QolY1pCYLZn zn1#)6I+S7MsxfL12tPjt#;Y%^p@tsbn)@Xuy%@PSb6>A7Y~MW;Hr=LUIPjsUpQ_Sb zS_ljRt?Wh+Eaba~xuz9F&go^&Id!k;b?|60K?$A=_qC;Eok){y{rBtDpboMHC*Wk(0M&|?#blTxm(r&AKC~#7ReglKc;^j7YCuoR1>V{zV8sH5n0H_p@a0of=#!9p2P# zD+{Ihe2|X@-k-0Y5)1XnGEb!_nQ+yF5$hv5MPwDDLz2ZxFr~wS_AGNU59Azmh)e(p zL-=b<;ZlU328@N5`ke#aS!R?Jq$HJS`pq{en$F~iA}Y!kCw~4^DtHKV0u!zwaMOw^ z5Xw(u6N&!WrsHMeDwGt!PsJnR9e+V76zD}gk}g!JW+xDb{PoE+eM(mtN`yJ*Cv_qk zbkmi#5Y44gy)oqmj8OZtYCI6%@|Cr?JD+bRS4S!5AEh`=dLCvuE% z0&a=p`9SVchseS%>_JZ*4v~|bUi6d=oG`p@YKD8T2jpWq$GFG(pHrnzW+&9ULkN8< zG~mI7-yKEr4gw^T-&bYc2hR$9gSR#!+bJxxM8eb;YxaGc$%pkcBcZLkd9E3Wo=pMk zGoQLFGfe`Axi>xGNrmn7p^A*^byRbvXxJ~xu!fIHPgDR9w;GcHt4&qGJta!%jqwz- zk5g>Ul#`LGeL(nb1F7cqZ~5xJvnzn9jlMKml;obzwC4n85aOS`<3=#fC&!0XghYQ` zs_ZHrV5(kB^+6Oe`5V40S0Z4INeY3&S;VMtM10VW1vuHa3afC@z7;`XgZAiNzvxCE z8;eWl@ob}zS?TUVi+R7W*e`ifna=A`H0q3nLKO9il%%y35=^d0m;qn(#^``{IqoI& zXsY>3mFi>snp`mL9GQEt)GKS2+CQ8oWH zmK3o)4;zH5C9*QrI5KGorcb1oMuv_e5TLVb|8(}85Hy4Cc#L>XzH)G`gm@@(pdS_In`O{{M#d{s*iV=Me|syr;*bF{vVoioc4!XkI9SX{v$Ye-^iH!3;z|ZSWw_HRshbV4|PmN!J#$J4lo0^f{8+(ou)$%JRw4fg_`SOmkruXaEAIh|{A#_WuPgCycafVi`p{=IQFQ7=KB{ zb>+83Kp5G0eO6js((CiPPO(MN5ZXd&kxJ8A;LMXuz)QektMihkN!!0yf?ZA5pz!Whi-2iMKW1KF5BfYfsF|1t^MQ&vFF357_`+LaD6=|W1Cng8u2n5 zo2VuU?@js^&MDs`pPPlxaH6yA5{yeCotLY&rO)xUEICb%g3Xbdr^QWVi8gHHs^}}5 zHEv|C%%C@vQ40gip7~or9`=VfV!-v1qsfuB=@R9&81QAH{ca9GXk^hV=yh#21sHR_2 z&dA0Oce#{g$n^>scvRFVgN;KYFN zlhqj=wECyX^27&|P{N(S#1Q8vAq%g* zZ)_b6o?SJ+wzGf=PM|4OLFJilpw`a^6#n4aYpbTr3Y(q`ou1a-IFM+ZXNT?V9us4> z-0-w*GI_oZFr?%$P}s`hJr1gzR871H|6XGqhZ=a+;{UejpJ^`RMj5cZ^=Ruw^* z9vvBPP04Y)V&Y-Nh~|C%$MSjErO7&O`$n&1LS%B7N6IB7lBtlTyX)kC?TVov8##@M? z^bB!Xemv4&i!-*kaFLC>`E=K>!I*N#f2VQi5hN2mhohwty`;49TFOrO{CTJAm$yw8 zENIA7FaCa&QB#RL^XM0D29^vBuL(2%D+a5li!Ye6RmhBZ!&f)c4E)5Ws-Y)!^2AyJ zE(Mpa{8#<3gFajyZ`55Tm>#0>LeySt1w#}1bX*d@#JbpH|6B%v77l+5T||pSHOxz$%p{@ zB~VSmZ|FCe^{lhC4ewRai`$t3ISXx#nyg|H2=qO_ePnh>&hz1Q7JO%LFgi+biX9@n z-&Pa|l*+xRN8`*+Xf{cII>8MhOLjS)6N!2_)+#P4F>!##zcEG z$!O`_gAoQIC^a-tS=k^~Eoqh1EZN!m;rKelmMT&!c=UWxtClJeEJHni2L$*MPFC#c zohfT5H)0094#|d(a&oX5CtV-M+TAJhp~-Ag99z!un6GzB6heJ5q!1U|SmA{mFhycf znB(IfGtzr25=o!0*PHfOm%$gSDI{l(O2~dQLrST~QZ;~c1iiyY2R%MnwrVTc8{W7$ zBbQf~Oo8|ZvkwXJ46(eWmh?FOrca-_#^aiIRd1(PajvElu_RbP>}`#o1^*N1rl+7Z zJ1FeXW|k+w)gze*=M{4+f91w{eES28XNhpWNz1vx;=s!d42TX!>7B#sEs@UpgOI;z z1&XDIHTugDdbH}_(d-%a<4+t;1bVh{L-kGV?KPQJ2CQfy4bXGaxcYxZdjFmxs1gF| z!jRm&6VOm80BTB%j889&Ps`NqZgh)F$#f?rk~#N=g|)_SZLN`wh@gwsEk%UFf>_y@ zfq6=$I24Syv08GpwN~kP{Du_`s4K(#iT?|kY%1j%aQH@dM&kbpoBl0L@lQ+@Dbu^% zx^O(zZ#Eh7<6yQG%_oe+ouY&hY~An{uU+H#m;BQ9Vayolcs^8&%GotsmEoFN$m|^iKeb%j*Fok7o$fPks@w-`fF^t9bz0; z>KO5h=f;z(BBxjoa!Ao{(bU^Rj1PP=I1!OevlZ=Q+)~Y+lsO0QevxW0rZ${p=CFo5 zTnmn8G1VCA0N(O9JzSF}=aAAdx9e#l?i`!t;QJCE`_6hxI21wAalbT=rW~_G!8`uk zSmXaUpxn)WW-`t8k}sd34~s9`WA0#o;y>GPOR`(aR5kZN`&XcGs+jK?fN)Yf@oy#)u0zD#iXatm511lF2gwd1z z>C$KB_vblp<%bNK0t8HOvPTNkip>mX901;x6k+ZaFEe7Z@aR5X$Ki?iafGw_{_!&F z!9uRYuR6HgXtI%k3-Qj0U!m76BmlN_i`1oaP)MD{^ ziGjMN{8-4SmThb6XPON-Lk7xs2x+moV?MP;G6I(LetU65@Fjl6fomWwkcn_%-eTnY z^z##ci-H0F?6B`o`RBtM*9^EPlmqLs=&hEUbN-L@O4ef-JS?PrE_s38>sA(C^gL<4R5jyt5g zKhXE^GQEr3+oAK=+z6VPG&PV~S<6RyXwVJh0dS@D?##muUeZN) zf%VXaNh>YN0xOg&DT@`-2^Wx&`&rS!U>aM~pqW8Esyj1$n733(8Ea}vkB_g`KAkz% zCCou_?Yjw4Z^<5ndt`$56N$#L>$4^}07IrGhdz(|Q5(e%UxxhDJXsC*tXkcfSh9|g zfdXM8BjYC}37dOUnI#W&!^Yo@BAGJsdafUY4Le7bG%PU|SGAyH)wKUvRVT>GT;hNl z8_U-Br)`rrA3Rv!BO>6aZ)bmlCNX~p@_oE7QyQ){#Kbxf6kzi1aDUG_-fiU0zM>8c z8oicVftQQ#A|uFTA?GWH{?rjEOA2G3p?s;NF?~VNREQf7KJ=sj@%fvH;Mpj9my^#e zJocB>)P>u{^}f`-R&D}2mSETP-dk3ab;J`@Jdc3LfQ(h+fTw0Xvlrcz7kk=4)BB{Z z925cy(yWbK*Ljv-tKchp^PZ{Nquqh?kdd|dy7DXcp!s6IvxFEsUSmnY{OMop>qt$( zU;P>!z>KNl?9QZS0qPNxZp(}7VS7@UG_0(&%x@;f9dsoDkaIXI>2r3-m~K|b`mz~( z=l7>WqG^}g&1wm2yWhG$&w4CC)_8q-*6s=h4nz1$DZ%Vk>BPL2f-Z$p~ z--HDwuq-vY+RT#Gv=90f7wDWtWEK|aq^JLcqVx3pbY1SzMD>L&{ljh@+YIg)=Ri26 zw*r#3_n965Y_kyJK&Hqp63NgqCZ${VND*$2D%|qdZJ_=BN2#PlrpS@RsGuIMoEElUWz6 zh=Os;buJh6wQe{RJ;$@BgovaejBwf`T+-#sntDien=E%l+Tpn*h98;w0Wti4yQ|K><%R+-(H>$6E&*%DV=az>)1VeG!$$v6r3vHEHWY!%!q(BrX6OwIc zEO{m~Z4cI3#Q-3m1=&og*5>Z6rdJHWb2JrMststL2ES*wKyL=CsZX50Ip z2^7w-m#>IivSxomY%tsHq8^;}ob|VW#LG`JRXMPvr_J zI_Q%Jyk}WL!{cCdOay@3`;7n(u_XNAIp8Jw`$rE7f7oBa^FdM!oAEx)H?!qAE~|Jk znV-hIwn~dgBxJJR1g1E;_Ue~irM%k1e6gmwuqAGtuz5sS)(Nz&>_fE}T=Q;j^V(cx z`H`r79iyoLDc7rF(ZX-@I`}?8U>1L^6e*=7Oto0(7W_cV)ZRa>MgziT?6cL`TSzT@ zu;oz3cK%5=qy7@NtI`MpZLhZ1rr3MtBn&)6T6d&*=?y4x-Mfu%Joo;H4wkW}BhKEo zWwBj$5=`l*H^WCu7M`4rl-?1Cta{#RK}8fTdh|w53}|W?ytTXqXlJh02(H4aIysEeU32AVPRd!k2nvO z72bl9$96jr3%qyO-si%XSO5@PuG;R?nd1&@t@JZX%6Z|L^NVx+(*JyjN_2lX0rtvs z4#E}U@m9V{$h6kOaj0O<@L^rC;CtBa_bVztboQ_)NY|~UoUj@B%1N1^T8rBa z|9nkgw)@+`G}?HF;r7>(GwA!G@82+N|~2K>+I%z zYP&Osp_iC3;;GCKyi&E%`2c;zoU#u&PjP)~AB)mk~#If-6#=H(hZ>Z#`wH}U&ZoCky z(1sMyB#hm4DDW0U8<=+7-f$u7%jpM8Na8?)5SSo41|3^bWKW?ChlNn@wX}k1A@BA> zH|khBDz7U|dFGbj{G}PCr(1L1*hNrutJ0fQ~l`KbX1~=LjT+^=2 zPbOU9K@CAAF;8)@ZOugfkIB80m7k*wgrXLflR#7nI`PSQZF4>;-p;4Rd4Z%SX||5L zV^}42mMHkPc^LY{#>7|9M5+}H$YvXY6J776L!B|4GF(a@1(!(sTOX{Hj^R+!Zc zm3Ui5^ytW!wsS|mA1W4UxOk64HNqNXexAAAiM&N1oHjlgn!h~FL~I<9+_eB{@h3(eV zDgp>3mkM!$7T$_RrN`8n1Y&$pnKQ#bfcq`3mvvDjS6+mRxvw<>-B^oKxLYlj5Nlb!Zn+hZ0!J~=MSgt#yb}E z*c+W2F-}vt0bjAZx`W@E8_uMEmq}L9B_a~1C(s=4ZJAy1UH%*oRF+p|R_ti8d^+24 zm~r8d3HiZvdQ_BtbD&;zMy3Z%U-3T4dX6UJSH_N8izRMPI{!`GEWX)Ov$mhb7VF%z zkxt;5P~I+DPm}!R4EcW%57nnXC3igmTfg8}JG#{$B}JO!yr#5I_kX-(^mLSxfPW2n zUJEo_y-?lv2IrK-MH=+P^rnkuHv)Y>B=6b9c2+9Wn0njAV4GZ)xUKZOwnKCw#BSo9 zAt66ikCtRonDWfkG&LzxMaJ8d8IS^BA`fU$W6)~ayf{__U5jVtO9*gE(o1vVZ9)NU z6D`du2bHj%7DHQIl}K(46`hvcNB+C9awn_^>=@*@dS;{9(1t0PVdc?!rZkuI9_1qr zwH*WWCY6uVU?{>jZ7dxE>wDF=fBb)K_{RjFu1d{UT%rJa5sEO`=uqs{N1DGBrKQ;@Xz%2lz zHmJ5wX?T2A{CBOAjPjglPOTS$;Ap?F^_e&oDjE1K%H z9H9^c&j@Ns4tsgoTdaj$E{!&Yx?%a1^;+q#6-!re3+IL}1ov~JAmS{PhIJ=%T0TBL z7Tca*TnsV>rPPl#-1iFVFc%gxuv`+Udu`%&Y|g)ZE#FAE5k)EK7&gEPZjKsdp8FeS zhNY!h>&|7kJNnj4li%Kq<73k!0}lke|8qW(Fmtcxir^@#>t-i9j;Qf1t;_S&zNThM z-ZOdAmDhIU#o5ku#1g!anzK;lO*ZT!lcxVcAXZHkz`*s_xHaG0h*cOdo86I>ET z-&t97;Gd&>F_d#(W>2{xzaP8f_*&DD9*8a~{2i<2?9V~@hBc3m;t^a!CoZVC%O24S zeO(^{Kh@f;R2w)edNZ(^?c#tvxyY`sKcC>LtC1J7J@r&dRp(J-DJ z*BO3_z+qr9wkr{i&1md%`5G~ZY1sJ%IvY~>ct(}G&Mq794wyoM+KA>9mTvAiS3yc^*xVvGb?>8O^&EdHcj+X zh39?F`0z0KP7kW%rn^hmceK#XUv0m*x?ElAU_GR|tqJBFWTA>rQ0_z~;Yr}LXGgd` z2&QhWG`NXDM};t>`+3$!=!%D7sq*ErvQlT*WeEGEM@Me}bi^}CS#?)e9#GIyyFCGC z|8^{O9{9=ITeU=Sw%3Cg$%n0_f!<^02Y>rh1z$Sq9^;zJRO3l{UUIWG7cJ;=xY>L~ z>Q?T$&^XP2_A>ZGoH`Mdn`#INk@hM|hQ!rur=Mu$z#m_-Q>{nq) zCAXtevnDWJC&{qLPD+o@N~!xnLQBL9uO6%f&Bkm$&JUvii>0)`H!`g$rD#otwj@XI z0pFI|QDpKWV200xlybG&k@hlofD8gL(U!h0t0x`*<)&%aL%G%>oY{Ml^>m{Il%rIS z2z3P9^?Dj!Ngq98fTO<{a#6U3{VyESOy*7d^opwScjlv}HS@%B!^@@_vOA#I)ifin zQ*-@o=A~N{)V6w>vtq%X@{JxaRph`SzGinFXgpBnY0uAbE2h!5{WzC0 zhn2MSovGmEVC(&Mo-KAX&=14w(BtjBot2V;zEJl5H<_qT%6J0H?7ew4Wpzz?gO`&h zJfKclG`)$6mNvh%%}l0!CS7cV#VjHiV}V4-lcIP?XoG(o#3( ziemD@7Jo%LlKi$08ejuI!^to!&%#l&poKeCR@T-Z2-&ZOV1l$z1!OAhaNcOV{56TD zAvboNh*sBZ54J~F@TBf5=nZro4~&5?CE*LI~s)3 zV#ITLb>5pctO;ba86JOCV!xzCP0h4sQi0hRjBiXPJ0Do9>@+i^>edB zuQjPR<_tYfed-Y@b}8G`dPtzokjQeot#K|WtyeOEBo5LUGyI<=Q!FPxa?na&^EusE zAGY$obbn~)R)1afzmh?vl_n-N2L|7giDVU4>dwB`l(5Z{We1De?)_8(fow+l6seiC z_gNc**D=cAOTk^RzlK^|&VD>JA@LnxI=$dDkxhS<#NE13n=a;Nl?V8exeG!8H@{Tw zht9{y@%fCdA+JC-jDp|k>Xcz3Y`=<596{S?s}vWhqA@`SjXX8Cp}DKERka<#r(4bV z{s$kmquh|&Vsnp#B*f1d7WHtw%JW9$L<9MIazeY(?F%8JawSQH( z-j!?edN>ePfqbj}_IUKg=j516V;TN-S!x$b@6Vmg3$=y6g_Y6E9ZNB0tNg%JX$Q`|4x zJApA{pz<~gw?W!+>BucFO9Zoa`wa;#8?BJwJpo47bZUcLEbmJI%+1poq7h5W8y?;+ zQ34Rpe-%)T2Q`Oy$Yk%B2%<<&o!Nu?rVjWDBYq7@W+1r0Tn&x7QPq+)W~j1X$?1$e z3(6N|nXVOmL{0^apFkSz`rolI*#;bq`TnZS-Z|?08;SPqCOLghp_{&h@i3f>w{k27 zcIbbWcug4Z58ABtyFS+pEvHQ=97hZQ0Zi4A6U6D8FVjUeURt<0yVr;O7NhOUNXvo+ zki`})rDnMQ5Kfh8SAG3=eU+9^I!zcLJhV=0U3W)COsx%)lDZt$H^!Y-AjhxiEA;Z85jQ5}A74{)pT=-T-8mQ( z`kQUPP_czvMtTal9q#p$B%!5|RyScqlBT`1{(YX2V#~uE1DgrYsqs?_P>1vo4K0Ot zbbdotx+!|U^D5A^wqo3PJId0->D~_xG^Ak^?ms*#W0Xw9CL`vc#z-%8up@wYO5(%F z%g*b!53ZZFaQYZIJx_^-=|Ib`3lCB{+V^2MA4~KtQr&HAsm30_*C@GhFBv2oBcvj| zdd~VUG7hLgJ89pmxvzRRr)tAA$3mYb%r!h@q_U8{17ts^rNr_+skQyYpa8gH$j$Z#Z9|^TH~=mq_IxHHYK39^%oKP-pJ8 z-8H&6fFhSWbC@@)tsZl_Zn?3NuWKR%6ZK|Q(|xWgnylM|L7gu%WyK2xhZBcYvX)2H zM|RKE+LG02&PS^j*{OAng&~Ai1sOrf`kN8|WDYI_a}#4be_cFSGb<^vRf?JK#3BiR zAMPt;TV<9gMKl8QWSW0H4GD`0MrZ@nsI*!9J>F#9_*iq)=Xu9 zWh(;>8~c5&q(`}GarSU>;V=w^Id1wcL}IHDPOVsk0;3%1|I~v)gQGb<{6bZJ7L^Wb za*x|nMxrD27bmyEJCzsXs|e#5XjGDX8w|*4ZY}KqFEi!t{MB^u-sE$FB^){v_*sTi z%2?D9YVvaSg5__0iH#AtgALAKvafC}qdC-nA^$ybn|qwO@m&?447}agbM;%+d3$jDZGhBgyBos)K0E_lMpj6Jn6}au9qzs$ zS?8QZ)f@|tX&{yjkN$BMDY)R6+1zr78)YBRM;#I-bi~3MSRg!WzwuyFsA%4Az5a%A zkyJWrz(Mh$ZMDuhFWFBh3@EE^ArZt}sC2z&9@M{=au54hNFyCXF@(NdGFI2V07YnW zMIV)&Z{b!bArr)m{to4?*md08`v0lyEyLnkws2jN6_P*#1P`7x?(P=crEw>C@HFlu zB)A24cL?qt+}+*XrD>cyS!fuMwNil|PbWM;0Z24Q{}n$@FAPIY~*jkjY% z=!HOm)RM0q*p>>tWG-GM6cZKY<6xOAd$7jFQ?0}N5St7ATC9N7SYHOo%OFFbC%%qL zuT)lU-6)U%^(kgYez^dzQ%4HdkA7H1qcThM8729oe<~w%Po=}Mi&00OQ+7EEBx$)} zipLYaPd8>TQ_T;9O0d)3Y6Is1{GN^l&EG5rdCH*GwoLH_O}cr9$Lm0;^wP29>qU#h zn#WVo?;qVqoPoy_HHtMv8q&0V<$9LyUl2)@Uu;(~rYB&?$x{i3YQzAoMDrikhNGzu zW^9>oP*^XJ7{x%C0{O~tEwkg7` zlNC5fiHQktfFpoCA$kD{jFeR5pK5FS|AkEdaX*2l_Bb(TZo$Dm0xlwmWw+SI2k$5;JTUtkF^KZJ{=_M%;V9jqR0_v7`k{Rie zEDHx{wX;(^x}FH!bd&bheK~WA^OnV923lgQDaUi>9k~J6L3<$#4hixc4u@Ss)3quF zlE&4UGTijCAdO+AD4}utuc@XK1&^Ai$g6GFa}6C0ZIR^rYdMXw-ClMua;v}Gv%nCV z!2-k9)>s62C!?8F4)dvt{(OSdZ+Evg#`7Vep}bo-rTUJ51MwGiTR5RNDk?E<_zyX+ zeciPGuOb>e?v)?g5Jcj^=q^Gi zK2PC3+wyr(nT|+?K1o%mH~OLUf>8a(kfJj2y(`x5`tt8ex$ZSMKVvf`-$pK8``stp zUO=d1R|o3q)}KwZW*Z4T)CDYL24}cB(IC&Fa{Tn|I$e-z)MR{XW=16pO?sPOca4h( zM@sK?k%NqHTKvnxWaNIJvP4d+>>b-JSi^Wzb*7<*Y9jZ^UlMVjzML0plBj&ui+Ebz zcGp-xr)H10>`0vS!VU*WD>uG5X5w=jrD$Nh%UP7AYcg-;JNrT>6uF!G)jGeI*VN&0 z2-JTSGws|EwmjX}2jKPt2F5a2%Kae~692QfyXb;Bov$!Xe-@U)t*4D7E6cG6u;+)m z2dPiLU!;DqFO)-egucCD?*ekJL=N8M(3y!;|JpL8j9Dm_)5TUN>%?K_9JhPYw;6-k z$DR`G=g+Zh=eZd${1gYsBl@q3EX*+mlodC&6lI zVi`%@1$i`9`aM(e^pF@u?rfDQ0e1{zzUy{dhSur=Dlqf063ug2U|(W5YF+v^1Oyx& zBE|nG^w$f2*}A~ww84-|*a(BQi*5^7b^;N@=viFR!-Mq^-*4$PRUBVZX0$&95WJDX z=qIC(a zGJFizom$yrFG?6K?b{Ac>x+!i3}4XYH~V(5tG$-U-(V7r&A8+_A11Am?DlD^a&lol z*d8M|-wqX;a7Hag!xz^Q&O!-dYMVqpLXMJrLHUFH`^8}sh|O30dfmcFG9#M}lqSP7 zvGdU_Hh+;1%OtdY;|I@%jj+bI&PlI?^m*anEDc0$Ej;+j*H>5Tx?<;WL1le2Lg%z2#1ZoUj{6Bh2Y{3d6)JyYgcT&e5IDMMcH#i;%%)yW$Z@NB4y zV#>-=C=ugmA0%H|r*bS^_qIRkz<_jkOO0n(0MP}8KW20ME1Fw9T=SZCZqr2ekDsF- zFdM8!$l>IdFvgv~iPw-37^$^{*2pai{kDnVXk|pnQ;1iu~uE9h?=_|fAQ}&vSZz4)fyo1Dgt@g>! z@Zq<-!UYDT)`t#wfN^?CMF;$WEe`>_tC}wXz7nb;MqF6bT4s4djjl z2-D_ge}E|TM6_k!R64*upY7Lzm;y=FTYZv4F5q`Hl@F^}wYQr$a}f`MQNRxU*G~?3 z-MOo*=bZN!eFc-#XM;3|xWo{CTvN)=gBr|G=@eTvn`G!NPaGdmPRh zwVBN4%60o;sePF>#+z595~Fg_E(GlzR3UYmRrbr2Xp`rYj9e4~l10&@q+(GqKf;ds z=O;LUyDULmZwkd(R3s6zDJs(I&I%~jL} zY!p5}WUXAfnHpK1GOZy=iOyJ`(OS|#4Fk-upzWL2lbn)Ydc&V~6hV5q@=s*oa^$;x zk6Wa3{~YKvSJTn53*KFZrGadKZChy5Bjl z{{uYe!gSCp*mjb)%x(LexvS6_4?D$D_Dqy@B!V(ueLQH3HNNdbLL6e=fj|R=Ym_LP2lSb@Kc-S@oC< z@)`|XqKl0Y=K|{6qu$EHiDZkz2}j8HDH}O-!d7rtbUNM6Q}~o@6-3EtP>FfQJ1VOa zWVBt!#||L*s1Xn^Q}6H^xX~FJh`HODXqt0snS7o-3P_?Cw-WhL+DNFVVJw^CHI`Gg zSPk$QzX2C>zOBl|Jy6vM^xJi+#5xLbp@U%@V@%Bce525ZdzY?)D86z{-yUM-l?dB) z_%6SXn<+OQ42-1;QFTNNvZxw}Z`BYAkwmzn>^@Nrwc{t%sP`ae$#Z)HlN=l6&zQNK zeBoSBbMnr1WAsuW?(d>r{k5qfO!LEn`up5y)@zw842s(OSI571alq;7`1Z^9^@Y={ zx$%6wMM~6WVA2ZE2%$Gd_aT)m)zHrS!EY`VTJ-GZOco29&p{y2)maYZ#P^bPd#S<` z)RA95z!_u?=V5O*{1Z35pKs5tp|P5H-#dh$5n?Ko6vgQ)e0Q{5Z(gN3cn5CLO$vD< z6J>{lMnNkYtU|I2riIrxis-HDTP>jr3?v+|l;{U9BvQy#aPn;eOvOE~hN?-v%>T@^ zaEumhyuKR)8(@a({rhxJa#P{t;o+g65XTmqOgcXPtf{Gq6#DTC_!XkUaZj_+moBQv|}<}xxj*ZjP>%h;a_4^K&vq!$vZCrV2pE}%DCTmvlk14V=X z=Pw-NaW5~7$U%m#E`|&&jLcMqhS1-wjFJ}6EV|TH0N4Bc1QQz$?#ruq4;v~nGEwTS z_tHR~TapDqq}A5FA%~*EV-;?vFK4-Mp0u`N&}63C;VNs%5t)EMy;JM&1SfkYSA>Ed znEuj|7IITHND_~GA`7P%-{XgblGFg#`3_D3*kt~Ho7_Ua;{Ny628Yx3Z5p zDQTqP7Afia!*ewt8T1Pw{!PBRdC!;ovX0e2R8=LV3{Nhz?DV2DUpIeQ*;!*D-KPG% z73H8$F{4Kd`g%`~b2Gj3Fc!Q1&D#INebC6QafZ>+T)H*nEF4cleH?ZwxAYMUt{5ef zosF;O;b2E~E_6CXSF(6w&lqUbV&MccARX?m=8)xLb)l_9nQ8p1W#1oIjAPk49$6=X z{oKv;>mjggKV>m?_A7JIX`#667k_-QRnccDQGx1a+#g-$Ciybm2LaWex+)gcjcTbo zGOgh^>be6q(;s0|Rx-pSLq&w!47GJ4K= zcy_rpp>09SN5>Ubnuh==2K^M?viQp_$0zI##aX|WTCs^ZhToeWE(Z2`ybeSyr8H&r zvAkzH|0oxjf}HG5JtRI*Xr}!2JqV1sw_B{BcyfyQ0q3LM)0aQ8F=J5u)i|X95mH^Z zP@+=xP1}r~|2*C=R&oH3ohh2P=DvNfs}xNLv`0pp&*Se|WarNmeJHaV;NNdHr!3MH|h) z9*kmJ=>>W=PFDk?qx$^ygrnIoM^l=O{KccgA~j~RpX<~3z>Amgv?YfR}&9I+9JhX+Y33lvkPvG~IDtM%ENcQX8(BAD^goE&x= znHx#in@q3G&8GB^i47O9@~MnLVlBY+>@)>IhmbHH{~26B?dK~dR8x5f zFo7%N+Po#=JymA<+6+1~{c#DO!6Cj6QzTprvy1>w!a_+sB;kQ*_3 zXjX0{2l*MIpbYJf)=J?$&~GKnL|`}8$3(aWh9u2%U$Y&ZZ=9Eqk1`Q{1?pyALejQ* ze>FLIbY1o8*bz8Z=5(YBzN@FH-qBEO>X)G2CYjFw-i34D1z(Zn+)dEcj$2*Bu&JHH z(e_-~-1fvsu{zt{{AyankYO|%-^Y)=Q>~G_^Q?6;U9yJ++_jDtQ^Tg)gMty-mEP*J zs4rik+kFrlg5Zn|W$N+Yo=D&C@D7-dZlE(d3k#D3bK`_*FJvxbv7N1#YEH5t_&!Kp zeSSNEQh>U1_hP)JN!Z~5bu70icDjOp(`R(Nk(|!zAj*DKzqH^B_za=zo(t3d+F~5o zHuSYRbSt5Z@cqqdy!_0UQkkNAn})?8{!m}-*?H(83U{%Df0^71`o^_o@X->9kr?e; zsDa86Qp`PaD5v8m@3xMk=v!#`ofEF7ND|-K6h}m1T8@r_$PwhucB;7kT(;T(-X)8N zRvL0wJSmpQ2V8c%!`@xn?S8HnQPIf+6U%r<447*NW6GXd*)zYwoViuC5!F_VG_I3> zx>L_}^JkpR^8_2Q%Lz}Iu6=I)=z6{L?SzooL=tQNc2E7Gc+XFg!@{GrxZK3rnS^nW zJSVI#=5hk9v9`*rKau!~^&pXUY$iZhc(m*0+oq*!fA3Ove6SkUs;;Ki>dv~>Qu*!{ zF&M={XNbd@oGY)WXQs1lG0An_;fI{EsPNYUA<2duaG?s}1PXamcaw)j<*wLA-IN7~ z3H$M%y(-~G&FH8-Oc9E3S3**qWSZl*dm`6M_4S$lwVr;s?z{!UZ%9SG!n znrGlR`^*k{CD)as*Tu)n5w!=B>5^?hnzzHFS@4?_rR3!-9(Ch`UnGw%9*#3*K{a-lX44qG@tRA}?xD zJCJtLI++NI$PWg{Qq3GD?c z()O%Paxxyv<@x@|?*3Zu3`W$R!=)PN@%X6D-Sxv(!oW#c%n-Ab`$^z{)4O67U67n} z`|LP6dZ~i?VWWx4lGU`mopOCF{89?#wT*mBz$xzT zlx`ko#LKu7F%%5SQ}^Cyw_aCL7Z%~WXbX4iDlYqh6^_}eaxRw#?*_#rPog|=trH78e8{u#uY_F z$7q-aEv?OpAr)YL+2UPE)fejs`TUS2qawU#>eFHkA)Y!#V|op7tGr6Uv>^nBM}=H< zE%wAA=K{40G!26@EK^T2G_q~Zy204rCdJ1Rq4;p`WaS&;W^xL9uo_I2;Gqf&u&{Es z%ruQloqe8!+@817-in$&&AgEHFzFe+4hPv_bUKkneg6SJe~+>0JRfl5b>-pGC3yk` z_4R468wicvuGG`%PytuULS?*;2$7dYmBeb?^%>q^8-k<*^;B4$=K(<+`@H)`AS-*ITM+B~SM z%1xYV*}ALqaH628?gFW*Y{RSoC9u zns+=5b#gvP&22ga_xqD(OHTEa(+zA-nOBD?tuxrD52Q9R!4IkU6G-= z#F=3{c)C}oL{~iT(VHJ)OcJ~>Y_i)TMxUG=aN~ODkWgi?gFT^T;xnjL^--aos9fZ= zxyF#|c6_RpF$r;RsQT2x12$t=Q0YFv-KM~!ZY(v}3^ge$Rz|$oP<2*|1%`oZvPPE9 zPUzPM-qhAsnhikKq`vCEy*m5m<%MXCh=?fZj?V2O`Eez9PZgh`JePrVyPy?M_1Qsq zLxmuo?0{j)D_pnKyXc7~jr(QL&GmV^9=5A>%2r`7Gu)f-9~1oI1(}p|)%)z6YOz`^ zt>0IBD;NrC6EV!G7s1O(aj@^;blpjv{?T2>)tR5K%pD~}Lt(A?&*fiWk$p-j(pC16 zGsw5ss;pegfl<#1P`eDx?iOx*d;0bbjr1@V&);3A)Sa6eY&ua zCOIppC4iWdI`hs0cRA|IeQHasO$#FKh==KQFwdj>B>U{lu6zGc(3A* z_r&a=d{+16XLu&TZ;vOw6gS~AU3E31*-r3;(CE#vk0EE3i?Efc=dd3&(HrO{-#RZD zaDsJKPkxjN`5+Uq8IO$kI>W-UIJESP2ujp0P;cTt>&8f~Q@2jfL zP1B>}vUYOAn>e+LOHKn##r7tia-$TdU!j_8#YysK>)8=IG^IE>Y`3Di4<+_}XwB8w zMcEMMSROIjd}zvY;as)#Qj>e%?HAEP?iyA73RoBPSq|m9UjJ!~>yoeLmM2}jN;9p* zE5ULO?NtZryed{V&Qd=#=c9#S_F%VX*sM29r=e1nqIWf>XVD!8ghw10^b?CSFVUAM zKvv?MLpy6b3D zvLbFsp|%`S-jH>=&G!j=)83*kU&dD9me8spp?sEIgA>N*5mXb>(^EN~_rcdiS53jr zo&Hyf4%*gth0JZ)+2gTIo74rb0!=huUoV{+cdBm*HO;vf4pdupk`l~iN>r~LDO7DC zznxs|jMi=cc3u6(q_Hr`ZlyPnj-a2YeO}g69Tv>Pyy5LEdyZ1h&hKI{qW`O1ps?nx zTvLW|&-eGBnYN5B_2w(2*CzxnR*LW5rz4V`f?OFf`5L|XpW+jBX4+t)gL++^*451B z-d}LVRy_@h_Y=oOzaD5MjIXk}E-iG~?*}oreb}Fj-uyaW!;B^uzIVjwyFF&VQzbAhkt*1#cT-710gg&`+ zLL<;WHsC4NumrWUT z({e61hK2)!)cM=X??qdDF#|1k4EKHwBzP;YImh z1;qFo71;RnJD`AAb}=g|G$-U<&`I&s>Ya_K%^ds`!*M)hgl%9o`Z(sRmfvl69QaeP z>tAh4HKGOCi**w?UTZVx@6CF*RW%!kWZTan z#q+n11mwHfyH5mJ!=y4h2Slaq!-joPwiz)SeUO`~^d~Qg(c*Fimhz%Se5G-#4MvxL z(+8ocJH(YXzzXcZ3ue+D?9;6_#H?$T`;*i$&Y)Z2cJqglF9 z5_gDMfnU0OocMHOEGVXcl-cwCHZ?W1HNnfvOJS)!BpFT3Y3&;GML!LdG{SZMaVMrg z{U~~HFND4QV6Iwoc9YmFjjh+=SBx|pSIX7kQ>@(kpn9#Z9d*f$Pz z7xm0|Jo+$G?bKqgpJHMwU9j2oRJ0rORr1N=vT^b_amsKt-}jFUs=qJ@m(rOz?HbX9 zkyC0nX3al?OY!l};|HF*M%i@b7bttTz!IMCP1%sH`)HnR{cg~9c?dPz)g+q4gE~v& zQE!ey_@WLIF9XTRUB*Y})2}mKMYqto%L!Pfvm6XLiE!A%A(X9r3tv0}vx#zV@cLKJ z!mJF+=LB>cDqubJ1|{`H2?NykqL}~#9o=@V6RD6W`Zqc{x(*G&{K0qWSjr5{w6~w) zWLy)$I|tCXhZzr8R4r_uF`{0@_R^}v_E(Y6K|B%v4hMVxL}7T-^9N@e{Jjb z8e~Qzy)fL-I_}ZY2w4-=*G~t+gp;8^6<1bPgflNLaSg|RO2@|>XYRsoVkSK zt*DaQF_aVW-&ibfb@9Nyv#6d<*vx#}liPR0r)CtMl+jSSqQ2*x+ibH073`9kRHIyr zC<(-eUl)yOM)y{fAThxkeT-c*_yIQ1Gy58Xl`ZS>q^+wfXBzh7J3R#j1tTLkg7_`) z1gpeV<~KLe?TT%;*KV*FB+V=%u%P5-4a}pEbH9+4&6ZrMpg141dVO{@HMD^;X+5oM zf7udzfq8g;-yv=<+kLDc&*sC)=Y!3Pm5sIiNpIC%OCC>gW^U%zVtoNWJf~eOXJ%TR zc|WjwBQ0t(5%**1MNBgy3>zDjI;vN~Vyf)$8*{Zzd3*LkZL;!E8rIng*;OC*%w(`x zJa{oFV+!qTovA_hSt>G(^a?O=l_dZ_Zi~n2INlfh%J{PCa^*8Fb=e+pRB{VRHDgoK zLj~gD)FRD|;Biy+(z&I{-u=SyRKYiEL-(5EOt7Nwc8cpj5aS`|=kOCE-Ru5+RHq^V zaUztpQ1*^l=n=_{o@}udwP*FoUV@|937eZN+VU7Ml>FX1fZ?Iy8W;Bm#z*OMR2(5G z&ic2_4;S~4EOu1X30yArpTi!~NJo{9h*oL8WZsJaXSggS#6RHPtVGk#p`wmdF!NSA z+{104VPzAx*#28AXSrxq(z@R<&thzNadELJOj~gYp_xAtZ6hUNISu&}kYCC*3PN~uq$H*$qrx8+7Ze7|Ls zEAG*=FZKdYAHHek*@8HCX4kHeu@HXx1INb7;30qxzT92k_EGs3vboD4zcQ)}#JQuW zG{5@}olw>g6DdYctD!KdSUy;{xjix{xLyc3m~?aw1;_VVKfPMaST@l9!j!_p zlRM#hZeKZ^bIZV5^k-hPL1zdFrx5`XV((j5GP=XfBW;jzabTDkW47EHwHXjq3sJjn zHjgUoASIlBw+(_1h>(n;ZEc^re1jUQ7+IVK2IMfow%Dw##;{mwG2b=|Hr0LuYv} zr-n0xYdGlC%B`7WXHPw8ifGh^;#xA0GB%wduU~l@^cg=CCG1@`MHjf4*ArAab~!kK z1BQOZW8WbGHD-)qlvtOo8*fUFxSq0}TsjlM78dx2nruGjDNi*Cb(LXQ$^ETPB zj~dhDuW^`#FD@|fXLq$W8Qp)|h`~~P2aZ(=7?Xa7E@zJ)8u}rdq8i+%NXTDsdr{2z zu(Lct_xT>-Ytz&(#-!|bM=~ldPM<%RW9O(We7l#7Mo&Qx9s7qUe_lU*dFc+!DEc61 zeF%Lz{)wbqa^57f;Yt+3om-VgwZ&ZKSO4|9g6Etkn$MAWm)5p;2_-$cG-I3sH102B zYI7Q}@k=mur+A+8=9FVl$Wkd+y+oED$APfYF`~?VPgLq!d$#Q>Hu5?dEI20GnaeCb{Rwa5SO$K8d)k zL)LAEEJTKAi(0?d&dxl3vmw$CBXT5(h$1JNbuP4LcQp7PSjziEEj%!ndK|Y5=CB{t z7elW*&((c$_CUZgZBr1rNhzPGly5PDtrZk1{#h|HLcK)^dQr;ZNrQ?UH2wF|Ajhc@A?Ziopr&M9uUrJUASO3Ll!z zLUdAc_MOV+%beigP@s=2z`Pt!Ln;73{W+rdA1L7u4B(A?hw`kX7lx(~g=u(3S0l%G zutszVkpAfY^;#vW-cknpHuM59D4)1^I$NTW_F6V!Edzx&67duBF8kCZipA9YL~Wiy zJ)6kzT2yIK!XuM!B;(h&UFnf4)3N5<%O6zS9?SmX3hnL9 zP0TGTgu);3KXq?-RdtH#x=>&aD~wyrp4j^yMCb_uiTzxq>M{_ZDm1U z6$wAm{YBB@tAb{o0UM;B#cgdW7mP@x+sMKei%UnQxm@K*XfQcZFILxMy-JaC`MDmZ zYB45knBN+GD$;f`Qcoxe&k`74x?4^f0|a;`2LJgd^`;*deq8|j6X6ZeBI{Am(A zk%mMpb%jr^X0uZSzIyFW2RJ&Hzaa#z9>UCR7+JntKl)O$=dtnr;!(64fj(;)j*mUV zMeRp%w9gVV3w0H{bDZfG-PnhJBO6HxPb$MKpiSZp#aAtj`YvexvV)BCXA^8yo_g1- zepBR-WAWoJ%Dq#!%BlC5qKm1f_HO5?z6wt)MJ@O1Z$(G@b4kA%U|s7Vo-j<#NW{?N zW`2OgBkX}Ff0|34sG#*~3SpQdkxasL@XbWDkI#A;#ab=v2P>Bk#-WL3#MU)KG8OE6 z*CRrCU!v2abF{fVa_X8@c_9aQ2R%#gn>AP=a!gk{11SqTWPkJ9nL17Di8(*h=)Nj&davNVur3~3ecop&aF&<)ZpOwBix{OZcHa;_vfpMbyyG!=7MtlN-I;$It5grfC$!%^-g;ckA zgaZ&|>Q;@Uri9Wl{UV!h5N5*qHiRA-DV@xJE<84)j<3Z=sv}HwG`mOAoAz4nzjMXR zhM=6=EXZoG9AA+izBfB=m@eFl64)02ZUgrpK*c|nNx0CTYG`KYf z(pZ&>-9#ar3OYjet`)-@x*^j+3+2(B;xiQlXDS_ietwIKvrNC5%wmSuVeUn?_cKi0 zZ7m}+dC(X<>y0<4K(N;1WBq2)pbWuW@XXnVUXhvypI@ruf7OuKE|c(D=5|p}cfRpv zG{(=cmpkrr99-IRQqHw5wJQ>YuC=XUY|xwi3GQ6@ph{)}Jf@6!WO-y$)^MHsZ@B6B z`0jOXPdpIDwg{aI2L&=Aj@@ac!Y()O*W`7j&o8?BO-l#)kUEa6O%y#78YP*w-1Z0j zSfP&n(_thgVa5u^rD~bg1Q19paeP$2=!WU1OKWJR;5k8WymDdrd$S1BbD=`c@ww z9a;aPjz|DR4KpEmvaH9(Nkq`DOLgVq-8-549qm9T>_+E9pQXWt;xn67qS6?mnK*pM z#LX$a9ghdGf`+!GBVs8@C;hF7Y$_pX+h&zeHsQ^n*ZY4mhCGdtgOKrsMZ=zfek4a$ z>WmHy>774G)o%TshD)qY$+v*&n-4)LZH-OG7rP_;ui zm|~*KSu;+xr^{}S*JSu1VQ`G?wxq#2VX%C1;AJi?T3kW`79kge*VhMj&vPQ(*;I=| z7#A6T9Y6}|<>ns9asZ+|587;HVKv{|_TF}Zw;FpSQBaUS9juP$yAJF27$x;>gac}n z@h_?s6xNCb5|HMZfbj4=jZ5e0M+E&J991;y9ziH=?2oP_%o&1Yfu-szrauYWO!hY| z$?b8)E5t3$=#L%s(ahoWq>z`e{uY_cqgZIt-AavqePvN;HB%+=8@fb`(crY>Rd8k; zSn6pP&3Plns@_jM3L{{YqQw-O4COE7)R&Vuh=lT$w-s6K7^gY2xJZ~#cKa_eecrwz ztGo4Q?GN)*#_ zDt3W92eS}hDDFDp5;55?+XkFsvi=u%^|^j?`J0tOs$u*I`-zjz!#{EPJ=O+N_|K(J z_qg2dI_(v?9B>*L?c%~)g(Ck(;_o~@G_X?oKT58f>l6tp7rE;#$D0P zPCCw4FCx?1oXYIG(9CjG>4n*ZdpZl;81qZ|iEFtoU!X5#k@_c0Vr=*T__`VsfRPlu zeHf=5}_c@XO?7i=q7D$h>>}A*gLj?Q6igfR~Hu3iu~SvwmXg zkBsR*prC(2@e888xSO~*>(VAtGiu#LG7$kJm{4FB%nu(x@D*cDd50g`mGiN>wNE}7 zpKYPIvfekrWC$}%T`)Nf0D?syKx4lln_%F;o6HbI|6_s&ctIiAIHNO399x zw0&78;e$h4esuIPlGL{P8EhtU_Nwu32nFin!r2|TZUS*pl<;1=s7h!8JY9F&LN5?( zSrcr;kE~*e#tih9n5t8kqkO=xGM@zMP%9$(1=v5MO4C;6RH!>Uajw0tEl)I0?315TqSJ~D*#3sqnC*cG)T%IB zB>9lX%9lki6s?BlggGiOF3+}0D=h$hN#*|^(4BST2YCu|{nhcxn_~C}tVs)Uw6Tk-xuj*!rX-{&mk`F+pXEgMF-^ zVo0c_aHY?FBky-VaTlX#n9I;N7q+#D!1#59H23Dd_-d1Yi1pw7#*8bgU+G;+QZ0|E zI=944Z+CTpW9OLK95SOQ==D8d+E(>bc741RXBvA?3gBm+slHOeAtLP?CZTUWIQu~? z(dMJsc@L?x@qMoh-9`0}2k%V)O;yr}4Nw)7e0e}yVV?0JM?@zvUWU_&*IXtKyK)(~ zWSbd~hnd7bGM(14ZKSeJz-m67$zwu^FQA`lb*v^0r|SlgE}IXdf?dGWrpHEEOV|{I zdErkVI~d^sQ`Pi^KjOU^+Xe-PVF4*>VTD_~M)pMR2(dW9ZqA2N;vc1g^;CB`Etd`a zznc4glpdCG{Or>wNXla~Wi&HFYFZv+5&!zdarGnr*np&iT?*jIAA4@*84KeW#PJZWc@tWC8KmZZ3?O*ZE)E)91Af8`gL)Z!|wvY8p3hX!> z%K0#piU6p63wda2VFc1G|48#5JAy)WS5tN=QC8};Aw0DAd3?Keg(H6UNs z6alxK;spT3FBArbr^Ko#K-}Hj>^!(0T>_V}nT@)AHGYH{c{hWQB_aKYIOtq(SAh literal 0 HcmV?d00001 diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 6cc3181d..592e4f9d 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -4,19 +4,67 @@ CSCS uses the [SLURM](https://slurm.schedmd.com/documentation.html) as its workload manager to efficiently schedule and manage jobs on Alps vClusters. SLURM is an open-source, highly scalable job scheduler that allocates computing resources, queues user jobs, and optimizes workload distribution across the cluster. It supports advanced scheduling policies, job dependencies, resource reservations, and accounting, making it well-suited for high-performance computing environments. -## Accounting +## Accounts and resources -!!! todo - document `--account`, `--constraint` and other generic flags. +Slurm associates each job with a CSCS project in order to perform accounting. +The project to use for accounting is specified using the `--account/-A` flag. +If no job is specified, the primary project is used as the default. -[Confluence link](https://confluence.cscs.ch/spaces/KB/pages/794296413/How+to+run+jobs+on+Eiger) +??? example "Which projects am I a member of?" + Users often are part of multiple projects, and by extension their associated `groupd_id` groups. + You can get a list of your groups using the `id` command in the terminal: + ```console + $ id $USER + uid=12345(bobsmith) gid=32819(g152) groups=32819(g152),33119(g174),32336(vasp6) + ``` + Here the user `bobsmith` is in three projects (`g152`, `g174` and `vasp6`), with the project `g152` being their **primary project**. + +??? example "What is my primary project?" + In the terminal, use the following command to find your **primary group**: + ```console + $ id -gn $USER + g152 + ``` + +```console title="Specifying the account on the command line" +srun -A g123 -n4 -N1 ./run +srun --account=g123 -n4 -N1 ./run +sbatch --account=g123 ./job.sh +``` + +```bash title="Specifying the account in an sbatch script" +#!/bin/bash + +#SBATCH --account=g123 +#SBATCH --job-name=example-%j +#SBATCH --time=00:30:00 +#SBATCH --nodes=4 +... +``` + +!!! note + The flag `--account` and `-Cmc` that were required on the old Eiger cluster are no longer required. + +## Prioritization and scheduling + +Job priorities are determined based on each project's resource usage relative to its quarterly allocation, as well as in comparison to other projects. +An aging factor is also applied to each job in the queue to ensure fairness over time. + +Since users from various projects are continuously submitting jobs, the relative priority of jobs is dynamic and may change frequently. +As a result, estimated start times are approximate and subject to change based on new job submissions. + +Additionally, short-duration jobs may be selected for backfilling — a process where the scheduler fills in available time slots while preparing to run a larger, higher-priority job. [](){#ref-slurm-partitions} ## Partitions -At CSCS, SLURM is configured to accommodate the diverse range of node types available in our HPC clusters. These nodes vary in architecture, including CPU-only nodes and nodes equipped with different types of GPUs. Because of this heterogeneity, SLURM must be tailored to ensure efficient resource allocation, job scheduling, and workload management specific to each node type. +At CSCS, SLURM is configured to accommodate the diverse range of node types available in our HPC clusters. +These nodes vary in architecture, including CPU-only nodes and nodes equipped with different types of GPUs. +Because of this heterogeneity, SLURM must be tailored to ensure efficient resource allocation, job scheduling, and workload management specific to each node type. -Each type of node has different resource constraints and capabilities, which SLURM takes into account when scheduling jobs. For example, CPU-only nodes may have configurations optimized for multi-threaded CPU workloads, while GPU nodes require additional parameters to allocate GPU resources efficiently. SLURM ensures that user jobs request and receive the appropriate resources while preventing conflicts or inefficient utilization. +Each type of node has different resource constraints and capabilities, which SLURM takes into account when scheduling jobs. +For example, CPU-only nodes may have configurations optimized for multi-threaded CPU workloads, while GPU nodes require additional parameters to allocate GPU resources efficiently. +SLURM ensures that user jobs request and receive the appropriate resources while preventing conflicts or inefficient utilization. !!! example "How to check the partitions and number of nodes therein?" You can check the size of the system by running the following command in the terminal: @@ -165,9 +213,9 @@ See [Scientific Applications][ref-software-sciapps] for information about recomm The "default" mode is used to avoid issues with certain containers. Unlike "exclusive process" mode, "default" mode allows multiple processes to submit work to a single GPU simultaneously. This also means that different ranks on the same node can inadvertently use the same GPU leading to suboptimal performance or unused GPUs, rather than job failures. - + Some applications benefit from using multiple ranks per GPU. However, [MPS should be used][ref-slurm-gh200-multi-rank-per-gpu] in these cases. - + If you are unsure about which GPU is being used for a particular rank, print the `CUDA_VISIBLE_DEVICES` variable, along with e.g. `SLURM_LOCALID`, `SLURM_PROCID`, and `SLURM_NODEID` variables, in your job script. If the variable is unset or empty all GPUs are visible to the rank and the rank will in most cases only use the first GPU. @@ -187,7 +235,7 @@ The examples below launch jobs on two nodes with four ranks per node using `sbat srun ``` - + Omitting the `--gpus-per-task` results in `CUDA_VISIBLE_DEVICES` being unset, which will lead to most applications using the first GPU on all ranks. [](){#ref-slurm-gh200-multi-rank-per-gpu} @@ -258,5 +306,39 @@ The configuration that is optimal for your application may be different. ## AMD CPU Nodes Alps has nodes with two AMD Epyc Rome CPU sockets per node for CPU-only workloads, most notably in the [Eiger][ref-cluster-eiger] cluster provided by the [HPC Platform][ref-platform-hpcp]. -!!! todo - document how slurm is configured on AMD CPU nodes (e.g. eiger) + +For a detailed description of the node hardware, see the [AMD Rome node][ref-alps-zen2-node] hardware documentation. + +The typical Slurm workload that we want to schedule will distribute `NR` MPI ranks over nodes, with `NT` threads per . + +Each node has 128 cores: so we can reasonably expect to run a maximum of 128 MPI ranks per node. + +Each node has 2 sockets, and each socket contains 4 NUMA nodes. + +Each MPI rank is assigned a set of cores on a specific node - to get the best performance you want to follow some best practices: + +* don't spread MPI ranks across multiple sockets +* and it might be advantageous to have 8 ranks per node, with 16 cores each - the sweet spot is application specific + +!! todo "table of basic flags nodes, cores-per-task, etc" + +Here we assign 64 cores to each, and observe that there are 128 "cores" (2 per core): +```console title="One MPI rank per socket" +$ OMP_NUM_THREADS=64 srun -n2 -N1 -c64 ./affinity.mpi +``` + +If you want only 64, consider the `--hint=nomultithreading` option. +```console title="One MPI rank per socket" +$ OMP_NUM_THREADS=64 srun -n2 -N1 -c64 --hint=nomultithreading ./affinity.mpi +``` + +```console title="One MPI rank per NUMA region" +$ OMP_NUM_THREADS=64 srun -n16 -N1 -c8 ./affinity.mpi +``` + +In the above examples all threads on each -- we are effectively allowing the OS to schedule the threads on the available set of cores as it sees fit. +This is often gives the best performance, however sometimes it is beneficial to bind threads to explicit cores. + +```console title="One MPI rank per NUMA region" +$ OMP_BIND_PROC=true OMP_NUM_THREADS=64 srun -n16 -N1 -c8 ./affinity.mpi +``` From edbebe5c0ef06ab5075f65ad8359fae60cb0284b Mon Sep 17 00:00:00 2001 From: bcumming Date: Wed, 25 Jun 2025 15:37:40 +0200 Subject: [PATCH 03/13] finish first draft --- docs/alps/hardware.md | 13 ++- docs/running/slurm.md | 213 ++++++++++++++++++++++++++++++++++++------ 2 files changed, 193 insertions(+), 33 deletions(-) diff --git a/docs/alps/hardware.md b/docs/alps/hardware.md index cfd43d80..ffa80faf 100644 --- a/docs/alps/hardware.md +++ b/docs/alps/hardware.md @@ -3,8 +3,8 @@ Alps is a HPE Cray EX3000 system, a liquid cooled blade-based, high-density system. -!!! todo - this is a skeleton - all of the details need to be filled in +!!! under-construction + This page is a work in progress - contact us if you want us to prioritise documentation specific information that would be useful for your work. ## Alps Cabinets @@ -100,12 +100,11 @@ A schematic of a *standard memory node* below illustrates the CPU cores and [NUM * The two sockets are labelled Package L#0 and Package L#1. * Each socket has 4 NUMA nodes, with 16 cores each, for a total of 64 cores per socket. -Each core supports [simultaneous multi threading (SMT)](https://www.amd.com/en/blogs/2025/simultaneous-multithreading-driving-performance-a.html), whereby each core can execute two threads concurrently, which are presented as two PU per physical core. - -* The first PU on each core are numbered 0:63 on socket 0, and 64:127 on socket 1; -* The second PU on each core are numbered 128:191 on socket 0, and 192:256 on socket 1. -* Hence core `n` SMT +Each core supports [simultaneous multi threading (SMT)](https://www.amd.com/en/blogs/2025/simultaneous-multithreading-driving-performance-a.html), whereby each core can execute two threads concurrently, which are presented as two processing units (PU) per physical core: +* the first PU on each core are numbered 0:63 on socket 0, and 64:127 on socket 1; +* the second PU on each core are numbered 128:191 on socket 0, and 192:256 on socket 1; +* hence, core `n` has PUs `n` and `n+128`. Each node has two Slingshot 11 network interface cards (NICs), which are not illustrated on the diagram. diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 592e4f9d..4365aa63 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -90,7 +90,7 @@ The following sections will provide detailed guidance on how to use SLURM to req ## Affinity The following sections will document how to use Slurm on different compute nodes available on Alps. -To demonstrate the effects different Slurm parameters, we will use a little command line tool [affinity](https://github.com/bcumming/affinity) that prints the CPU cores and GPUs that are assinged to each MPI rank in a job, and which node they are run on. +To demonstrate the effects different Slurm parameters, we will use a little command line tool [affinity](https://github.com/bcumming/affinity) that prints the CPU cores and GPUs that are assigned to each MPI rank in a job, and which node they are run on. We strongly recommend using a tool like affinity to understand and test the Slurm configuration for jobs, because the behavior of Slurm is highly dependent on the system configuration. Parameters that worked on a different cluster -- or with a different Slurm version or configuration on the same cluster -- are not guaranteed to give the same results. @@ -127,14 +127,14 @@ The build generates the following executables: $ uenv start prgenv-gnu/24.11:v2 --view=default $ srun -n8 -N2 -c72 ./affinity.mpi affinity test for 8 MPI ranks - rank 0 @ nid006363: threads [ 0:71] -> cores [ 0: 71] - rank 1 @ nid006363: threads [ 0:71] -> cores [ 72:143] - rank 2 @ nid006363: threads [ 0:71] -> cores [144:215] - rank 3 @ nid006363: threads [ 0:71] -> cores [216:287] - rank 4 @ nid006375: threads [ 0:71] -> cores [ 0: 71] - rank 5 @ nid006375: threads [ 0:71] -> cores [ 72:143] - rank 6 @ nid006375: threads [ 0:71] -> cores [144:215] - rank 7 @ nid006375: threads [ 0:71] -> cores [216:287] + rank 0 @ nid006363: thread 0 -> cores [ 0: 71] + rank 1 @ nid006363: thread 0 -> cores [ 72:143] + rank 2 @ nid006363: thread 0 -> cores [144:215] + rank 3 @ nid006363: thread 0 -> cores [216:287] + rank 4 @ nid006375: thread 0 -> cores [ 0: 71] + rank 5 @ nid006375: thread 0 -> cores [ 72:143] + rank 6 @ nid006375: thread 0 -> cores [144:215] + rank 7 @ nid006375: thread 0 -> cores [216:287] ``` In this example there are 8 MPI ranks: @@ -306,39 +306,200 @@ The configuration that is optimal for your application may be different. ## AMD CPU Nodes Alps has nodes with two AMD Epyc Rome CPU sockets per node for CPU-only workloads, most notably in the [Eiger][ref-cluster-eiger] cluster provided by the [HPC Platform][ref-platform-hpcp]. - For a detailed description of the node hardware, see the [AMD Rome node][ref-alps-zen2-node] hardware documentation. -The typical Slurm workload that we want to schedule will distribute `NR` MPI ranks over nodes, with `NT` threads per . +??? info "Node description" + - The node has 2 x 64 core sockets + - Each socket is divided into 4 NUMA regions + - the 16 cores in each NUMA region have faster memory access to their of 32 GB + - Each core has two processing units (PUs) -Each node has 128 cores: so we can reasonably expect to run a maximum of 128 MPI ranks per node. + ![Screenshot](../images/slurm/eiger-topo.png) -Each node has 2 sockets, and each socket contains 4 NUMA nodes. -Each MPI rank is assigned a set of cores on a specific node - to get the best performance you want to follow some best practices: +Each MPI rank is assigned a set of cores on a node, and Slurm provides flags that can be used directly as flags to `srun`, or as arguments in an `sbatch` script. +Here are some basic flags that we will use to distribute work. -* don't spread MPI ranks across multiple sockets -* and it might be advantageous to have 8 ranks per node, with 16 cores each - the sweet spot is application specific +| flag | meaning | +| ---- | ------- | +| `-n`, `--ntasks` | The total number of MPI ranks | +| `-N`, `--nodes` | The total number of nodes | +| `--ntasks-per-node` | The total number of nodes | +| `-c`, `--cpus-per-task` | The number of cores to assign to each rank. | +| `--hint=nomultithread` | Use only one PU per core | -!! todo "table of basic flags nodes, cores-per-task, etc" +!!! info "Slurm is highly configurable" + These are a subset of the most useful flags. + Call `srun --help` or `sbatch --help` to get a complete list of all the flags available on your target cluster. + Note that the exact set of flags available depends on the Slurm version, how Slurm was configured, and Slurm plugins. -Here we assign 64 cores to each, and observe that there are 128 "cores" (2 per core): +The first example assigns 2 MPI ranks per node, with 64 cores per rank, with the two PUs per core: ```console title="One MPI rank per socket" -$ OMP_NUM_THREADS=64 srun -n2 -N1 -c64 ./affinity.mpi +# one node +$ srun -n2 -N1 -c64 ./affinity.mpi +affinity test for 2 MPI ranks +rank 0 @ nid002199: thread 0 -> cores [ 0: 31,128:159] +rank 1 @ nid002199: thread 0 -> cores [ 64: 95,192:223] + +# two nodes +$ srun -n4 -N2 -c64 ./affinity.mpi +affinity test for 4 MPI ranks +rank 0 @ nid001512: thread 0 -> cores [ 0: 31,128:159] +rank 1 @ nid001512: thread 0 -> cores [ 64: 95,192:223] +rank 2 @ nid001515: thread 0 -> cores [ 0: 31,128:159] +rank 3 @ nid001515: thread 0 -> cores [ 64: 95,192:223] ``` -If you want only 64, consider the `--hint=nomultithreading` option. -```console title="One MPI rank per socket" -$ OMP_NUM_THREADS=64 srun -n2 -N1 -c64 --hint=nomultithreading ./affinity.mpi +!!! note + In the above example we use `--ntasks/-n` and `--nodes/-N`. + It is possible to achieve the same effect using `--nodes` and `--ntasks-per-node`, for example the following both give 8 ranks on 4 nodes: + + ```bash + srun --nodes=4 --ntasks=8 + srun --nodes=4 --ntasks-per-node=2 + ``` + +It is often more efficient to only run one task per core instead of the default two PU, which can be achieved using the `--hint=nomultithreading` option. +```console title="One MPI rank per socket with 1 PU per core" +$ srun -n2 -N1 -c64 --hint=nomultithread ./affinity.mpi +affinity test for 2 MPI ranks +rank 0 @ nid002199: thread 0 -> cores [ 0: 63] +rank 1 @ nid002199: thread 0 -> cores [ 64:127] ``` +!!! note "Always test" + The best configuration for performance is highly application specific, with no one-size-fits-all configuration. + Take the time to experiment with `--hint=nomultithread`. + +Memory on the node is divided into NUMA (non-uniform memory access) regions. +The 256 GB of a standard-memory node are divided into 8 NUMA nodes of 32 GB, with 16 cores associated with each node: + +* memory access is optimal when all the cores of a rank are on the same NUMA node; +* memory access to NUMA regions on the other socket are significantly slower. + +??? info "How to investigate the NUMA layout of a node" + Use the command `numactl -H`. + + ```console + $ srun -n1 numactl -H + available: 8 nodes (0-7) + node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 + node 0 size: 63733 MB + node 0 free: 62780 MB + node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 + node 1 size: 64502 MB + node 1 free: 61774 MB + node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 + node 2 size: 64456 MB + node 2 free: 63385 MB + node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 + node 3 size: 64490 MB + node 3 free: 62613 MB + node 4 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 + node 4 size: 64502 MB + node 4 free: 63897 MB + node 5 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 + node 5 size: 64502 MB + node 5 free: 63769 MB + node 6 cpus: 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 + node 6 size: 64502 MB + node 6 free: 63870 MB + node 7 cpus: 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 + node 7 size: 64428 MB + node 7 free: 63712 MB + node distances: + node 0 1 2 3 4 5 6 7 + 0: 10 12 12 12 32 32 32 32 + 1: 12 10 12 12 32 32 32 32 + 2: 12 12 10 12 32 32 32 32 + 3: 12 12 12 10 32 32 32 32 + 4: 32 32 32 32 10 12 12 12 + 5: 32 32 32 32 12 10 12 12 + 6: 32 32 32 32 12 12 10 12 + 7: 32 32 32 32 12 12 12 10 + ``` + The `node distances` table shows that the cores have the fastest memory access to memory in their own region (`10`), and fast access (`12`) to NUMA regions on the same socket. + The cost of accessing memory of a NUMA node on the other socket is much higher (`32`). + + Note that this command was run on a large-memory node that has 8 x 64 GB NUMA regions, for a total of 512 GB. + +The examples above placed one rank per socket, which is not optimal for NUMA access. +To constrain + +!!! Note "Always test" + It might still be optimal for applications that have high threading efficiency and benefit from using fewer MPI ranks to have one rank per socket or even one one rank per node. + Always test! + ```console title="One MPI rank per NUMA region" -$ OMP_NUM_THREADS=64 srun -n16 -N1 -c8 ./affinity.mpi +$ srun -n8 -N1 -c16 --hint=nomultithread ./affinity.mpi +affinity test for 8 MPI ranks +rank 0 @ nid002199: thread 0 -> cores [ 0: 15] +rank 1 @ nid002199: thread 0 -> cores [ 64: 79] +rank 2 @ nid002199: thread 0 -> cores [ 16: 31] +rank 3 @ nid002199: thread 0 -> cores [ 80: 95] +rank 4 @ nid002199: thread 0 -> cores [ 32: 47] +rank 5 @ nid002199: thread 0 -> cores [ 96:111] +rank 6 @ nid002199: thread 0 -> cores [ 48: 63] +rank 7 @ nid002199: thread 0 -> cores [112:127] ``` In the above examples all threads on each -- we are effectively allowing the OS to schedule the threads on the available set of cores as it sees fit. -This is often gives the best performance, however sometimes it is beneficial to bind threads to explicit cores. +This often gives the best performance, however sometimes it is beneficial to bind threads to explicit cores. -```console title="One MPI rank per NUMA region" -$ OMP_BIND_PROC=true OMP_NUM_THREADS=64 srun -n16 -N1 -c8 ./affinity.mpi +### OpenMP + +The OpenMP threading runtime provides additional options for controlling the pinning of threads to the cores assinged to each MPI rank. + +Use the `--omp` flag with `affinity.mpi` to get more detailed information about OpenMPI thread affinity. +For example, four MPI ranks on one node with four cores and four OpenMP threads: + +```console title="No OpenMP binding" +$ export OMP_NUM_THREADS=4 +$ srun -n4 -N1 -c4 --hint=nomultithread ./affinity.mpi --omp +affinity test for 4 MPI ranks +rank 0 @ nid001512: threads [0:3] -> cores [ 0: 3] +rank 1 @ nid001512: threads [0:3] -> cores [ 64: 67] +rank 2 @ nid001512: threads [0:3] -> cores [ 4: 7] +rank 3 @ nid001512: threads [0:3] -> cores [ 68: 71] ``` + +The status `threads [0:3] -> cores [ 0: 3]` is shorthand "there are 4 OpenMP threads, and the OS can schedule them on cores 0, 1, 2 and 3". + +Allowing the OS to schedule threads is usually efficient, however to get the most you can try pinning threads to specific cores. +The [`OMP_PROC_BIND`](https://www.openmp.org/spec-html/5.0/openmpse52.html) environment variable can be used to tune how OpenMP sets thread affinity. +For example, `OMO_PROC_BIND=true` will give each thread exclusive affinity with a core: + +```console title="OMP_PROC_BIND=true" +$ export OMP_NUM_THREADS=4 +$ export OMP_PROC_BIND=true +$ srun -n4 -N1 -c4 --hint=nomultithread ./affinity.mpi --omp +affinity test for 4 MPI ranks +rank 0 @ nid001512 + thread 0 -> core 0 + thread 1 -> core 1 + thread 2 -> core 2 + thread 3 -> core 3 +rank 1 @ nid001512 + thread 0 -> core 64 + thread 1 -> core 65 + thread 2 -> core 66 + thread 3 -> core 67 +rank 2 @ nid001512 + thread 0 -> core 4 + thread 1 -> core 5 + thread 2 -> core 6 + thread 3 -> core 7 +rank 3 @ nid001512 + thread 0 -> core 68 + thread 1 -> core 69 + thread 2 -> core 70 + thread 3 -> core 71 +``` + +!!! note + There are many OpenMP variables that can be used to fine tune affinity. + See the [OpenMP documentation](https://www.openmp.org/spec-html/5.0/openmpch6.html) for more information. + +!!! warning + The `OMP_*` environment variables only affect thread affinity of applications that use OpenMP for thread-level parallelism. + Other threading runtimes will be configured differently, and the `affinity.mpi` tool will only be able to show the set of cores assigned to the rank. From 598c0825169ab0244f23387239b219976c74e504 Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 15:43:50 +0200 Subject: [PATCH 04/13] Update docs/running/slurm.md Co-authored-by: Mikael Simberg --- docs/running/slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 4365aa63..f7e24006 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -303,7 +303,7 @@ The configuration that is optimal for your application may be different. [NVIDIA's Multi-Process Service (MPS)]: https://docs.nvidia.com/deploy/mps/index.html [](){#ref-slurm-amdcpu} -## AMD CPU Nodes +## AMD CPU nodes Alps has nodes with two AMD Epyc Rome CPU sockets per node for CPU-only workloads, most notably in the [Eiger][ref-cluster-eiger] cluster provided by the [HPC Platform][ref-platform-hpcp]. For a detailed description of the node hardware, see the [AMD Rome node][ref-alps-zen2-node] hardware documentation. From 9238ea3d83ac1a6a076b980469739e704da347ac Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 15:44:49 +0200 Subject: [PATCH 05/13] Update docs/alps/hardware.md Co-authored-by: Rocco Meli --- docs/alps/hardware.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/alps/hardware.md b/docs/alps/hardware.md index ffa80faf..9923ca7e 100644 --- a/docs/alps/hardware.md +++ b/docs/alps/hardware.md @@ -82,8 +82,8 @@ Each node contains four Grace-Hopper modules and four corresponding network inte These nodes have two [AMD Epyc 7742](https://en.wikichip.org/wiki/amd/epyc/7742) 64-core CPU sockets, and are used primarily for the [Eiger][ref-cluster-eiger] system. They come in two memory configurations: -* *Standard-memory*: 256 GB in 16x16 GB DDR4 Dimms. -* *Large-memory*: 512 GB in 16x32 GB DDR4 Dimms. +* *Standard-memory*: 256 GB in 16x16 GB DDR4 DIMMs. +* *Large-memory*: 512 GB in 16x32 GB DDR4 DIMMs. !!! note "Not all memory is available" The total memory available to jobs on the nodes is roughly 245 GB and 497 GB on the standard and large memory nodes respectively. From 348be3c822203feeeacf17c6257bf30de601bab4 Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 15:46:00 +0200 Subject: [PATCH 06/13] Update docs/running/slurm.md Co-authored-by: Rocco Meli --- docs/running/slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index f7e24006..24cae2fd 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -43,7 +43,7 @@ sbatch --account=g123 ./job.sh ``` !!! note - The flag `--account` and `-Cmc` that were required on the old Eiger cluster are no longer required. + The flags `--account` and `-Cmc` that were required on the old Eiger cluster are no longer required. ## Prioritization and scheduling From 6ddff4a4f673090e313f7f5e5b4d9f7f62d006de Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 15:46:17 +0200 Subject: [PATCH 07/13] Update docs/running/slurm.md Co-authored-by: Rocco Meli --- docs/running/slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 24cae2fd..2a58d2fe 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -62,7 +62,7 @@ At CSCS, SLURM is configured to accommodate the diverse range of node types avai These nodes vary in architecture, including CPU-only nodes and nodes equipped with different types of GPUs. Because of this heterogeneity, SLURM must be tailored to ensure efficient resource allocation, job scheduling, and workload management specific to each node type. -Each type of node has different resource constraints and capabilities, which SLURM takes into account when scheduling jobs. +Each type of node has different resource constraints and capabilities, which Slurm takes into account when scheduling jobs. For example, CPU-only nodes may have configurations optimized for multi-threaded CPU workloads, while GPU nodes require additional parameters to allocate GPU resources efficiently. SLURM ensures that user jobs request and receive the appropriate resources while preventing conflicts or inefficient utilization. From 2bcea80bcd85289d4ad3b58f7e431f95ffe0756b Mon Sep 17 00:00:00 2001 From: bcumming Date: Wed, 25 Jun 2025 16:28:03 +0200 Subject: [PATCH 08/13] document --cpu-bind=verbose --- docs/running/slurm.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 4365aa63..af9b56a7 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -145,8 +145,6 @@ The build generates the following executables: * all threads on each rank have affinity with the same 72 cores; * each rank gets 72 cores, e.g. rank 1 gets cores `72:143` on node `nid006363`. - - ??? example "Testing GPU affinity" Use `affinity.cuda` or `affinity.rocm` to test on GPU-enabled systems. @@ -197,6 +195,22 @@ The build generates the following executables: 2. Test GPU affinity: note how the `--gpus-per-task=1` parameter assings a unique GPU to each rank. +!!! info "Quick affinity checks" + + The Slurm flag [`cpu-bind=verbose`](https://slurm.schedmd.com/srun.html#OPT_cpu-bind) prints information about MPI ranks and their thread affinity. + + The mask it prints is not very readable, but it can be used with the `true` command to quickly test Slurm parameters without building the Affinity tool. + + ```console title="hello" + $ srun --cpu-bind=verbose -c32 -n4 -N1 --hint=nomultithread -- true + cpu-bind=MASK - nid002156, task 0 0 [147694]: mask 0xffffffff set + cpu-bind=MASK - nid002156, task 1 1 [147695]: mask 0xffffffff0000000000000000 set + cpu-bind=MASK - nid002156, task 2 2 [147696]: mask 0xffffffff00000000 set + cpu-bind=MASK - nid002156, task 3 3 [147697]: mask 0xffffffff000000000000000000000000 set + ``` + + You can also check GPU affinity by inspecting the value of the `CUDA_VISIBLE_DEVICES` environment variable. + [](){#ref-slurm-gh200} ## NVIDIA GH200 GPU Nodes From af549e09df1ff9200aaca8861b6c624bcff3c547 Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 17:25:08 +0200 Subject: [PATCH 09/13] Update docs/running/slurm.md Co-authored-by: Mikael Simberg --- docs/running/slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index f9d6cb63..92cabbdd 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -464,7 +464,7 @@ This often gives the best performance, however sometimes it is beneficial to bin The OpenMP threading runtime provides additional options for controlling the pinning of threads to the cores assinged to each MPI rank. -Use the `--omp` flag with `affinity.mpi` to get more detailed information about OpenMPI thread affinity. +Use the `--omp` flag with `affinity.mpi` to get more detailed information about OpenMP thread affinity. For example, four MPI ranks on one node with four cores and four OpenMP threads: ```console title="No OpenMP binding" From 5500cc6e2fe36da8e6513ba42ea2d74c45cb5e9e Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 17:25:20 +0200 Subject: [PATCH 10/13] Update docs/running/slurm.md Co-authored-by: Rocco Meli --- docs/running/slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 92cabbdd..e4edff34 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -64,7 +64,7 @@ Because of this heterogeneity, SLURM must be tailored to ensure efficient resour Each type of node has different resource constraints and capabilities, which Slurm takes into account when scheduling jobs. For example, CPU-only nodes may have configurations optimized for multi-threaded CPU workloads, while GPU nodes require additional parameters to allocate GPU resources efficiently. -SLURM ensures that user jobs request and receive the appropriate resources while preventing conflicts or inefficient utilization. +Slurm ensures that user jobs request and receive the appropriate resources while preventing conflicts or inefficient utilization. !!! example "How to check the partitions and number of nodes therein?" You can check the size of the system by running the following command in the terminal: From d0be829bb0d4da6df1d2597915aab5a84122dee3 Mon Sep 17 00:00:00 2001 From: Ben Cumming Date: Wed, 25 Jun 2025 17:26:21 +0200 Subject: [PATCH 11/13] Update docs/running/slurm.md Co-authored-by: Mikael Simberg --- docs/running/slurm.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index e4edff34..3ce97295 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -27,9 +27,9 @@ If no job is specified, the primary project is used as the default. ``` ```console title="Specifying the account on the command line" -srun -A g123 -n4 -N1 ./run -srun --account=g123 -n4 -N1 ./run -sbatch --account=g123 ./job.sh +$ srun -A g123 -n4 -N1 ./run +$ srun --account=g123 -n4 -N1 ./run +$ sbatch --account=g123 ./job.sh ``` ```bash title="Specifying the account in an sbatch script" From 38929e7285deae59aeb75297353338cdb9b03cd6 Mon Sep 17 00:00:00 2001 From: bcumming Date: Wed, 25 Jun 2025 17:30:05 +0200 Subject: [PATCH 12/13] pr tweaks --- docs/running/slurm.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 3ce97295..2943c8c5 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -437,12 +437,9 @@ The 256 GB of a standard-memory node are divided into 8 NUMA nodes of 32 GB, wit Note that this command was run on a large-memory node that has 8 x 64 GB NUMA regions, for a total of 512 GB. -The examples above placed one rank per socket, which is not optimal for NUMA access. -To constrain +The examples above placed one rank per socket, which is not optimal for NUMA access, because cores assigned to each rank are spread over the 4 NUMA nodes on the socket. +To constrain tasks to NUMA nodes, use 16 cores per task: -!!! Note "Always test" - It might still be optimal for applications that have high threading efficiency and benefit from using fewer MPI ranks to have one rank per socket or even one one rank per node. - Always test! ```console title="One MPI rank per NUMA region" $ srun -n8 -N1 -c16 --hint=nomultithread ./affinity.mpi @@ -457,11 +454,15 @@ rank 6 @ nid002199: thread 0 -> cores [ 48: 63] rank 7 @ nid002199: thread 0 -> cores [112:127] ``` -In the above examples all threads on each -- we are effectively allowing the OS to schedule the threads on the available set of cores as it sees fit. -This often gives the best performance, however sometimes it is beneficial to bind threads to explicit cores. +!!! Note "Always test" + It might still be optimal for applications that have high threading efficiency and benefit from using fewer MPI ranks to have one rank per socket or even one one rank per node. + Always test! ### OpenMP +In the above examples all threads on each -- we are effectively allowing the OS to schedule the threads on the available set of cores as it sees fit. +This often gives the best performance, however sometimes it is beneficial to bind threads to explicit cores. + The OpenMP threading runtime provides additional options for controlling the pinning of threads to the cores assinged to each MPI rank. Use the `--omp` flag with `affinity.mpi` to get more detailed information about OpenMP thread affinity. From 7e805f87d4a2475cd0c381f2903752a81952215d Mon Sep 17 00:00:00 2001 From: bcumming Date: Wed, 25 Jun 2025 18:16:32 +0200 Subject: [PATCH 13/13] micro tweak linky link text --- docs/running/slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/running/slurm.md b/docs/running/slurm.md index 2943c8c5..2aef9c37 100644 --- a/docs/running/slurm.md +++ b/docs/running/slurm.md @@ -197,7 +197,7 @@ The build generates the following executables: !!! info "Quick affinity checks" - The Slurm flag [`cpu-bind=verbose`](https://slurm.schedmd.com/srun.html#OPT_cpu-bind) prints information about MPI ranks and their thread affinity. + The Slurm flag [`--cpu-bind=verbose`](https://slurm.schedmd.com/srun.html#OPT_cpu-bind) prints information about MPI ranks and their thread affinity. The mask it prints is not very readable, but it can be used with the `true` command to quickly test Slurm parameters without building the Affinity tool.