Hosting Cloud Tests in pangeo resources instead of GHA

[jbusecke]

Circling back to a discussion we had at the pangeo meeting with @WesleyTheGeolien about running the CI on pangeo resources (instead of the ones provided by GHA).

The main reason to do this is that we could theoretically? run more concurrent jobs and might be able to get more performant nodes.

I have attempted a naive performance test with the following steps:

1. Get a large server on the pangeo gcp deployment. This gives us 2 cores/4 threads to run on?

(notebook) jovyan@jupyter-jbusecke:~/cmip6_preprocessing$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              2
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           63
Model name:                      Intel(R) Xeon(R) CPU @ 2.30GHz
Stepping:                        0
CPU MHz:                         2300.000
BogoMIPS:                        4600.00
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        512 KiB
L3 cache:                        45 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1
                                 gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse
                                 4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ssbd ibrs ibpb stibp 
                                 fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat md_clear arch_capabilities

2. Set up the cloud testing environment with mamba env create -f ci/environment-cloud-test.yml
3. Run the cloud specific tests with pytest -n auto --maxfail 5 --reruns 1 --reruns-delay 10 tests/test_preprocessing_cloud.py. This runs two parallel processes and takes ~14 minutes (I also tried to run it with -n 4 and it took ~12 minutes)
4. These times are however much slower than what I am getting for the GHA workflows (~7-9 minutes example)

I am wondering if it is even worth trying to port this to pangeo? I am not sure what kind of increased performance we could expect. Even if we manage to squeeze out higher performance, the fact that we have this working in a decent state on GHA right now, makes me think we should maybe focus our efforts elsewhere? It seems that GHA allows for a decent amount of concurrent jobs (~8?) and most of the test can be run in parallel. This gives us ~30 min for the full CI, which is not great, but also not terrible.

Any thoughts?

[jbusecke]

I am going to close this for now. If performance becomes an issue again, we can reopen.