GCP Integration tests failing with latest jupyterhub

[TomAugspurger]

https://app.circleci.com/pipelines/github/pangeo-data/pangeo-cloud-federation/1638/workflows/9ead26f7-994d-43cd-9c4a-8bd5dcee52d4/jobs/1805 failed. We tried to start the test before the singleuser pod was ready:

[Staring singleuser server]
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
waiting for pod
waiting for pod
waiting for pod
waiting for pod
waiting for pod
waiting for pod
[Running tests]
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"/srv/conda/envs/notebook/bin/pytest\": stat /srv/conda/envs/notebook/bin/pytest: no such file or directory": unknown
command terminated with exit code 126
[Cleaning up]
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100    70  100    70    0     0    418      0 --:--:-- --:--:-- --:--:--   419
{"status": 400, "message": "pangeo-bot is pending spawn, please wait"}

In https://github.com/pangeo-data/pangeo-cloud-federation/blob/032b35884c57806084e0d45ab4af04a62ebce205/run_tests.sh, we should be waiting until the pod is "Ready". Perhaps the set of states has changed?

[tjcrone]

I'm not familiar with this particular test. Happy to troubleshoot as needed though I may not be much help here. I did test the latest deployment to staging on Azure, and it seems to be working fine. Dask is working fine on the OOI staging deployment.

[TomAugspurger]

This is the integration test that runs on each deploy to staging. Things are working fine on GCP, it’s just the test itself that apparently needs updating.

…

On Nov 28, 2020, at 6:34 PM, Tim Crone ***@***.***> wrote: I'm not familiar with this particular test. Happy to troubleshoot as needed though I may not be much help here. I did test the latest deployment to staging on Azure, and it seems to be working fine. Dask is working fine on the OOI staging deployment. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#892 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIR7DSG6YBFWV7U36X3SSGJIBANCNFSM4UGD6TGQ>.

[scottyhq]

@TomAugspurger - looks like this is still failing, but the deploy succeeds as far as I can tell. https://app.circleci.com/pipelines/github/pangeo-data/pangeo-cloud-federation/1653/workflows/8e4cddbe-e25f-48bd-80a8-d3de53fe017e/jobs/1820

I also haven't really dug into what the test does, but I notice this line in the CI log, so it seems the patching can be removed now that we're running dask-gateway 0.9

pangeo-cloud-federation/.circleci/config.yml

Lines 192 to 193 in 1a29fbd

	# Patch gateway pods to set affinity till dask-gateway>0.8 is out.
	python deployments/gcp-uscentral1b/patches/patch.py gcp-uscentral1b ${CIRCLE_BRANCH}