ci: send Slack notification on GKE node pool creation failure

[dzhu]

Description

In order to prevent quota failures from showing up as CI failures, this makes node pool creation failure send a Slack notification and mark the job as successful.

I couldn't figure out how to use the Slack orb while distinguishing this particular situation from the general failure case, so I just slapped in a direct request to the already-configured Slack webhook for sending messages to #ci-bots.

circleci-agent step halt marks the job as successful, which is why we want a notification at all. For some reason, CircleCI fails to provide an equivalent for marking the job as canceled or some other state besides success/failure; we could make a call to the CircleCI API to cancel the current job, but that would rely on having a CircleCI token available, which we're trying to get away from.

Test Plan

• run the command in CI and make sure it makes a message appear

[netlify]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	f844377
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/64068ecb75d1b10008c8d653

[dannysauer]

Code LGTM. Optional style suggestion included.

[dannysauer]

Maybe this would be a nice place for the : > yaml multi-line block, where it replaces newlines and the initial indentation with spaces. That way this wouldn't have to be a kajillion character long string, but could instead be wrapped however makes it more readable. :)

For one example which is by no means the "right-est" way:

Suggested change

	command: |
	gcloud container node-pools create accel --cluster ${CLUSTER_ID} --region <<parameters.region>> --num-nodes <<parameters.num-machines>> --accelerator type=<<parameters.gpu-type>>,count=<<parameters.gpus-per-machine>> --machine-type=<<parameters.machine-type>> --scopes cloud-platform --node-taints=<<parameters.accel-node-taints>> || (curl "$SLACK_WEBHOOK" -H 'Content-Type: application/json' -d "{\"text\":\"GKE node pool creation failed! $CIRCLE_BUILD_URL\"}"; circleci-agent step halt)
	command: >
	gcloud container node-pools create accel
	--cluster ${CLUSTER_ID}
	--region <<parameters.region>>
	--num-nodes <<parameters.num-machines>>
	--accelerator type=<<parameters.gpu-type>>,count=<<parameters.gpus-per-machine>>
	--machine-type=<<parameters.machine-type>>
	--scopes cloud-platform
	--node-taints=<<parameters.accel-node-taints>>
	|| (
	curl "$SLACK_WEBHOOK"
	-H 'Content-Type: application/json'
	-d "{\"text\":\"GKE node pool creation failed! $CIRCLE_BUILD_URL\"}";
	circleci-agent step halt
	)

That will render with some extra spaces in the commands (because the indentation varies), but I think you get the gist.

Approving either way, because this is purely cosmetic.