Someone left the lights on

[iterative-olivaw]

Found instances in region us-west-1

Instance i-064c95a8ba90dbe34 on for 06h 04m 11s

[0x2b3bfa0]

Information

Unkown run with identifier = tpi-smoke-test-1kvc5kjt-685p0we1 and:

• oldData = 10d9e1d1-6aee-4ae6-8eb3-9960a19f59a8
• newData = 3d9a31cd-d3e1-41e8-9d37-8a1f6297f3c7

Created with the AWS access key identifier ASIAU7UXIWDIYW2QQUGB

Error

$ journalctl --unit tpi-task
-- Logs begin at Mon 2022-08-29 11:58:05 UTC, end at Mon 2022-09-19 21:57:53 UTC. --
Sep 15 17:49:46 ip-172-31-14-196 systemd[1]: Started tpi-task.service.
Sep 15 17:49:47 ip-172-31-14-196 systemd[1]: tpi-task.service: Service reached runtime time limit. Stopping.
Sep 15 17:50:07 ip-172-31-14-196 tpi-task-shutdown[19551]: /usr/bin/tpi: line 1: Not: command not found
Sep 15 17:50:07 ip-172-31-14-196 systemd[1]: tpi-task.service: Control process exited, code=exited, status=127/n/a
Sep 15 17:50:07 ip-172-31-14-196 systemd[1]: tpi-task.service: Failed with result 'timeout'.

$ xxd /usr/bin/tpi
00000000: 4e6f 7420 466f 756e 64                   Not Found

Cause

As pointed out by @tasdomas in a separate conversation that led to iterative/terraform-provider-iterative#672, there is a ~20 minute time window during the release process where the latest binary download URL points to a release where build artifacts haven't been uplaoded yet,¹ corrupting a binary that's crucial to terminate compute resources once the task script finishes.

Footnotes

1. See https://github.com/iterative/setup-dvc/issues/9#issuecomment-789712644 for a similar issue experienced in the pasr with the DVC release process. ↩

[0x2b3bfa0]

@iterative/platform, could you kindly run aws blame on ASIAU7UXIWDIYW2QQUGB using CloudTrail?¹ 😈 I have reasons to believe that the resources above weren't created from GitHub Actions as expected, but from a developer's computer.

Footnotes

1. Nevermind, we have access to CloudTrail in the sandbox account. 🙈 ↩

[0x2b3bfa0]

I've turned off the lights

[dacbd]

Session duration is 43200 so not from an action, and the source IP is from LT.

So resolved with iterative/terraform-provider-iterative#672 correct? /close