TestBot_Run_CARotation flakiness

[nklaassen]

Failure

Link(s) to logs

• https://console.cloud.google.com/cloud-build/builds;region=us-west1/a5d8a355-0c28-43f0-9c84-06c60978387d?project=ci-account

Relevant snippet

=== CONT  TestBot_Run_CARotation
    tbot_test.go:86: teleport process did not instantiate in 10 seconds
2022-07-14T14:59:15Z INFO             Shutting down long running test processes.. tbot/tbot_test.go:140
2022-07-14T14:59:15Z INFO [AUDIT]     role.created cluster_name:02b75d1401d6 code:T9000I ei:0 event:role.created expires:0001-01-01T00:00:00Z name:db-role time:2022-07-14T14:59:15.484Z uid:5080872e-24e8-45de-b50b-0e92a1c142df user:273f7c04-d92b-4a3a-a3c9-3ba85dba808d.02b75d1401d6 events/emitter.go:263
2022-07-14T14:59:15Z WARN [PROC:4]    Failed to write Admin identity: disk I/O error. pid:53002.4 service/connect.go:648
    tbot_test.go:80: 
        	Error Trace:	tbot_test.go:80
        	            				asm_amd64.s:1571
        	Error:      	Received unexpected error:
        	            	initialization failed
        	            		database is in readonly mode
        	Test:       	TestBot_Run_CARotation
--- FAIL: TestBot_Run_CARotation (10.06s)

[strideynet]

Oerr thanks for raising this. I'll try and take a look tonight or tomorrow.

[ibeckermayer]

A new error on this one:

• https://console.cloud.google.com/cloud-build/builds/bf669ff3-c24a-41d2-a567-a2687508ed29?project=ci-account

        	Test:       	TestBot_Run_CARotation
        	            	can only switch to phase update_clients from init, current phase is 
        	Error:      	Received unexpected error:
        	            				/workspace/lib/tbot/tbot_test.go:177
        	Error Trace:	/workspace/lib/tbot/tbot_test.go:59

[GavinFrazar]

A new error on this one:

• https://console.cloud.google.com/cloud-build/builds/bf669ff3-c24a-41d2-a567-a2687508ed29?project=ci-account

        	Test:       	TestBot_Run_CARotation
        	            	can only switch to phase update_clients from init, current phase is 
        	Error:      	Received unexpected error:
        	            				/workspace/lib/tbot/tbot_test.go:177
        	Error Trace:	/workspace/lib/tbot/tbot_test.go:59

Got the same failure today

[zmb3]

https://console.cloud.google.com/cloud-build/builds;region=global/d9879abe-a8a5-4089-be03-8bd29b254ef9;step=0?project=ci-account

[ibeckermayer]

https://console.cloud.google.com/cloud-build/builds/887d9ac4-3204-40fd-b340-b61500c5739c?project=ci-account

[ibeckermayer]

https://console.cloud.google.com/cloud-build/builds/82df7966-5310-4c28-ac60-8d1afe20abd3?project=ci-account

[r0mant]

@jakule Was fix test fixed? Just hit it on v11: https://github.com/gravitational/teleport/actions/runs/3989701570/jobs/6842456228. If we disabled it in master/v12, should we disable it in v11 as well until we can get a proper fix in?