Runner Deployment Pod keeps Terminating

[gvisca]

I have deployed the following RunnerDeployment on EKS 1.18

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runner
  namespace: actions-runner-system
spec:
  replicas: 1
  template:
    spec:
      organization: {my-org}
      image: summerwind/actions-runner-dind
      dockerdWithinRunnerContainer: true

It starts and keep working during several minutes (5-10 min) but then starts to be killed and never goes up ...

controller logs:

2020-10-22T14:30:37.328Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:30:39.889Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:30:39.889Z	INFO	controllers.RunnerReplicaSetdebug	{"runner": "actions-runner-system/github-actions-runner-2zbv7", "desired": 1, "available": 1}
2020-10-22T14:30:39.904Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerreplicaset", "request": "actions-runner-system/github-actions-runner-2zbv7"}
2020-10-22T14:30:39.906Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerdeployment", "request": "actions-runner-system/github-actions-runner"}
2020-10-22T14:30:39.906Z	INFO	controllers.RunnerReplicaSetdebug	{"runner": "actions-runner-system/github-actions-runner-2zbv7", "desired": 1, "available": 1}
2020-10-22T14:30:39.906Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerreplicaset", "request": "actions-runner-system/github-actions-runner-2zbv7"}
2020-10-22T14:30:40.148Z	ERROR	controllers.Runner	Failed to check if runner is busy	{"runner": "actions-runner-system/github-actions-runner-2zbv7-pxwg5", "error": "runner not found"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/summerwind/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
	/workspace/controllers/runner_controller.go:212
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190913080033-27d36303b655/pkg/util/wait/wait.go:88
2020-10-22T14:30:40.148Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:30:40.270Z	DEBUG	controllers.HorizontalRunnerAutoscaler	Calculated desired replicas	{"computed_replicas_desired": 1, "spec_replicas_min": 1, "spec_replicas_max": 5, "workflow_runs_completed": 150, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0}
2020-10-22T14:30:40.270Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "horizontalrunnerautoscaler", "request": "actions-runner-system/github-actions-runner-autoscaler"}

# Here it works during 30 minutes but then start failing

2020-10-22T14:50:19.113Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerdeployment", "request": "actions-runner-system/github-actions-runner"}
2020-10-22T14:50:19.113Z	INFO	controllers.RunnerReplicaSetdebug	{"runner": "actions-runner-system/github-actions-runner-2zbv7", "desired": 1, "available": 1}
2020-10-22T14:50:19.113Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerreplicaset", "request": "actions-runner-system/github-actions-runner-2zbv7"}
2020-10-22T14:50:35.921Z	INFO	controllers.RunnerReplicaSetdebug	{"runner": "actions-runner-system/github-actions-runner-2zbv7", "desired": 1, "available": 1}
2020-10-22T14:50:35.921Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerreplicaset", "request": "actions-runner-system/github-actions-runner-2zbv7"}
2020-10-22T14:50:36.011Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerdeployment", "request": "actions-runner-system/github-actions-runner"}
2020-10-22T14:50:36.208Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"github-actions-runner-2zbv7-pxwg5","uid":"eed46ffa-2da8-46e9-af95-dba35bf8bd16","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"2111908"}, "reason": "PodDeleted", "message": "Deleted pod 'github-actions-runner-2zbv7-pxwg5'"}
2020-10-22T14:50:36.208Z	INFO	controllers.Runner	Deleted runner pod	{"runner": "actions-runner-system/github-actions-runner-2zbv7-pxwg5", "repository": ""}
2020-10-22T14:50:36.208Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:36.212Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:36.218Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:37.115Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:40.519Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:40.572Z	INFO	controllers.Runner	Created runner pod	{"runner": "actions-runner-system/github-actions-runner-2zbv7-pxwg5", "repository": ""}
2020-10-22T14:50:40.572Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:40.572Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"github-actions-runner-2zbv7-pxwg5","uid":"eed46ffa-2da8-46e9-af95-dba35bf8bd16","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"2111908"}, "reason": "PodCreated", "message": "Created pod 'github-actions-runner-2zbv7-pxwg5'"}
2020-10-22T14:50:40.591Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:40.591Z	INFO	controllers.RunnerReplicaSetdebug	{"runner": "actions-runner-system/github-actions-runner-2zbv7", "desired": 1, "available": 1}
2020-10-22T14:50:40.603Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerreplicaset", "request": "actions-runner-system/github-actions-runner-2zbv7"}
2020-10-22T14:50:40.606Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerdeployment", "request": "actions-runner-system/github-actions-runner"}
2020-10-22T14:50:40.607Z	INFO	controllers.RunnerReplicaSetdebug	{"runner": "actions-runner-system/github-actions-runner-2zbv7", "desired": 1, "available": 1}
2020-10-22T14:50:40.607Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runnerreplicaset", "request": "actions-runner-system/github-actions-runner-2zbv7"}
2020-10-22T14:50:40.845Z	INFO	controllers.Runner	Deleted runner pod	{"runner": "actions-runner-system/github-actions-runner-2zbv7-pxwg5", "repository": ""}
2020-10-22T14:50:40.845Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:40.845Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"github-actions-runner-2zbv7-pxwg5","uid":"eed46ffa-2da8-46e9-af95-dba35bf8bd16","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"2118812"}, "reason": "PodDeleted", "message": "Deleted pod 'github-actions-runner-2zbv7-pxwg5'"}
2020-10-22T14:50:40.914Z	DEBUG	controllers.HorizontalRunnerAutoscaler	Calculated desired replicas	{"computed_replicas_desired": 1, "spec_replicas_min": 1, "spec_replicas_max": 5, "workflow_runs_completed": 150, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0}
2020-10-22T14:50:40.914Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "horizontalrunnerautoscaler", "request": "actions-runner-system/github-actions-runner-autoscaler"}
2020-10-22T14:50:41.091Z	INFO	controllers.Runner	Deleted runner pod	{"runner": "actions-runner-system/github-actions-runner-2zbv7-pxwg5", "repository": ""}
2020-10-22T14:50:41.091Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:41.091Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}
2020-10-22T14:50:41.091Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Runner","namespace":"actions-runner-system","name":"github-actions-runner-2zbv7-pxwg5","uid":"eed46ffa-2da8-46e9-af95-dba35bf8bd16","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"2118812"}, "reason": "PodDeleted", "message": "Deleted pod 'github-actions-runner-2zbv7-pxwg5'"}
2020-10-22T14:50:43.500Z	DEBUG	controller-runtime.controllerSuccessfully Reconciled	{"controller": "runner", "request": "actions-runner-system/github-actions-runner-2zbv7-pxwg5"}

It happened with version 0.9.3 and 0.9.4

[mumoshu]

@gvisca Thanks for reporting! I believe this relates to #138. Would you mind trying v0.9.5 that should fix the issue?

[gvisca]

Thanks a lot fot you answer @mumoshu
I've just tested with v0.10.0 and the issue is fixed indeed