Runner scale set dind container not starting

[sravula84]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

gha-runner-scale-set-0.9.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1> installed runner scale set i can dind container but its failing with 
time="2024-04-11T17:36:43.038675049Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby                                                                                                                                                                                                                                                                               │
│ time="2024-04-11T17:36:43.039373640Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd                                                                                                                                                                                                                                                                                                             │
│ time="2024-04-11T17:36:43.040084974Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby                                                                                                                                                                                                                                                            │
│ failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1                                     │
│  (exit status 4))                                                                                                                                                                                                                                                                                                                                                                                                                        │
│ Stream closed EOF for arc-runners/prosper-linux-1-lpbvp-runner-fk69f (dind)                                                                                                                                                                                                                                                                                                                                                              │
│

Describe the bug

1> installed runner scale set i can dind container but its failing with
time="2024-04-11T17:36:43.038675049Z" level=info msg="stopping event stream following graceful shutdown" error="" module=libcontainerd namespace=moby │
│ time="2024-04-11T17:36:43.039373640Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd │
│ time="2024-04-11T17:36:43.040084974Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby │
│ failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain: (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables): RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1 │
│ (exit status 4)) │
│ Stream closed EOF for arc-runners/prosper-linux-1-lpbvp-runner-fk69f (dind) │

Describe the expected behavior

docker:dind container should up and running

Additional Context

1> installed runner scale set i can dind container but its failing with 
time="2024-04-11T17:36:43.038675049Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby                                                                                                                                                                                                                                                                               │
│ time="2024-04-11T17:36:43.039373640Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd                                                                                                                                                                                                                                                                                                             │
│ time="2024-04-11T17:36:43.040084974Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby                                                                                                                                                                                                                                                            │
│ failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1                                     │
│  (exit status 4))                                                                                                                                                                                                                                                                                                                                                                                                                        │
│ Stream closed EOF for arc-runners/prosper-linux-1-lpbvp-runner-fk69f (dind)                                                                                                                                                                                                                                                                                                                                                              │

Controller Logs

1> installed runner scale set i can dind container but its failing with 
time="2024-04-11T17:36:43.038675049Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby                                                                                                                                                                                                                                                                               │
│ time="2024-04-11T17:36:43.039373640Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd                                                                                                                                                                                                                                                                                                             │
│ time="2024-04-11T17:36:43.040084974Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby                                                                                                                                                                                                                                                            │
│ failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1                                     │
│  (exit status 4))                                                                                                                                                                                                                                                                                                                                                                                                                        │
│ Stream closed EOF for arc-runners/prosper-linux-1-lpbvp-runner-fk69f (dind)                                                                                                                                                                                                                                                                                                                                                              │

Runner Pod Logs

1> installed runner scale set i can see dind container, but its failing with 
time="2024-04-11T17:36:43.038675049Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby                                                                                                                                                                                                                                                                               │
│ time="2024-04-11T17:36:43.039373640Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd                                                                                                                                                                                                                                                                                                             │
│ time="2024-04-11T17:36:43.040084974Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby                                                                                                                                                                                                                                                            │
│ failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.10 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1                                     │
│  (exit status 4))                                                                                                                                                                                                                                                                                                                                                                                                                        │
│ Stream closed EOF for arc-runners/prosper-linux-1-lpbvp-runner-fk69f (dind)                                                                                                                                                                                                                                                                                                                                                              │

[nikola-jokic]

Hey @sravula84,

This is not an ARC issue. The error starts showing up when ARC applies your configuration. I would suggest you inspect the node and see why it can't register the bridge driver. When this stage is reached, the error is out of ARCs control.

Closing this one.

[marc-barry]

@sravula84 I'm hitting the same issue you describe in this when running on a Google Cloud Platform GKE cluster. The documentation offers no help into how to investigate this. I'm using the dind container mode. Perhaps it simply won't work on GKE. Did you happen to solve your issue?

[marc-barry]

Looks like #3257 is related.