arc 0.4.0 with service container or image fails on Failed to update oom_score_adj #2791

jandrunas · 2023-08-02T00:16:24Z

jandrunas
Aug 2, 2023

running a normal job works fine, but when trying to use a service container or specifying a container image in a work flow causes the job to fail with

Failed to update oom_score_adj for PID: 70. [WORKER 2023-08-01 23:48:59Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/70/oom_score_adj' is denied. [WORKER 2023-08-01 23:48:59Z INFO ProcessInvokerWrapper] ---> System.IO.IOException: Permission denied [WORKER 2023-08-01 23:48:59Z INFO ProcessInvokerWrapper] --- End of inner exception stack trace ---

I even copied the exact example from the deep dive video in https://www.youtube.com/watch?v=_F5ocPrv6io

I can connect to the runner and the permissions on /proc look ok, but any attempt to do anything in /proc gives permissions denied even as root or with sudo

the jobs run just fine on hosted runners, and were running just fine using 0.27.4 with runner deployments

.github/workflows/stupid_test.yml

this fails

---
name: stupid_test
on:
  workflow_dispatch

jobs:
  build-and-test:
    runs-on: arc-runner-set
    container:
      image: ghcr.io/actions/actions-runner:latest
    steps:
      - name: Trials
        run: |
          echo "Hello World" >> $GITHUB_STEP_SUMMARY
          sleep 720
        env:
          REDIS_HOST: redis
          REDIS_PORT: 6379

this does not

---
name: stupid_test
on:
  workflow_dispatch

jobs:
  build-and-test:
    runs-on: arc-runner-set
    steps:
      - name: Trials
        run: |
          echo "Hello World" >> $GITHUB_STEP_SUMMARY
          sleep 720
        env:
          REDIS_HOST: redis
          REDIS_PORT: 6379

values.yaml

githubConfigUrl: "https://github.com/PollyEx"
githubConfigSecret: "controller-manager"
maxRunners: 20
minRunners: 1
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "ebs"
    resources:
      requests:
        storage: 10Gi
        cpu: "1000m"
        memory: "2Gi"
template:
  spec:
    securityContext:
      fsGroup: 1001
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        imagePullPolicy: Always
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
        volumes:
          - name: work
            ephemeral:
              volumeClaimTemplate:
                spec:
                  accessModes: ["ReadWriteOnce"]
                  storageClassName: "ebs"
                  resources:
                    requests:
                      storage: 10Gi

ashishkurmi · 2023-09-27T02:09:34Z

ashishkurmi
Sep 27, 2023

I am experiencing this issue as well with the latest version of ARC when I run a GitHub Action step that uses a container:

[WORKER 2023-09-27 01:54:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '6cb15225-bace-5236-3d64-8b6d88bf381f', success rate: 1/1.
[WORKER 2023-09-27 01:54:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record '0bd5d730-eae6-5b54-0101-80d8a124934a', success rate: 1/1.
[WORKER 2023-09-27 01:54:41Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'c64be972-a956-57ab-3a81-fb6cad8060ba', success rate: 1/1.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Starting process:
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   File name: '/home/runner/externals/node16/bin/node'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Arguments: '/home/runner/k8s/index.js'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Working directory: '/home/runner/_work/link-arc/link-arc'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'True'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 681.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/681/oom_score_adj' is denied.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.StreamWriter.Dispose(Boolean disposing)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteAllText(String path, String contents)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Process started with process id 681, waiting for process exit.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] Close STDIN after the first redirect finished.
[WORKER 2023-09-27 01:54:41Z INFO ProcessInvokerWrapper] STDIN stream write finished.

1 reply

kmeratia Nov 25, 2023

Were you able to resolve this issue? any help is appreciated.

kmeratia · 2023-11-25T02:27:24Z

kmeratia
Nov 25, 2023

Were you able to resolve this issue? any help is appreciated.

0 replies

alexgnq · 2024-01-08T11:09:40Z

alexgnq
Jan 8, 2024

I'm also getting the same error using 0.8.1.

However Nikola seems to hint at it not being an issue:
#3132 (comment)

0 replies

asoltesz · 2024-02-02T17:58:19Z

asoltesz
Feb 2, 2024

I am also getting this error on 0.8.1 and the job fails with some kind of timeout.

Error on the GitHUB UI:

Error: connect ETIMEDOUT 172.30.0.1:443

It is possible that this is not the main reason.

What I can confirm that node with the index.js starts up and runs for a while.
Probably, index.js experiences some kind of problem.

It seems that there are no logs coming from index.js so we never learn the actual reason.

0 replies

sfiligoi · 2024-03-01T21:38:53Z

sfiligoi
Mar 1, 2024

Just noticed the same problem on our on-prem K8s cluster:

[2024-03-01 21:31:32Z INFO SelfUpdater] Validated Runner Hash matches /mnt/scratch/builder/_update/runner1.tar.gz : 56910d6628b41f99d9a1c5fe9df54981ad5d8c9e42fc14899dcc177e222e71c4
[2024-03-01 21:31:33Z INFO SelfUpdater] Which: 'tar'
[2024-03-01 21:31:33Z INFO SelfUpdater] Location: '/usr/bin/tar'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper] Starting process:
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   File name: '/usr/bin/tar'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   Arguments: '-xzf "/mnt/scratch/builder/_update/runner1.tar.gz"'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   Working directory: '/mnt/scratch/builder/_update'
[2024-03-01 21:31:33Z INFO ProcessInvokerWrapper]   Require exit code zero: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 73.
[2024-03-01 21:31:34Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/73/oom_score_adj' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
   at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
   at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)
   at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.File.WriteAllText(String path, String contents)
   at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
[2024-03-01 21:31:36Z INFO ProcessInvokerWrapper] Process started with process id 73, waiting for process exit.

4 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

arc 0.4.0 with service container or image fails on Failed to update oom_score_adj #2791

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

arc 0.4.0 with service container or image fails on Failed to update oom_score_adj #2791

Replies: 5 comments · 5 replies

Replies: 5 comments 5 replies