Failed to use act_runner with cache with error `failed: socket hang up`

[catfishlty]

Description

Overview

Version

All my services are on Docker.
Gitea Instance: gitea/gitea:1.22.1, name: gitea, port: 3000
Act_runner Instance: gitea/act_runner:0.2.10, name: gitea-runner, port: 9040(for cache server)
Job Container: catthehacker/ubuntu:act-latest

Network

All these 3 containers are in one Docker network(172.18.0.0/24),
And I follow this to setup the act runner cache setting, and I'm pretty sure that Job container can connect to the cache server port testing by curl

Problems

When I use this workflow to start a action job, the job will be succeed at the end, but the cache save/restore steps always failed.
Then I catch the logs form act_runner container, even if I set the log.level to debug , I can't get the logs with more details.

Workflow

name: Test
run-name: test 🚀
on: 
  [push]

jobs:
  Release:
    runs-on: linux_amd64
    steps:
      - name: 🔎 Check out repository code
        uses: actions/checkout@v4
      - name: 💡 Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.22
          cache-dependency-path: "**/*.sum"
      - name: 🔎 Install dependencies
        run: |
          go mod tidy
      - name: 🔎 Build
        run: |
          go build -v

Config for Act Runner

# Example configuration file, it's safe to copy this as the default config file without any modification.

# You don't have to copy this file to your instance,
# just run `./act_runner generate-config > config.yaml` to generate a config file.

log:
  # The level of logging, can be trace, debug, info, warn, error, fatal
  level: info

runner:
  # Where to store the registration result.
  file: .runner
  # Execute how many tasks concurrently at the same time.
  capacity: 1
  # Extra environment variables to run jobs.
  # envs:
  # Extra environment variables to run jobs from a file.
  # It will be ignored if it's empty or the file doesn't exist.
  env_file: .env
  # The timeout for a job to be finished.
  # Please note that the Gitea instance also has a timeout (3h by default) for the job.
  # So the job could be stopped by the Gitea instance if it's timeout is shorter than this.
  timeout: 3h
  # Whether skip verifying the TLS certificate of the Gitea instance.
  insecure: true
  # The timeout for fetching the job from the Gitea instance.
  fetch_timeout: 5s
  # The interval for fetching the job from the Gitea instance.
  fetch_interval: 2s
  # The labels of a runner are used to determine which jobs the runner can run, and how to run them.
  # Like: "macos-arm64:host" or "ubuntu-latest:docker://gitea/runner-images:ubuntu-latest"
  # Find more images provided by Gitea at https://gitea.com/gitea/runner-images .
  # If it's empty when registering, it will ask for inputting labels.
  # If it's empty when execute `daemon`, will use labels in `.runner` file.
  labels:
    #- "ubuntu-latest:docker://gitea/runner-images:ubuntu-latest"
    - "linux_amd64:docker://catthehacker/ubuntu:act-latest"

cache:
  # Enable cache server to use actions/cache.
  enabled: true
  # The directory to store the cache data.
  # If it's empty, the cache data will be stored in $HOME/.cache/actcache.
  dir: ""
  # The host of the cache server.
  # It's not for the address to listen, but the address to connect from job containers.
  # So 0.0.0.0 is a bad choice, leave it empty to detect automatically.
  host: "gitea-runner"
  # The port of the cache server.
  # 0 means to use a random available port.
  port: 9040
  # The external cache server URL. Valid only when enable is true.
  # If it's specified, act_runner will use this URL as the ACTIONS_CACHE_URL rather than start a server by itself.
  # The URL should generally end with "/".
  external_server: ""

container:
  # Specifies the network to which the container will connect.
  # Could be host, bridge or the name of a custom network.
  # If it's empty, act_runner will create a network automatically.
  network: "local_net"
  # Whether to use privileged mode or not when launching task containers (privileged mode is required for Docker-in-Docker).
  privileged: false
  # And other options to be used when the container is started (eg, --add-host=my.gitea.url:host-gateway).
  options:
  # The parent directory of a job's working directory.
  # NOTE: There is no need to add the first '/' of the path as act_runner will add it automatically. 
  # If the path starts with '/', the '/' will be trimmed.
  # For example, if the parent directory is /path/to/my/dir, workdir_parent should be path/to/my/dir
  # If it's empty, /workspace will be used.
  workdir_parent:
  # Volumes (including bind mounts) can be mounted to containers. Glob syntax is supported, see https://github.com/gobwas/glob
  # You can specify multiple volumes. If the sequence is empty, no volumes can be mounted.
  # For example, if you only allow containers to mount the `data` volume and all the json files in `/src`, you should change the config to:
  # valid_volumes:
  #   - data
  #   - /src/*.json
  # If you want to allow any volume, please use the following configuration:
  # valid_volumes:
  #   - '**'
  valid_volumes: []
  # overrides the docker client host with the specified one.
  # If it's empty, act_runner will find an available docker host automatically.
  # If it's "-", act_runner will find an available docker host automatically, but the docker host won't be mounted to the job containers and service containers.
  # If it's not empty or "-", the specified docker host will be used. An error will be returned if it doesn't work.
  docker_host: ""
  # Pull docker image(s) even if already present
  force_pull: true
  # Rebuild docker image(s) even if already present
  force_rebuild: false

host:
  # The parent directory of a job's working directory.
  # If it's empty, $HOME/.cache/act/ will be used.
  workdir_parent:

Logs from Act Runner Container

[Test/Release] ::debug::Reserving Cache
[Test/Release]   | ::debug::Reserving Cache
[Test/Release] ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release]   | ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release] ::debug::reserveCache - Attempt 1 of 2 failed with error: socket hang up
[Test/Release]   | ::debug::reserveCache - Attempt 1 of 2 failed with error: socket hang up
[Test/Release] ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release]   | ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release] ::debug::reserveCache - Attempt 2 of 2 failed with error: socket hang up
[Test/Release]   | ::debug::reserveCache - Attempt 2 of 2 failed with error: socket hang up
[Test/Release] ::warning::Failed to save: reserveCache failed: socket hang up
[Test/Release]   | ::warning::Failed to save: reserveCache failed: socket hang up

Logs form Gitea - Actions - Log View

Setup go version spec 1.22
Found in cache @ /opt/hostedtoolcache/go/1.[2](https://{domain}/{org}/{repo}/actions/runs/5#jobstep-2-2)2.5/x64
Added go to the path
Successfully set up Go version 1.22
[command]/opt/hostedtoolcache/go/1.22.5/x6[4](https://{domain}/{org}/{repo}/actions/runs/5#jobstep-2-4)/bin/go env GOMODCACHE
[command]/opt/hostedtoolcache/go/1.22.[5](https://{domain}/{org}/{repo}/actions/runs/5#jobstep-2-5)/x64/bin/go env GOCACHE
/root/go/pkg/mod
/root/.cache/go-build
::warning::Failed to restore: getCacheEntry failed: socket hang up
Cache is not found
##[add-matcher]/run/act/actions/actions-setup-go@v5/matchers.json
go version go1.22.5 linux/amd[6](https://{domain}/{org}/{repo}/actions/runs/5#jobstep-2-6)4

Gitea Version

1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Running in Docker Engine Community

Database

PostgreSQL

[wolfogre]

Job containers are on a different network from the runner, so it will fail to visit gitea-runner:9040 from job containers, please check https://docs.gitea.com/usage/actions/act-runner#configuring-cache-when-starting-a-runner-using-docker-image

[catfishlty]

Job containers are on a different network from the runner, so it will fail to visit gitea-runner:9040 from job containers, please check https://docs.gitea.com/usage/actions/act-runner#configuring-cache-when-starting-a-runner-using-docker-image

Thanks for the reply, but it's not like you said actually.
I set the job container to the custom network local_net and my gitea instance and act_runner are both linked to local_net, then I use curl gitea-runner:9400, the response will be 204 NO Content, so it means it's not the network issue, the connection is correct, But the caching function doesn't seem to work.

I think with this config, there will be 3 containers in the same network, gitea instance, gitea act runner, job container.

container:
  # Specifies the network to which the container will connect.
  # Could be host, bridge or the name of a custom network.
  # If it's empty, act_runner will create a network automatically.
  network: "local_net"

[wolfogre]

Sorry, I overlooked that. Thanks for pointing it out.

You just did what came to my mind: run curl in job containers to check the network. To make sure, you have run curl gitea-runner:9040/_apis/artifactcache/cache, not 9400, not the root path, and it returned 204, right?

If so, I would say it's weird. Could you please set the log level to debug (in the config file for runner), and search logs with key word artifactcache?

[catfishlty]

Sorry, it's a typo for the port number(Configs has been changed several times), it should be 9040.

Steps how I test it

1. trigger the action by push
2. Container started and the workflow start.
3. Action call cache functions but failed.

Setup go version spec 1.22
Found in cache @ /opt/hostedtoolcache/go/1.[2](https://{domain}/{org}/{repo}/actions/runs/6#jobstep-2-2)2.5/x64
Added go to the path
Successfully set up Go version 1.22
[command]/opt/hostedtoolcache/go/1.22.5/x6[4](https://{domain}/{org}/{repo}/actions/runs/6#jobstep-2-4)/bin/go env GOMODCACHE
[command]/opt/hostedtoolcache/go/1.22.[5](https://{domain}/{org}/{repo}/actions/runs/6#jobstep-2-5)/x64/bin/go env GOCACHE
/root/go/pkg/mod
/root/.cache/go-build
::warning::Failed to restore: getCacheEntry failed: socket hang up
Cache is not found
##[add-matcher]/run/act/actions/actions-setup-go@v5/matchers.json
go version go1.22.5 linux/amd[6](https://{domain}/{org}/{repo}/actions/runs/6#jobstep-2-6)4

4. Run curl gitea-runner:9040/_apis/artifactcache/cache in Job Container, Return 204 NO CONTENT
5. Catch the logs in runner container.

[Test/Release]   | ::debug::Search path '/root/go/pkg/mod'
[Test/Release] ::debug::Search path '/root/.cache/go-build'
[Test/Release]   | ::debug::Search path '/root/.cache/go-build'
[Test/Release] ::debug::Matched: ../../../root/go/pkg/mod
[Test/Release]   | ::debug::Matched: ../../../root/go/pkg/mod
[Test/Release] ::debug::Matched: ../../../root/.cache/go-build
[Test/Release]   | ::debug::Matched: ../../../root/.cache/go-build
[Test/Release] ::debug::Cache Paths:
[Test/Release]   | ::debug::Cache Paths:
[Test/Release] ::debug::["../../../root/go/pkg/mod","../../../root/.cache/go-build"]
[Test/Release]   | ::debug::["../../../root/go/pkg/mod","../../../root/.cache/go-build"]
[Test/Release] ::debug::Archive Path: /tmp/73b4df17-4636-4fea-bf66-8c8b20762b75/cache.tzst
[Test/Release]   | ::debug::Archive Path: /tmp/73b4df17-4636-4fea-bf66-8c8b20762b75/cache.tzst
[Test/Release]   | [command]/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /workspace/chc/test-action --files-from manifest.txt --use-compress-program zstdmt
[Test/Release] ::debug::File Size: 8078016
[Test/Release]   | ::debug::File Size: 8078016
[Test/Release] ::debug::Reserving Cache
[Test/Release]   | ::debug::Reserving Cache
[Test/Release] ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release]   | ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release] ::debug::reserveCache - Attempt 1 of 2 failed with error: socket hang up
[Test/Release]   | ::debug::reserveCache - Attempt 1 of 2 failed with error: socket hang up
time="2024-07-19T10:56:22+08:00" level=debug msg="GET /_apis/artifactcache/cache" func="[func2]" file="[handler.go:345]" module=artifactcache
time="2024-07-19T10:56:22+08:00" level=debug msg="skip gc: 2024-07-19 10:54:06.351556786 +0800 HKT m=+0.004439592" func="[gcCache]" file="[handler.go:428]" module=artifactcache
[Test/Release] ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
[Test/Release]   | ::debug::Resource Url: http://gitea-runner:9040/_apis/artifactcache/caches
time="2024-07-19T10:56:27+08:00" level=debug msg="GET /_apis/artifactcache/cache" func="[func2]" file="[handler.go:345]" module=artifactcache
time="2024-07-19T10:56:27+08:00" level=debug msg="skip gc: 2024-07-19 10:54:06.351556786 +0800 HKT m=+0.004439592" func="[gcCache]" file="[handler.go:428]" module=artifactcache
[Test/Release] ::debug::reserveCache - Attempt 2 of 2 failed with error: socket hang up
[Test/Release]   | ::debug::reserveCache - Attempt 2 of 2 failed with error: socket hang up
[Test/Release] ::warning::Failed to save: reserveCache failed: socket hang up
[Test/Release]   | ::warning::Failed to save: reserveCache failed: socket hang up
[Test/Release]   ✅  Success - Post 💡 Setup Go

I hope it can help.

[wolfogre]

Thanks! According to the log, it seems stuck in the reserve method.

You could test it with

- run: |
     curl -v -X POST ${ACTIONS_CACHE_URL}_apis/artifactcache/caches -H "Content-Type: application/json" -d '{"key": "test-key", "version": "test", "cacheSize": 100}' | jq

and it should fail.

This step doesn't do many things. It just opens the bolt database file, inserts a new record, and returns.

Since it's OK to call /_apis/artifactcache/cache, it should be fine to open the database file. So there may be some problem writing to the bolt database.

Could you please provide more information about the disk that stores the bolt database file? Is it a SD card or something similar? Is it full?

[catfishlty]

Unfortunately, the curl command execute successfully.
Here's the log for executing this command.

Note: Unnecessary use of -X or --request, POST is already inferred.
* Uses proxy env variable NO_PROXY == '127.0.0.1,localhost,172.18.0.0/16,192.168.0.0/8'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.18.0.42:9040...
* Connected to gitea-runner (172.18.0.42) port 9040 (#0)
> POST /_apis/artifactcache/caches HTTP/1.1
> Host: gitea-runner:9040
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 56
> 
} [56 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Date: Mon, 22 Jul 2024 09:45:[3](https://{domain}/{org}/{repo}/actions/runs/7#jobstep-3-3)6 GMT
< Content-Length: 13
< 
{ [13 bytes data]
100    69  100    13  100    56   1289   5552 --:--:-- --:--:-- --:--:--  862[5](https://{domain}/{org}/{repo}/actions/runs/7#jobstep-3-5)
* Connection #0 to host gitea-runner left intact
{
  "cacheId": 1
}

And for the devices, I host it on intel x64 platform with SSD with enough disk spaces, ESXI as the infrastructure and virtual debian-12 OS, and docker engine inside. so I don't think it's a storage issue.
Bolt DB file is also here.

bolt.zip

[wolfogre]

Hmm, that's so weird.

So, there's no problem with the network, bolt db or act_runner. And I think maybe there's some problem with Node.js ...

Then I found this:

• "socket hang up" / ECONNRESET on consecutive requests with Node.js 19 and Node.js 20 node-fetch/node-fetch#1735

However, it's a bug of node-fetch, but Node.js doesn't need it any longer since v18, so maybe you are using a old version of node?

Could you please use gitea/runner-images:ubuntu-latest as the job container images? Or add "setup-node" before "setup-go":

      - uses: actions/setup-node@v4
        with:
          node-version: 20

---

Update:

I'm not an expert in Node.js, but now I'm quite sure the problem is not related to Gitea or act_runner. So if upgrading Node didn't help, could you please provide your complete workflow file?

[catfishlty]

T_T, still failed to get the cache

NodeJS Version: v20.15.1, using setup-node@v4 to setup node env.
I changed the image to run this job.

runner:
  # Where to store the registration result.
  file: .runner
  # Execute how many tasks concurrently at the same time.
  capacity: 1
  # Extra environment variables to run jobs.
  envs:
    NO_PROXY: 127.0.0.1,localhost,172.18.0.0/16,192.168.0.0/8
  # Extra environment variables to run jobs from a file.
  # It will be ignored if it's empty or the file doesn't exist.
  env_file: .env
  # The timeout for a job to be finished.
  # Please note that the Gitea instance also has a timeout (3h by default) for the job.
  # So the job could be stopped by the Gitea instance if it's timeout is shorter than this.
  timeout: 3h
  # Whether skip verifying the TLS certificate of the Gitea instance.
  insecure: true
  # The timeout for fetching the job from the Gitea instance.
  fetch_timeout: 5s
  # The interval for fetching the job from the Gitea instance.
  fetch_interval: 2s
  labels:
    # - "linux_amd64:docker://catthehacker/ubuntu:act-latest"
    - "linux_amd64:docker://gitea/runner-images:ubuntu-latest"

Logs

Restore

Setup go version spec 1.22
Found in cache @ /opt/hostedtoolcache/go/1.[2](https://{domain}/{org}/{repo}/actions/runs/9/jobs/0#jobstep-4-2)2.5/x64
Added go to the path
Successfully set up Go version 1.22
[command]/opt/hostedtoolcache/go/1.22.5/x6[4](https://{domain}/{org}/{repo}/actions/runs/9/jobs/0#jobstep-4-4)/bin/go env GOMODCACHE
[command]/opt/hostedtoolcache/go/1.22.[5](https://{domain}/{org}/{repo}/actions/runs/9/jobs/0#jobstep-4-5)/x64/bin/go env GOCACHE
/root/go/pkg/mod
/root/.cache/go-build
::warning::Failed to restore: getCacheEntry failed: socket hang up
Cache is not found
##[add-matcher]/run/act/actions/actions-setup-go@v5/matchers.json
go version go1.22.5 linux/amd[6](https://{domain}/{org}/{repo}/actions/runs/9/jobs/0#jobstep-4-6)4

Save

Exec command '[node /var/run/act/actions/actions-setup-go@v5/dist/cache-save/index.js]'
Working directory '/workspace/chc/test-action'
[command]/opt/hostedtoolcache/go/1.22.5/x64/bin/go env GOMODCACHE
[command]/opt/hostedtoolcache/go/1.22.5/x64/bin/go env GOCACHE
/root/go/pkg/mod
/root/.cache/go-build
[command]/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /workspace/chc/test-action --files-from manifest.txt --use-compress-program zstdmt
::warning::Failed to save: reserveCache failed: socket hang up
::warning::Failed to save: reserveCache failed: socket hang up
  ✅  Success - Post 💡 Setup Go

Here's the workflow file

name: Test
run-name: test 🚀
on: 
  [push]

jobs:
  Release:
    runs-on: linux_amd64
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Check NodeJS version
        run: |
          node -v
      - name: 🔎 Check out repository code
        uses: actions/checkout@v4
      - name: 💡 Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: 1.22
          cache-dependency-path: "**/*.sum"
      - run: |
          curl -v -X POST ${ACTIONS_CACHE_URL}_apis/artifactcache/caches -H "Content-Type: application/json" -d '{"key": "test-key", "version": "test", "cacheSize": 100}' | jq
      - name: 🔎 Install dependencies
        run: |
          go env -w GOPROXY=https://goproxy.io,direct
          go mod tidy
      - name: 🔎 Build
        run: |
          go build -v

[catfishlty]

I found the dependencies that node-fetch is used in @actions/cache

cache@4.0.2
+-- @actions/cache@3.2.4
| +-- @azure/ms-rest-js@2.7.0
| | +-- node-fetch@2.6.7
| | | +-- UNMET OPTIONAL DEPENDENCY encoding@^0.1.0
| | | `-- whatwg-url@5.0.0
| | |   +-- tr46@0.0.3
| | |   `-- webidl-conversions@3.0.1
| +-- @azure/storage-blob@12.13.0
| | +-- @azure/core-http@3.0.4
| | | +-- @types/node-fetch@2.6.2
| | | | +-- @types/node@16.18.3 deduped
| | | | `-- form-data@3.0.1
| | | |   +-- asynckit@0.4.0 deduped
| | | |   +-- combined-stream@1.0.8 deduped
| | | |   `-- mime-types@2.1.34 deduped
| | | +-- node-fetch@2.6.7 deduped

The node-fetch version is 2.6.7, but socket hang up issue is fixed after 3.3.2

Is there any other way to cache build dependencies using cache plugin?
As far as I know, many build environment setting plugins use @actions/cache, and reusing it may be a better solution.

Beside this, I also tried nodejs versions 16, 18, 20, but all failed.

[wolfogre]

It shouldn't be, Node.js has builtin fetch doesn't use node-fetch any longer since v18.

We have act runners to run setup-go with cache every day, we never hit the "socket hang up" error.

I still think there's some problem with your node env ...