Files excluded in .gitignore files?

[cwilby]

I'm currently building a workflow using self-hosted runners for a Laravel application to automatically:

1. Run npm run build on a self hosted actions runner, populating ./public/* folders.
2. Sync ./public/* from the runner to /var/www/public/* on the web server using this rsync action.

The issue at the moment is that the build output is not being transferred to the web server.

To go back a bit - to make this work, I added a .gitignore file to each folder in ./public/*. This is because the server pulls the latest changes via Git, which would create conflicts if the build output were included in source control.
This keeps the structure of the public directory in source control, but not the files. When a build succeeds, those files are "rsync-ed" to the server.

A strange thing is that the .gitignore files are being copied to the web server, just not the files that would be ignored.

Verification that runner has the build output

# start a shell on the self hosted runner
$ docker-compose exec github-runner bash 

# open the source directory
$ cd /tmp/runner/work/project-name/project-name/

# open the public folder
$ cd public

# list everything inside - `mix-manifest.json` is present, asserting the runner has created the build output successfully.
$ ls -lah

total 44K
drwxr-xr-x  7 root root 4.0K Sep 24 21:52 .
drwxr-xr-x 15 root root 4.0K Sep 24 21:51 ..
drwxr-xr-x  6 root root 4.0K Sep 24 21:50 css
-rw-r--r--  1 root root    0 Sep 24 18:29 favicon.ico
drwxr-xr-x  3 root root 4.0K Sep 24 21:52 fonts
-rw-r--r--  1 root root  698 Sep 24 18:29 .htaccess
drwxr-xr-x  3 root root 4.0K Sep 24 21:52 images
-rw-r--r--  1 root root 1.8K Sep 24 18:29 index.php
drwxr-xr-x  6 root root 4.0K Sep 24 18:29 js
-rw-r--r--  1 root root 3.5K Sep 24 21:52 mix-manifest.json   <---------------
-rw-r--r--  1 root root   24 Sep 24 18:29 robots.txt
drwxr-xr-x  4 root root 4.0K Sep 24 18:29 vendor                    

rsync output

I am expecting to see mix-manifest.json in this output.

/usr/bin/docker run 
    --name c3f2612037611bedbc4af49af158a513e1d677_8278ef 
    --label c3f261 
    --workdir /github/workspace 
    --rm 
    -e NVMRC 
    -e INPUT_SWITCHES 
    -e INPUT_PATH 
    -e INPUT_REMOTE_PATH 
    -e INPUT_REMOTE_HOST 
    -e INPUT_REMOTE_USER 
    -e INPUT_REMOTE_KEY 
    -e INPUT_RSH 
    -e INPUT_REMOTE_PORT 
    -e INPUT_REMOTE_KEY_PASS 
    -e HOME 
    -e GITHUB_JOB 
    -e GITHUB_REF 
    -e GITHUB_SHA 
    -e GITHUB_REPOSITORY 
    -e GITHUB_REPOSITORY_OWNER 
    -e GITHUB_RUN_ID 
    -e GITHUB_RUN_NUMBER 
    -e GITHUB_RETENTION_DAYS 
    -e GITHUB_RUN_ATTEMPT 
    -e GITHUB_ACTOR 
    -e GITHUB_WORKFLOW 
    -e GITHUB_HEAD_REF 
    -e GITHUB_BASE_REF 
    -e GITHUB_EVENT_NAME 
    -e GITHUB_SERVER_URL 
    -e GITHUB_API_URL 
    -e GITHUB_GRAPHQL_URL 
    -e GITHUB_WORKSPACE 
    -e GITHUB_ACTION 
    -e GITHUB_EVENT_PATH 
    -e GITHUB_ACTION_REPOSITORY 
    -e GITHUB_ACTION_REF 
    -e GITHUB_PATH 
    -e GITHUB_ENV 
    -e RUNNER_OS 
    -e RUNNER_NAME 
    -e RUNNER_TOOL_CACHE 
    -e RUNNER_TEMP 
    -e RUNNER_WORKSPACE 
    -e ACTIONS_RUNTIME_URL 
    -e ACTIONS_RUNTIME_TOKEN 
    -e ACTIONS_CACHE_URL 
    -e GITHUB_ACTIONS=true 
    -e CI=true 
    -v "/var/run/docker.sock":"/var/run/docker.sock" 
    -v "/tmp/runner/work/_temp/_github_home":"/github/home" 
    -v "/tmp/runner/work/_temp/_github_workflow":"/github/workflow" 
    -v "/tmp/runner/work/_temp/_runner_file_commands":"/github/file_commands" 
    -v "/tmp/runner/work/project-name/project-name":"/github/workspace" 
    c3f261:2037611bedbc4af49af158a513e1d677

Identity added: (stdin) (project_deploy)
Warning: Permanently added '***' (ED25519) to the list of known hosts.
sending incremental file list
./
.htaccess
favicon.ico
index.php
robots.txt
deleting css/marketing/.gitignore
deleting images/.gitignore
css/app-a/
css/app-a/.gitignore
css/app-b/
css/app-b/.gitignore
css/app-c/
css/app-c/.gitignore
css/marketing/
fonts/
fonts/.gitignore
images/
js/app-a/
js/app-a/.gitignore
js/app-b/
js/app-b/.gitignore
js/app-c/
js/app-c/.gitignore
js/marketing/
js/marketing/.gitignore

sent 2,562 bytes  received 18,609 bytes  8,468.40 bytes/sec
total size is 3,247,432  speedup is 153.39

[cwilby]

Ok, I managed to get my CI process working, but no solution to this problem yet.

When rsync runs on the runners runtime, it is able to find and sync those files.

[Burnett01]

Ok, I managed to get my CI process working, but no solution to this problem yet.

When rsync runs on the runners runtime, it is able to find and sync those files.

Hello @cwilby, thank's for reaching out. It is interesting to know that all your intended files are synced when running the action on the runners runtime but not on your self-hosted runner.
Are there any rsync switches active or filters?

[cwilby]

Thanks for responding - here is the build step, the switches are -avzr --delete.

      - name: Deploy build artifact
        if: github.ref == 'refs/heads/develop'
        uses: burnett01/rsync-deployments@5.1
        with:
          switches: -avzr --delete
          path: public/
          remote_path: ${{ secrets.SSH_PATH }}
          remote_host: ${{ secrets.SSH_HOST }}
          remote_user: ${{ secrets.SSH_USER }}
          remote_key: ${{ secrets.SSH_KEY }}

[cwilby]

I've discovered the issue is related to running self hosted runners using Docker, where the runner container shares a Docker daemon with the host.

The issue is that the when the volume is mounted to the container that runs this action, the mount point is from the host operating systems perspective, not the runner containers perspective (which contains the build output).

This is only a problem when running self hosted runners using Docker.

[cwilby]

FYI - I found a solution I'm happy with 🎉

A GitHub workflow has many Jobs, and each Job can specify which runner it should run on.

Instead of one job, I now run Test/Build on the self hosted runner, and Deploy on a GitHub hosted runner.

The Build script uses https://github.com/actions/upload-artifact to upload the build output to the workflow.

The Deploy script uses https://github.com/actions/download-artifact to download the build output from the workflow, and then uses this rsync action successfully to push to environments.

[Burnett01]

Congrats @cwilby for solving this issue and for investigating. I am glad you figured it out and thank you for your contribution! I have pinned this issue and added the documentation label, in case anyone else runs into the same problem.
Using separate runners on each job combined with the artifacts actions is a straightforward way of dealing with your usecase.

Have a great week,
greetings
Steve

[cwilby]

Additionally using https://github.com/marketplace/actions/delete-artifact to delete artifacts at the end of each build (I don't need to keep them around) to prevent taking up space I don't need.