Checkout step of build fails after release on 19 May 2022

[markwhitfeld]

Our build started failing after the latest release a few hours ago of cimg/base:current.
It is failing on the checkout step:

Using SSH Config Dir '/home/circleci/.ssh'
git version 2.36.1
Cloning git repository
Cloning into '.'...
Warning: Permanently added the ECDSA host key for IP address '140.82.113.3' to the list of known hosts.
remote: Enumerating objects: 43931, done.        
remote: Counting objects: 100% (1363/1363), done.        
remote: Compressing objects: 100% (568/568), done.        
remote: Total 43931 (delta 959), reused 1107 (delta 782), pack-reused 42568        
Receiving objects: 100% (43931/43931), 38.30 MiB | 39.98 MiB/s, done.
Resolving deltas: 100% (30259/30259), done.
Checking out branch
fatal: unsafe repository ('/mnt/ramdisk' is owned by someone else)
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/ramdisk

exit status 128

Here are the steps in this job that get to this point:

  build-and-push:
    docker:
      - image: cimg/base:current
    working_directory: /mnt/ramdisk
    steps:
      - gcp-gcr/gcr-auth:
          registry-url: gcr.io,us-west1-docker.pkg.dev
      - checkout
      - ......

How do you suggest we resolve this issue?
Are we doing something wrong in our job, or is there a bug in this release?

Environment Details

Environment details from "Spin up environment" step:

Build-agent version 1.0.123804-6379e05c (2022-05-18T21:49:50+0000)
System information:
 Server Version: 20.10.15
 Storage Driver: overlay2
  Backing Filesystem: xfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Kernel Version: 5.13.0-1021-aws
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64

Starting container cimg/base:current
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/base:current
current: Pulling from cimg/base
Digest: sha256:087e0c2f4f3702c9080b726e240ec4e649eb3242cc2b453c8a49476a40bc50d1
Status: Downloaded newer image for cimg/base:current
cimg/base:current:
  using image cimg/base@sha256:087e0c2f4f3702c9080b726e240ec4e649eb3242cc2b453c8a49476a40bc50d1
  pull stats: Image was already available so the image was not pulled
  time to create container: 740ms
Time to upload agent and config: 509.777606ms
Time to start containers: 375.07039ms

It worked a few hours ago when based on cimg/base@sha256:a394761a8abe3dbb25170529be8687c5d47eea2cbe118e37a7b49adb48692d45

[FelicianoTech]

@JalexChen are you able to take a look at this? A quick fix would be to tag the second most recent commit to main as 'monthly' and push it up.

Assuming you find an issue with this release

[markwhitfeld]

It is related to a vulnerability fix in GIT:
https://github.blog/2022-04-12-git-security-vulnerability-announced/

Github actions had to make adjustments to fix their checkout step.
Have a look at these issues (and related PRs):
actions/checkout#759
actions/checkout#760

It was reported on CircleCI discuss about a month ago, but it looks like nobody took note:
https://discuss.circleci.com/t/unable-to-checkout-code/43672

What is your process around testing these cimg releases?
I don't see much in the way of automated checks in your CI for this repo before publishing.

[FelicianoTech]

So this then isn't a regression in the image so to speak. Setting a working_directory outside of /home/circleci isn't technically supported in Convenience Images. Considering the type of change git made and the fact that our images already allow passwordless sudo, I don't think it's unreasonable to set the default git behavior back to the old one. I believe I saw that was possible. I will double check.

In the meantime, possible solutions forward here:

1. Change (or don't set) your working directory.
2. Use the previous base image release, 2022.04.

As for testing, we have some testing in this repository and some in external repositories. We are gearing up to quadruple down on the amount of testing that all images under cimg/ get in the next couple of months.

---

I'm going to keep this issue open to track the git behavior change request.

[Dunedan]

So this then isn't a regression in the image so to speak. Setting a working_directory outside of /home/circleci isn't technically supported in Convenience Images.

The CircleCI documentation explicitly suggests using /mnt/ramdisk as working_directory: https://circleci.com/docs/2.0/executor-types/#ram-disks

So how is it no regression that official CircleCI images don't work with suggested configuration?

[FelicianoTech]

The CircleCI documentation explicitly suggests using /mnt/ramdisk as working_directory:

If you want to use that feature yes. It doesn't say it works with a Convenience Image. Even in the example provided in that doc it uses the Docker Library alpine image, which is not one of our images.

So how is it no regression that official CircleCI images don't work with suggested configuration?

We shipped an updated version of git that works as git intended. This is the change they wanted.

---

As I mentioned above, I've been following this. Git introduced a config flag to add individual directories to be ignored by this new security feature. Then, in a version of git newer than what the May update provides, they allow an * to be used, basically allowing us to have git work like it did before the CVE. I have a PR in place for this change. Please see #171 for info on when you can use it.

In the meantime, you can run the following in your config BEFORE - checkout to get around this now, that's if you don't want to use one of my previous two suggestions:

- run: git config --global --add safe.directory '*'
# or
- run: git config --global --add safe.directory '/mnt/ramdisk'
- checkout

I hope that helps.