Github Action checkout v3 fails on fetching - git error - The process '/usr/bin/git' failed with exit code 128 #9632

yardensade · 2024-04-04T10:23:22Z

Description

Many actions are all of a sudden failing today on checkout, absolutely nothing changed on my side.

This is the action step which I had for months:

runs:
  using: "composite"
  steps:
    - name: Check out ****-github-actions
      uses: actions/checkout@v3
      with:
        repository: ****-github-action
        ref: main
        path: ./.github/actions/****

Attaching logs:

2024-04-04T07:33:22.8383374Z ##[group]Run actions/checkout@v3
2024-04-04T07:33:22.8383775Z with:
2024-04-04T07:33:22.8384133Z   repository: ****
2024-04-04T07:33:22.8384632Z   ref: main
2024-04-04T07:33:22.8384970Z   path: ./.github/actions/****
2024-04-04T07:33:22.8385641Z   token: ****
2024-04-04T07:33:22.8385991Z   ssh-strict: true
2024-04-04T07:33:22.8386320Z   persist-credentials: true
2024-04-04T07:33:22.8386662Z   clean: true
2024-04-04T07:33:22.8386998Z   sparse-checkout-cone-mode: true
2024-04-04T07:33:22.8387342Z   fetch-depth: 1
2024-04-04T07:33:22.8387633Z   fetch-tags: false
2024-04-04T07:33:22.8388019Z   lfs: false
2024-04-04T07:33:22.8388302Z   submodules: false
2024-04-04T07:33:22.8388649Z   set-safe-directory: true
2024-04-04T07:33:22.8388989Z ##[endgroup]
2024-04-04T07:33:23.0326554Z Syncing repository: ****
2024-04-04T07:33:23.0328818Z ##[group]Getting Git version info
2024-04-04T07:33:23.0330105Z Working directory is '/home/runner/work/****'
2024-04-04T07:33:23.0331231Z [command]/usr/bin/git version
2024-04-04T07:33:23.0367581Z git version 2.43.2
2024-04-04T07:33:23.0392396Z ##[endgroup]
2024-04-04T07:33:23.0414859Z Temporarily overriding HOME='/home/runner/work/_temp/d405e998-3c6c-4296-9916-fbb53712203f' before making global git config changes
2024-04-04T07:33:23.0417435Z Adding repository directory to the temporary git global config as a safe directory
2024-04-04T07:33:23.0420012Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work****
2024-04-04T07:33:23.0457392Z ##[group]Initializing the repository
2024-04-04T07:33:23.0479936Z [command]/usr/bin/git init /home/runner/work/****
2024-04-04T07:33:23.0542829Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-04-04T07:33:23.0545017Z hint: is subject to change. To configure the initial branch name to use in all
2024-04-04T07:33:23.0546604Z hint: of your new repositories, which will suppress this warning, call:
2024-04-04T07:33:23.0547984Z hint: 
2024-04-04T07:33:23.0549297Z hint: 	git config --global init.defaultBranch <name>
2024-04-04T07:33:23.0550537Z hint: 
2024-04-04T07:33:23.0552992Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-04-04T07:33:23.0555023Z hint: 'development'. The just-created branch can be renamed via this command:
2024-04-04T07:33:23.0556710Z hint: 
2024-04-04T07:33:23.0557845Z hint: 	git branch -m <name>
2024-04-04T07:33:23.0559837Z Initialized empty Git repository in /home/runner/work/****
2024-04-04T07:33:23.0564373Z [command]/usr/bin/git remote add origin https://github.com/****
2024-04-04T07:33:23.0599952Z ##[endgroup]
2024-04-04T07:33:23.0601276Z ##[group]Disabling automatic garbage collection
2024-04-04T07:33:23.0603564Z [command]/usr/bin/git config --local gc.auto 0
2024-04-04T07:33:23.0632098Z ##[endgroup]
2024-04-04T07:33:23.0633569Z ##[group]Setting up auth
2024-04-04T07:33:23.0639491Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-04-04T07:33:23.0667251Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-04-04T07:33:23.0965730Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-04-04T07:33:23.0978576Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-04-04T07:33:23.1155656Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-04-04T07:33:23.1195621Z ##[endgroup]
2024-04-04T07:33:23.1197545Z ##[group]Fetching the repository
2024-04-04T07:33:23.1206635Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/main*:refs/remotes/origin/main* +refs/tags/main*:refs/tags/main*
2024-04-04T07:33:23.1258870Z /usr/lib/git-core/git-remote-https: error while loading shared libraries: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header
2024-04-04T07:33:23.1266961Z The process '/usr/bin/git' failed with exit code 128
2024-04-04T07:33:23.1268196Z Waiting 16 seconds before trying again
2024-04-04T07:33:39.1303048Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/main*:refs/remotes/origin/main* +refs/tags/main*:refs/tags/main*
2024-04-04T07:33:39.1341658Z /usr/lib/git-core/git-remote-https: error while loading shared libraries: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header
2024-04-04T07:33:39.1352537Z The process '/usr/bin/git' failed with exit code 128
2024-04-04T07:33:39.1353281Z Waiting 19 seconds before trying again
2024-04-04T07:33:58.1456060Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +refs/heads/main*:refs/remotes/origin/main* +refs/tags/main*:refs/tags/main*
2024-04-04T07:33:58.1494241Z /usr/lib/git-core/git-remote-https: error while loading shared libraries: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header
2024-04-04T07:33:58.1571200Z ##[error]The process '/usr/bin/git' failed with exit code 128

Platforms affected

Azure DevOps
GitHub Actions - Standard Runners
GitHub Actions - Larger Runners

Runner images affected

Image version and build link

ubuntu-20.04 20240403.1.0

Is it regression?

yes

Expected behavior

git should not fail on checkout

Actual behavior

git fails on checkout

Repro steps

simply use checkout v3 with ubuntu 20:04 runner and it will fail, sporadically

The text was updated successfully, but these errors were encountered:

shamil-mubarakshin · 2024-04-04T11:06:11Z

Hello @yardensade, I couldn't reproduce the issue, first job, second job. I would suggest contacting https://github.com/actions/checkout for further troubleshooting

yardensade · 2024-04-04T11:09:35Z

As I said this happens sporadically, the fact you ran it twice does not mean there is no issue.
I am running thousands of actions like that a day, and it happened to me dozens of times today, while most of the time it works.

I would not have taken my time to contact you if it was not very important, unlike the ubuntu version change that started this, checkout was not changed recently.

I am not the only one to experience this, I know for a fact it happens to others as well.

chris-griffin · 2024-04-04T15:58:31Z

We are also seeing this as new behavior. It is also sporadic, and happening about 1/100 runs. The first time I observed it was around the incident GitHub had last night so perhaps that incident is really ongoing?


Fetching the repository
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +18e82eac24081b13bbf6774a58e4ff02c2c7341a:refs/remotes/pull/44861/merge
  /usr/lib/git-core/git-remote-https: error while loading shared libraries: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header
  The process '/usr/bin/git' failed with exit code 128
  Waiting 16 seconds before trying again
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +18e82eac24081b13bbf6774a58e4ff02c2c7341a:refs/remotes/pull/44861/merge
  /usr/lib/git-core/git-remote-https: error while loading shared libraries: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header
  The process '/usr/bin/git' failed with exit code 128
  Waiting 14 seconds before trying again
  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=1 origin +18e82eac24081b13bbf6774a58e4ff02c2c7341a:refs/remotes/pull/44861/merge
  /usr/lib/git-core/git-remote-https: error while loading shared libraries: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header
  Error: The process '/usr/bin/git' failed with exit code 128

jmuleiro · 2024-04-05T17:15:38Z

I'm also seeing this error sporadically. It occurred in one out of 25 identical ubuntu-20.04 containers created from a matrix strategy.

Topherhindman · 2024-04-05T17:24:03Z

I'm also seeing this error randomly. It's happening about one in 20 for me on runs created by a job using a matrix.

mikhailkoliada · 2024-04-05T17:35:50Z

I'll check linkage on Monday, some ideas crossed my mind

mikhailkoliada · 2024-04-05T18:23:16Z

@Topherhindman @jmuleiro @chris-griffin @yardensade can someone play around for me and if build does not fail for you, run the following command apt list --installed (output gonna be long), I suspect there might be dependencies change that caused this, and I wanna study its objdump

yardensade · 2024-04-06T05:56:50Z

@Topherhindman @jmuleiro @chris-griffin @yardensade can someone play around for me and if build does not fail for you, run the following command apt list --installed (output gonna be long), I suspect there might be dependencies change that caused this, and I wanna study its objdump

Just to confirm we are aligned, you mean to apt list prior to the checkout step?

mikhailkoliada · 2024-04-06T10:12:36Z

@yardensade Yep, better even 2 runs to compare, 1 apt list with green checkout and 1 apt list with red checkout (before the actual checkout happens surely), I need to compare some deep dependencies installed in the runner image but I was unable to catch the problem myself.

yardensade · 2024-04-07T06:52:23Z

@yardensade Yep, better even 2 runs to compare, 1 apt list with green checkout and 1 apt list with red checkout (before the actual checkout happens surely), I need to compare some deep dependencies installed in the runner image but I was unable to catch the problem myself.

I will add it and send here once I get the logs from the broken build as well, should not take too long

Artifact from a working build:
installed-packages.zip

yardensade · 2024-04-07T09:20:35Z

Artifact from broken build:
installed-packages (6).zip
@mikhailkoliada

The contents seem to be identical actually 🤔

mnylensc · 2024-04-08T07:56:37Z

Happens also with actions/checkout@v4 using ubuntu-latest runner.

mnylensc · 2024-04-08T11:00:52Z

Maybe related, but also managed to get invalid ELF header error from Python installed using actions/setup-python@v5 running on ubuntu-latest:

/opt/hostedtoolcache/Python/3.10.14/x64/bin/pip cache dir
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 69, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 5, in <module>
    import subprocess
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/subprocess.py", line 75, in <module>
    import select
ImportError: /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so: invalid ELF header

VinceCarollo · 2024-04-08T16:09:45Z

FYI: We're currently unable to deploy due to this.

erik-bershel · 2024-04-09T08:03:25Z

Heads up! @mnylensc @yardensade @chris-griffin @jmuleiro @Topherhindman
May I ask you to check your actions today. New images were rolled out couple hours ago.

yardensade · 2024-04-09T08:04:22Z

So far seems good to me, I will keep monitoring it in the next hours to make sure it is stable. (I moved to ubuntu 22 tho, which had a docker error but since your new version seems to be solved)

mnylensc · 2024-04-09T09:46:30Z

@erik-bershel thank you. Will monitor if this happens. If possible, is there an explanation of what could've caused this, if you think it's now fixed?

erik-bershel · 2024-04-12T09:47:51Z

Heads up!
Seems that the deployment resolved the issue - we see no new cases during last days.

Unfortunately, we cannot provide details of what happened at this time. If and when such an opportunity arises, we will publish this information here with a mention of those interested. We are now focused on developing tests that will allow us to catch similar problems at an earlier stage in the future.

yardensade added bug report needs triage labels Apr 4, 2024

shamil-mubarakshin closed this as completed Apr 4, 2024

Alexey-Ayupov added OS: Ubuntu and removed needs triage labels Apr 4, 2024

mikhailkoliada self-assigned this Apr 5, 2024

mikhailkoliada reopened this Apr 5, 2024

actions deleted a comment from acaudle10 Apr 6, 2024

actions deleted a comment from acaudle10 Apr 7, 2024

erik-bershel added bug Something isn't working awaiting-deployment Code complete; awaiting deployment and/or deployment in progress labels Apr 8, 2024

erik-bershel mentioned this issue Apr 9, 2024

Damageddocker-proxy #9654

Closed

13 tasks

actions deleted a comment from acaudle10 Apr 9, 2024

tonistiigi mentioned this issue Apr 9, 2024

Action sporadically fails with exec /usr/bin/buildctl: exec format error docker/setup-buildx-action#313

Closed

3 tasks

erik-bershel closed this as completed Apr 12, 2024

cory-miller mentioned this issue May 17, 2024

Actions fails to run on public runner: /lib/x86_64-linux-gnu/libgssapi_krb5.so.2: invalid ELF header actions/checkout#1673

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Github Action checkout v3 fails on fetching - git error - The process '/usr/bin/git' failed with exit code 128 #9632

Github Action checkout v3 fails on fetching - git error - The process '/usr/bin/git' failed with exit code 128 #9632

yardensade commented Apr 4, 2024

shamil-mubarakshin commented Apr 4, 2024

yardensade commented Apr 4, 2024

chris-griffin commented Apr 4, 2024

jmuleiro commented Apr 5, 2024

Topherhindman commented Apr 5, 2024

mikhailkoliada commented Apr 5, 2024 •

edited

mikhailkoliada commented Apr 5, 2024

yardensade commented Apr 6, 2024 •

edited

mikhailkoliada commented Apr 6, 2024

yardensade commented Apr 7, 2024 •

edited

yardensade commented Apr 7, 2024 •

edited

mnylensc commented Apr 8, 2024 •

edited

mnylensc commented Apr 8, 2024

VinceCarollo commented Apr 8, 2024

erik-bershel commented Apr 9, 2024

yardensade commented Apr 9, 2024

mnylensc commented Apr 9, 2024

erik-bershel commented Apr 12, 2024

Github Action checkout v3 fails on fetching - git error - The process '/usr/bin/git' failed with exit code 128 #9632

Github Action checkout v3 fails on fetching - git error - The process '/usr/bin/git' failed with exit code 128 #9632

Comments

yardensade commented Apr 4, 2024

Description

Platforms affected

Runner images affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

shamil-mubarakshin commented Apr 4, 2024

yardensade commented Apr 4, 2024

chris-griffin commented Apr 4, 2024

jmuleiro commented Apr 5, 2024

Topherhindman commented Apr 5, 2024

mikhailkoliada commented Apr 5, 2024 • edited

mikhailkoliada commented Apr 5, 2024

yardensade commented Apr 6, 2024 • edited

mikhailkoliada commented Apr 6, 2024

yardensade commented Apr 7, 2024 • edited

yardensade commented Apr 7, 2024 • edited

mnylensc commented Apr 8, 2024 • edited

mnylensc commented Apr 8, 2024

VinceCarollo commented Apr 8, 2024

erik-bershel commented Apr 9, 2024

yardensade commented Apr 9, 2024

mnylensc commented Apr 9, 2024

erik-bershel commented Apr 12, 2024

mikhailkoliada commented Apr 5, 2024 •

edited

yardensade commented Apr 6, 2024 •

edited

yardensade commented Apr 7, 2024 •

edited

yardensade commented Apr 7, 2024 •

edited

mnylensc commented Apr 8, 2024 •

edited