v1.65.0: action fails with exit code 141 pipefail #269

baumac · 2023-05-18T18:08:20Z

Summary

We updated the action from v.1.62.0 -> v1.65.0 and now the action is failing ~50% of the time with Docker Action run completed with exit code 141 which we believe is the result of a pipe error occurring on line 85 of entrypoint.sh.

Debug Logs

##[debug]Evaluating: needs.bump-version.outputs.new-version
##[debug]Evaluating Index:
##[debug]..Evaluating Index:
##[debug]....Evaluating Index:
##[debug]......Evaluating needs:
##[debug]......=> Object
##[debug]......Evaluating String:
##[debug]......=> 'bump-version'
##[debug]....=> Object
##[debug]....Evaluating String:
##[debug]....=> 'outputs'
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'new-version'
##[debug]=> '0.0.416'
##[debug]Result: '0.0.416'
##[debug]Evaluating: secrets.GH_TOKEN
##[debug]Evaluating Index:
##[debug]..Evaluating secrets:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'GH_TOKEN'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Evaluating condition for step: 'Create GH Tag'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Create GH Tag
##[debug]Loading inputs
##[debug]Loading env
Run anothrNick/github-tag-action@1.65.0
  env:
    DOCKER_BASE_IMAGE_NAME: baumac/test
    DOCKER_OCI_ARCHIVE: docker-archive
    GITHUB_TOKEN: ***
    CUSTOM_TAG: 0.0.416
    VERBOSE: true
/usr/bin/docker run --name ed866e737379fcd0fc48c5941d83db8a81dea6_9f19db --label ed866e --workdir /github/workspace --rm -e "DOCKER_BASE_IMAGE_NAME" -e "DOCKER_OCI_ARCHIVE" -e "GITHUB_TOKEN" -e "CUSTOM_TAG" -e "VERBOSE" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_DEBUG" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/baumac/test":"/github/workspace" ed866e:737379fcd0fc48c5941d83db8a81dea6
++ git rev-parse --abbrev-ref HEAD
*** CONFIGURATION ***
	DEFAULT_BUMP: minor
	DEFAULT_BRANCH: 
	WITH_V: false
	RELEASE_BRANCHES: master,main
	CUSTOM_TAG: 0.0.416
	SOURCE: .
	DRY_RUN: false
	INITIAL_VERSION: 0.0.0
	TAG_CONTEXT: repo
	PRERELEASE: false
	PRERELEASE_SUFFIX: beta
	VERBOSE: true
	MAJOR_STRING_TOKEN: #major
	MINOR_STRING_TOKEN: #minor
	PATCH_STRING_TOKEN: #patch
	NONE_STRING_TOKEN: #none
	BRANCH_HISTORY: compare
pre_release = false
+ current_branch=main
+ pre_release=false
+ IFS=,
+ read -ra branch
+ for b in "${branch[@]}"
+ [[ main == \m\a\s\t\e\r ]]
+ '[' master '!=' master ']'
+ for b in "${branch[@]}"
+ [[ main == \m\a\i\n ]]
+ pre_release=false
+ '[' main '!=' main ']'
+ echo 'pre_release = false'
+ git fetch --tags
From https://github.com/baumac/test
   e51e64f..bde24f1  main                    -> origin/main
 * [new branch]      prod-us                 -> origin/main-deploy
 * [new tag]         0.0.4                   -> 0.0.4
 * [new tag]         0.0.414                 -> 0.0.414
+ tagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+$'
+ preTagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+(-beta\.[0-9]+)$'
+ case "$tag_context" in
++ head -n 1
++ grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$'
++ git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)'
++ true
+ tag=0.0.414
##[debug]Docker Action run completed with exit code 141
##[debug]Finishing: Create GH Tag

The text was updated successfully, but these errors were encountered:

sbe-arg · 2023-05-19T02:08:25Z

whats the result of this?

preTagFmt="^v?[0-9]+\.[0-9]+\.[0-9]+(-$suffix\.[0-9]+)$"
git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)' | (grep -E "$preTagFmt" || true) | head -n 1
echo $?

also can you share more context of how your workflow operates? I see you run based on customtags instead of auto discovery

also what do you mean that fails 50% of the time?

sbe-arg · 2023-05-19T02:11:09Z

have a look at this https://unix.stackexchange.com/questions/580117/debugging-sporadic-141-shell-script-errors regarding head command and number of filters... I see u have 400 patch tags do they all exist in the repo history?

sbe-arg · 2023-05-19T02:12:27Z

To me sounds the head command is signaling 141 after line 84 not necessarily after line 85 (85 does not even start)

sbe-arg · 2023-05-19T02:13:40Z

if this is the case we need to figure out a way to handle SIGPIPE as feature request (in between clearup some tags) or remain in 1.62 or lower for now.

Interesting scenario

baumac · 2023-05-19T08:25:38Z

whats the result of this?

preTagFmt="^v?[0-9]+\.[0-9]+\.[0-9]+(-$suffix\.[0-9]+)$"
git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)' | (grep -E "$preTagFmt" || true) | head -n 1
echo $?

When ran locally the result of this is 0. I'd imagine that it's not zero when the pipeline fails, but I can't confirm this. I'll fork this repo see if I can capture the output of this today in the pipeline.

also can you share more context of how your workflow operates? I see you run based on customtags instead of auto discovery

Yup in short our workflow:

Checks out the source code
Creates the new version and stores it in ${{ bump-version.outputs.new-version }} (uses actions-ecosystem/action-bump-semver@v1.0.0)
Creates and pushes a docker image using the new version
Creates the git tag using the new version (anothrNick/github-tag-action@1.65.0)
Pushes the new git tag

also what do you mean that fails 50% of the time?

The github-tag-action is flakey. Approximately half the time the action will succeed, and the other half of the time it will fail.

I see u have 400 patch tags do they all exist in the repo history?

Yup they all exist in the repo history, the real output of git fetch --tags is all 400 tags in the semvar format. For readability I removed them from the job output.

baumac · 2023-05-19T08:28:08Z

@sbe-arg thanks for taking a look at this so quickly, it's very much appreciated 🙇

sbe-arg · 2023-05-19T10:24:28Z

Ill try to debug further next week, need to simulate 500 tags or so to get a realistic scenario XD

baumac · 2023-05-19T12:25:10Z

@sbe-arg I setup a repo with 998 tags to repro the error. Note, I also forked the action, updated it to not exit immediately on piperrors, and to log the pipe exit status for the command causing the error.

Test repo with 998 tags: https://github.com/baumac/actions-test-repo
github-tag-action fork: https://github.com/baumac/github-tag-action/tree/1.65.1

Here's a GH actions run that produces the Pipefailure in the logs: https://github.com/baumac/actions-test-repo/actions/runs/5024209249/jobs/9009683359

Relevant snippet:

tagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+$'
+ preTagFmt='^v?[0-9]+\.[0-9]+\.[0-9]+(-beta\.[0-9]+)$'
+ git for-each-ref --sort=-v:refname --format '%(refname:lstrip=2)'
+ head -n 1
+ grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$'
+ true
+ printf '%s\n' 141 141 0 0

sbe-arg · 2023-05-19T22:52:54Z

Maybe add a sleep between tag and pretag greps to make sure is happening in tag.

But we can probably do a $? 141 || true as dirty fix to ignore sigpipe

sbe-arg · 2023-05-20T00:39:43Z

Another simple way to fix this is split the tag pretag one lines into 2 functions

Where
1- get tags into a var
2- once tags are in var, grep
3- once grep finish, head

Because pipe with grep is making the git command to still piping info when grep already finishes with 0 so the pipe returns a 141

baumac · 2023-05-20T15:37:09Z

Created #270 which implements your suggested fix of using vars.

Was able to test it out, and everything appears to be working 😄

baumac changed the title ~~Action fails with exit code 141 pipefail for v1.65.0~~ Action fails with exit code 141 pipefail after upgrading to v1.65.0 May 18, 2023

baumac changed the title ~~Action fails with exit code 141 pipefail after upgrading to v1.65.0~~ v1.65.0: action fails with exit code 141 pipefail May 18, 2023

sbe-arg self-assigned this May 19, 2023

baumac mentioned this issue May 20, 2023

Fix pipefail error caused by piping output to head #270

Merged

sbe-arg closed this as completed May 22, 2023

zvecr mentioned this issue May 26, 2023

Bump anothrNick/github-tag-action from 1.62.0 to 1.66.0 qmk/qmk_firmware#21050

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.65.0: action fails with exit code 141 pipefail #269

v1.65.0: action fails with exit code 141 pipefail #269

baumac commented May 18, 2023 •

edited

sbe-arg commented May 19, 2023

sbe-arg commented May 19, 2023

sbe-arg commented May 19, 2023

sbe-arg commented May 19, 2023

baumac commented May 19, 2023 •

edited

baumac commented May 19, 2023

sbe-arg commented May 19, 2023

baumac commented May 19, 2023 •

edited

sbe-arg commented May 19, 2023

sbe-arg commented May 20, 2023

baumac commented May 20, 2023

v1.65.0: action fails with exit code 141 pipefail #269

v1.65.0: action fails with exit code 141 pipefail #269

Comments

baumac commented May 18, 2023 • edited

Summary

Debug Logs

sbe-arg commented May 19, 2023

sbe-arg commented May 19, 2023

sbe-arg commented May 19, 2023

sbe-arg commented May 19, 2023

baumac commented May 19, 2023 • edited

baumac commented May 19, 2023

sbe-arg commented May 19, 2023

baumac commented May 19, 2023 • edited

sbe-arg commented May 19, 2023

sbe-arg commented May 20, 2023

baumac commented May 20, 2023

baumac commented May 18, 2023 •

edited

baumac commented May 19, 2023 •

edited

baumac commented May 19, 2023 •

edited