[1.6.2] Fails with exit code 22

[ldiqual]

Orb version

1.6.2

What happened

Right after upgrading to 1.6.2, queue step failed with the following output (staging is the branch in question):

staging queueable

Exited with code exit status 22
CircleCI received exit code 22

Config:

- queue/until_front_of_line:
          time: '30'
          circleci-api-key: CIRCLE_TOKEN

Expected behavior

Queue step should not fail

Full logs

#!/bin/bash -eo pipefail

load_variables(){
  # just confirm our required variables are present
  : ${CIRCLE_BUILD_NUM:?"Required Env Variable not found!"}
  : ${CIRCLE_PROJECT_USERNAME:?"Required Env Variable not found!"}
  : ${CIRCLE_PROJECT_REPONAME:?"Required Env Variable not found!"}
  : ${CIRCLE_REPOSITORY_URL:?"Required Env Variable not found!"}
  : ${CIRCLE_JOB:?"Required Env Variable not found!"}
  # Only needed for private projects
  if [ -z "${CIRCLE_TOKEN}" ]; then
    echo "CIRCLE_TOKEN not set. Private projects will be inaccessible."
  else
    fetch "https://circleci.com/api/v2/me" "/tmp/me.cci"
    me=$(jq -e '.id' /tmp/me.cci)
    echo "Using API key for user: ${me}"
  fi
  VCS_TYPE="github"
}

fetch(){
  url=$1
  target=$2
  http_response=$(curl -f -s -X GET -H "Circle-Token:${CIRCLECI_API_KEY}" -o "${target}" -w "%{http_code}" "${url}")
  if [ $http_response != "200" ]; then
      echo "ERROR: Server returned error code: $http_response"
      cat ${target}
  else
      echo "DEBUG: API Success"
  fi
}

fetch_filtered_active_builds(){
  if [ "true" != "true" ];then
    echo "Orb parameter 'consider-branch' is false, will block previous builds on any branch." 
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
  elif [ -n "${CIRCLE_TAG:x}" ] && [ "" != "" ]; then
    # I'm not sure why this is here, seems identical to above?
    echo "CIRCLE_TAG and orb parameter tag-pattern is set, fetch active builds"
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}?filter=running"
  else
    : ${CIRCLE_BRANCH:?"Required Env Variable not found!"}
    echo "Only blocking execution if running previous jobs on branch: ${CIRCLE_BRANCH}"
    jobs_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/tree/${CIRCLE_BRANCH}?filter=running"
  fi

  if [ ! -z $TESTING_MOCK_RESPONSE ] && [ -f $TESTING_MOCK_RESPONSE ];then
    echo "Using test mock response"
    cat $TESTING_MOCK_RESPONSE > /tmp/jobstatus.json
  else
    echo "Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLE_TOKEN  is set."
    fetch "$jobs_api_url_template" "/tmp/jobstatus.json"
    if [ -n "${CIRCLE_TAG:x}" ] && [ "" != "" ]; then
      jq '[ .[] | select(.vcs_tag | . != null) | select(.vcs_tag | test("") ) ]' /tmp/jobstatus.json > /tmp/jobstatus_tag.json
      mv /tmp/jobstatus_tag.json /tmp/jobstatus.json
    fi
    echo "API access successful"
  fi
}

fetch_active_workflows(){
  cp /tmp/jobstatus.json /tmp/augmented_jobstatus.json
  for workflow in `jq -r ".[] | .workflows.workflow_id //empty" /tmp/augmented_jobstatus.json | uniq`
  do
    echo "Checking time of workflow: ${workflow}"
    workflow_file=/tmp/workflow-${workflow}.json
    if [ ! -z $TESTING_MOCK_WORKFLOW_RESPONSES ] && [ -f $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json ]; then
      echo "Using test mock workflow response"
      cat $TESTING_MOCK_WORKFLOW_RESPONSES/${workflow}.json > ${workflow_file}
    else
      fetch "https://circleci.com/api/v2/workflow/${workflow}" "${workflow_file}"
    fi
    created_at=`jq -r '.created_at' ${workflow_file}`
    echo "Workflow was created at: ${created_at}"
    cat /tmp/augmented_jobstatus.json | jq --arg created_at "${created_at}" --arg workflow "${workflow}" '(.[] | select(.workflows.workflow_id == $workflow) | .workflows) |= . + {created_at:$created_at}' > /tmp/augmented_jobstatus-${workflow}.json
    #DEBUG echo "new augmented_jobstatus:"
    #DEBUG cat /tmp/augmented_jobstatus-${workflow}.json
    mv /tmp/augmented_jobstatus-${workflow}.json /tmp/augmented_jobstatus.json
  done
}

update_comparables(){     
  fetch_filtered_active_builds

  fetch_active_workflows

  load_current_workflow_values

  # falsey parameters are empty strings, so always compare against 'true' 
  if [ "false" = "true" ] ;then
    echo "Orb parameter block-workflow is true."
    echo "This job will block until no previous workflows have *any* jobs running."
    oldest_running_build_num=`jq 'sort_by(.workflows.created_at)| .[0].build_num' /tmp/augmented_jobstatus.json`
    oldest_commit_time=`jq 'sort_by(.workflows.created_at)| .[0].workflows.created_at' /tmp/augmented_jobstatus.json`
  else
    echo "Orb parameter block-workflow is false."
    echo "Only blocking execution if running previous jobs matching this job: ${CIRCLE_JOB}"
    oldest_running_build_num=`jq ". | map(select(.workflows.job_name==\"${CIRCLE_JOB}\")) | sort_by(.workflows.created_at)|  .[0].build_num" /tmp/augmented_jobstatus.json`
    oldest_commit_time=`jq ". | map(select(.workflows.job_name==\"${CIRCLE_JOB}\")) | sort_by(.workflows.created_at)|  .[0].workflows.created_at" /tmp/augmented_jobstatus.json`
  fi
  if [ -z "$oldest_commit_time" ]; then
    echo "API Error - unable to load previous job timings. Report to developer."
    exit 1
  fi
  echo "Oldest job: $oldest_running_build_num"
  if [ -z $oldest_commit_time ];then
    echo "API Call for existing jobs failed, failing this build.  Please check API token"
    echo "All running jobs:"
    cat /tmp/jobstatus.json || exit 0
    echo "All running jobs with created_at:"
    cat /tmp/augmented_jobstatus.json || exit 0
    echo "All worfklow details."
    cat /tmp/workflow-*.json
    exit 1
  fi
}

load_current_workflow_values(){
   my_commit_time=`jq '.[] | select( .build_num == '"${CIRCLE_BUILD_NUM}"').workflows.created_at' /tmp/augmented_jobstatus.json`
}

cancel_current_build(){
  echo "Cancelleing build ${CIRCLE_BUILD_NUM}"
  cancel_api_url_template="https://circleci.com/api/v1.1/project/${VCS_TYPE}/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/${CIRCLE_BUILD_NUM}/cancel?circle-token=${CIRCLE_TOKEN}"
  curl -s -X POST $cancel_api_url_template > /dev/null
}



#
# We can skip a few use cases without calling API
#
if [ ! -z "$CIRCLE_PR_REPONAME" ]; then
  echo "Queueing on forks is not supported. Skipping queue..."
  # It's important that we not fail here because it could cause issues on the main repo's branch
  exit 0
fi
if [ "*" = "*" ] || [ "*" = "${CIRCLE_BRANCH}" ]; then
  echo "${CIRCLE_BRANCH} queueable"
else
  echo "Queueing only happens on * branch, skipping queue"
  exit 0
fi

#
# Set values that wont change while we wait
# 
load_variables
max_time=30
echo "This build will block until all previous builds complete."
echo "Max Queue Time: ${max_time} minutes."
wait_time=0
loop_time=11
max_time_seconds=$((max_time * 60))

#
# Queue Loop
#
confidence=0
while true; do
  update_comparables
  echo "This Workflow Timestamp: $my_commit_time"
  echo "Oldest Workflow Timestamp: $oldest_commit_time"
  if [[ ! -z "$my_commit_time" ]] && [[ "$oldest_commit_time" > "$my_commit_time" || "$oldest_commit_time" = "$my_commit_time" ]] ; then
    # API returns Y-M-D HH:MM (with 24 hour clock) so alphabetical string compare is accurate to timestamp compare as well
    # recent-jobs API does not include pending, so it is posisble we queried in between a workfow transition, and we;re NOT really front of line.
    if [ $confidence -lt 1 ];then
      # To grow confidence, we check again with a delay.
      confidence=$((confidence+1))
      echo "API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API."
      echo "Rerunning check ${confidence}/1"
    else
      echo "Front of the line, WooHoo!, Build continuing"
      break
    fi
  else
    # If we fail, reset confidence
    confidence=0
    echo "This build (${CIRCLE_BUILD_NUM}) is queued, waiting for build number (${oldest_running_build_num}) to complete."
    echo "Total Queue time: ${wait_time} seconds."
  fi

  if [ $wait_time -ge $max_time_seconds ]; then
    echo "Max wait time exceeded, considering response."
    if [ "false" == "true" ];then
      echo "Orb parameter dont-quit is set to true, letting this job proceed!"
      exit 0
    else
      cancel_current_build
      sleep 10 # wait for API to cancel this job, rather than showing as failure
      exit 1 # but just in case, fail job
    fi
  fi

  sleep $loop_time
  wait_time=$(( loop_time + wait_time ))
done

staging queueable

Exited with code exit status 22
CircleCI received exit code 22

[karlparry]

Also having this issue, pinned our version back to 1.6.1 for now

[eddiewebb]

Thanks for report @karlparry, @ldiqual

Can you share what image you're using? Based on the lack of any output besides "Branch Queueable" , It is surely the introduction of the API wrapper/token check fetch which happens next, but I'm not sure what since this works locally and my images, but version of curl or shell might matter.

        fetch(){
          url=$1
          target=$2
          http_response=$(curl -f -s -X GET -H "Circle-Token:${CIRCLECI_API_KEY}" -o "${target}" -w "%{http_code}" "${url}")
          if [ $http_response != "200" ]; then
              echo "ERROR: Server returned error code: $http_response"
              cat ${target}
          else
              echo "DEBUG: API Success"
          fi
        }

[eddiewebb]

Hmm, when I copy and paste your debug into a container I sshed into I get reasonable output, so it does make me think curl/shell compatibility.

main queueable
This build will block until all previous builds complete.
Only blocking execution if running previous jobs on branch: main
Attempting to access CircleCI api. If the build process fails after this step, ensure your CIRCLE_TOKEN  is set.
DEBUG: API Success
API access successful
Checking time of workflow: bd272146-1517-4026-8126-d1e32786b929
DEBUG: API Success
Workflow was created at: 2021-11-11T00:08:08Z
Orb parameter block-workflow is false.
Only blocking execution if running previous jobs matching this job: build-auditor/check_context_usage
Oldest job: 2614
This Workflow Timestamp: "2021-11-11T00:08:08Z"
Oldest Workflow Timestamp: "2021-11-11T00:08:08Z"
API shows no previous jobs/workflows, but it is possible a previous workflow has pending jobs not yet visible in API.
Rerunning check 1/1

[ldiqual]

Image is cimg/node:16.13.0 fyi.

[eddiewebb]

Ahh I think I see, I assume you're both overriding the default token name? My new method was old merge that missed the switch to parameter. eddiewebb/queue:73 has "fix" if you'd liek to validate

[ldiqual]

Confirmed that this fixed the issue we had with 1.6.2. Thank you so much!

[eddiewebb]

Thank all. Fixed version is released as 1.6.3