Ignore build step ignores content changes too

[nickluger]

It is recommended to set the ignore-build-step-configuration for monorepos to something like git diff HEAD^ HEAD --quiet frontend/ to run only on changes, that affect the directory frontend.

Similar provisions are necessary for some CI/CD pipelines too.

Now if the latest commit, say for example "API change xyz..." was in the folder /api, the build is skipped. If we now have a content update in our CMS or API-Backend and trigger a SSG-rebuild at Vercel via deploy hook, this deployment is skipped, because of the last code update being in another part of the monorepo and all subsequent deploy hook calls (content changes) will be skipped until a new commit affects frontend again.

This can't be intentional, can it?

[nickluger]

Or could there be maybe a system environment variable that one could use to detect in the ignore-script whether this was triggered by a certain deployment hook and then always build, ignoring the type of the latest code changes?

[bryanrsmith]

Same issue here. I expected a deploy hook to skip the check, and am now having issues with stale content when CMS changes fail to trigger a deployment in my monorepo.

If the existing behavior is desirable, maybe adding a parameter to the deploy hook endpoint would be another option?

[nickluger]

Yes, that would be the easiest solution.

[dimosmera]

Is there any update on this? Have you guys figured any workarounds?

[nickluger]

We're doing pretty weird stuff as a workaround:

We set a flag in Airtable via the Airtable API before deployment with our CI/CD system or our CMS and read the Airtable flag from Vercel during the deployment step. If the flag is set, then either the CI/CD system or the CMS have toggled it, and it's ok to deploy. If it's not set, then it's a regular trigger via GitHub push, and we block the deployment.

😅

[frankly-elevensiesed-out]

and Vercel doesn't have any interest in fixing this developer experience?
Looking like time to roll our own next server and cancel this service.

[webkreation]

We have the same problem with our monorepo.
A deployment hook should definitely trigger a deployment.
How can this be solved?

[webkreation]

I have contacted vercel support.
Today the support ticket was marked as done (unfortunately without feedback?).
I think the problem has been fixed.
Our deployment hooks are now working again.

[sreetamdas]

Is there a proposed solution to this? I'm not able to find any 😅

[samcx]

@sreetamdas There is no fix to this yet. We may need a System Environment Variable to detect whether a build was triggered by Git or a Deploy Hook.

[sreetamdas]

@samsisle yep, that would be ideal!

[srweeks]

@sreetamdas There is no fix to this yet. We may need a System Environment Variable to detect whether a build was triggered by Git or a Deploy Hook.

has there been any movement on this yet?

[samcx]

@srweeks Unfortunately, no significant movement yet on this.

[nickluger]

This seems to be still an issue, making Vercel + Monorepo and Vercel + CI/CD really difficult to use. I wonder why not more people run into this. Are there so few monorepo-fans? 😅 Also, I don't understand why so few people seem to be relying on their CI/CD pipeline to run tests before deploying even a preview.

Of course one could rely on the Vercel CLI only, but I find the GitHub deploy hooks very approachable, when called from CMS.

We have a different workaround now. We just spun up a small Redis DB at upstash.com, which is free and set/check the flag there, calling ./control-vercel-deployments.sh CHECK my-app in the ignore build step command.... what a hassle.😄

#!/bin/sh
# control-vercel-deployments.sh
if [ $# -lt 2 ]
  then
    echo "Not enough arguments provided. Provide \"ENABLE\" or \"CHECK\" commands and the Vercel project to control deployment."
    # Vercel considers exit code 0 as "no deployment necessary"
    # See here: https://vercel.com/docs/platform/projects#ignored-build-step
    exit 0
fi

case $1 in
  CHECK)
    echo "Checking if Vercel deployment for project \"$2\" is currently allowed."
    DEPLOYMENT_FLAG=$(curl -s "$UPSTASH_DATABASE_URL/get/$2" -H "Authorization: Bearer $UPSTASH_BEARER_TOKEN")
    echo "DEPLOYMENT_FLAG: $DEPLOYMENT_FLAG"
    if [ "$DEPLOYMENT_FLAG" = '{"result":null}' ]; then
      echo "DEPLOYMENT_FLAG is not set. Deployment not allowed."
      exit 0
    elif [ "$DEPLOYMENT_FLAG" = '{"result":"true"}' ]; then
      echo "DEPLOYMENT_FLAG is set. Deployment allowed. Deleting flag."
      DELETION_RESPONSE=$(curl -s "$UPSTASH_DATABASE_URL/del/$2" -H "Authorization: Bearer $UPSTASH_BEARER_TOKEN")
      if [ "$DELETION_RESPONSE" = '{"result":1}' ]; then
            echo "Flag successfully deleted after check."
      fi
      # Vercel considers exit code 1 as "deployment needed"
      exit 1
    else
      echo "DEPLOYMENT_FLAG contains invalid value. Deleting it and aborting."
      curl -s "$UPSTASH_DATABASE_URL/del/$2" -H "Authorization: Bearer $UPSTASH_BEARER_TOKEN"
      exit 0
    fi
    ;;

  ENABLE)
    echo "Enabling Vercel deployment for project \"$2\" by setting DEPLOYMENT_FLAG."
    RESPONSE=$(curl -s "$UPSTASH_DATABASE_URL/set/$2/true" -H "Authorization: Bearer $UPSTASH_BEARER_TOKEN")
    echo "RESPONSE: $RESPONSE"
    exit 0
    ;;

  *)
    echo "First argument must be CHECK or ENABLE."
    exit 0
    ;;
esac

[bryanrsmith]

It is really painful. I suspect we all have hacky workarounds like this. I'm currently using scripts that commit changes to dummy files in my monorepo just to trigger vercel deployments, but I've been through several iterations of monorepo-related workarounds as vercel functionality has evolved.

[criess]

Basically its just not only monorepo but everything that is asynchrnous builds were build hook is issued from a concurrent system. We build some of our assets already on CI. Failing builds is not an option, skipping builds non-deploy-hook triggered would be …

[robertschneiderman]

This solution works well for us! You just need a Vercel API access token.

// vercel-ignore-build.js
const https = require("https");
 
let vercelEnv = process.env.VERCEL_ENV;

const options = {
  hostname: 'api.vercel.com',
  port: 443,
  path: '/v5/now/deployments?limit=5&teamId=[YOUR_TEAM_ID]',
  method: 'GET',
  headers: {
    'Authorization': `Bearer ${process.env.VERCEL_API_ACCESS_TOKEN}`
  }
}
let data = '';

const req = https.request(options, res => {
  res.on('data', d => {
    data += d.toString();
  })
  res.on('end', d => {
    let parsedData = JSON.parse(data);
    let prodRunningFromDeployHook;

    try {
      prodRunningFromDeployHook = parsedData.deployments.find(({state, meta, target}) => state === 'BUILDING' && target === 'production' && meta.deployHookId === '[YOUR_DEPLOY_HOOK_ID]')
    } catch(e) {
      console.log('e: ', e);
      process.exit(0);
    }
    
    if (vercelEnv === 'production' && !prodRunningFromDeployHook) {
      console.log('🛑 - Build cancelled');
      process.exit(0);
    } else {
      console.log('✅ - Build can proceed');
      process.exit(1);
    }
  })
})

req.on('error', error => {
  console.error(error)
})

req.end()

Alternatively you could use deployHookName instead of deployHookId if you're going to be revoking the hook and don't want to update the script.

[thomas391]

thank you sir for sharing this, saved me a ton today!

tip: for ppl using "node jsfile.js" in the vercel ignore deploy rather than sh file
process.env.ENV_NAME will use vercel's environment variables as your js file runs, i thought it was using the same naming with bash: &ENV_NAME

[dannyhw]

This was also really useful for me and I wanted to share some changes I made that made it work better for me.

I updated to use the deployements/{id_or_url} endpoint and I use vercel_url env to get the current build by url instead of using .find on the list of the last 5. Also added a check for changes to git in the relevant folder.

// vercel-ignore-build.js
const https = require("https");
const { exec } = require("child_process");

exec("git diff --quiet HEAD^ HEAD .", (error, stdout, stderr) => {
  if (error) {
      console.log(`detected changes in the project`);
      console.log("✅ - Build can proceed");
      process.exit(1);
  } else {
    console.log("no changes detected");
  }
});

let vercelEnv = process.env.VERCEL_ENV;

const options = {
  hostname: "api.vercel.com",
  port: 443,
  path: `/v13/deployments/${process.env.VERCEL_URL}?teamId=[YOUR_TEAM_ID]`,
  method: "GET",
  headers: {
    Authorization: `Bearer ${process.env.VERCEL_API_ACCESS_TOKEN}`,
  },
};

let data = "";

const req = https.request(options, (res) => {
  res.on("data", (d) => {
    data += d.toString();
  });
  res.on("end", (d) => {
    let parsedData = JSON.parse(data);
    let prodRunningFromDeployHook;

    prodRunningFromDeployHook =
      parsedData.target === "production" &&
      parsedData.meta.deployHookName === "Contentful Blog CMS"; // replace this with the name of your deploy hook

    console.log({
      target: parsedData.target,
      hook: parsedData.meta.deployHookName,
    });

    if (vercelEnv === "production" && !prodRunningFromDeployHook) {
      console.log("🛑 - Build cancelled");
      process.exit(0);
    } else {
      console.log("✅ - Build can proceed");
      process.exit(1);
    }
  });
});

req.on("error", (error) => {
  console.error(error);
});

req.end();
 

[benthejack-vuw]

thanks, I've used this to create smaller bash script that just detects if the current branch build is from a webhook or not (works on any branch with a webhook).

#any other branches that you have webhooks for can go here
currentBranch=$(echo $VERCEL_BRANCH_URL | grep -Eo '(dev|staging|main)')
if [ -z $currentBranch ]; then
    exit 0
fi

result=$(curl 'https://api.vercel.com:443/v6/deployments?limit=5&teamId=[YOUR_TEAM]' -H "Authorization: Bearer [ACCESS_TOKEN]")

echo "
    const items=$result.deployments;
    const buildTriggeredByHook=items.findIndex((itm) => (
        itm.state === 'BUILDING' && itm?.meta?.deployHookRef?.includes('$currentBranch')
    ));
    console.log(buildTriggeredByHook != -1);
" > node_run_script.js

buildTriggeredByHook=$(node node_run_script.js)

echo "this build was triggered by a hook: $buildTriggeredByHook"

if [ $buildTriggeredByHook = true ]; then
    exit 1
else
    exit 0
fi

[zachgoll]

There has got to be a simpler solution to this? This seems like a fairly common use case and would be solved with an environment variable such as VERCEL_TRIGGER_SOURCE = 'deploy-hook' | 'api' | 'console' (oversimplified, but something along those lines).

Going with the highlighted workaround for now, but would love to have a native ability to do this!

[ThomasMarcelo]

Any updates on this?
Suffering from the same. Used npx turbo-ignore website-frontend to skip unnecessary deployments (we are using Turborepo), but now my Strapi application wants to use a hook to deploy after changes on it, and it can't, because no file was changed on website-frontend itself, only on the CMS.

[amyegan]

I haven't heard any news about this yet, but I'm asking the team to see if they have anything to share!

[amyegan]

Sorry about the wait, Thomas. Because turbo-ignore has no way of knowing about CMS changes, it will always skip deployments when content is the only thing that changed.

To work around this, you could use Incremental Static Regeneration to pull in content changes from a CMS without requiring a full site redeploy. If you use Next.js or Web Output API, this is probably the best option. There is an ISR Quickstart guide available to walk you through the setup.

Someone else recently shared that they found success in a similar situation by adding ?buildCache=false to their deploy webhook to force the build and deployment.

[adelindanaila]

I couldn't find anything similar like ?buildCache=false

[amyegan]

@adelindanaila You can find more info here: https://vercel.com/docs/concepts/deployments/deploy-hooks#build-cache