Metadata 'apply' failing only in Heroku CI/CD - error in converting sdl to metadata

[mikepaszkiewicz]

I'm attempting to simply use the Hasura CLI during the build process to apply the metadata to my Hasura Cloud instance, e.g:

hasura metadata apply --admin-secret=XXX --endpoint=https://XXX.hasura.app --project=hasura --skip-update-check --log-level DEBUG

This is working perfectly fine on my machine, but failing on Heroku (right after the message about cli-ext plugin being installed:

time="2021-01-13T22:34:11Z" 
level=fatal 
msg="failed to apply metadata: 
   cannot apply metadata on the database: 
       cannot build actions from metadata: 
           error in converting sdl to metadata: exit status 4: "

I have read through all issues related to sdl/CI/metadata/CLI, made sure to set HASURA_GRAPHQL_CLI_ENVIRONMENT=default, etc. but still can't get it to work. I was also unable to find any issues that referenced Exit Status 4, nor was I able to trace it down in the CLI source code.

Any ideas? Thanks in advance!

[domsim1]

I had the same error. I fixed it by ensuring the Hasura CLI version was the same as the Hasura GraphQL Engine.
In my case, Hasura was version 1.3.0, but Hasura CLI was version 1.3.3. Changed Hasura CLI to version 1.3.0, and it worked.

[mikepaszkiewicz]

@domsim1 Thanks for the response!

I'm currently on Hasura's paid cloud-hosted offering - and when I run the apply in Heroku's build step, it does indeed compare CLI version vs. server - showing a difference:

"msg":"versions: cli: [v1.3.3] server: [v1.3.3-cloud.3]",

Although the versions are different, I get the same version difference when I run the exact command locally, where the apply works fine. There's also no CLI version v1.3.3-cloud.3 - so that would be off the table...

Still stumped here - anything else you'd recommend trying?

[mikepaszkiewicz]

So, after another day, have made some discoveries:

• I can run the script fine inside the Heroku instance's shell (hasura-apply.sh below). This is super strange to me..
• The 'exit code 4' is also related to other issues around cli-ext / permissions related stuff
• Made sure NODE_OPTIONS is not set (Bug when exporting metadata through the CLI #4953)
• the rm -rf ~/.hasura workaround (cli: v1.3.3 cannot fetch plugin manfiest cli-ext error happens intermittently #6219 (comment))
• downgrading to 1.3.2 (since i'm on Hasura Cloud - which is on 1.3.3, this shouldn't work anyways since different versions are incompatible AFAIK)

My best guess: it's related to permissions or context differences in the build process compared to after deploy.

Ideally would like to fix the issue in the context of the Heroku build process, vs settings up Github actions, Dockerizing, etc

---

Details:

• node: 14.15.4
• heroku stack: same on 18 and 20

package.json:

  "scripts": {
    "build": "sh hasura-apply.sh && DEBUG=1 graphql-codegen --config codegen.yml && webpack --config=configs/webpack/prod.js",

  },

hasura-apply.sh (previously had it installed as a devDependency, but opted to switch to global for debugging purposes)

yarn global add hasura-cli@1.3.3 --prefix /app && \
/app/bin/hasura plugins install cli-ext --version=v1.3.3 && \
/app/bin/hasura plugins list && \
rm -rf ~/.hasura && \
/app/bin/hasura metadata apply  \
    --admin-secret=XXXX \
    --endpoint=https://XXXX.hasura.app/ \
    --project=hasura --skip-update-check --log-level DEBUG

[mikepaszkiewicz]

Solved! Heroku's build process actually sets NODE_OPTIONS environment variable during the build process - which messes with the cli-ext plugin (see #4953). Setting it to empty fixed the issue!

Full working script:

export NODE_OPTIONS= &&
yarn global add hasura-cli@1.3.3 --prefix /app && \
/app/bin/hasura plugins install cli-ext --version=v1.3.3 && \
/app/bin/hasura plugins list && \
rm -rf ~/.hasura && \
/app/bin/hasura metadata apply  \
    --admin-secret=XXXX \
    --endpoint=https://XXXX.hasura.app/ \
    --project=hasura --skip-update-check --log-level DEBUG

Huge thanks to @scriptonist for the support - keep up the good work!!