NX_* Environment Variables not working on prod builds in our test/staging deployments from gitlab, but only sometimes

[KevinVandy]

Current Behavior

We use environment variables to point to different aws resources in our dev, test, and prod environments.

Locally, we use a .env file, and prefix all of the environment variables with NX_, like we are supposed to, and it all works fine.

In a Dev environment deployment that we set up that is built in a gitlab pipeline, and deployed to an S3 bucket, it also works fine with the env variables we set up in our gitlab CI/CD settings that are scoped to the dev environments.

However, in the exact same kind of pipeline for our Test (Staging) environment, it seems as if the NX build just simply won't recognize our environmental variables from our gitlab CI/CD settings that we have set there, but sometimes it does. We've tried adding the --skip-nx-cache to see if that was messing it up or not. Lately it seems like if we commit and push directly to our main branch, the build will include our env variables just fine, but if we make a merge request, and have a merge commit that results in the pipeline, none of our env variables get used and they are all undefined.

Here are our gitlab-ci steps for both of those pipelines. In the pipeline, we echo some of the variables that have been set as a sanity check to make sure gitlab itself is not misconfigured during the pipeline.

# store node_modules in a cache
cache:
  untracked: true
  key: '*******_portals_node_cache'
  paths:
    - node_modules/

# build apps dev
build_ap_dev:
  stage: build
  dependencies: []
  environment:
    name: ap_dev
    action: prepare
  only:
    - develop
    - /(^\d)*-feature-.*/
    - /(^\d)*-bugfix-.*/
  script:
    - echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
    - yarn build-ap --prod --skip-nx-cache
    - echo "Build Successful"
  artifacts:
    expire_in: 4 hour
    paths:
      - dist/apps/agent-portal-frontend

build_ap_test:
  stage: build
  dependencies: []
  environment:
    name: ap_test
    action: prepare
  only:
    - main
    - /(^\d)*-hotfix-.*/
  script:
    - echo "cognito user pool - $NX_COGNITO_USER_POOLS_ID" # successfully echos here, so gitlab variable not a problem
    - yarn build-ap --prod --skip-nx-cache
    - echo "Build Successful"
  artifacts:
    expire_in: 4 hour
    paths:
      - dist/apps/agent-portal-frontend

package.json build scripts are kept simple, the pipeline adds the flags we need above

Other things from googling around, we've tried making sure our environment.prod.ts is like this:

export const environment = {
  production: false,
};

Might this be related to some kind of Nx caching issue that gets stored in our node_modules cache for our pipelines?

Expected Behavior

Nx environment variables should work in all environments (dev, test, prod) when all of them have the same build command.

yarn build-appname --prod

Steps to Reproduce

1. Set up environment variables on gitlab CI/CD that are scoped to specific environments
2. Develop locally with a .env file in the root of the monorepo with variables that are prefixed with NX_*
3. Make Merge Request to develop branch and merge - DEV environment build works fine.
4. Make Merge Request from develop to main branch and merge - TEST environment build is missing env variables.
5. Panic and directly commit some console.logs of env variables on the main branch and run a new pipeline in TEST env - It sometimes magically works, but sometimes not.

Failure Logs

When we download the artifacts from the build to look to see if the variables got used by Nx, we don't see them in a grep, is this normal?

Environment

env for my local machine, but problem happens in our pipelines that run on a node:15.14 docker image

nx report

>  NX  Report complete - copy this into the issue template

  Node : 15.14.0
  OS   : linux x64
  yarn : 1.22.5
  
  nx : Not Found
  @nrwl/angular : Not Found
  @nrwl/cli : 12.6.4
  @nrwl/cypress : 12.6.4
  @nrwl/devkit : 12.6.4
  @nrwl/eslint-plugin-nx : 12.6.4
  @nrwl/express : Not Found
  @nrwl/jest : 12.6.4
  @nrwl/linter : 12.6.4
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : Not Found
  @nrwl/nx-cloud : Not Found
  @nrwl/react : 12.6.4
  @nrwl/schematics : Not Found
  @nrwl/tao : 12.6.4
  @nrwl/web : 12.6.4
  @nrwl/workspace : 12.6.4
  @nrwl/storybook : 12.7.2
  @nrwl/gatsby : Not Found
  typescript : 4.3.5

[marckassay]

I noticed updating from nx 12.6.3 to 12.8.0, dotenv got updated. And looking at the releases, it seems to be in your interest to at least do this update.

[KevinVandy]

update, we upgraded to the Nx 12.8 releases last week, and the first few deploys were successful. But now a few days later, with no package upgrades at all, we have had the exact same problem pop up again where we know that our gitlab env variables are being set correctly, but every other Nx build in our dev environment has undefined env vars

[sediatis]

Hi, any solution so far? Our team is having the same problem on production builds with env variables being undefined

[KevinVandy]

Hi, any solution so far? Our team is having the same problem on production builds with env variables being undefined

Yes, but it is a slow workaround. We added a line in our pipeline scripts to delete node_modules/.cache/nx so that is guaranteed to rerun the entire build step during the build step in our gitlab pipeline.

[StallionV]

Hi, can this be prioritized please? Currently environment variables are not being read and it is a big issue

[istvandesign]

+1

[crates]

I'm not certain if it's an identical issue, but it sure seems similar: My project, running Nx v13.0.2 presently, has zero problems whatsoever with environment issues when started or built normally. However, when I run my Cypress e2e tests, they all disappear completely. This makes very little sense to me, since I assumed the same Webpack config was being used to build both versions out... but when I check my variables while running the watcher used to execute Cypress tests, all I see are these:

{
  "NODE_ENV": "development",
  "NX_CLI_SET": "true",
  "NX_INVOKED_BY_RUNNER": "true",
  "NX_WORKSPACE_ROOT": "<dir>/parfait-nx",
  "NX_TASK_TARGET_PROJECT": "parfait-e2e",
  "NX_TASK_HASH": "<hash>"
}

There are around a dozen environment variables (usually read from my .env file) missing from that readout.

[nurbashanghai]

have u find any solution? Im also facing this issue right now

[crates]

@nurbashanghai - That depends. If you're having Kevin Vandy's problem, I can't help you. However, if you're having the same issue I was having (where the env vars don't populate for Cypress tests), you are in luck my friend!

I worked around this issue by manually bootstrapping the environment variables already available in my project, using the env-cmd NPM package. Here's the updated package.json command I use to launch my end-to-end tests:
env-cmd -f ./apps/myapp/.env nx e2e myapp-e2e

Eventually, I imagine the Nx team will find and fix this bug. The good news is that regardless of if/when/how they do that, this fix should remain working regardless.

[Graveheart]

We're having the same issue right now, using Azure pipelines. All environment variables set with a .env file are gone, it would be great if this gets addressed.

[rlenoir-codepoint]

We have a very similar issue @KevinVandy, the env variables are cached between builds when going through a complete Git flow (push branch for PR review -> merge to dev -> PR dev to main -> merge to main).

If we push directly from the local main branch from the CLI to the remote main, the values are properly set.

This have a big impact for us because we control the meta robots index/follow through env variables, where we allow some content pages indexation only on production. At the moment, every time we push to main after going through a full Git flow, we have to do an extra & empty "magic" push directly from the main local branch (described above) to get the good values...

We are using Vercel for the deployments and we thought it was an issue on their end with the builds cache, but it might be more complex than that, based on several discussions I could find (including this one):

• NX cache ignores environment variable value changes #4662
• Nx Cache Inputs should include NX_* env vars and some others #4455

We are using Nx 12.7.2, we did not try to upgrade to the latest yet

[Graveheart]

Just FYI, upgrading to the latest NX 13.3.1 didn't fix the issue. It happens to us when using Dockerfile in the pipeline but it doesn't happen when running nx build from the pipeline itself.

[Graveheart]

At last, I found a solution by using dotenv-webpack plugin and passing a .env file to it! I tried it with env-cmd too but it didn't help while running nx build

[crates]

@vsavkin ~ Any thoughts? It looks like env vars are broken for quite a few folks. Is this on the roadmap for a fix?

[KevinVandy]

Nx why have you forsaken us?

[kryptus36]

This is a major problem. I just recently started to use NX and now I'm wondering if I should back out. My prod builds are simply broken atm. It's kind of worst possible case.

[KevinVandy]

This is a major problem. I just recently started to use NX and now I'm wondering if I should back out. My prod builds are simply broken atm. It's kind of worst possible case.

We have a solution to be clear, it just makes the builds go a lot slower.

Yes, but it is a slow workaround. We added a line in our pipeline scripts to delete node_modules/.cache/nx so that is guaranteed to rerun the entire build step during the build step in our gitlab pipeline.

[kryptus36]

This is a major problem. I just recently started to use NX and now I'm wondering if I should back out. My prod builds are simply broken atm. It's kind of worst possible case.

We have a solution to be clear, it just makes the builds go a lot slower.

Yes, but it is a slow workaround. We added a line in our pipeline scripts to delete node_modules/.cache/nx so that is guaranteed to rerun the entire build step during the build step in our gitlab pipeline.

That didn't work for me. Oddly my nestjs app picks up my environment from Kubernetes, but react doesn't. My issue may be different. But I'm not sure what to do either way.

[niek-dev]

Same problem here deploying to Vercel. I have multiple CRA & Nextjs apps in the workspace.

[nichwch]

Can confirm that this is a problem for builds. I haven't tested it on Vercel, but running the built version of my app locally shows that the environment variables aren't being passed in.

[JamesGrom]

This is ridiculous, how has this not been resolved by now

[richtera]

I have not really had problems with environment variables, but found today that NX_INVOKED_BY_RUNNER doesn't seem to be set in some cases. I can't even find the code that sets it anymore. This breaks a whole bunch of our targets. Is this a known issue?

[Lonli-Lokli]

Have you all just investigate pure JavaScript files you have? As it's converted during compilation time.

[sohailykhan94]

I was able to get this working using runtimeCacheInputs something like:

"tasksRunnerOptions": {
    "default": {
      "runner": "@nrwl/workspace/tasks-runners/default",
      "options": {
        "cacheableOperations": [
          "build",
          "lint",
          "test",
          "e2e"
        ],
        "runtimeCacheInputs": [
          "echo $NX_ENV_VARIABLE",
          "echo $NX_ENV_VARIABLE_1",
          "echo $NX_ENV_VARIABLE_2",
          "echo $NX_ENV_VARIABLE_3"
        ]
      }
    }
  },

Maybe this might help someone else

[hoevelmanns]

+1

[tanint]

I try to use Runtime Hash Inputs by watching some ENV changed and NX still not invalidate cache. But if I use

  "targetDefaults": {
    "build": {
      "dependsOn": ["^build"],
      "inputs": [
        {
          "runtime": "cat .env"
        }
      ]
    }
  },

NX will invalidate cache, not sure this is proper way?

[sediatis]

Any help with this? We still cannot seem to read the env vars on production.

[l1qu1d]

I was having the same issue as well. My situation is that I'm using environment variables to determine if the environment is a preview or production. I would build out on staging and if the staging build is good to go, I'd rebase that onto production (which triggers a rebuild on Vercel). But, because of Nx caching, the env. vars would still be on staging, even though it was just rebuilt on production.

As soon as I implemented #6891 (comment) solution, it now works as expected.

[benjaminhobbs]

I was having the same issue as well. My situation is that I'm using environment variables to determine if the environment is a preview or production. I would build out on staging and if the staging build is good to go, I'd rebase that onto production (which triggers a rebuild on Vercel). But, because of Nx caching, the env. vars would still be on staging, even though it was just rebuilt on production.

As soon as I implemented #6891 (comment) solution, it now works as expected.

@l1qu1d
Interesting, that did not work for me, in fact, the only thing on this issue that worked was #6891 (comment). Thanks @tanint

I don't think this should be necessary, though nor am I sure what the proper solution is. It's very surprising that such an important issue is open for so long.

[l1qu1d]

I was having the same issue as well. My situation is that I'm using environment variables to determine if the environment is a preview or production. I would build out on staging and if the staging build is good to go, I'd rebase that onto production (which triggers a rebuild on Vercel). But, because of Nx caching, the env. vars would still be on staging, even though it was just rebuilt on production.
As soon as I implemented #6891 (comment) solution, it now works as expected.

@l1qu1d Interesting, that did not work for me, in fact, the only thing on this issue that worked was #6891 (comment). Thanks @tanint

I don't think this should be necessary, though nor am I sure what the proper solution is. It's very surprising that such an important issue is open for so long.

@benjaminhobbs
I was stating my situation and what I believe is the cause, not a solution to the problem. The solution that you referenced is the same one that I did as well.

[enchorb]

Solution from #6891 (comment) works for now.

@FrozenPandaz / @vsavkin - this is a fairly important and missing feature, any plans to officially support it soon?

NOTE: This also fails for developers on Windows, so we have to have Linux/Mac devs use cat .env and Windows devs to change it to type .env

[WinmezzZ]

Hope these code help to you, it works for me

next.config.js

const webpack = require('webpack');

/**
 * @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
 **/
const nextConfig = {
  webpack(config) {
    // add bellow code
    if (process.env.NODE_ENV === 'production') {
      config.plugins.push(
        new webpack.DefinePlugin({
          'process.env': JSON.stringify(process.env),
        })
      );
    }

    return config;
  },
};

[FrozenPandaz]

I believe the issue is that the builds are cached from other environments where it was run.

Environment variables are generally not included in the hash of a task. But in cases like when using the DefinePlugin from webpack or other tools which use it under the hood like NextJS, the environment variables are inlined into the build artifacts. Hence, this means that the environment variables change the behavior of the build and should also be included as an input of the task. Here is a guide: https://nx.dev/more-concepts/customizing-inputs#customizing-inputs-and-named-inputs

To add an environment variable to the input of a task, add the following:

"inputs": [
  {
    "env": "NX_VARIABLE_NAME",
  }
]

We currently don't have a built in way to add a group of environment variables. You can use a workaround like the following.

      {
        "runtime": "node -e "console.log(Object.entries(process.env).filter(([name]) => name.startsWith('NX_')).map(([k, v]) => k + '=' + v).join(','))"
      }

cc @jaysoo

If this is the default behavior of NextJS/webpack should we also configure this for them?

[marko-hologram]

To add an environment variable to the input of a task, add the following:

"inputs": [
  {
    "env": "NX_VARIABLE_NAME",
  }
]

Thank you for this part, this helped me hopefully resolve an issue I had. I just have one more question.

If these inputs are defined in project.json, do I have to repeat defaults set in nx.json or are these project specific ones just appended to those defaults? In docs I've found (https://nx.dev/reference/project-configuration#using-^), I see "default" and "^production" defined again on project level so I wasn't really sure.

So my question is, do we have to do this (JSON below) or is repeating those redundant and even potentially problematic?

"build" {
  "inputs": [
    "production",
    "^production",
    { "env": "ENV_VAR_1" },
    { "env": "ENV_VAR_2" },
  ]
}

[smasala]

NX environment variables for a project are ignored for me only in the CI (github actions) with v15.9.2 (locally - all builds and tests run fine).

For your webpack.config{.prod}.js add the following line if you are referencing environment variables (the error will be something great like The "path" argument must be of type string. Received undefined with no further output)

const path = require('path');
require('dotenv').config({ path: path.resolve(__dirname, '.env') });

If your application accesses env vars via process.env['NX_{VARNAME}'] (via apps/my_app/src/environments/environment.ts for example); then for your Jest tests you'll need:

jest.config.ts

export default {
   ...
   setupFilesAfterEnv: ['<rootDir>/jest.setup.ts'],
}

jest.setup.ts

if (process.env['NX_INVOKED_BY_RUNNER']) {
  config({ path: __dirname + '/.env' });
}

[gabriel-calin-lazar]

Any updates on this one ?
Are only NX_ prefixed variables supposed to be supported ?
Nx 16 seems to not be able to pick up env values - env values set and env values not set appear to be hashed to the same value - indicating that env values set - have the undefined value
even tough I edit the env variable - I get "local-cache-kept-existing' - but the process.env variable value is different
it might be that only the result of getDotenvVariablesForTask is being used

[charleskoehl]

I consistently experience something similar with any environment variables, not just NX_*:

This sequence of events happens every time with 4 separate next.js projects within an NX monorepo. It happens whether deploying with the CLI or by triggering deployments by pushing commit(s). I'll use the vercel CLI for this example:

1. Deploy a project to preview environment by executing vercel from the project directory.
2. Verify that process.env.VERCEL_ENV and process.env.NEXT_PUBLIC_VERCEL_ENV both equal 'preview' when logged to console in node.js and browser, respectively.
3. Deploy same project to production environment by executing vercel --prod from the project directory.
4. See that process.env.VERCEL_ENV and process.env.NEXT_PUBLIC_VERCEL_ENV both still equal 'preview' when logged to respective consoles.
5. Redeploy the project from the Deployments list in the Vercel dashboard.
6. See that process.env.VERCEL_ENV and process.env.NEXT_PUBLIC_VERCEL_ENV are now correctly set to 'production' when logged to respective consoles.

I compared the Vercel build logs, scrolling backwards through the last 2000 lines of each, and found that the first build used the nx cache while 2nd (the redeploy) did not.

[11:13:36.820]    Nx read the output from the cache instead of running the command for 1 out of 1 tasks.
[11:13:36.820]    Nx Cloud made it possible to reuse admin: https://cloud.nx.app/runs/F77ZqhtWpI

I'm going to try this workaround.

nx.json:

{
  "$schema": "./node_modules/nx/schemas/nx-schema.json",
  "npmScope": "instant-roofer-nx-monorepo",
  "affected": {
    "defaultBase": "main"
  },
  "tasksRunnerOptions": {
    "default": {
      "runner": "nx-cloud",
      "options": {
        "cacheableOperations": [
          "build",
          "lint",
          "test",
          "e2e"
        ],
        "accessToken": "**************************************************************="
      }
    }
  },
  "targetDefaults": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "inputs": [
        "production",
        "^production"
      ]
    },
    "test": {
      "inputs": [
        "default",
        "^production",
        "{workspaceRoot}/jest.preset.js"
      ]
    },
    "e2e": {
      "inputs": [
        "default",
        "^production"
      ]
    },
    "lint": {
      "inputs": [
        "default",
        "{workspaceRoot}/.eslintrc.json"
      ]
    }
  },
  "namedInputs": {
    "default": [
      "{projectRoot}/**/*",
      "sharedGlobals"
    ],
    "production": [
      "default",
      "!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)",
      "!{projectRoot}/tsconfig.spec.json",
      "!{projectRoot}/jest.config.[jt]s",
      "!{projectRoot}/.eslintrc.json",
      "!{projectRoot}/src/test-setup.[jt]s"
    ],
    "sharedGlobals": [
      "{workspaceRoot}/babel.config.json"
    ]
  },
  "generators": {
    "@nx/react": {
      "application": {
        "babel": true
      },
      "library": {
        "unitTestRunner": "jest"
      }
    },
    "@nx/next": {
      "application": {
        "style": "@emotion/styled",
        "linter": "eslint"
      }
    }
  },
  "defaultProject": "booking"
}

the project.json for one of the projects:

{
  "name": "admin",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "apps/admin",
  "projectType": "application",
  "targets": {
    "build": {
      "executor": "@nx/next:build",
      "outputs": [
        "{options.outputPath}"
      ],
      "defaultConfiguration": "production",
      "options": {
        "outputPath": "dist/apps/admin"
      },
      "configurations": {
        "development": {
          "outputPath": "apps/admin"
        },
        "production": {}
      }
    },
    "serve": {
      "executor": "@nx/next:server",
      "defaultConfiguration": "development",
      "options": {
        "buildTarget": "admin:build",
        "dev": true
      },
      "configurations": {
        "development": {
          "buildTarget": "admin:build:development",
          "dev": true,
          "port": 4500
        },
        "production": {
          "buildTarget": "admin:build:production",
          "dev": false
        }
      }
    },
    "export": {
      "executor": "@nx/next:export",
      "options": {
        "buildTarget": "admin:build:production"
      }
    },
    "test": {
      "executor": "@nx/jest:jest",
      "outputs": [
        "{workspaceRoot}/coverage/{projectRoot}"
      ],
      "options": {
        "jestConfig": "apps/admin/jest.config.ts",
        "passWithNoTests": true
      }
    },
    "lint": {
      "executor": "@nx/linter:eslint",
      "outputs": [
        "{options.outputFile}"
      ],
      "options": {
        "lintFilePatterns": [
          "apps/admin/**/*.{ts,tsx,js,jsx}"
        ]
      }
    }
  },
  "tags": [
    "scope:admin"
  ]
}

[diginikkari]

@charleskoehl your problem is propably related to not havin ENV variables defined as an inputs in your nx.json. See: #6891 (comment)

[FrozenPandaz]

@jaysoo @ndcunningham

Please take a look at this issue when you have the chance. I'm going to relabel this as next.

[ndcunningham]

Sorry for taking a while to get to this.
As of current (Nx 19) all built-in NX_ variables will now be NX_PUBLIC_. Should you want to include this in your build artifact it has to be named as such:

https://nx.dev/blog/2024-05-08-nx-19-release#breaking-change-updating-bundled-environment-variables-to

tl:dr; is that being more explicit should help to prevent unwanted values (like secrets) to be available in the bundle.

Similarly, NEXT_PUBLIC_ variables are also available in the client bundle as a feature from Next.js
https://nextjs.org/docs/app/building-your-application/configuring/environment-variables#bundling-environment-variables-for-the-browser