Nx Agents command sometimes doesn't propagate error in tasks

[thekip]

Current Behavior

We noticed this few times since implemented DTE, the main task hangs for hours. I checked the logs of each agent and that what i've found:

The log shows that task failed, but runner finished with non-zero code and didn't propogate this error to the cloud and main task. On the moment i've done this screenshot this run in NX was in pending status, however all tasks in the agents was executed.

Expected Behavior

It should fail in every case when error is happend.

GitHub Repo

No response

Steps to Reproduce

Unfortunately, it is hard to find steps to reproduce, since it happens sometimes on CI

Nx Report

Node   : 20.9.0
   OS     : darwin-arm64
   yarn   : 3.4.1
   
   nx (global)        : 17.2.7
   nx                 : 16.10.0
   @nx/js             : 16.10.0
   @nx/jest           : 16.10.0
   @nx/linter         : 16.10.0
   @nx/workspace      : 16.10.0
   @nx/angular        : 16.10.0
   @nx/cypress        : 16.10.0
   @nx/devkit         : 16.10.0
   @nx/esbuild        : 16.10.0
   @nx/eslint-plugin  : 16.10.0
   @nx/node           : 16.10.0
   @nrwl/tao          : 16.10.0
   @nx/webpack        : 16.10.0
   nx-cloud           : 16.5.2
   typescript         : 5.1.6
   ---------------------------------------
   Community plugins:
   @nebular/theme               : 11.0.1
   @simondotm/nx-firebase       : 2.1.2
   ng2-charts                   : 4.1.1
   ngx-daterangepicker-material : 6.0.4

Failure Logs

No response

Package Manager Version

No response

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[rarmatei]

could you send an email please to cloud-support@nrwl.io and include your CI config (main + agents config) + a link to one of your CI executions on nx.app where the main job hanged?

[kdawgwilk]

We also just experienced this with jest getting OOM killed and then the nx task just hangs

[faileon]

Same issue here. Reported it to the cloud-support as per @rarmatei suggestion

[meeroslav]

I am closing this issue since it's related to the Nx Cloud SAAS, not the Nx open-source solution.

If you are experiencing the same issue please follow @rarmatei's suggestion:

#21235 (comment)