Failed DTE tasks in Agents not replayed on the main job

[markwhitfeld]

Current Behavior

Something very odd started happening today. It seems like failed nx tasks on the agent job are not being replayed on the main job. It was working yesterday.

We are using Distributed Task Execution (NX 18.0.8) with our Playwright (v1.42.1) E2E tests.
I can see that the tests are failing on the Agents and are even being retried on another agent.
Now, if I look at the logs for the main job, the failed tasks are not being replayed, but all the other tasks are. The main job step with the nx run is reporting a non successful completion, but the failed tests are not listed, and their outputs are not available among the outputs of the other successful tasks.

The failures are getting sent to NX cloud, because they can be seen in the run, but the results are missing from the main job.
I can also confirm that no github workflow has changed in the last 2 weeks and that we have not upgraded any packages.

PS. I do notice that we have just crossed the allocated 300k credits of the pro plan. I wasn't sure if that was related, so I created a brand new NX org to test if using that org would fix the issue. It didn't fix the issue.

Expected Behavior

Failed DTE tasks in Agents should be replayed on the main job, making their outputs available to the main job.

GitHub Repo

No response

Steps to Reproduce

1. n/a

Nx Report

n/a

Failure Logs

No response

Package Manager Version

npm 10.2.4

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[markwhitfeld]

Possibly related to #21235.
PS. I have sent an email to cloud-support@nrwl.io with the job logs attached.

[markwhitfeld]

@philipjfulcher I'm not sure if you are the same Philip that responded to my email to support, but this is the related issue I logged on GitHub for tracking and for reference for other users.

[meeroslav]

Hi, @markwhitfeld @KwintenP, can you confirm whether this issue is still happening or has been resolved?

[markwhitfeld]

@meeroslav It appears that the issue is now resolved.
Will you be adding acceptance tests for this critical part of the NX product workflow?
This was quite a costly bug for us.

[meeroslav]

Sorry to hear that this bug has affected you significantly. Perhaps @StalkAltan or @rarmatei can chime in on the tests.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.