[CI] : Intermittent failure in E2E tests while restarting workspace

[rohanKanojia]

Description

Related to #1469

The CI pipeline for the devworkspace-operator is experiencing intermittent failures in the devworkspace_restart_tests suite on OpenShift. The specific failure occurs within the [It] Start DevWorkspace test, where the workspace transitions from Running to Stopped and then fails to restart, entering a Failed state.

The test suite successfully creates and starts a DevWorkspace. The workspace briefly reaches a Running state before being unexpectedly stopped. When the test attempts to restart the workspace, it fails with the following error message:

Error creating DevWorkspace deployment: Container dev has state Error:

This suggests an issue with the underlying container or deployment during the restart process. The failure is not consistent, indicating a potential race condition, resource constraint, or intermittent cluster instability.

Failure Log

2025/09/24 16:21:38 Pod 'devworkspace-webhook-server-5dd7f57bb9-h9b5z' created in namespace devworkspace-controller... Checking startup data.
2025/09/24 16:21:38 Pod started after 35.895207 seconds
�[38;5;10m•�[0m2025/09/24 16:21:42 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:44 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:46 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:48 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:50 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:52 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:54 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:56 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:21:58 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:00 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:02 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:04 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:06 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:08 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:10 Now current status of developer workspace code-latest is: Running. Message: Workspace is running
�[38;5;10m•�[0m�[38;5;10m•�[0m�[38;5;10m•�[0m2025/09/24 16:22:12 Now current status of developer workspace code-latest is: Stopped. Message: Stopped
�[38;5;10m•�[0m2025/09/24 16:22:14 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:16 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:18 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:20 Now current status of developer workspace code-latest is: Starting. Message: Waiting for workspace deployment
2025/09/24 16:22:22 Now current status of developer workspace code-latest is: Failed. Message: Error creating DevWorkspace deployment: Container dev has state Error: 

�[38;5;243m------------------------------�[0m
�[38;5;9m• [FAILED] [10.019 seconds]�[0m
�[0m[Create DevWorkspace and ensure data is persisted during restarts] �[38;5;9m�[1m[It] Start DevWorkspace�[0m
�[38;5;243m/go/src/github.com/devfile/devworkspace-operator/test/e2e/pkg/tests/devworkspace_restart_tests.go:87�[0m

  �[38;5;9m[FAILED] DevWorkspace didn't start properly. Error: workspace has been failed unexpectedly. Message: Error creating DevWorkspace deployment: Container dev has state Error: �[0m
  �[38;5;9mIn �[1m[It]�[0m�[38;5;9m at: �[1m/go/src/github.com/devfile/devworkspace-operator/test/e2e/pkg/tests/devworkspace_restart_tests.go:94�[0m �[38;5;243m@ 09/24/25 16:22:22.732�[0m
�[38;5;243m------------------------------�[0m
�[38;5;14mS�[0m�[38;5;14mS�[0m2025/09/24 16:22:22 Cleaning up test resources are disabled


�[38;5;9m�[1mSummarizing 1 Failure:�[0m
  �[38;5;9m[FAIL]�[0m �[0m[Create DevWorkspace and ensure data is persisted during restarts] �[38;5;9m�[1m[It] Start DevWorkspace�[0m
  �[38;5;243m/go/src/github.com/devfile/devworkspace-operator/test/e2e/pkg/tests/devworkspace_restart_tests.go:94�[0m

�[38;5;9m�[1mRan 10 of 12 Specs in 61.865 seconds�[0m
�[38;5;9m�[1mFAIL!�[0m -- �[38;5;10m�[1m9 Passed�[0m | �[38;5;9m�[1m1 Failed�[0m | �[38;5;11m�[1m0 Pending�[0m | �[38;5;14m�[1m2 Skipped�[0m
--- FAIL: TestWorkspaceController (61.87s)
FAIL
make: *** [Makefile:109: test_e2e] Error 1

Acceptance Criteria

• Intermittent failure is investigated and test is fixed to avoid intermittent failures

[rohanKanojia]

It seems during restarts workspace is updated with transient errors that get picked up by devworkspace operator. There is already logic in place to ignore these type of errors:
I think this issue could be fixed by adding below config before running e2e tests:

devworkspace-operator/.ci/openshift_e2e.sh

Line 73 in e0ed8d2

make test_e2e

kubectl patch devworkspaceoperatorconfig devworkspace-operator-config -n openshift-operators --type merge -p '{"config":{"workspace":{"ignoredUnrecoverableEvents":["Error"]}}}'

[rohanKanojia]

Closed via #1528