[BUG] Running new e2e test pipeline always failed with error not ready

[PhanLe1010]

The error message is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 196, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 245, in _main
    config.hook.pytest_collection(session=session)
  File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 255, in pytest_collection
    return session.perform_collect()
  File "/usr/local/lib/python3.9/site-packages/_pytest/main.py", line 456, in perform_collect
    hook.pytest_collection_modifyitems(
  File "/usr/local/lib/python3.9/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.9/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.9/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/integration/tests/conftest.py", line 160, in pytest_collection_modifyitems
    check_longhorn(core_api)
  File "/integration/tests/common.py", line 3745, in check_longhorn
    assert ready
AssertionError

Ref:

• https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5705/testReport/junit/(root)/pytest/internal/
• https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5706/testReport/(root)/pytest/internal/
• https://ci.longhorn.io/job/private/job/longhorn-tests-regression/5707/testReport/(root)/pytest/internal/

Context:

• Longhorn version master-head
• Test version master-head

[PhanLe1010]

RCA

1. The test pipeline failed because one of Longhorn components are not running https://github.com/longhorn/longhorn-tests/blob/9b36237782194019e154a6bce2419a2a16015b8d/manager/integration/tests/common.py#L3720-L3736
2. I tried to deploy Longhorn master-head manually and see that the instance-manager pod are not running
3. Looking into instance manager CR, they are there but not in running state
4. Look into the logs of Longhorn manager to see why and see

   time="2023-12-28T21:15:33Z" level=error msg="Dropping Longhorn instance manager out of the queue" func=controller.handleReconcileErrorLogging file="utils.go:72" InstanceManager=longhorn-system/instance-manager-4d524aa41ebc7564d820c4eb786183ad controller=longhorn-instance-manager error="failed to sync instance manager for longhorn-system/instance-manager-4d524aa41ebc7564d820c4eb786183ad: unknown data engine " node=phan-v607-engine-image-pool2-e1b8ff09-cmchx
   

   and

   W1228 21:15:05.979968       1 warnings.go:70] unknown field "spec.dataEngine"
   

5. This suggest that the CRD definition are not up to date.
6. Indeed, this CRD PR has not been merged which may be the root cause of the problem chart: rename backendStoreDriver to dataEngine #7470

[chriscchien]

Close this ticket because

• chart: rename backendStoreDriver to dataEngine #7470 merged.
• Manually deploy longhorn master success.
• Perform regression on private pipeline, the test setup success and test case completed.