[FEATURE] Consolidate Instance Manager Engine & Replica for resource consumption reduction

[innobead]

Is your improvement request related to a feature? Please describe (👍 if you like this request)

Currently, we will have an instance manager engine & replica pods on each node, and each of them will have a default 12% Guaranteed CPU request which is configurable. After upgrading Longhorn, there will be another two new instance manager pods running, so there will be four instance managers pods before all volumes are migrated to the new version. This causes high resource requirements for the fresh install and upgrade as well.

Engine/Replica processes running in the instance manager are process-based, so they will not impact the instance manager container. To simplify the architecture to decrease resource usage, the goal here is to consolidate the instance manager engine & replica to one pod, but continue serving all data plane operations on each node without any change. For volume migration, the same flow is as usual.

Describe the solution you'd like

• Introduce the new architecture of the instance manager
• Revamp the current automatic engine upgrade implementation
• Benchmark the resource consumption based on the number/size of volume. Have a theoretical explanation for the resource usage by instance manager.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

rancher/image-mirror#329 (comment)

cc @longhorn/dev

[c3y1huang]

Note: #1691 (comment)

[c3y1huang]

Initial test results:

• core tests: https://ci.longhorn.io/job/private/job/longhorn-tests-regression/3112/
• core tests (upgrade): https://ci.longhorn.io/job/private/job/longhorn-tests-regression/3120/

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:

  • feat: consolidate instance managers #5493
    The PR for the chart change is at:
  • feat: consolidate instance managers #5493

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at:

  • feat: consolidate instance managers longhorn-manager#1731
  • feat: consolidate instance managers longhorn-instance-manager#191

• Which areas/issues this PR might have potential impacts on?
  Area manager, instance manager
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at:

  • feat(lep): consolidate instance managers #5457

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at:

  • doc: consolidate instance managers website#671

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at:

  • test: consolidate instance managers longhorn-tests#1277
    The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[c3y1huang]

@longhorn/qa , for this feature, please also do some tests using the UI to see if the frontend needs to adjust, thanks!

cc @smallteeths

[yangchiu]

Tested on master-head, although there are volumes created and attached, the Ref Count Volume in Instance Manager Image page is always 0, and the Volumes modal of each instance manager is always empty:

Also the Instance Manager Image in Components modal of each node is N/A:

Not sure if it's normal. @c3y1huang

[innobead]

This is not right.

@yangchiu btw, do you know if we have any e2e test cases related to the reference count of instance image?

[yangchiu]

This is not right.

@yangchiu btw, do you know if we have any e2e test cases related to the reference count of instance image?

We only have test cases for engine image reference count, and there's no test case for instance manager reference count.

[c3y1huang]

Tested on master-head, although there are volumes created and attached, the Ref Count Volume in Instance Manager Image page is always 0, and the Volumes modal of each instance manager is always empty:

Also the Instance Manager Image in Components modal of each node is N/A:

Not sure if it's normal. @c3y1huang

Thanks @yangchiu , this should be related to the UI as we have not yet implemented support for the new aio instance managers, for example: https://github.com/longhorn/longhorn-ui/blob/master/src/models/host.js#L122-L127.

Let's track the UI implementation in #5876

[yangchiu]

Verified passed on master-head (longhorn-manager 6631855, longhorn-ui 51e912d). Operations work well on Longhorn ui, and it presenta the correct information about the new aio type of instance manager. The above mentioned ui issue has been corrected.