adding deployment spec overrides via node label selector

[EraKin575]

What type of PR is this?
Adding deployment spec overrides via node label selector to have granular node characteristics.

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

A part of #5755

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[WillardHu]

You can add some description in the "What this PR does / why we need it" part.

[EraKin575]

@WillardHu can you please review if I am going to the right track?

[EraKin575]

question: 1. How can I update the edgeapplication crd yaml to test these changes?
2.
If both TargetNodeGroups and TargetNodeLabels are defined for the same workload, and each specifies different overrides for the same property (e.g., different NodeSelector values), there could be conflicts. What should I prioritize here?

[WillardHu]

question: 1. How can I update the edgeapplication crd yaml to test these changes? 2. If both TargetNodeGroups and TargetNodeLabels are defined for the same workload, and each specifies different overrides for the same property (e.g., different NodeSelector values), there could be conflicts. What should I prioritize here?

Sorry, I've been busy recently, so I'm replying late.

1. You can execute make generate to update crds.
2. Each item configuration of the nodeSelector and nodeGroup generates a new Deployment. If a user configuration error causes multiple Pods of the same application to be deployed to one node, resulting in resource conflict, We can reflect the operation result of deployment in the edgeapplication status first. Then we can judge whether verification is necessary based on user feedback. I think validation becomes expensive when there are a large number of nodes and overriders configurations. What do you think? @EraKin575 @Shelley-BaoYue @wbc6080

By the way, for deployment generated by nodeSelector, we need to define their names in other ways. For example, <edgeapplication>-<random number>.

[EraKin575]

question: 1. How can I update the edgeapplication crd yaml to test these changes? 2. If both TargetNodeGroups and TargetNodeLabels are defined for the same workload, and each specifies different overrides for the same property (e.g., different NodeSelector values), there could be conflicts. What should I prioritize here?

Sorry, I've been busy recently, so I'm replying late.

1. You can execute `make generate` to update crds.

2. Each item configuration of the `nodeSelector` and `nodeGroup` generates a new Deployment. If a user configuration error causes multiple Pods of the same application to be deployed to one node, resulting in resource conflict, We can reflect the operation result of deployment in the edgeapplication status first. Then we can judge whether verification is necessary based on user feedback. I think validation becomes expensive when there are a large number of nodes and overriders configurations.  What do you think? @EraKin575  @Shelley-BaoYue  @wbc6080

By the way, for deployment generated by nodeSelector, we need to define their names in other ways. For example, <edgeapplication>-<random number>.

make generate is causing an error when running:

I have also updated the OpenAPI schema before make generate

[wbc6080]

You can try to run hack/update-codegen.sh to pass the make verify CI test and run hack/generate-crds.sh to generate new crds

[wbc6080]

What do you think? @EraKin575 @Shelley-BaoYue @wbc6080

Maybe we can discuss more details in community meeting

[EraKin575]

You can try to run hack/update-codegen.sh to pass the make verify CI test and run hack/generate-crds.sh to generate new crds

Its still generating the same error. I tried to upgrade the controller gen version to latest version and it apparently worked. Should I go ahead with it?

[wbc6080]

@EraKin575 Do you mean that after upgrading the controller gen version, hack/generate-crds.sh was executed successfully? Based on this success, you can run hack/update-codegen.sh again, and then execute git status. You should be able to see that some files have been automatically modified. Push these modifications up again, and you should be able to solve the CI error.

[EraKin575]

Hi! I have a question. Can you please tell me about the matching logic for targetnodegroup overriders? How does it find the node group to override the deployment spec?

[wbc6080]

Hi! I have a question. Can you please tell me about the matching logic for targetnodegroup overriders? How does it find the node group to override the deployment spec?

Sorry, I didn't understand your question clearly. Are you asking how to build nodes into node groups? Or how to fully populate edge application deployment in different node groups?
If it is the former, we define the NodeGroup CRD to describe a node group. Nodes with specific labels can be add to a node group management through the matchlable field.

If it is the latter, in edge application CRD , the Workload Templates field can define the Deployment Template of the edge application, and WorkloadScopes can be configured differently according to the needs of different node groups, such as configuring an image repository address. Workload Templates and WorkloadScopes can be added together to form a complete deployment for deployment.

You can also read nodegroup's proposal for more information https://github.com/kubeedge/kubeedge/blob/master/docs/proposals/node-group-management.md

[WillardHu]

Hi! I have a question. Can you please tell me about the matching logic for targetnodegroup overriders? How does it find the node group to override the deployment spec?

1. Assign a instance overridemanager.OverrideManager to Overrider field in controller management.

   kubeedge/cloud/pkg/controllermanager/controllermanager.go

   Line 63 in 841368a

   Overrider: &overridemanager.OverrideManager{

2. Loops overrider-infos to overrides the Deployment yaml

   kubeedge/cloud/pkg/controllermanager/edgeapplication/edgeapplicationcontroller.go

   Line 130 in 841368a

   for _, info := range overriderInfos {

3. Loops overriders to executes func ApplyOverrides(..) of each Overrider implementation(argsoverrider.go, envoverrider.go, imageoverrider.go, etc...).
   

   kubeedge/cloud/pkg/controllermanager/edgeapplication/overridemanager/overridemanager.go

   Line 35 in 841368a

   func (o *OverrideManager) ApplyOverrides(rawObjs *unstructured.Unstructured, overrideInfo OverriderInfo) error {

4. Updates, creates or deletes k8s resources.

[EraKin575]

Hi! I have a question. Can you please tell me about the matching logic for targetnodegroup overriders? How does it find the node group to override the deployment spec?

1. Assign a instance `overridemanager.OverrideManager` to `Overrider` field in controller management.https://github.com/kubeedge/kubeedge/blob/841368a8b412169dff3cd7bb34b12f3fe8cfa9a7/cloud/pkg/controllermanager/controllermanager.go#L63

2. Loops overrider-infos to overrides the Deployment yaml https://github.com/kubeedge/kubeedge/blob/841368a8b412169dff3cd7bb34b12f3fe8cfa9a7/cloud/pkg/controllermanager/edgeapplication/edgeapplicationcontroller.go#L130

3. Loops overriders to executes func `ApplyOverrides(..)` of each Overrider implementation(argsoverrider.go, envoverrider.go, imageoverrider.go, etc...).
   https://github.com/kubeedge/kubeedge/blob/841368a8b412169dff3cd7bb34b12f3fe8cfa9a7/cloud/pkg/controllermanager/edgeapplication/overridemanager/overridemanager.go#L35

4. Updates, creates or deletes k8s resources.

thanks for the help @WillardHu , but already I knew this part. I am sorry I couldn't explain my problem better. I am struggling to find the logic for finding the correct targetnodegroup that implements the deployments on. Example;

targetNodeGroups:
      - name: beijing
        overriders:
          resourcesOverriders:
            - containerName: container-1
              value: {}
      - name: shanghai
        overriders:
          resourcesOverriders:
            - containerName: container-1
              value: {}

where is the code to match the name of targetnodegroup with the actual NodeGroup where it does the deployment?

[EraKin575]

Hi, @WillardHu can you review this PR ? I have made the required changes and it seems to be working. For now, I have added tests for command overrider and it passed. If you approve the changes, I will add test for every overrider
cc @wbc6080

[WillardHu]

A consistent version is required here, and the latest version will make a difference in comparison checks

[WillardHu]

This code can be put in the nodeselectoroverrider.go

[WillardHu]

It would be best to write the message to the EdgeApplication Status. And the historical error handling doesn't look good, the Reconcile() function returns an error only make it constant retried.

[WillardHu]

Here, the processing performance is better with map, but the readability is worse. Is it better to use jsonpath?

[WillardHu]

The nameoverrider.go file is also to be modified

[WillardHu]

where is the code to match the name of targetnodegroup with the actual NodeGroup where it does the deployment?

Refer to nodeselectoroverrider.go, when each node is bound by a node group, the node automatically adds a label apps.kubeedge.io/belonging-to @EraKin575

[EraKin575]

You need to restore the changes related to the device @EraKin575

Should I also restore those crd files which have been modified and not added with 'v1alpha2' device versioning?

[EraKin575]

@WillardHu please review the PR
CC @wbc6080

[WillardHu]

Please rebase your commits to one commit. @EraKin575

[WillardHu]

/lgtm

[fisherxu]

/approve

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fisherxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [fisherxu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment