Metac - use case - mutating admission web hook

[burningalchemist]

Hi @AmitKumarDas,

Just a quick question, if you have time for it.
I'm trying to use metac/metacontroller so it could patch the existing object's specs (Service). It works with labels/annotations, but what I need is to inject an additional configuration parameter (in my case - externalIPs). The problem is that metac doesn't update the existing object (not created by itself), but rather tries to create a new one, which already exists. I suspect, that although it seemed to be easy at first to use metac for that, it's a wrong tool and I have no other chance but to implement a custom MutatingAdmissionWebhookController. What's your opinion?

Kind regards,
Sergei

[AmitKumarDas]

Hi @burningalchemist ,
IMO GenericController from metac should solve your requirements. Did you try metac already? Can you provide the yamls to look into further?

I guess, in your specific case both watch & attachment are same. So watch & attachment resources can be configured to Service. One can further filter the attachment to refer to same watch via advanced filters. One need to be a bit careful to avoid getting into hot loops since updating the attachment will trigger reconciliation of watch, since both are same. However if fixed values are being updated (IPs in your case), this should be safe.

Getting back to the topic of MutatingWebHooks, I would prefer the one with low maintenance overhead.

References

• You may want to have a look at https://github.com/AmitKumarDas/metac/tree/master/examples/gctl/uninstall-openebs where its able to update as well as delete resources that were created by someone else.
• Refer UpdateAny & DeleteAny fields: https://github.com/AmitKumarDas/metac/blob/master/examples/gctl/uninstall-openebs/config/config.yaml#L5-L11
• Further needs are being tracked under add granular update & delete options to GenericController #89
• Advanced filters API:
  • https://github.com/AmitKumarDas/metac/blob/master/apis/metacontroller/v1alpha1/types_generic.go#L176
  • https://github.com/AmitKumarDas/metac/blob/master/apis/metacontroller/v1alpha1/selector.go#L31
• Advanced filters example: https://github.com/mayadata-io/d-operators/blob/master/config/metac.yaml

[burningalchemist]

@AmitKumarDas Thank you for a quick response!

Yeah, I've implemented the whole stuff but with DecoratorController for the moment (I've come from Metacontroller implementation) and I bumped into that kind of issue.

I'm on the same page with you, MutatingAdmissionWebhook for such a small thing seems like an overkill. I'll try to provide more details if you don't mind, but I'll also look into what you've provided already. Thanks again! :)

[AmitKumarDas]

@burningalchemist you may try something like below:

apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
metadata:
  name: sync-services
  namespace: my-namespace
spec:
  # set update any to true since these Services are created
  # by some other tools, workflows, etc
  updateAny: true
  # kind Service is watched
  watch:
    apiVersion: v1
    resource: services
    # you may want to add label or ann selector to filter the watches further
  attachments:
  - apiVersion: v1
    resource: services
    advancedSelector:
      selectorTerms:
      - matchReferenceExpressions:
        # select the Service (i.e. attachment) that is same as Watch
        - key: metadata.uid
          operator: EqualsWatchUID
  hooks:
    sync:
    # fill in your details

[burningalchemist]

Hi @AmitKumarDas ! Thanks a lot for the snippet, I'll try it out as soon as I can and share the feedback. I had to shift my gears to something different this week, but will get back and let you know. :)

[burningalchemist]

@AmitKumarDas I have a related question, which I can't get so far.
This is the config.yml which gets loaded by GenericController:

apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
metadata:
  name: sync-services
  namespace: metallb-system
spec:
  # set update any to true since these Services are created
  # by some other tools, workflows, etc
  updateAny: true
  # kind Service is watched
  watch:
    apiVersion: v1
    resource: services
    annotationSelector:
      matchExpressions:
      - {key: metacontroller, operator: Exists}
  attachments:
  - apiVersion: v1
    resource: services
    updateStrategy:
      method: InPlace
    advancedSelector:
      selectorTerms:
      - matchAnnotationExpressions:
        - key: metacontroller
          operator: Exists
      - matchReferenceExpressions:
        # select the Service (i.e. attachment) that is same as Watch
        - key: metadata.uid
          operator: EqualsWatchUID
  hooks:
    sync:
      inline:
        funcName: update/service

The logic is empty yet (took it from the example):

func updateService(request *generic.SyncHookRequest, response *generic.SyncHookResponse) error {
	if response == nil {
		response = &generic.SyncHookResponse{}
	}

	if request.Attachments.IsEmpty() {
		response.Finalized = true
		return nil
	}

	for _, attachment := range request.Attachments.List() {
		if attachment.GetKind() == "Service" {
			println(attachment)
			response.Attachments = append(response.Attachments, attachment)
			continue
		}

	}

	return nil
}

I have only a single service which contains annotation metacontroller: "true" - nginx. So I expect it to be filtered among all the services. But actually I get more than a single service observed:

I0325 18:05:46.779493       1 controller.go:716] WatchGCtl metallb-system/sync-services: Will apply attachments: Observed Resource Instances:-
	Service.v1:nginx f4f80ba1-7313-4a46-82f5-e1d7fcbb8094
	Service.v1:speaker 2d1278d4-456d-45b5-bc89-e8c0a2260ec0
	Service.v1:kube-dns b13ab268-eeed-4c52-bc0d-1a7a50bf856d
	Service.v1:redis d4ac93a4-4f16-43f1-bd42-4fca81b8ad9d
: Desired Resource Instances:-
	Service.v1:nginx f4f80ba1-7313-4a46-82f5-e1d7fcbb8094
	Service.v1:speaker 2d1278d4-456d-45b5-bc89-e8c0a2260ec0
	Service.v1:kube-dns b13ab268-eeed-4c52-bc0d-1a7a50bf856d
	Service.v1:redis d4ac93a4-4f16-43f1-bd42-4fca81b8ad9d

All of them get updated (with last-applied-configuration annotation).

Service yaml:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    external-ip: enabled
  annotations:
    metacontroller: "true"
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    external-ip: enabled
  type: LoadBalancer

Do I use Selectors incorrectly? Currently, I don't feel brave enough to proceed with the logic as I don't understand yet how the filtering in metacworks, so I may harm the existing environment accidentally. 😢

[AmitKumarDas]

@burningalchemist can you try this?

You may probably want to try things in different environment other than production before applying the same in production. I really don't have any other idea, other than implementing a DryRun stuff etc (a long term / future stuff) for this particular experience you had so far.

I have removed one selector from attachments. This additional selector was a OR operation & hence multiple services were considered as attachments.

apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
metadata:
  name: sync-services
  namespace: metallb-system
spec:
  # set update any to true since these Services are created
  # by some other tools, workflows, etc
  updateAny: true
  # kind Service is watched
  watch:
    apiVersion: v1
    resource: services
    annotationSelector:
      matchExpressions:
      - {key: metacontroller, operator: Exists}
  attachments:
  - apiVersion: v1
    resource: services
    updateStrategy:
      method: InPlace
    advancedSelector:
      selectorTerms:
      - matchReferenceExpressions:
        # select the Service (i.e. attachment) that is same as Watch
        - key: metadata.uid
          operator: EqualsWatchUID
  hooks:
    sync:
      inline:
        funcName: update/service

In above yaml, the attachment will get selected based on the watch. The attachment uid is matched against the watch uid. Since both attachment & watch refer to same Kind, they are same instance only.

[burningalchemist]

Hi @AmitKumarDas,

Yeah, or course I develop things in my own local cluster. To proceed further and avoid any unexpected situation, I need to make sure that I get the selection I request.

I guess you've removed the part:

     - matchAnnotationExpressions:
        - key: metacontroller
          operator: Exists

I have tried it already as well (initially I picked the configuration you've shared above as is), but so far the result is the same - for some reason all the services get observed, including kube-dns, which is definitely not my case.

From the logs I can tell that watch rules work correctly as my service is the only one selected and watched. But attachment list is basically all the services available. So either I don't understand how to use rules for attachments properly, or it doesn't work as intended. My current case implies that I watch a single service and apply attachment to a single service.

[AmitKumarDas]

Ok. Are you using the latest Metac binary?
Meanwhile I shall test the same on some setup

[burningalchemist]

Oh, that's a good point. I'll check go.mod, as I build the binary myself. Let me try to check it first. :)

[AmitKumarDas]

@burningalchemist if this is any opensource stuff or if repo is public, then you may provide the link.

[burningalchemist]

@AmitKumarDas
Yeah, exactly, that was an issue. go.mod from the example contained an outdated reference and I wasn't attentive enough to check it. I see that exactly one service is picked up now after I've updated it. 🎉 Thanks for you help! 👍

[AmitKumarDas]

Some of the repos that uses metac and are being using in production might help you to refer as well. I will add these are references/adopters one day.

• https://github.com/mayadata-io/cstorpoolauto
• https://github.com/mayadata-io/d-operators/

[burningalchemist]

Awesome, I'll take a look. Thanks again and have a great day. :)

[burningalchemist]

Hi @AmitKumarDas, just for your information - I've successfully achieved the goal 👍
So far for quick development, I've used GenericController with an external webhook in Python, but once the logic is approved and stabilised, I'm going to move it to an inline function in Go.
I've also found a small quirk and happy to share as soon as I can. Thanks a lot for your help! 😃