Best practice to handle a kubernetes job that needs to connect to the mesh

[harscoet]

Actually to be able to create a kubernetes job that needs to call some services from the mesh, I need to register the job itself as a service in consul. Is it the only solution? Do you have some best practices to handle this really common use case?

Related to this issue #692 I now need to create a kubernetes service, as I don't have any port, what manifest I need to create?
Is it enough?

apiVersion: v1
kind: Service
metadata:
  name: dummy-service
spec:
  clusterIP: None
  selector:
    name: dummy-service

I have a secondary issue related to this kubernetes behavior kubernetes/kubernetes#25908
When my jobs ends, consul sidecars are still running so the job is not considered terminated. Do you have a good solution to handle this? Linkerd for example added a special endpoint to be able to shutdown linkerd/linkerd2-proxy#811

The workaround actually for me is to use shareProcessNamespace: true run my main container as root and run pkill consul at the end of my script

[kschoche]

HI @harscoet, thanks for posting this question.
You are correct in that in order to access the mesh you would need to add a headless kubernetes service for your Job to get registered in the service mesh.
You are also correct in that due to the envoy sidecar not being terminated by Kubernetes the pod will be left around even after your container completes. One possible option might be to have your container issue a POST to envoy's admin port /quitquitquit at it's completion, that might avoid having to try to issue a pkill through namespaces?
https://www.envoyproxy.io/docs/envoy/latest/operations/admin#post--quitquitquit

[wmorgan]

Linkerd also provides the linkerd-await wrapper that will call the shutdown hook for you when the job terminates, as a convenience. C'mon over and give it a try!

[harscoet]

One possible option might be to have your container issue a POST to envoy's admin port /quitquitquit at it's completion, that might avoid having to try to issue a pkill through namespaces?

Thanks for the reply @kschoche!
It seems to be a good solution to exit the first envoy-sidecar! What about the second one consul-sidecar ?

[kschoche]

Hi @harscoet!
We no longer attach the consul-sidecar sidecar in the latest releases, so you should just need to send it to the envoy-sidecar. The only time it gets added anymore is when you're using metrics merging.
Edit: removing the part about the preStop hook as after a closer look it would not work!

[lkysow]

Related to Kyle's response, are you using metrics merging?

[nicholasjackson]

Adding an additional option to this @harscoet, this code repository:

https://github.com/nrmitchi/k8s-controller-sidecars

contains a controller that you can deploy to your cluster that automatically kills sidecar pods when the main job container exits.

You will need to build this from the go code and bundle it into a Docker container as there is no release (if you just want to test you can use: nicholasjackson/k8s-controller-sidecars:latest). But it works really well, in addition to the injection annotations for Consul you simply add an additional annotation "nrmitchi.com/sidecars": "consul-sidecar,envoy-sidecar". This is a comma-separated list of containers to kill when the main container exits.

I think you are using the transparent proxy features so many of the below annotations may not be required, my cluster is not yet using that.

Example annotations

              "consul.hashicorp.com/connect-inject": "true"
              "consul.hashicorp.com/connect-service": "cron-job"
              "consul.hashicorp.com/connect-service-port": 9090
              "consul.hashicorp.com/connect-service-upstreams": "minecraft-bot:3000"
              "nrmitchi.com/sidecars": "consul-sidecar,envoy-sidecar"

Kind regards,

Nic

[david-yu]

@harscoet Could you share your example yaml file with your job injected? We're looking into how to solve this now.

[david-yu]

This is now documented here: https://developer.hashicorp.com/consul/docs/k8s/connect#kubernetes-jobs. Will go ahead and close this issue. Thanks for your feedback. If you're on Consul K8s 1.2.x, you'll need to wait until the next patch release to get the graceful shutdown endpoint.