-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pod controlled by a Job does not exit after after main container completes #1869
Comments
We need something from kubernetes/kubernetes#25908 to move forward with this. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions. |
whoop, reopen. |
Should this really be closed? |
I still deem this problematic, too |
@christianhuening , by a chance, do you know any workarounds better than this one: |
We just don't inject Jobs atm. This is not an issue right now, since we only use jobs for initialization when deploying environments. |
Do you know if it's possible to blacklist jobs on the namespace level or somewhere in L5d configuration? |
provide the annotation and set it to disabled on the job's pod spec |
This requires direct access to each Job specification (or a custom admission controller). Thank you for your feedback. |
I believe I found a solution to this that doesn't require waiting for a new k8s feature or significantly altering the main job process. Pods have a The solution: Assume you can identify the process id for the main workload in your job/cronjob. Then you can add your own sidecar container that checks to see if your job process is running, sleeps, and repeats until the job process exits. Once it exits, you kill the linkerd2-proxy process, which makes that container exit, and successfully ends the job/cronjob. Here's an example which assumes your job process is called
For context, we are running k8s version 1.15.7. |
@alexklibisz I tried this approach with a deployment for a similar issue (#3751), and the kill command is not permitted:
The base image is did you encounter any permissions issues you had to work around? |
IIRC, I saw some permission errors when I forgot to add the |
I do have that setting, the termination process can see the pid of the other process. Afaict, the issue is that |
In my case, the |
Yeah I didn't set up the termination process to run as root.
…On Mon, Mar 9, 2020, 4:57 PM Alex Klibisz ***@***.***> wrote:
I do have that setting, the termination process can see the pid of the
other process. Afaict, the issue is that linkerd-proxy and the
termination process are run by different users.
In my case, the java process was definitely owned by a non-root user
(user id 1001 IIRC) and the kill process, definitely owned by the root
user in the termination container, and I believe that linkerd2-proxy was
owned by a non-root user as well. But I would have to double check on the
last one and don't have my work computer right now.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1869?email_source=notifications&email_token=AABHMIXJLVAVHPVMV4EXOS3RGWGBNA5CNFSM4GF3VQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOJPSAA#issuecomment-596834560>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABHMITEYAXHKPAR3FNH25LRGWGBNANCNFSM4GF3VQAQ>
.
|
Got it. So it's working now? |
I'll update once I try 😉
…On Mon, Mar 9, 2020, 5:59 PM Alex Klibisz ***@***.***> wrote:
Yeah I didn't set up the termination process to run as root.
Got it. So it's working now?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1869?email_source=notifications&email_token=AABHMIWH7TXQM6MTUY5RK7TRGWNJHA5CNFSM4GF3VQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOJTDFQ#issuecomment-596849046>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABHMIXNFXJ3JRU3RHP2VALRGWNJHANCNFSM4GF3VQAQ>
.
|
Any updates on this? I am new to using cron jobs and linkerd both. If you could spare a moment to share progress on this issue will be great. |
@kumargauravin the upstream Kubernetes issue is still open and there's not anything we can do on the Linkerd side. |
Can confirm the original solution I posted is still working fine after about three months. We also have some some crons running on Argo Workflows with linkerd sidecars.
The key part is the |
@alexklibisz thanks for the workaround! |
I updated the comment above. |
For future reference. A shutdown hook was added in linkerd/linkerd2-proxy#811 |
Is there an example or documentation for this feature? |
@laukaichung Good catch, I don't think we documented this very well. Would you mind filing an issue to https://github.com/linkerd/website so that we can track this? |
@wmorgan @laukaichung I didn't find an existing one, so I created an issue about the missing documentation. |
Bug Report
What is the issue?
Pods that are controlled by Jobs are not terminating when the main container exits
How can it be reproduced?
Create a job with linkerd sidecar container
Logs, error output, etc
main container logs are as usual, sidecar container logs are as usual
linkerd check
outputEnvironment
Possible solution
I think that when a container within a pod controlled by a Job completes, the sidecar should exit as well.
Additional context
The sidecar was created using
linkerd inject
The text was updated successfully, but these errors were encountered: