Add option to create one-off job from cron job

[muenchdo]

Summary

Add a button to create a one-off job from a cron job.

Motivation

We often deploy cron jobs using Argo CD, but want to do test run of the job after deployment. Using kubectl you can easily create a job from a cron job using kubectl create job --from=cronjob/foo foo. As most of our engineers don't actually have cluster access using kubectl, integrating this into Argo CD (including RBAC, easily browsable logs, etc.) would be tremendously useful.

Proposal

Similar to the Restart option for deployments, this could be included in the dropdown when clicking a cron job object in the resource tree.

[jessesuen]

This isn't currently possible with the current action feature, which only manipulates the resource that you are performing the action on.

However I do see the value in generating new resources as part of an action. This would need to be a new feature and potentially affect RBAC.

[RichiCoder1]

This would be an interesting feature standalone to be able to provide the ability to perform manual, but defined cluster actions via the ArgoCD UI/API

[genslein]

bump here, there is a real desire to run things like seed data migrations as one-off jobs in real environments from the UI.

This unblocks regular deployment pipelines and allows better management of multiple environments. Any updates?

[stephaneetje]

Hello, also very interested in that feature that would simplify a LOT our processes

[crenshaw-dev]

If anyone's willing to put up a PR, I'd be happy to help w/ dev setup and review!

My rough idea of the way this could work is that the custom action could produce a list of manifests (strings of JSON or YAML) and actions (for now just apply, but maybe later delete, etc.). The API server would then loop over that list and send the manifests to the controller to sync. The controller would make sure that the application actually has permissions to deploy the resource.

I'm hazy on the exact mechanic behind "send the manifests to the controller." The API server currently has the ability to trigger and "override" sync, where manifests are sent to the controller to be applied once, ignoring the usual application source of truth. But I'm not sure the override mechanism currently has a "just add this one thing (or these few things)" option. The developer would probably have to build this new sync mechanism.

@jessesuen does that line up with how you'd envision this feature being built?

[reggie-k]

So upon a new "create-from" action, performed on the

apiVersion: batch/v1
kind: CronJob
metadata:
  ...

resource, which is part of some Application, a new

apiVersion: batch/v1
kind: Job
metadata:
  ...

resource (which is not part of that Application) has to be created?

[crenshaw-dev]

@reggie-k yep, something like that is what I'd imagine!

[bygui86]

sorry guys, I don't understand the status of this new feature :)
can someone summarise it?
thanks!!

[SergeyLadutko]

Can i make this button with Lua ?

[bygui86]

@SergeyLadutko very good question! I'm looking into it, but I don't have Lua expertise at the moment :(
Any update here?

[SergeyLadutko]

@bygui86
This is a very important button as it is the last thing that makes our developers use Lens or dashboard

[SergeyLadutko]

You created a button to restart deploymenta, but did not create a cronjob to launch (((

[reggie-k]

I am working on the issue, but it it will take time, not a trivial one. If anyone wants to join me, it can be a team work. There are tasks at 3 different levels here.

…

On Thu, Dec 1, 2022, 13:23 SergeyLadutko ***@***.***> wrote: You created a button to restart deploymenta, but did not create a cronjob to launch ((( — Reply to this email directly, view it on GitHub <#4116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVDWBHGN52UEFEBXP2755DWLCDBRANCNFSM4QDK6PPA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bygui86]

@SergeyLadutko it would be a very useful feature for us as well, so I prepared a prototype in Lua script and I will test it in the next days.

@reggie-k Maybe I can open a MR adding it to the resource_customizations folder for now and then let's see :)

[SergeyLadutko]

@bygui86 This is very good news, thank you!!!

[crenshaw-dev]

@bygui86 I'd be interested to see the PoC. How are you getting the Lua script to create a new resource? iirc, those can only modify the existing resource.

[bygui86]

@crenshaw-dev It's only a PoC for now, will test it in the next days 🤞🏼 and let you know

[bygui86]

@crenshaw-dev I was able to test it in minikube and unfortunately it doesn't work :( I tried to simply create a new Job object starting from CronJob definition (obj in Lua), but of course I was wrong!
As you said, it seems that ArgoCD can only modify the existing resource on which you trigger the action.
Actually it looked strange that nobody was able to come up with such feature so easily, but it's not written anywhere in the documentation that "ArgoCD can only modify the existing resource on which you trigger the action".
So sad...
Sorry to all for a false hope 👎🏼
@reggie-k let me know if you need my help and how you want to proceed :)

[bygui86]

ArgoCD error in logs:

{
    "error": "rpc error: code = NotFound desc = the server could not find the requested resource",
    "grpc.code": "NotFound",
    "grpc.method": "RunResourceAction",
    "grpc.service": "application.ApplicationService",
    "grpc.start_time": "2022-12-01T14:03:32Z",
    "grpc.time_ms": 35.879,
    "level": "info",
    "msg": "finished unary call with code NotFound",
    "span.kind": "server",
    "system": "grpc",
    "time": "2022-12-01T14:03:32Z"
}

[reggie-k]

@crenshaw-dev I was able to test it in minikube and unfortunately it doesn't work :( I tried to simply create a new Job object starting from CronJob definition (obj in Lua), but of course I was wrong! As you said, it seems that ArgoCD can only modify the existing resource on which you trigger the action. Actually it looked strange that nobody was able to come up with such feature so easily, but it's not written anywhere in the documentation that "ArgoCD can only modify the existing resource on which you trigger the action". So sad... Sorry to all for a false hope 👎🏼 @reggie-k let me know if you need my help and how you want to proceed :)

@bygui86
What I did on my local branch so far was adding a new dummy button for this CronJob resource type.
Next two things are :

1. Generating a new k8s Job resource using a template file from Lua code
2. Making the API support syncing this resource. The resource is not part of what exists in the Git repo for the app, so the direction would be something like how sync --local works.

I can take the first part, unless you have progressed with that, in that case, I will take the second one.

[bygui86]

@reggie-k actually I have a Lua script to generate a Job from a CronJob :)
here the code:

local os = require("os")
job = {}
job.apiVersion = "batch/v1"
job.kind = "Job"
job.metadata = {}
job.metadata.name = cronjob.spec.jobTemplate.metadata.name .. os.date("!%Y%m%d%H%M")
job.spec = {}
job.spec = cronjob.spec.jobTemplate.spec
-- optional
job.metadata.annotations = {}
job.metadata.annotations["kubectl.kubernetes.io/createdAt"] = os.date("!%Y-%m-%dT%XZ")
return job

do you think you can use it?

[SergeyLadutko]

Hi @bygui86, I added this code to the configmap , but I didn't see the cronjob restart button (((Maybe I'm doing something wrong?

    discovery.lua: |
      local os = require("os")
      job = {}
      job.apiVersion = "batch/v1"
      job.kind = "Job"
      job.metadata = {}
      job.metadata.name = cronjob.spec.jobTemplate.metadata.name .. os.date("!%Y%m%d%H%M")
      job.spec = {}
      job.spec = cronjob.spec.jobTemplate.spec
      -- optional
      job.metadata.annotations = {}
      job.metadata.annotations["kubectl.kubernetes.io/createdAt"] = os.date("!%Y-%m-%dT%XZ")
      return job```

[reggie-k]

Thanks! I think it would be better to have a template file and just fill it with values. A template file is more readable, easier to test with and easier to change if k8s api changes. Going on a vacation, will continue working on filling the template file next week.

…

On Sat, Dec 10, 2022, 16:49 Matteo Baiguini ***@***.***> wrote: @reggie-k <https://github.com/reggie-k> actually I have a Lua script to generate a Job from a CronJob :) here the code: local os = require("os") job = {} job.apiVersion = "batch/v1" job.kind = "Job" job.metadata = {} job.metadata.name = cronjob.spec.jobTemplate.metadata.name .. os.date("!%Y%m%d%H%M") job.spec = {} job.spec = cronjob.spec.jobTemplate.spec-- optional job.metadata.annotations = {} job.metadata.annotations["kubectl.kubernetes.io/createdAt"] = os.date("!%Y-%m-%dT%XZ")return job do you think you can use it? — Reply to this email directly, view it on GitHub <#4116 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVDWBERNGDILU27LIKMCL3WMSJ7TANCNFSM4QDK6PPA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bygui86]

It depends how you would like to implement this feature.
For me no difference between a lua script or a yaml template to fill ;)

[reggie-k]

@jessesuen @crenshaw-dev a question for after the holidays :)
Looks like in order to work with files from Lua actions I need the io library enabled (to create the new manifest).
In util.lua.oslib_safe.go it is stated explicitly that working with files is not exposed to custom actions for security reasons.

Another option would be returning the new object from the custom action and then saving it as a temp file from the code that invokes lua, but not sure how would this align with the vision of lua scripts returning only the object they act upon?
Maybe a new kind of custom action, allowing to return a different object, should be defined?

Another workaround (not a pretty one) might be in lua putting the newly created manifest yaml string in the annotation on the original object, returning the original object as usual and then dealing with the new manifest creation outside of lua.

Or in lua annotating the original cronjob object with something like "create-job-from-cron-job", and then the code that invokes the action will create the k8s resource imperatively by invoking k8s api, without going through ArgoCD sync at all.

[reggie-k]

@muenchdo @genslein
Basically, the result of performing a "kubectl create job --from=cronjob/foo foo" is : k8s creating a job resource immediately.
The job manifest is yet another k8s resource.

My suggestion: handling the imperative "kubectl create" result as a declarative representation of this one-time job resource and dealing with it in a regular GitOps way.

Can the following work for you:

• Running kubectl create job --from=cronjob/foo foo on a test cluster you have access to
• Taking the created manifest and cleaning it from k8s runtime-added fields
• Putting it in the repo alongside the original CronJob resource
• Letting ArgoCD sync it regularly
• After tests finished, deleting it from the repo

This worked for me, attaching the original CronJob and the Job manifest which I copied from the cluster and cleaned up:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "* * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox:1.28
            imagePullPolicy: IfNotPresent
            command:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure`

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-27863954
spec:
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        job-name: hello-27863954
    spec:
      containers:
        - command:
            - /bin/sh
            - '-c'
            - date; echo Hello from the Kubernetes cluster
          image: 'busybox:1.28'
          imagePullPolicy: IfNotPresent
          name: hello
      restartPolicy: OnFailure

[bygui86]

@SergeyLadutko exactly the same as I commented above.
I think that changing the repo adding another yaml file is not the best. It would be better just to create a Job in the target K8s cluster and that's it.

@reggie-k A good example is the button "suspend/resume" right for CronJob, you can find the example here. The custom action doesn't change anything in the source repo, acts just on the CronJob deployed in K8s cluster.

[Talador12]

I was about to create a ticket Submit a kubernetes CronJob resource in ArgoCD UI

This is not a complicated ask and should be prioritized

[otherguy]

I think #12174 is what is needed first. Is there any progress on this?

[otherguy]

@reggie-k actually I have a Lua script to generate a Job from a CronJob :) here the code:

local os = require("os")
job = {}
job.apiVersion = "batch/v1"
job.kind = "Job"
job.metadata = {}
job.metadata.name = cronjob.spec.jobTemplate.metadata.name .. os.date("!%Y%m%d%H%M")
job.spec = {}
job.spec = cronjob.spec.jobTemplate.spec
-- optional
job.metadata.annotations = {}
job.metadata.annotations["kubectl.kubernetes.io/createdAt"] = os.date("!%Y-%m-%dT%XZ")
return job

do you think you can use it?

I tried that but I'm getting:

Unable to execute resource action: error patching resource: the server could not find the requested resource

I assume it's because it can't create new resources, right?

[bygui86]

@otherguy exactly, I got the same error. That's why @reggie-k created the task #12174

[SergeyLadutko]

Hi guys, the development seems to be going well, can anyone tell me when it will be released ?

[crenshaw-dev]

Barring anything wild going wrong, we'll release this with 2.8-rc1 next Monday (June 26).

It will also include the ability to create Workflows from CronWorkflows and WorkflowTemplates.

[crenshaw-dev]

Round of applause for @reggie-k for knocking out this feature and closing the 3rd highest 👍'd issue in the repo!

[otherguy]

That is great news!

Thank you @reggie-k (and @crenshaw-dev for getting this merged)!

Looking forward to testing this!

[otherguy]

I had to try out the new release as soon as I saw it 😄

Unfortunately, I found a small bug with it 🐞 We rely on an annotation pod.kubernetes.io/sidecars on our jobs for correct termination (provided by otherguy/k8s-controller-sidecars). This is applied to jobs via jobTemplate like so:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: testjob
  namespace: test
spec:
  jobTemplate:
    spec:
      template:
        metadata:
          annotations:
            pod.kubernetes.io/sidecars: cloud-sql-proxy
        spec:
          containers:
            ...

Unfortunately, when creating a Job from a CronJob with the new, built in create-job action, metadata from the jobTemplate is ignored.

[crenshaw-dev]

@otherguy good catch! We copy over the spec but not the metadata.

argo-cd/resource_customizations/batch/CronJob/actions/create-job/action.lua

Line 50 in 97906bb

job.spec.template.spec = deepCopy(obj.spec.jobTemplate.spec.template.spec)

Would you mind opening an issue? Also, if you're up for a PR, I'd be happy to review. :-)

[otherguy]

@otherguy good catch! We copy over the spec but not the metadata.

argo-cd/resource_customizations/batch/CronJob/actions/create-job/action.lua

Line 50 in 97906bb

job.spec.template.spec = deepCopy(obj.spec.jobTemplate.spec.template.spec)

Would you mind opening an issue? Also, if you're up for a PR, I'd be happy to review. :-)

Of course: #14232

I don't think I will have time for a PR because it should also have test coverage and probably not copy all metadata.

I did mention these concerns in the issue so I hope it's a quick fix and maybe can even go into 2.8.0 final!

[ArieLevs]

what would be the argocd policy rbac equivalent to:

rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["create"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get"]

my motivation here is to create a policy that the only thing it allows a user/group to do is to be able to use the Create Job feature (and nothing else)

[SergeyLadutko]

Hi @ArieLevs, if I understand you correctly, you should add:
action/batch/CronJob/create-job

To policy rulles in argoCD

[ArieLevs]

@SergeyLadutko thanks a lot! this totally did the trick 👍

btw, I'm controlling roles via the argocd helm chart, so I added

p, role:role-name-here, applications, action/batch/CronJob/create-job, project-name/*, allow

then attached this role to the relevant group, but I do not see this role in argocd UI similar (under the project-name) guess a sync bug between the argocd-rbac-cm configmap and the UI

[vyrwu]

@ArieLevs Did you manage to solve this? I'm hitting the same issue.

[ArieLevs]

@vyrwu
yes what @SergeyLadutko suggest totally worked,
I'm adding the policy under this key, which will basically add it to the argocd-rbac-cm config map.

I'm not doing any updates via the UI to keep on a GitOps approach,
while if I browse to the roles location under the relevant project here https://argocd.server.address.example/settings/projects/project_name_here?tab=roles this policy is not there

[crenshaw-dev]

RBAC configured in argocd-rbac-cm, even if it references a particular project, will not appear in the project interface.

Only RBAC configured directly on the AppProject resource will appear in the settings/projects/project_name UI.

[ArieLevs]

@crenshaw-dev thanks, was not even aware of that, thought its a bug 😅
@vyrwu perhaps you can add it directly to the project as described here to make it work for you

[vyrwu]

It worked 🥳 Thx so much for support @ArieLevs and @crenshaw-dev.

Really wish the RBAC set via rbac.csv on the chart was inspectable via UI, but it works. ✌️