Triggering long running background jobs when events occurs

[pierDipi]

Problem

Usually event processing combined with a Knative Service is expected to complete in a relative short period of time (minutes) as it requires the HTTP connection to stay open as otherwise the service is scaled down and keeping long running connections open increases the possibility of failing and so the processing needs to restart as the request is retried.

This limitation is not ideal, therefore providing a resource (JobSink) which will trigger a long running job when an event occurs might be a good alternative vehicle.

Example JobSink API:

apiVersion: sinks.knative.dev/v1alpha1
kind: JobSink
metadata:
  name: job-sink-success
spec:
  job:
    apiVersion: batch/v1
    kind: Job
    spec:
      completions: 12
      parallelism: 3
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: main
              image: docker.io/library/bash:5
              command: [ "bash" ] 
              args:
                - -c
                - echo "Hello world!" && sleep 5
      backoffLimit: 6
      podFailurePolicy:
        rules:
          - action: FailJob
            onExitCodes:
              containerName: main      # optional
              operator: In             # one of: In, NotIn
              values: [ 42 ]
          - action: Ignore             # one of: Ignore, FailJob, Count
            onPodConditions:
              - type: DisruptionTarget   # indicates Pod disruption

Possible Client UX

We could provide kn client UX which would take a k8s job and turn it into a jobsink as creating a Job might be easier with existing IDE tooling:

kn jobsink create --from-job-file=job.yaml -oyaml > jobsink.yaml

Persona:
Which persona is this feature for?

Developers

Exit Criteria
A measurable (binary) test that would indicate that the problem has been resolved.

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

10

Additional context (optional)
Add any other context about the feature request here.

[fsedano]

/assign

[fsedano]

@pierDipi as discussed in Kubecon -- This is my main use case, so I'd like to work on solving this issue.

[pierDipi]

Thanks @fsedano feel free to go ahead, here is my PoC branch on my fork https://github.com/pierDipi/eventing/tree/jobsink

[pierDipi]

/triage accepted

[fsedano]

Before working on this, I think we should fix an underlying issue being discussed here:

knative/serving#13075

Because of this, the premise "as it requires the HTTP connection to stay open as otherwise the service is scaled down " is not 100% true -- Keeping the HTTP open won't guarantee the pod is not killed.

[Cali0707]

@fsedano I think the idea of this proposal was to create a k8s Job resource, instead of a Knative service (correct me if I'm wrong @pierDipi ). That way, this limitation would not be an issue

[pierDipi]

Yes, that's what the POC also does, it creates a job for each received event. A received event is mounted as a JSON file volume to each job

[mroberts91]

@fsedano, Are you still actively working on this feature? If so, I'd like to know your timeframe.

I see a lot of value in this feature and am trying to determine whether it is worth waiting for its release or writing a one-off job handler for a current use case.

[fsedano]

@mroberts91 before working on this I want to handle the regular UX described here: knative/serving#13075 (which is impacting my use case)

Would that also help you? Or do you need the job route as discussed here?