Difficult to use parallel Tasks that share files using workspace

[jlpettersson]

Expected Behavior

A key feature for a CI-system is to provide fast feedback. Parallelization is one way to provide faster feedback.

I would be interested in an easy way to run parallel "commands" (e.g. steps or tasks), sharing the same workspace volume.

Actual Behavior

Tasks can be run parallel today. But there is a lot of complexity when using a PVC based workspace (using PVC workspace should be the common case).

Clusters can be regional crossing multiple zones. While PVC often is zonal. This is not a problem for most applications, but with Tekton we want to mount the same PVC in multiple Tasks, sequential or parallel.

PVC access mode ReadWriteOnce is most common, and with that pods can only execute in parallel if scheduled to the same node. An access mode ReadWriteMany would be best for Tekton workload, but that is the least common and those solutions are commonly not "cloud native", e.g. based on NFS servers, or custom file system view of buckets.

Proposal

SOLUTION ALTERNATIVES

• Investigate if we could support parallel steps within a Task (a DAG of steps per Task). That way, all containers in the Task is scheduled to the same Node and can share the volume when executed parallel.

  This can still be compatible with the Tasks and Pipelines as how they are today, but it is an additional feature that also allow parallel steps (or a DAG) within the Task, instead of strictly sequential as of today.

  This was the initially proposed solution but it has a major drawback in that it makes Tasks less reusable. In addition, it is a bit tricky to implement. Most comments below is in context of this alternative.

• Introduce a concept of TaskGroup and execute all Tasks part of the same group (a DAG) within a single Pod. This keeps the reusability of Tasks. Is it a major API change and is not easy to implement. A special case, to schedule all Task pods in a PipelineRun within a single Pod was also considered.

• Using Kubernetes Pod affinity was considered difficult to use, since Task pods only lives during task execution and then terminates.

/kind design
/kind feature

[afrittoli]

Results is a feature that has been introduce to allow for PVC-less data share across tasks, at least for parameter / text type of data.

One concern for parallel steps is scheduling, since a Task with multiple parallel steps might be tougher to schedule, even though we could have a maximum parallel steps setting to ease that issue.

Another small concern I have is re-usability of tasks. Since the smallest reusable unit is the Task, a Task with parallel steps may lead to building tasks that accomplish many different tasks and are thus less re-usable.

That said I think it would be really useful to explore this - define concrete use cases and assess benefits and issues.

[GregDritschler]

I'm confused. Steps in a Task map to containers in a Pod. I assume that "parallel steps within a Task" then would map to parallel containers in a Pod. There wouldn't be any scheduling involved for parallel steps. Right?

[bobcatfish]

@jlpettersson could you give some examples of steps you'd want to run in parallel?

[jlpettersson]

Since the smallest reusable unit is the Task

I also see the step container image as a reusable unit. But a full step including parameters is also reusable, but only via copy-and-paste.

define concrete use cases and assess benefits and issues.

The typical use-case is a CI-pipeline that starts with a git-clone and the following steps analyze or generate outputs depending on all files that was cloned from git - hence the usefulness with a shared workspace volume.

e.g. I have a pipeline similar to this in a project - highly parallel for fast feedback:

git-clone -> install dependencies -> build app       -> build artifact -> upload artifact
                                  -> test js
                                  -> lint js
                                  -> validate swagger file

It is beneficial to do all work that use the files from git-clone on the same node (most easily done when within the same Pod), also hopefully parallel to some extent.

I assume that "parallel steps within a Task" then would map to parallel containers in a Pod.

Yes. More exactly, I would love to be able to execute the steps in a DAG within the Pod.

There wouldn't be any scheduling involved for parallel steps.

Only the Pod is scheduled to a Node - once. But something e.g. the entrypoint binary, must coordinate in what order the containers within the pod should execute.

You probably would like to dedicate more CPU resources to this kind of fat Task, but I let that to be a different question.

Tekton is a distributed system compared to older single-node CI-systems. But Tekton would still be a distributed system with fat Tasks. It is just that bigger "container groups" is co-located - but this is beneficial when sharing resources (e.g. workspace files)

[bobcatfish]

Interesting! Thanks for the details. I haven't thought this through very far but a couple of intial reactions:

1. git-clone in particular being a separate task is a shortcoming we've run into as a result of seeing how things go without pipeline resources, but i think we'll have other options here eventually (+ @sbwsg )
2. it seems from your description like what you really want is the graph expressive-ness of Pipelines, but you want the "executing with access to the same disk" features of Tasks. I wonder if "fat tasks" are the best answer here, or if instead we could look into an execution mode for pipelines where all Tasks can more easily share the same disk (e.g. be in teh same node) - that way you can get the reusability of Tasks as they currently exist, the expressiveness of Pipelines and avoid PVCs

[bobcatfish]

Re. (2) -> #617 might be relevant to your interests

[jlpettersson]

Linking task output (result) to task input (params) is a great feature.

But this is more about sharing all files in the workspace, typically a larger number of files from a git-clone operation.

The closest alternative to this is to use pod-affinity as in my comment

I am curious if we can use some kind of pod affinity to get tasks co-located on the same node.

Possibly co-locate all pods belonging to a single PipelineRun so they perfectly fine can use the same PVC as a workspace and perfectly fine can execute parallel. (this is essentially what any single-node CI/CD system does).

We would still be a distributed system where different PipelineRuns possibly scheduled to different nodes. Using different PVCs is "easier" for fan-out, but not for fan-in (e.g. git-clone and then parallel tasks using the same files)

My idea was to co-locate later pods with the first pods scheduled to a node. But I have come to the conclusion that this is very hard to do, especially in a DAG and when some pods terminate.

Therefore I am proposing to investigate if we could execute a Pipeline DAG within a pod instead. That would do it trivial to share a volume/files (e.g. emptyDir or PVC) and also trivial to schedule to the same node, as the whole Pod is located in one node.

it seems from your description like what you really want is the graph expressive-ness of Pipelines, but you want the "executing with access to the same disk" features of Tasks.

Yes, exactly. Essentially it is a way to co-located a group of processes that operate on the same set of files (from git-clone), but potentially executed in order or in parallel specified in a DAG (pipeline).

[jlpettersson]

A use-case from Tekton Catalog is the three golang-tasks: build, lint and test. For fastest feedback, it would be good to execute all three in parallel. They are all three operating on the same set of files, typically originated from a git-clone operation in a CI-pipeline.

However, it is possible alternative ways to implement this. One could be to introduce a TaskGroup so when I put these four tasks (including git-clone) in a TaskGroup in my pipeline, they are all executed within the same Pod - but in a DAG (managed by e.g. entrypoint binary or other solution).

[jlpettersson]

A full example of a Golang CI-pipeline that uses a proposed "TaskGroup":

git-clone -> go-lint -> kaniko-build
          -> go-test
          -> go-build

Example Pipeline definition. I have excempted params for clarity. The taskGroup syntax could be designed differently.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: golang-ci-pipeline
spec:
  tasks:
  - name: git-clone
    taskRef:
      name: git-clone
    taskGroup: ci
  - name: go-build
    runAfter: [git-clone]
    taskRef:
      name: golang-build
    taskGroup: ci
  - name: go-lint
    runAfter: [git-clone]
    taskRef:
      name: golang-lint
    taskGroup: ci
  - name: go-test
    runAfter: [git-clone]
    taskRef:
      name: golang-test
    taskGroup: ci
  - name: kaniko
     runAfter:
     # possibly use runAfter: ['ci'] here instead of individual task names.
     - go-build
     - go-lint
     - go-test
    taskRef:
      name: kaniko

Since all tasks about belong to the same taskGroup: ci they will all be executed within the same Pod.

[jlpettersson]

/assign

[jlpettersson]

I wonder if "fat tasks" are the best answer here, or if instead we could look into an execution mode for pipelines where all Tasks can more easily share the same disk (e.g. be in teh same node) - that way you can get the reusability of Tasks as they currently exist, the expressiveness of Pipelines

@bobcatfish you are right.

With a small trick, by introducing a new component (not user-facing), I found a working solution for this problem using Inter-pod affinity, implemented in #2630