Multiple pipeliens per repository

[laszlocph]

Find a way that repositories can have multiple pipelines triggered in a webhook.

For shorter feedback loop back to Github.

Today in a typical linting - building - testing pipeline the result of a lint only makes back to Github if all steps are done. The goal is to be able to lint in one pipeline, build and test in another one. This way developers can see linting results way quicker.

[laszlocph]

The design that I'm implementing:

The pipeline is either in the .drone.yml file or the .drone folder.

• pipeline location may be configurable in a later iteration
• with the defualt pipeline location, the .drone.yml takes precedence if both the .drone.yml file and the .drone folder exists

For multiple pipelines you must use separate files

• the pipeline file format does not change, each file describes one pipeline with the well known control flows: parallelism, branch and pipeline triggers
• an example .drone folder

.drone
├── lint.yml
├── test.env
├── build.yml
└── deploy.yml

• the different pipelines run in parallel on separate agents
• with custom status lines feeding back to Github
• pipelines can have a depends_on triggers to block the deploy.yml until the prerequisites are finished

With this approach we tackle two problems:

• Drone files growing too big
• Feedback loop is slow on purely lint/test errors

[AkiraNorthstar]

the different pipelines run in parallel on separate agents

Even if a limitation is set to an agent / process?

I understand the thoughts behind and this is very useful to accelerate everything.
However, a specified limitation should not be ignored (maximum parallel instances).
Or do I understand something wrong?

But your idea is thought through.
I like it!

[laszlocph]

If you ask about the per agent limit of DRONE_MAX_PROCS then yes, it is respected.

Continuing on the above example, if you have lint.yml, test.yml and build.yml, and you have a single agent with DRONE_MAX_PROCS=2, then one of the 3 pipelines will wait until there is an available agent. If you have two agents with DRONE_MAX_PROCS=2, then they can all start, and you have spare room for one additional pipeline.

The behavior is exactly the same how matrix builds work today.

[AkiraNorthstar]

That's great! - Thanks for the answer.

[AkiraNorthstar]

I've still have a question or maybe a feature request.
It is possible to have "Named Agent" for different build tasks?

Example:
Fileserver : QuadCore + 8 GB RAM (Intel(R) Pentium(R) CPU G4560 @ 3.50GHz)
Workstation: QuadCore + 16 GB RAM (Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz)

drone server and drone agent are always running on Fileserver...

Default = Fileserver

Some bash logic for demonstrating:

if [ "Workstation" == "online" ] ; then
    DRONE_USE_AGENT="Workstation"
else
    DRONE_USE_AGENT="Fileserver"
fi

I hope that I have shown understandable.
Edit: If this not the right topic, i can create a new issue for this...

[laszlocph]

Distinguishing between agent capabilities is not in scope for this task.

I believe that it's best if agents have identical capabilities in general. Now some cases would require named agents, like when you attach some special hardware to your agent nodes. But even then, the mapping would be static, so the presented if-else logic would not be a feature I would implement.

[AkiraNorthstar]

The if-else logic I have just introduced for more clarity.
It is true what you say that actually all agents should have the same conditions to deliver a reproducible result. I was more concerned with the temporary and faster execution of a build, which could be reduced to a quarter of the time if you run the agent on an another computer.
Thanks again!

[laszlocph]

Docs:
https://github.com/laszlocph/docs/blob/master/content/usage/concepts/multi-pipeline.md

How to try it out on a branch while keep existing builds running?

• Change the "Pipeline Path" in the repo settings to .drone/
• Click "Fallback to .drone.yml if path not exists"
• Add your yamls in the .drone/ folder

It will search for the pipeline files under .drone/ and it will find it on your branch. While for existing branches where there is the existing .drone.yml it will fall back to that file.