Explore `ytt` for our Concourse pipelines #209

gcapizzi · 2021-04-22T17:51:16Z

At the moment our Concourse pipelines are built on top of Aviator which in turn is based on Spruce.

The current solution features a lot of indirection, with aviator.yml files glueing lots of small files together and no place showing the structure of the pipeline. We load config values by merging a config section to the YAML (which is ignored by Concourse) and then grab them.

I propose we try to use ytt to simplify our pipelines without losing the possibility to reuse jobs and resources. Interesting features to look at:

data.read() to inject the content of a text file (could be used instead of file e.g. here).
@data/values to load config values (instead of having a config block and grab like here).
Functions to abstract YAML fragments with parameters. I suspect this could replace many config values.

The text was updated successfully, but these errors were encountered:

wip

[cloudfoundry/eirini-release#209] wip

wip

[cloudfoundry/eirini-release#209] Co-authored-by: Kieron Browne <kbrowne@vmware.com>

instance_index_env_injector and resource_validator [#206] Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: #206 Issue: #209

[cloudfoundry/eirini-release#209] wip Co-authored-by: Kieron Browne <kbrowne@vmware.com>

[cloudfoundry/eirini-release#209] wip Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io>

[cloudfoundry/eirini-release#209] wip Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

[cloudfoundry/eirini-release#209] wip Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Mario Nitchev <marionitchev@gmail.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209 Co-authored-by: Mario Nitchev <marionitchev@gmail.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Co-authored-by: Mario Nitchev <marionitchev@gmail.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209 Co-authored-by: Mario Nitchev <marionitchev@gmail.com> Issue: cloudfoundry/eirini-release#209

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

Issue: cloudfoundry/eirini-release#209

Issue: cloudfoundry/eirini-release#209

In CI and Acceptance pipelines Issue: cloudfoundry/eirini-release#209

Issue: cloudfoundry/eirini-release#209

And export those libs in structs to make loading require a single resource id. Issue: cloudfoundry/eirini-release#209

It was accidentally removed Issue: cloudfoundry/eirini-release#209

This controls whether CF is automatically deleted and redeployed on a timer. Issue: cloudfoundry/eirini-release#209

Issue: cloudfoundry/eirini-release#209

kieron-dev · 2021-05-14T15:23:40Z

All pipelines migrated to use YTT templating. Old aviator config removed.

[cloudfoundry/eirini-release#209] Co-authored-by: Kieron Browne <kbrowne@vmware.com>

gcapizzi added the chore label Apr 22, 2021

gcapizzi self-assigned this Apr 22, 2021

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue Apr 27, 2021

[cloudfoundry/eirini-release#209]

1beb848

wip

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue Apr 27, 2021

[cloudfoundry/eirini-release#209]

596e2e3

wip

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue Apr 27, 2021

[cloudfoundry/eirini-release#209]

2f6694c

[cloudfoundry/eirini-release#209] wip

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue Apr 27, 2021

[cloudfoundry/eirini-release#209]

30c28f8

[cloudfoundry/eirini-release#209] wip

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue Apr 27, 2021

[cloudfoundry/eirini-release#209]

5cd4c3d

wip

gcapizzi assigned kieron-dev Apr 27, 2021

gcapizzi added a commit to eirini-forks/cf-for-k8s that referenced this issue Apr 28, 2021

Rename Eirini Secret to use the "eirini-" prefix

5645e9e

[cloudfoundry/eirini-release#209] Co-authored-by: Kieron Browne <kbrowne@vmware.com>

gcapizzi added a commit that referenced this issue Apr 28, 2021

Rename certificate secrets prefixing with eirini-

7383f80

instance_index_env_injector and resource_validator [#206] Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: #206 Issue: #209

gcapizzi unassigned gcapizzi and kieron-dev Apr 28, 2021

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 6, 2021

[cloudfoundry/eirini-release#209]

5f9276f

[cloudfoundry/eirini-release#209] wip Co-authored-by: Kieron Browne <kbrowne@vmware.com>

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 6, 2021

[cloudfoundry/eirini-release#209]

5b54996

[cloudfoundry/eirini-release#209] wip Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io>

gcapizzi assigned gcapizzi and mnitchev May 10, 2021

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

[cloudfoundry/eirini-release#209]

dd270d2

[cloudfoundry/eirini-release#209] wip Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

[cloudfoundry/eirini-release#209]

43dd443

[cloudfoundry/eirini-release#209] wip Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

wip

5ff8b6b

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

Complete the CI pipeline

0dad206

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

Move component list to config.yml

dbf2911

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

Move directories to share common libs

974e857

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

Add the ci-images pipeline as ytt

c98e4fa

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

Finish ci-images pipeline

e46dfc5

Co-authored-by: Mario Nitchev <marionitchev@gmail.com> Issue: cloudfoundry/eirini-release#209

gcapizzi added a commit to cloudfoundry/eirini-ci that referenced this issue May 10, 2021

Move git config back to the pipeline

4044237

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

gcapizzi removed their assignment May 11, 2021

mnitchev added a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Move git config back to the pipeline

06de2bc

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

mnitchev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Add the CI pipeline as ytt templates

4e1d95c

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

mnitchev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Move component list to config.yml

981dbac

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Issue: cloudfoundry/eirini-release#209

mnitchev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Move directories to share common libs

30dd760

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

mnitchev added a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Add the ci-images pipeline as ytt

04cf8f0

Co-authored-by: Kieron Browne <kbrowne@vmware.com> Co-authored-by: Mario Nitchev <marionitchev@gmail.com> Issue: cloudfoundry/eirini-release#209

mnitchev added a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Move git config back to the pipeline

f7186c5

Co-authored-by: Giuseppe Capizzi <gcapizzi@pivotal.io> Issue: cloudfoundry/eirini-release#209

mnitchev added a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Fix shellcheck issues

dcfbe34

Issue: cloudfoundry/eirini-release#209

mnitchev added a commit to cloudfoundry/eirini-ci that referenced this issue May 11, 2021

Fix next set of shellcheck issues

aa8f007

Issue: cloudfoundry/eirini-release#209

mnitchev added a commit to cloudfoundry/eirini-ci that referenced this issue May 12, 2021

Add the q-check pipeline as ytt templates

5a7d028

Issue: cloudfoundry/eirini-release#209

kieron-dev assigned kieron-dev and unassigned mnitchev May 12, 2021

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 12, 2021

Convert egg pipeline to YTT

1d8f41e

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 12, 2021

Rename pipeline yamls to pipeline.yml

b2adc51

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 12, 2021

Tidy up egg pipeline

fdecb82

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 12, 2021

Convert acceptance pipeline to ytt

dc4dddc

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 13, 2021

Set gcp.worker_count from bash args

76cbf46

In CI and Acceptance pipelines Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 13, 2021

Ask for a fly login if we're not logged in

8a360c9

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 13, 2021

Convert the cf4k8s generic pipeline to ytt

4c42cf9

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Remove unused imports

a1aa8d5

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Move tasks.lib.yml to separate libs

64c5669

And export those libs in structs to make loading require a single resource id. Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Add back eirini resource to q-check pipeline

0b1af6b

It was accidentally removed Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Deal with the enable-redeploy flag correctly in dev pipelines

2094c6c

This controls whether CF is automatically deleted and redeployed on a timer. Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Allow setting branches of eirini and eirini-ci in ci pipeline

737922e

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Add the set dev-envs and all pipeline scripts

a1e6247

Issue: cloudfoundry/eirini-release#209

kieron-dev pushed a commit to cloudfoundry/eirini-ci that referenced this issue May 14, 2021

Replace aviator CI config with ytt config

a436fac

Issue: cloudfoundry/eirini-release#209

kieron-dev removed their assignment May 17, 2021

gcapizzi closed this as completed May 17, 2021

kieron-dev pushed a commit to eirini-forks/cf-for-k8s that referenced this issue Sep 1, 2021

Rename Eirini Secret to use the "eirini-" prefix

3dfb29a

[cloudfoundry/eirini-release#209] Co-authored-by: Kieron Browne <kbrowne@vmware.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Explore `ytt` for our Concourse pipelines #209

Explore `ytt` for our Concourse pipelines #209

gcapizzi commented Apr 22, 2021 •

edited

Loading

kieron-dev commented May 14, 2021

Explore ytt for our Concourse pipelines #209

Explore ytt for our Concourse pipelines #209

Comments

gcapizzi commented Apr 22, 2021 • edited Loading

kieron-dev commented May 14, 2021

Explore `ytt` for our Concourse pipelines #209

Explore `ytt` for our Concourse pipelines #209

gcapizzi commented Apr 22, 2021 •

edited

Loading