vdk-cicd: during a scheduled run publish_artifacts and release shouldn't run

[murphp15]

Why

We want to be able to run scheduled jobs.
But as part of the scheduled jobs we want to skip publish_artifact jobs and also release jobs.

This change should not have any impact on the pipeline and I will start running the scheduled jobs once we see the pipeline is still green for merge triggered builds.

In the future I might extract the rule into a shared abstract job. But that will be a later PR

What

Add a guard on all publish_artifact jobs and also release jobs so that they won't run if it is triggered by a scheduled operation.
I also upgraded helm if we are testing everything we should try upgrade that too. The hope here is that it fixes #1252.

How has this been tested

Not much, mostly visual inspection.

closes #1254

[antoniivanov]

Please assign the #1254 to yourself. And linked it in teh ticket.

[ivakoleva]

could explain in the PR description what are the changes expected?
for example, I am wondering why the vdk-heartbeat publish/release stages are handled, and the vdk-control-cli are not?

[murphp15]

could explain in the PR description what are the changes expected? for example, I am wondering why the vdk-heartbeat publish/release stages are handled, and the vdk-control-cli are not?

Fixed

[ivakoleva]

Still don't understand why some gitlab ci yaml files are changed, and some are not.....for example https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-control-cli/.gitlab-ci.yml#L86 ?

[ivakoleva]

for convenience, you can also link the gitlab schedule created, to save time (I went to find it on my own)

[murphp15]

for convenience, you can also link the gitlab schedule created, to save time (I went to find it on my own)

the schedule is not actually created yet.
that schedule you are looking at it for a branch and it does a pretty bad job of testing it.
I will create the schedule when I know I have not introduced regressions in the current builds.

[murphp15]

Still don't understand why some gitlab ci yaml files are changed, and some are not.....for example https://github.com/vmware/versatile-data-kit/blob/main/projects/vdk-control-cli/.gitlab-ci.yml#L86 ?

It is changed, but I only changed it after you pointed it out.
Do you see other examples where I am not doing it?

[antoniivanov]

Is it possible to configure the rule to be "opt-in" - run on schedule only the jobs marked as such.
Instead of opt-out as it's now (excluding jobs) ?

With opt-in , it's easier to start with smaller set of jobs and add more incrementally until we are happy

[murphp15]

Is it possible to configure the rule to be "opt-in" - run on schedule only the jobs marked as such. Instead of opt-out as it's now (excluding jobs) ?

With opt-in , it's easier to start with smaller set of jobs and add more incrementally until we are happy

IMO opt out is better so we are testing more by default.

I think the problem with current setup with both my solution and your suggested solution is that it touches soo many lines of code.
I think we should try make an abstract job for each stage. all jobs in that stage should extend it.
That way it is super easy to skip an entire stage.

However I only want to try that after I have identified all the edge cases etc..

Let me know what you think?

[antoniivanov]

Is it possible to configure the rule to be "opt-in" - run on schedule only the jobs marked as such. Instead of opt-out as it's now (excluding jobs) ?
With opt-in , it's easier to start with smaller set of jobs and add more incrementally until we are happy

IMO opt out is better so we are testing more by default.

I think the problem with current setup with both my solution and your suggested solution is that it touches soo many lines of code. I think we should try make an abstract job for each stage. all jobs in that stage should extend it. That way it is super easy to skip an entire stage.

However I only want to try that after I have identified all the edge cases etc..

Let me know what you think?

With opt-out if we add a new component/plugin build job, it would be automatically included in the scheduled nightly build. Which is a good thing.

But if we add a new release job then it would be automatically included which is not what we want.

But if we can extend a stage, as you suggest, then we can opt out "release" stage. Unless somebody releases in the wrong stages, there's no problem with wrong jobs being included.

So I like this..

OK. Let's go with the proposed plan. First let's identify the what jobs need to be included and what not.
Then we can refactor to support inheriting abstract job for each stage. Which would likely be useful for other purposes as well.

[antoniivanov]

Looks good to me.

Since there's some risk. But I'd suggest that You do not merge it soon before clocking out and monitor the pipeline afterward. It's very unlikely that the new rules would break something but you never know