Lower FCOS S3 bucket prefix into config.yaml #737
Conversation
There was a problem hiding this comment.
I wonder if there is a better way to achieve the goal here. To me if we change the top level s3_bucket to have a bucket+path then I'd expect the pipeline to not touch anything outside of that path in the bucket, which clearly isn't true (see sync-stream-metadata and also maybe the plume call in release).
Maybe what we should do here instead is separate things out a bit more:
s3:
bucket: fcos-builds
builds_path: prod/streams/${STREAM}
release_metadata_path: releases
Obviously the stuff that happens in sync-stream-metadata is FCOS only which is why we're ok breaking the cleanliness there, but it might be nice to have the relationship be clearer.
Yeah, I pretty much optimized it for the jobs which are in common between FCOS and RHCOS. It didn't seem valuable to try to extend the config schema right now for prod FCOS-specific things. Re. |
jlebon
force-pushed
the
s3-bucket-prefix
branch
from
738a54f
to
69a1bd4
Compare
jlebon
force-pushed
the
s3-bucket-prefix
branch
from
69a1bd4
to
86ae02c
Compare
jlebon
force-pushed
the
s3-bucket-prefix
branch
2 times, most recently
from
acb4862
to
74dd4e8
Compare
The `prod/streams/${params.STREAM}` is an FCOS thing. Let's have the
`s3_bucket` key in the `config.yaml` also incorporate the subpath.
This will allow the RHCOS config to specify its own subpath and is prep
for hotfix builds specifying a different subpath.
This parameter dates from the very start of kola jobs. At that time, we didn't pass in the stream itself as a separate parameter, so we passed the whole S3 stream dir instead. Nowadays, we do pass in the stream, and all kola jobs correctly query the pipecfg for that info. So clean things up by deleting the unnecessary parameter.
jlebon
force-pushed
the
s3-bucket-prefix
branch
from
74dd4e8
to
5bc2d0b
Compare
In the release job, we may not be releasing the latest build but our use of `cosa meta` assumed so. Pass `--build` to be more strict. We currently only call `run_cloud_tests()` from contexts where it's reasonable to expect the target version is the latest build, but as it's structured as a standalone function, also pass `--build` there to be safe.
The
prod/streams/${params.STREAM}is an FCOS thing. Let's have thes3_bucketkey in theconfig.yamlalso incorporate the subpath.This will allow the RHCOS config to specify its own subpath and is prep for hotfix builds specifying a different subpath.