Initial workflow (OWNERS, nodes, node channels, etc.)

[wking]

Still need to work out edge handling, but here's what I have so far for folks to look at. CC @lucab, @steveej

[lucab]

Two high level questions on this:

1. do you plan for Cincinnati to consume all the JSON blobs directly, or do you plan to have an additional aggregation/transformation step in-between? (I'll have further followups based on this)
2. to avoid confusion, it may be useful to replace the "cincinnati-" prefix in this repo name with something openshift specific. The equivalent of this for Fedora CoreOS (FCOS) is in a repo named "fedora-coreos-streams", under updates/

[wking]

do you plan for Cincinnati to consume all the JSON blobs directly, or do you plan to have an additional aggregation/transformation step in-between? (I'll have further followups based on this)

The latter. A postmerge prow hook will collect all of this, massage it into a single JSON blob, and publish to somewhere outside of GitHub for Cincinnati to consume (S3? An app-sre ConfigMap? Quay labels?).

to avoid confusion, it may be useful to replace the "cincinnati-" prefix in this repo name with something openshift specific.

The org name is OpenShift-specific, and folks who need to distinguish between multiple Cincinnati can rename of fork (e.g. my fork of openshift/release is wking/openshift-release).

But on both points, the repo is new. Push back if I'm not convincing ;).

[lucab]

All good.

On the former, it's nice to have a massaged format. It'd be good to keep that closer to the way we query the key-value quay API.
(Perhaps you can even think of making this format human-friendlier if people are supposed to write entries by hand, like YAML or TOML)

On the latter, it's just a minor naming concern. Cincinnati is originally the name of the protocol replacing Omaha. We are slowly conflating implementation AND metadata format under that name now. But only specifically for openshift API. However, there are already other implementations of the protocol (e.g. the endpoint for nightlies) and other metadata formats (e.g. the FCOS one).

[wking]

Perhaps you can even think of making this format human-friendlier if people are supposed to write entries by hand, like YAML or TOML

JSON looks like Python dicts and JavaScript objects, so it can't be too human-unfriendly ;). But sure, as long as a YAML/TOML parser is an acceptable dependency for the massaging/CI tools, I'd be ok with that. I'm fine deferring to the release managers; it should be easy to change later.

However, there are already other implementations of the protocol (e.g. the endpoint for nightlies)...

My long-term goal is to get nightlies into this flow and serve a graph for nightly-4.1, etc., out of this repo and the production OpenShift Cincinnati.

... and other metadata formats (e.g. the FCOS one).

I'd expect FCOS metadata to flow out of repos under other orgs. But no sense in diverging data formats. I'll take a closer look at the format in that repo.

[wking]

Checking the FCOS node definition, it looks like most of the time is spent distinguishing between multiple arches and formats. These rolls will eventually be managed in OpenShift with multi-arch images, so I don't think we need to be quite so elaborate, especially out of the gate when we are amd64-only. Do you think the FCOS feeder and this Cincinnati feeder are consuming different-enough artifacts to motivate different input formats? Or do you think my current nodes/ structure should be adjusted somehow?

[wking]

I've pushed up bbcf2d8 adding edge handling and doing some other polishing.

[wking]

I've pushed a WIP script for publishing this data to Quay labels, which we can use until we work out a better way to feed it into Cincinnati. Doesn't actually have POST/DELETE support yet, but you can run it to see what it would change:

$ hack/push-to-quay.py 
changing 4.1.0-rc.6 previous.add from [] to [u'4.1.0-rc.5']
(u'4.1.0-rc.6', 'FIXME post', 'https://quay.io/api/v1/repository/openshift-release-dev/ocp-release/manifest/sha256:1dabe42b5c94841fd8736d8f3a80afeaf5f5ad3833cef8d304c419a97b0efbc3/labels')
(u'4.1.0-rc.5', 'FIXME delete', 'https://quay.io/api/v1/repository/openshift-release-dev/ocp-release/manifest/sha256:dc67ad5edd91ca48402309fe0629593e5ae3333435ef8d0bc52c2b62ca725021/labels/io.openshift.upgrades.graph.next.add')
changing 4.1.0-rc.8 previous.add from [u'4.1.0-rc.7'] to []
(u'4.1.0-rc.8', 'FIXME delete', 'https://quay.io/api/v1/repository/openshift-release-dev/ocp-release/manifest/sha256:8250bbf79d4f567e24a7b85795aba97f9e75a9df18738891a0cb6ba42e422584/labels/io.openshift.upgrades.graph.previous.add')

That's replacing the next.add 4.1.0-rc.5 -> 4.1.0-rc.6 with a previous.add and removing the redundant 4.1.0-rc.7->4.1.0-rc.8 label, which makes sense.

[wking]

Ok, fixed up the POST and DELETE handling (although I don't have creds to actually run those, so there are a handful of untested lines in the script. That should be all we need to start using this repo to manage the production graph metadata, vs. directly editing Quay tags. I'll start a follow-up branch to add CI checks to catch things like "listed version does not exist", impossible edge, cyclic graph, version removed from channel, etc. And once this branch lands, I'll add a postsubmit job to Prow to automatically publish after a PR merges.

[lucab]

@wking the metadata you were looking at for FCOS is only used for boot-images. The relevant ones for Cincinnati are (example from the testing stream):

• release-index: describes the set of nodes in the graph
• updates-metadata: describes (cacheable) edges in the graph

Do you think the FCOS feeder and this Cincinnati feeder are consuming different-enough artifacts to motivate different input formats? Or do you think my current nodes/ structure should be adjusted somehow?

I think it is fine if they diverge, just let's try to keep the overall approach similar. Relevant semantic differences in FCOS are:

• there is a graph per stream. Clients cannot automatically jump across streams via auto-updates
• there is a single graph for all architectures in a stream. That is, update paths are not architecture dependent. Some nodes may be missing for a specific arch, though.

[wking]

• updates-metadata: describes (cacheable) edges in the graph

How does this represent edges? It has a releases array, but each entry in that array only mentions one version. Is that the to version? What is the from version?

[lucab]

Correct, a rollout only specifies the to version.
The from set is composed of all the releases that come before it (in the release-index array) and are not deadends.
(There is another special case for enforcing barriers/chokepoints, but we haven't finalized/tested it yet).

You can observe the current FCOS testing graph here:

curl -H 'Accept: application/json' 'https://updates.coreos.stg.fedoraproject.org/v1/graph?basearch=x86_64&stream=testing&rollout_wariness=0'

Note: all of these are moving pieces as of today, full details will be in a public repo once settled.

[lucab]

Additionally, we just triggered a 24h-spread rollout right now, procedure is coreos/fedora-coreos-streams#17.

[wking]

The from set is composed of all the releases that come before it (in the release-index array) and are not deadends.

Say a user upgrades from A->B, and then some flaw is discovered in B and you mark it a dead-end. How do they get off? With explicit from/to you can tune this sort of thing directly, and tooling can automate the tedium (e.g. see 1dafd8c).