Handle creating a mix that touches or skips a format bump in Clear Linux

[cmarcelo]

Mixer should be aware about the format bumps in the upstream (Clear Linux), since right now it might be producing a stream that is not usable (skipping a bump in upstream, leading to some invalid state).

"Handle" here might refer first to "be aware and fail if the input is invalid" and then to "create the intermediate mixes as needed", so might make sense implement this in steps. Just the warning would go great length to avoid issues.

See also #30 description that have a concrete example. That PR added more information with the intent of making life easy to identify this bumps.

For the record @mdhorn implemented this outside mixer for our test mix.

[cmarcelo]

@tmarcu I know this was in your Roadmap, added to GitHub so we have a way to keep track (and comment) on this, since we are doing the format bump work from outside the mixer at the moment. Any updates on this?

[kevincwells]

@tmarcu -- wanted to make an observation about how we might implement this, with regards to when we attempt to deal with format bumps and how we expose this to the user.

I was talking with @cmarcelo about steering the CLI away from "what tasks does the tool need to do" and more toward "what goals does the user want to accomplish". With that in mind, we run into the need for auto format bumps when the user wants to update the upstream version their mix is based on, and that version happens to be one or more formats above the old one. As such, I would recommend attempting to do the auto-bump at the time the user changes their upstream CLR version, rather than as a reaction to a new CLR version at a later event (like a build or a deliberate format bump).

This has the nice benefit of tying the format bump activities to the user's action that actually causes their need, as well as simplifying the already complicated output of a later build. (Imagine explaining to the user "You wanted to do a build, but at some other point you changed something else, so we tried to do these other builds you didn't ask for because you didn't know you needed them, and somewhere it all went wrong!" Something along the lines of "Warning: Updating to upstream version 19350 requires additional format bump builds to keep up with changes to upstream." is way easier to understand.)

Right now it's not possible easy to detect "the time the user changes their upstream CLR version", because it's just the .clearversion file; this is another reason why a command like mixer config set UpstreamVersion 19350 (or whatever it ends up looking like) is useful.

Not sure what your thoughts on this were already, but wanted to put that out there.

[cmarcelo]

Proposal on how we get this moving forward:

• Start with Fail when the user is trying to create a mix that breaks format bumps #145 so we stop producing "invalid" mixes.
• Teach mixer how to create a "bump" pair of versions assuming no bump in upstream (the bump is only in the mix). If we are going to mixversion N, this creates N+10 in format F+1, copy it back to N, doing some filesystem fixes.
• Teach mixer how to create a bump if we are at the edge of the bump. So if upstream bumps in U/U+1, and the mix is in U-1, this will automatically create a mix to U+1 with a mix bump. So will generate two versions N/N+1 that follow the upstream bump.

With these in place, it will be easier to think about a more generalized automatic format bumps, or how to improve the UI to direct the user better.

[cmarcelo]

@tmarcu @kevincwells makes sense?

[matthewrsj]

@tmarcu is putting together a proposal involving containerizing mixer and publishing a dockerfile as part of the build collateral. This approach will allow users to create any number of mixes to get to the latest format and handle the bumps for them via a simple script that loops running the dockerfile for each format. tmarcu should explain more when the proposal is fully hashed out.

[cmarcelo]

@tmarcu is putting together a proposal

OK. Let's wait on that for the big picture. Meanwhile I think that #145 (or a variant of it that just warns) would be something acceptable. What do you think?

...via a simple script that loops...

Just a note: we've done the approach of handling format bumps from the outside (Mark's work mentioned above), and we found out that we were reimplementing in bash some logic and information gathering already done by mixer.

[tmarcu]

Proposal and architecture have been reviewed, and issues are being tracked for this on github now (see #203). Adding this to the board.

[tmarcu]

Merged but awaiting major release for default usage.