Contents of release-v3.28.3.tgz don't match tagged files

[jcstanaway]

We recently upgraded to Calico v3.28.3. Our process is to download the release-v3.28.3.tgz archive and use the manifests contained therein.

Expected Behavior

The manifests in release-v3.28.3.tgz contain all of the manifest versions with the v3.28.3 tag.

Current Behavior

We found (after have a major cluster outage), that at a minimum, the calico.yaml and calico-typha.yaml manifests in the release-v3.28.3.tgz were missing the fix from #9788.

This PR was reported included in the v3.28.3 release as evidenced by the milestone reference in the PR and the list of commit diffs between v3.28.2 and v3.28.3 (v3.28.2...v3.28.3).

Possible Solution

Steps to Reproduce (for bugs)

Upon redownloading release-v3.28.3.tgz and comparing the calico.yaml and calico-typha.yaml manifests between the archive and the v3.28.3 tagged commits of those files, we observed that they were NOT the same and that specifically the fixes included by #9788 were not present in the manifests in the archive.

It is possible that there are other differences between the archive and the tagged commit, but we have not performed an exhaustive comparison.

Context

We had a major outage in a production cluster that severely impact many customers.

Your Environment

• Calico version: v3.28.3
• Calico dataplane (iptables, windows etc.)
• Orchestrator version (e.g. kubernetes, mesos, rkt):
• Operating System and version:
• Link to your project (optional):

[jcstanaway]

Does nobody care that the release archive was not packaged correctly?

[caseydavenport]

@radTuti @danudey could you please investigate this?

[danudey]

@jcstanaway Sorry for the late response on this. I did take a look into this this morning, and I wanted to share what I found. It doesn't seem to line up with your experiences, so I'm not certain what I'm missing in my investigation.

First, I downloaded the release archive from Github and looked into the manifests it contains. The template file which was updated with the permissions contains the comment "KubeControllersConfiguration is where it gets its config" just above the change, so I grepped for that string across the manifests and looked to see if it had the new - list permission, and it seems as though it does, based on this output.

I then checked out the tag for v3.28.3 and compared the manifests there to the manifests in the release archive and they were identical to me.

I'm wondering if you somehow received an incorrect version of the release archive, though I'm not sure how that would have happened. In case you still have a local copy of the tarball, please check that it has an md5 hash of 9c2fc3861ea2d31de4975af59340ac0d, and/or compare the manifest files you have locally to the md5sums I posted in that gist.

From everything I can see here, the manifest file should be correct; I'm very concerned that it, somehow, was not.

[jcstanaway]

Apparently the person on our team that was performing this activity had multiple different versions of the archive downloaded and ended up using the wrong version. I've gone back and verified that the v3.28.3 archive has the expected content.

We apologize for alarm this raised.

[danudey]

@jcstanaway That's a relief, thanks for keeping us updated!