The source tarball for podman 2.2.1 now has different checksum

[afbjorklund]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The checksum of the release tarball changed, causing verification in build scripts to fail.

Steps to reproduce the issue:

1. curl -RLOJ https://github.com/containers/podman/archive/v2.2.1.tar.gz

2. sha256sum podman-2.2.1.tar.gz

Describe the results you received:

3212bad60d945c1169b27da03959f36d92d1d8964645c701a5a82a89118e96d1 podman-2.2.1.tar.gz

Describe the results you expected:

bd86b181251e2308cb52f18410fb52d89df7f130cecf0298bbf9a848fe7daf60 podman-2.2.1.tar.gz

Additional information you deem important (e.g. issue happens only occasionally):

The actual code is functionally the same, but source verification fails.

ERROR: v2.2.1.tar.gz has wrong sha256 hash:
ERROR: expected: bd86b181251e2308cb52f18410fb52d89df7f130cecf0298bbf9a848fe7daf60
ERROR: got     : 3212bad60d945c1169b27da03959f36d92d1d8964645c701a5a82a89118e96d1
ERROR: Incomplete download, or man-in-the-middle (MITM) attack

Output of podman version:

(paste your output here)

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

Buildroot

---

It seems like one of the files in the distribution has some magic git attributes set:

$ more vendor/k8s.io/client-go/pkg/version/.gitattributes 
base.go export-subst

This causes it to replace some of the text, when the git archive is created:

$ tail vendor/k8s.io/client-go/pkg/version/base.go 
	// NOTE: The $Format strings are replaced during 'git archive' thanks to the
	// companion .gitattributes file containing 'export-subst' in this same
	// directory.  See also https://git-scm.com/docs/gitattributes
	gitVersion   string = "v0.0.0-master+$Format:%h$"
	gitCommit    string = "$Format:%H$" // sha1 from git, output of $(git rev-parse HEAD)
	gitTreeState string = ""            // state of git tree, either "clean" or "dirty"

	buildDate string = "1970-01-01T00:00:00Z" // build date in ISO8601 format, output of $(date -u +'%Y-%m-%dT%H:%M:%SZ')
)

And since the number of significant digits in podman changed, it now fails:

58c58
< 	gitVersion   string = "v0.0.0-master+a0d478edea"
---
> 	gitVersion   string = "v0.0.0-master+a0d478edea7"

Note that it is using the podman commit, and not the client-go commit.

	k8s.io/client-go v0.0.0-20190620085101-78d2af792bab

i.e. a0d478e not 78d2af792bab

commit a0d478edea7f775b7ce32f8eb1a01e75374486cb (HEAD, tag: v2.2.1)
Author: Matthew Heon <mheon@redhat.com>
Date:   Mon Dec 7 17:59:05 2020 -0500

commit 78d2af792babf2dd937ba2e2a8d99c753a5eda89 (tag: v12.0.0, tag: kubernetes-1.15.0)
Author: Kubernetes Publisher <k8s-publishing-bot@users.noreply.github.com>
Date:   Thu Jun 20 08:51:01 2019 +0000

[vrothberg]

@cevich PTAL

[cevich]

Whoa, I think I just barely understand what's happening here 😕 I'm not sure what/where the problem and what/where needs fixing (if it's even possible). Does podman just need to vendor a later commit of vendor/k8s.io/client-go (ref. the minikube issue)?

[cevich]

Update: Yeah, this seems like a github.com (source-archive via URI) facility that we have no control over. What am I missing/not understanding?

[afbjorklund]

The "best" would be to remove the vendor/k8s.io/client-go/pkg/version/.gitattributes file,
and do the expansion once and for all in the imported file (for each imported version).

	// NOTE: The $Format strings are replaced during 'git archive' thanks to the
	// companion .gitattributes file containing 'export-subst' in this same
	// directory.  See also https://git-scm.com/docs/gitattributes
	gitVersion   string = "v0.0.0-master+78d2af792"
	gitCommit    string = "78d2af792babf2dd937ba2e2a8d99c753a5eda89" // sha1 from git, output of $(git rev-parse HEAD)
	gitTreeState string = ""            // state of git tree, either "clean" or "dirty"

	buildDate string = "1970-01-01T00:00:00Z" // build date in ISO8601 format, output of $(date -u +'%Y-%m-%dT%H:%M:%SZ')

See https://github.com/kubernetes/client-go/archive/78d2af792babf2dd937ba2e2a8d99c753a5eda89.tar.gz

We can also just close the issue, since it will be "a while" before another digit is added.
But you might want to announce to everyone that all the old checksums were changed ?

---

We did this once before, when the "libpod" directory was changed to "podman":

See #6941

Sometimes I miss distfiles and FTP... You ran make dist and then you checksummed and pushed it

Other projects just delete the whole vendor dir, and git-ignore it. Then it's really "hope for the best", to reproduce.

[afbjorklund]

Here is the same thing in Fedora:

https://src.fedoraproject.org/rpms/podman/blob/f32/f/sources

SHA512 (v2.2.1.tar.gz) = 4c6eda9320daf18b5f0e892c0f3019a5658d983936c25a195bcb1fbc28c18a9a65e87043faddeb47e9f62c9b445efc2906a2b7ccedca877941fadb21b1b68367

It uses more bits and the wrong filename, but the fail is the same:

cf9c1c11d0a83de12a6840536040ab202e8b8d626ec6afd77f85629f72dd3a178a4911e02ffafbe9581df2b6d69524528eb2eb621e7d62ab3a261ada51f184e3 v2.2.1.tar.gz

Luckily, fedora also saves the .src.rpm with the original tarball.

[cevich]

So if I understand correctly, the problem is that the checksum of the source tarball changes depending on *when it's downloaded from the GitHub whiz-bang source-archiver URL This is because a file under the vendor tree is modified by git due to the special-magic .gitattributes file.

@vrothberg if we remove the .gitattributes file, and hard-code the changes into vendor/k8s.io/client-go/pkg/version/base.go will that wreck our 'make vendor' checking?

Seems it will also be a hassle when we need to vendor in a new version, someone will need to remember to manually edit that file 😕

[vrothberg]

So if I understand correctly, the problem is that the checksum of the source tarball changes depending on *when it's downloaded from the GitHub whiz-bang source-archiver URL This is because a file under the vendor tree is modified by git due to the special-magic .gitattributes file.

Wow!

@vrothberg if we remove the .gitattributes file, and hard-code the changes into vendor/k8s.io/client-go/pkg/version/base.go will that wreck our 'make vendor' checking?

Try it :) I think yes but I may be wrong.

[afbjorklund]

If you can live with it being wrong, then it can probably stay just the way it is for some time longer. :-)

Unfortunately it is not really deterministic, when git decides to add another significant digit to describe.

[afbjorklund]

I actually don't know where this version is used. But just like the 70's podman, it's probably ignored ?

// This is the fallback data used when version information from git is not
// provided via go ldflags. It provides an approximation of the Kubernetes
// version for ad-hoc builds (e.g. `go build`) that cannot get the version
// information from git.

	gitVersion   string = "v0.0.0-master+c640670e85c"
	gitCommit    string = "c640670e85c4aaaff92741691d6a854a90229d8d" // sha1 from git, output of $(git rev-parse HEAD)
	gitTreeState string = ""            // state of git tree, either "clean" or "dirty"

	buildDate string = "1970-01-01T00:00:00Z" // build date in ISO8601 format, output of $(date -u +'%Y-%m-%dT%H:%M:%SZ')

Version:      3.0.1
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Not sure exactly where podman would use the Kubernetes version. As far as I know, only tools/utils ?

package remotecommand // import "k8s.io/client-go/tools/remotecommand"

Package remotecommand adds support for executing commands in containers,
with support for separate stdin, stdout, and stderr streams, as well as TTY.

type Executor interface{ ... }
    func NewSPDYExecutor(config *restclient.Config, method string, url *url.URL) (Executor, error)
    func NewSPDYExecutorForProtocols(transport http.RoundTripper, upgrader spdy.Upgrader, method string, ...) (Executor, error)
    func NewSPDYExecutorForTransports(transport http.RoundTripper, upgrader spdy.Upgrader, method string, ...) (Executor, error)
type StreamOptions struct{ ... }
type TerminalSize struct{ ... }
type TerminalSizeQueue interface{ ... }

package homedir // import "k8s.io/client-go/util/homedir"

func HomeDir() string

[vrothberg]

Maybe we can prune the dependency :)

[vrothberg]

podman (master) $ go mod why k8s.io/client-go/pkg/version
warning: ignoring symlink /home/vrothberg/go/src/github.com/containers/podman/contrib/systemd/user
# k8s.io/client-go/pkg/version
github.com/containers/podman/v3/libpod
k8s.io/client-go/tools/remotecommand
k8s.io/client-go/rest
k8s.io/client-go/pkg/version

I think we can prune the dependency. By what I can see, we're pulling this dependency for remotecommand.TerminalSize which is a simple struct storing the width and height of a terminal. We should be able to move that into define and thereby prune the k8s tail.

@mheon, can you confirm?

[mheon]

If we can, that would be quite nice - it's a large dependency that we don't really need.