Create kubepkg tool to building debs/rpms #935

justaugustus · 2019-11-16T04:18:45Z

Refactors the debs builder into a package builder which will eventually build both debs/rpms.
This PR only attempts to convert the deb builder into a CLI tool that looks closer to how krel is structured and boost the test coverage a little.

rpm building will be handled in a separate PR once we think a little more about how the final tool should be shaped.

For reviewers: I'm explicitly punting on testing the following functions/methods, as they will likely be massively refactored or removed in a follow-up (and this code is not currently is use):

ConstructBuilds
WalkBuilds
buildPackage
(c cfg) run()

ref: #918
cc: @kubernetes/release-engineering

k8s-ci-robot · 2019-11-16T04:18:47Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

feiskyer · 2019-11-16T16:09:33Z

Cool. Are we able to extend this to subprojects, e.g. cri-tools?

cmd/kubepkg/main.go

justaugustus · 2019-11-16T19:19:02Z

@feiskyer --

Cool. Are we able to extend this to subprojects, e.g. cri-tools?

Yep! You'll actually be able to this today on the debs side with:

kubepkg debs --packages cri-tools --cri-tools-version 1.16.1

...and the output is as follows:

$ ls -lah bin/*
bin/nightly:
total 48M
drwxr-xr-x 2 augustus augustus 4.0K Nov 16 14:11 .
drwxr-xr-x 5 augustus augustus 4.0K Nov 15 23:31 ..
-rw-r--r-- 1 augustus augustus  11M Nov 16 14:09 cri-tools_1.16.1-0_amd64.deb
-rw-r--r-- 1 augustus augustus 9.2M Nov 16 14:10 cri-tools_1.16.1-0_arm64.deb
-rw-r--r-- 1 augustus augustus 9.2M Nov 16 14:09 cri-tools_1.16.1-0_armhf.deb
-rw-r--r-- 1 augustus augustus 9.1M Nov 16 14:10 cri-tools_1.16.1-0_ppc64el.deb
-rw-r--r-- 1 augustus augustus 9.9M Nov 16 14:11 cri-tools_1.16.1-0_s390x.deb

bin/release:
total 48M
drwxr-xr-x 2 augustus augustus 4.0K Nov 16 14:11 .
drwxr-xr-x 5 augustus augustus 4.0K Nov 15 23:31 ..
-rw-r--r-- 1 augustus augustus  11M Nov 16 14:09 cri-tools_1.16.1-0_amd64.deb
-rw-r--r-- 1 augustus augustus 9.2M Nov 16 14:10 cri-tools_1.16.1-0_arm64.deb
-rw-r--r-- 1 augustus augustus 9.2M Nov 16 14:09 cri-tools_1.16.1-0_armhf.deb
-rw-r--r-- 1 augustus augustus 9.1M Nov 16 14:10 cri-tools_1.16.1-0_ppc64el.deb
-rw-r--r-- 1 augustus augustus 9.9M Nov 16 14:11 cri-tools_1.16.1-0_s390x.deb

bin/testing:
total 48M
drwxr-xr-x 2 augustus augustus 4.0K Nov 16 14:11 .
drwxr-xr-x 5 augustus augustus 4.0K Nov 15 23:31 ..
-rw-r--r-- 1 augustus augustus  11M Nov 16 14:09 cri-tools_1.16.1-0_amd64.deb
-rw-r--r-- 1 augustus augustus 9.2M Nov 16 14:10 cri-tools_1.16.1-0_arm64.deb
-rw-r--r-- 1 augustus augustus 9.2M Nov 16 14:09 cri-tools_1.16.1-0_armhf.deb
-rw-r--r-- 1 augustus augustus 9.1M Nov 16 14:10 cri-tools_1.16.1-0_ppc64el.deb
-rw-r--r-- 1 augustus augustus 9.9M Nov 16 14:11 cri-tools_1.16.1-0_s390x.deb

This is strictly because cri-tools is one of the packages we currently publish through the Release Engineering (kubelet, kubectl, kubeadm, kubernetes-cni, cri-tools).

Eventually, I'd like to get to a place where this tool takes a yaml-based config with package definitions and builds the packages specified, though we won't have time to work on that piece probably until 1.18.

k8s-ci-robot · 2019-11-23T22:42:42Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cpanato · 2019-11-24T09:55:43Z

cmd/kubepkg/cmd/debs.go

+	builds, err := kpkg.ConstructBuilds(ro.packages, ro.channels, ro.kubeVersion, ro.revision, ro.cniVersion, ro.criToolsVersion)
+	if err != nil {
+		log.Fatalf("err: %v", err)
+	}


Instead do the log.Fatalf("err: %v", err) is not better log and return the error?

There should be no need to log since we fail errors with logrus.Fatal in any case. Just returning the error should be enough to avoid logging them twice.

Signed-off-by: Stephen Augustus <saugustus@vmware.com>

k8s-ci-robot · 2020-01-03T13:59:33Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Co-Authored-By: Sascha Grunert <sgrunert@suse.com> Signed-off-by: Stephen Augustus <saugustus@vmware.com>

justaugustus · 2020-01-03T15:30:08Z

@saschagrunert -- Thanks! I've addressed your feedback, if you have time for another sweep. :)

k8s-ci-robot · 2020-01-03T15:35:16Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

saschagrunert

Good, found just another three minor nits :)

pkg/kubepkg/kubepkg.go

cmd/kubepkg/cmd/debs.go

saschagrunert · 2020-01-03T17:03:19Z

cmd/kubepkg/cmd/debs.go

+	ro := rootOpts
+
+	// Replace the "+" with a "-" to make it semver-compliant
+	ro.kubeVersion = strings.TrimPrefix(ro.kubeVersion, "v")


Suggested change

ro.kubeVersion = strings.TrimPrefix(ro.kubeVersion, "v")

ro.kubeVersion = git.TrimTagPrefix(ro.kubeVersion)

k8s-ci-robot · 2020-01-03T17:29:27Z

@justaugustus: Adding label: do-not-merge/blocked-paths because PR changes a protected file.

Reasons for blocking this PR:

[Changes to certain release tools can affect our ability to test, build, and release Kubernetes. This PR must be explicitly approved by SIG Release repo admins.]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

justaugustus · 2020-01-03T17:35:30Z

@saschagrunert -- Fixed up!

saschagrunert · 2020-01-03T17:42:38Z

Cool, thank you!

/retest

Co-Authored-By: Sascha Grunert <sgrunert@suse.com> Signed-off-by: Stephen Augustus <saugustus@vmware.com>

saschagrunert · 2020-01-03T17:44:16Z

cmd/kubepkg/cmd/debs.go

+package cmd
+
+import (
+	"github.com/sirupsen/logrus"


I think we have to remove this line to fix the build error.

@saschagrunert -- Yep! Just fixed it.

Signed-off-by: Stephen Augustus <saugustus@vmware.com>

saschagrunert · 2020-01-03T17:56:41Z

/lgtm
/approve

k8s-ci-robot · 2020-01-03T17:57:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. labels Nov 16, 2019

k8s-ci-robot requested review from Bubblemelon and listx November 16, 2019 04:19

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 16, 2019

saschagrunert reviewed Nov 16, 2019

View reviewed changes

cmd/kubepkg/main.go Show resolved Hide resolved

justaugustus removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Nov 16, 2019

justaugustus force-pushed the kubepkg branch from 4beadc4 to 4a47fbd Compare November 23, 2019 22:42

k8s-ci-robot added the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Nov 23, 2019

cpanato reviewed Nov 24, 2019

View reviewed changes

justaugustus force-pushed the kubepkg branch from 4a47fbd to ffdb456 Compare November 29, 2019 11:52

justaugustus mentioned this pull request Dec 4, 2019

Deprecate kubernetes-cni deb/rpm packages #885

Closed

2 tasks

justaugustus force-pushed the kubepkg branch 3 times, most recently from 95d284d to 6c68c67 Compare December 14, 2019 03:58

justaugustus force-pushed the kubepkg branch from 6c68c67 to b7bfca3 Compare December 21, 2019 10:17

justaugustus force-pushed the kubepkg branch from b7bfca3 to 4a7d284 Compare January 1, 2020 22:03

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 1, 2020

justaugustus added 3 commits January 2, 2020 03:40

kubepkg: Move deb-builder into kubepkg tool

607056c

Signed-off-by: Stephen Augustus <saugustus@vmware.com>

kubepkg: Move debs subcommand to separate cmd directory

1cf6682

Signed-off-by: Stephen Augustus <saugustus@vmware.com>

kubepkg: Refactor common non-command vars and functions into pkg/kubepkg

bddf5c7

Signed-off-by: Stephen Augustus <saugustus@vmware.com>

k8s-ci-robot added the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Jan 3, 2020

cmd/kubepkg: Update copyright

061e478

Co-Authored-By: Sascha Grunert <sgrunert@suse.com> Signed-off-by: Stephen Augustus <saugustus@vmware.com>

justaugustus force-pushed the kubepkg branch 2 times, most recently from 6206087 to 7abb35e Compare January 3, 2020 15:28

justaugustus removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Jan 3, 2020

justaugustus force-pushed the kubepkg branch from 7abb35e to aabaa7a Compare January 3, 2020 15:35

k8s-ci-robot added the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Jan 3, 2020

justaugustus removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Jan 3, 2020

saschagrunert reviewed Jan 3, 2020

View reviewed changes

justaugustus force-pushed the kubepkg branch from aabaa7a to 7e29fd1 Compare January 3, 2020 17:29

k8s-ci-robot added the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Jan 3, 2020

justaugustus force-pushed the kubepkg branch from 4271071 to 7f55e12 Compare January 3, 2020 17:32

kubepkg: Clean up error handling

e14e860

Co-Authored-By: Sascha Grunert <sgrunert@suse.com> Signed-off-by: Stephen Augustus <saugustus@vmware.com>

saschagrunert reviewed Jan 3, 2020

View reviewed changes

kubepkg: Use long-form flags for dpkg-buildpackage

ddec037

Signed-off-by: Stephen Augustus <saugustus@vmware.com>

justaugustus force-pushed the kubepkg branch from 7f55e12 to ddec037 Compare January 3, 2020 17:45

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 3, 2020

saschagrunert approved these changes Jan 3, 2020

View reviewed changes

justaugustus removed the do-not-merge/blocked-paths Indicates that a PR should not merge because it touches files in blocked paths. label Jan 3, 2020

k8s-ci-robot merged commit b80729d into kubernetes:master Jan 3, 2020

justaugustus mentioned this pull request Jan 9, 2020

The release scripts of master branch can not work normally #1001

Closed

justaugustus mentioned this pull request Jan 19, 2020

[kubepkg] Create a go tool for building debs/rpms #1027

Open

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create kubepkg tool to building debs/rpms #935

Create kubepkg tool to building debs/rpms #935

justaugustus commented Nov 16, 2019 •

edited

Loading

k8s-ci-robot commented Nov 16, 2019

Reasons for blocking this PR:

feiskyer commented Nov 16, 2019

justaugustus commented Nov 16, 2019

k8s-ci-robot commented Nov 23, 2019

Reasons for blocking this PR:

cpanato Nov 24, 2019 •

edited

Loading

saschagrunert Jan 3, 2020

k8s-ci-robot commented Jan 3, 2020

Reasons for blocking this PR:

justaugustus commented Jan 3, 2020

k8s-ci-robot commented Jan 3, 2020

Reasons for blocking this PR:

saschagrunert left a comment

saschagrunert Jan 3, 2020

k8s-ci-robot commented Jan 3, 2020

Reasons for blocking this PR:

justaugustus commented Jan 3, 2020

saschagrunert commented Jan 3, 2020

saschagrunert Jan 3, 2020

justaugustus Jan 3, 2020

saschagrunert commented Jan 3, 2020

k8s-ci-robot commented Jan 3, 2020

	ro.kubeVersion = strings.TrimPrefix(ro.kubeVersion, "v")
	ro.kubeVersion = git.TrimTagPrefix(ro.kubeVersion)

Create kubepkg tool to building debs/rpms #935

Create kubepkg tool to building debs/rpms #935

Conversation

justaugustus commented Nov 16, 2019 • edited Loading

k8s-ci-robot commented Nov 16, 2019

Reasons for blocking this PR:

feiskyer commented Nov 16, 2019

justaugustus commented Nov 16, 2019

k8s-ci-robot commented Nov 23, 2019

Reasons for blocking this PR:

cpanato Nov 24, 2019 • edited Loading

Choose a reason for hiding this comment

saschagrunert Jan 3, 2020

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 3, 2020

Reasons for blocking this PR:

justaugustus commented Jan 3, 2020

k8s-ci-robot commented Jan 3, 2020

Reasons for blocking this PR:

saschagrunert left a comment

Choose a reason for hiding this comment

saschagrunert Jan 3, 2020

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 3, 2020

Reasons for blocking this PR:

justaugustus commented Jan 3, 2020

saschagrunert commented Jan 3, 2020

saschagrunert Jan 3, 2020

Choose a reason for hiding this comment

justaugustus Jan 3, 2020

Choose a reason for hiding this comment

saschagrunert commented Jan 3, 2020

k8s-ci-robot commented Jan 3, 2020

justaugustus commented Nov 16, 2019 •

edited

Loading

cpanato Nov 24, 2019 •

edited

Loading