Support new KEP-9 package format in installs #343

alenkacz · 2019-06-14T13:45:29Z

What type of PR is this?
/component kudoctl

What this PR does / why we need it:
kudo install is able to install frameworks packages in the new KEP-9 style package format. The old format is no longer supported.

Which issue(s) this PR fixes:
Fixes #349

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`kudo install` now accepts only packages in new KEP-9 style. Old packages are not supported anymore.

alenkacz · 2019-06-14T13:46:28Z

pkg/kudoctl/util/repo/package.go

+	GetInstallCRDs() (*InstallCRDs, error)
+}
+
+type V0Package struct {


as much as I love my interface here we can drop V0 packages immediately, up for discussion. Potentially we could drop it when we convert our official repo to the new format? But we cannot do that until just before 0.3.0 release, right? 🤔

gerred · 2019-06-14T13:49:39Z

pkg/kudoctl/util/repo/package.go

+
+const apiVersion = "kudo.k8s.io/v1alpha1"
+
+type InstallCRDs struct {


InstallCRDs is kind of a weird name for a type in my mind for some reason. I know it's technically right and I can't think of a better one, just wanted to observe that.

maybe

Suggested change

type InstallCRDs struct {

type BundleCRDs struct {

or

Suggested change

type InstallCRDs struct {

type BundleV1CRDs struct {

or

Suggested change

type InstallCRDs struct {

type V1Alpha1CRDs struct {

?

Also dislike the InstallCRDs name. It's CRDs now, but I'm sure we'll have more file types at some point.

I went with PackageCRDs - we can always change that :)

Just don't forget all the parameters and methods still named *InstallCRDs ;)

pkg/kudoctl/cmd/install/install.go

#347)

Co-Authored-By: Aleksey Dukhovniy <adukhovniy@mesosphere.io>

alenkacz · 2019-06-18T12:44:52Z

pkg/kudoctl/util/repo/package.go

+}
+
+// ReadFileSystemPackage reads package from filesystem and converts it to the CRD format
+func ReadFileSystemPackage(path string) (*InstallCRDs, error) {


this is technically not needed in this PR but I implemented it to ease the testing in the beginning and now since it's tested I decided to keep it :)

kensipe

I think I need to make a return visit... I want to comb thru the tests better.

nice work @alenkacz

pkg/kudoctl/util/repo/package.go

kensipe · 2019-06-18T19:43:22Z

pkg/kudoctl/util/repo/package.go

+}
+
+func fromFilesystem(packagePath string) (*v1Package, error) {
+	result := newV1Package()


I realize "package" is not ok for a var name... but not a fan of result... that could be anything and true for any result. thoughts?

no strong opinion but I tend to use result for variables containing the return type of the whole function, so the result of that function. I am open to changing that.

pkg/kudoctl/util/repo/package.go

kensipe · 2019-06-18T20:01:21Z

pkg/kudoctl/util/repo/package.go

+	if err != nil {
+		return nil, err
+	}
+	return &result, nil


questions that come to mind:

is there any file size limits? or any validation other than the rules of marshaling?
(the concern is byte arrays and sizes)

there is no validation after the fact. There seems to be a happy path of all the files should be there. What if there is a framework file, but not other file? or template file and no framework file?

We validate when we do getInstallCRDs. You cannot really do it much earlier because some parts of the validation can be done only AFTER you load everything and you start parsing the loaded YAML files. E.g. to find out that you're missing a template file that is included inside the package. All that will be caught in getInstallCRDs and it will return an error

On the size, we should at least validate the bundled package is not larger than 1MB (etcd's max key size). I know it's unlikely, but I'm not sure if we would get a sane/user understandable error back from Kubernetes.

pkg/kudoctl/util/repo/package.go

kensipe · 2019-06-18T20:09:24Z

pkg/kudoctl/util/repo/package.go

+		if err != nil {
+			return err
+		}
+		switch {


the code seems to conflate file walking and file processing. It is reasonably easy to follow... but what do you think of having a filewalker that returns a set of string paths... and a processor that takes a set of string paths and processes them? In my mind it simplifies the reasoning and testing of the 2 concerns.

I see your point, but the filewalker is actually what the library provides. If you return string paths then you basically need to iterate that tarball two times. I saw a lot of code snippets how people deal with tarballs in golang and everybody seems to be doing it this way...

kensipe · 2019-06-18T20:10:05Z

pkg/kudoctl/util/repo/package.go

+
+	tr := tar.NewReader(gzr)
+
+	result := newV1Package()


same concern with var name "result"

pkg/kudoctl/util/repo/package.go

gerred · 2019-06-19T13:40:07Z

pkg/kudoctl/util/repo/package.go

+	if err != nil {
+		return nil, err
+	}
+	return &result, nil


On the size, we should at least validate the bundled package is not larger than 1MB (etcd's max key size). I know it's unlikely, but I'm not sure if we would get a sane/user understandable error back from Kubernetes.

fabianbaier · 2019-06-19T19:19:35Z

pkg/kudoctl/util/repo/package.go

+
+const apiVersion = "kudo.k8s.io/v1alpha1"
+
+type InstallCRDs struct {


maybe

Suggested change

type InstallCRDs struct {

type BundleCRDs struct {

or

Suggested change

type InstallCRDs struct {

type BundleV1CRDs struct {

or

Suggested change

type InstallCRDs struct {

type V1Alpha1CRDs struct {

?

fabianbaier · 2019-06-19T19:22:54Z

pkg/kudoctl/util/repo/package.go

+)
+
+const (
+	frameworkV1FileName     = "framework.yaml"


difference between V1 and v1alpha1 and v0.2.0 ?

Does frameworkV1FileName represent the new file structure from kep-9 and we call it V1?

yeah I can drop the V1 if it's confusing. Basically I called the old one v0 and the kep-p v1

pkg/kudoctl/util/repo/package.go

fabianbaier · 2019-06-19T19:34:17Z

go.mod

@@ -48,7 +49,7 @@ require (
 	github.com/prometheus/common v0.2.0 // indirect
 	github.com/prometheus/procfs v0.0.0-20190209105433-f8d8b3f739bd // indirect
 	github.com/rogpeppe/go-internal v1.2.2 // indirect
-	github.com/spf13/afero v1.2.1 // indirect
+	github.com/spf13/afero v1.2.1


right now we are using afero just for our table tests in the package_test.go, right?

runyontr

LGTM

pkg/kudoctl/util/repo/package_test.go

zen-dog

Nice work! PTAL at my comments.

zen-dog · 2019-06-20T14:11:56Z

pkg/kudoctl/cmd/install/install.go

@@ -151,7 +151,7 @@ func installFramework(name, previous string, repository repo.FrameworkRepository

 	bundleName := bundleVersion.Name + "-" + bundleVersion.Version

-	bundle, err := repository.DownloadBundle(bundleName)
+	crds, err := repository.DownloadBundle(bundleName)


Shouldn't we rename this method to DownloadPackage for consistency? Also, it doesn't just download but also untars and parses so maybe GetPackage?

zen-dog · 2019-06-20T14:35:52Z

pkg/kudoctl/util/repo/package.go

+			var bf bundle.Framework
+
+			if err = yaml.Unmarshal(bytes, &bf); err != nil {
+				return errors.Wrap(err, "cannot unmarshal framework")


Suggested change

return errors.Wrap(err, "cannot unmarshal framework")

return errors.Wrap(err, "failed to unmarshal framework file %v", file.Name())

zen-dog · 2019-06-20T14:40:05Z

pkg/kudoctl/util/repo/package.go

+		}
+		switch {
+		case isFrameworkV1File(file.Name()):
+			var bf bundle.Framework


why not unmarshaling directly to result.Framework?

zen-dog · 2019-06-20T14:52:20Z

pkg/kudoctl/util/repo/package.go

+		case isParametersV1File(file.Name()):
+			var params map[string]map[string]string
+			if err = yaml.Unmarshal(bytes, &params); err != nil {
+				return errors.Wrapf(err, "unmarshalling %s content", file.Name())


Suggested change

return errors.Wrapf(err, "unmarshalling %s content", file.Name())

return errors.Wrapf(err, "failed to unmarshal parameters file %s", file.Name())

zen-dog · 2019-06-20T14:58:05Z

pkg/kudoctl/util/repo/package.go

+			}
+
+			switch {
+			case isFrameworkV1File(header.Name):


This part is identical to the unmarshaling files from func fromFilesystem()? We should extract it into a separate method.

zen-dog · 2019-06-20T17:36:31Z

pkg/kudoctl/util/repo/package_test.go

+	instanceV0FileName  = "-instance.yaml"
+)
+
+func TestReadFileSystemPackage(t *testing.T) {


Nice tests 👏

zen-dog · 2019-06-20T17:38:09Z

pkg/kudoctl/util/repo/testdata/zk-crd-golden/zookeeper-framework.yaml

@@ -0,0 +1,13 @@
+apiVersion: kudo.k8s.io/v1alpha1


I believe, the golden files in the go world have .golden extension.

alenkacz added 2 commits June 14, 2019 12:27

KEP9 install WIP

f999c2a

Kep-9 installs

6080c18

alenkacz requested review from fabianbaier and kensipe as code owners June 14, 2019 13:45

alenkacz commented Jun 14, 2019

View reviewed changes

gerred reviewed Jun 14, 2019

View reviewed changes

zen-dog reviewed Jun 14, 2019

View reviewed changes

pkg/kudoctl/cmd/install/install.go Outdated Show resolved Hide resolved

alenkacz mentioned this pull request Jun 17, 2019

Be able to install framework from a filesystem #309

Closed

alenkacz changed the title ~~Support new KEP-9 package format in installs~~ WIP: Support new KEP-9 package format in installs Jun 17, 2019

gerred added this to In progress in KUDO via automation Jun 17, 2019

gerred added this to the v0.3.0 milestone Jun 17, 2019

alenkacz and others added 8 commits June 18, 2019 10:38

Add converted ZK framework and test that it equals current package

c6a893d

Test also tarballed package

c15a685

Added more fields to zk test data. re-open tarball from aero for tests (

dc02b76

#347)

Make tarball tests pass

0a5493d

Formatting

f87e9f0

Update pkg/kudoctl/cmd/install/install.go

285d1b5

Co-Authored-By: Aleksey Dukhovniy <adukhovniy@mesosphere.io>

proper rename

e7acdd7

Fix test

e674f33

alenkacz changed the title ~~WIP: Support new KEP-9 package format in installs~~ Support new KEP-9 package format in installs Jun 18, 2019

alenkacz moved this from In progress to Review in progress in KUDO Jun 18, 2019

alenkacz mentioned this pull request Jun 18, 2019

Zookeeper framework converted to the new package structure kudobuilder/operators#21

Merged

alenkacz commented Jun 18, 2019

View reviewed changes

kensipe reviewed Jun 18, 2019

View reviewed changes

alenkacz added 2 commits June 19, 2019 10:50

Review comments

482c66e

Fix parameters deserialization

c2660b2

gerred requested changes Jun 19, 2019

View reviewed changes

gerred self-requested a review June 19, 2019 14:56

gerred approved these changes Jun 19, 2019

View reviewed changes

KUDO automation moved this from Review in progress to Reviewer approved Jun 19, 2019

fabianbaier reviewed Jun 19, 2019

View reviewed changes

runyontr self-requested a review June 20, 2019 11:28

runyontr reviewed Jun 20, 2019

View reviewed changes

zen-dog reviewed Jun 20, 2019

View reviewed changes

pkg/kudoctl/util/repo/package_test.go Show resolved Hide resolved

Rename

d2362a9

zen-dog approved these changes Jun 20, 2019

View reviewed changes

Review comments

2ba86bf

alenkacz merged commit 12babe2 into master Jun 21, 2019

KUDO automation moved this from Reviewer approved to Done Jun 21, 2019

alenkacz deleted the av/install-kep9 branch October 30, 2019 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support new KEP-9 package format in installs #343

Support new KEP-9 package format in installs #343

alenkacz commented Jun 14, 2019 •

edited

Loading

alenkacz Jun 14, 2019

gerred Jun 14, 2019

fabianbaier Jun 19, 2019

zen-dog Jun 20, 2019

alenkacz Jun 20, 2019

zen-dog Jun 20, 2019

alenkacz Jun 18, 2019

gerred Jun 19, 2019

kensipe left a comment

kensipe Jun 18, 2019

alenkacz Jun 19, 2019

kensipe Jun 18, 2019

alenkacz Jun 19, 2019

gerred Jun 19, 2019

kensipe Jun 18, 2019

alenkacz Jun 19, 2019

kensipe Jun 18, 2019

gerred Jun 19, 2019

fabianbaier Jun 19, 2019

fabianbaier Jun 19, 2019

alenkacz Jun 20, 2019

fabianbaier Jun 19, 2019

alenkacz Jun 20, 2019

runyontr left a comment

zen-dog left a comment

zen-dog Jun 20, 2019

zen-dog Jun 20, 2019

zen-dog Jun 20, 2019

zen-dog Jun 20, 2019

zen-dog Jun 20, 2019

zen-dog Jun 20, 2019

zen-dog Jun 20, 2019


		const apiVersion = "kudo.k8s.io/v1alpha1"

		type InstallCRDs struct {

	return errors.Wrap(err, "cannot unmarshal framework")
	return errors.Wrap(err, "failed to unmarshal framework file %v", file.Name())

	return errors.Wrapf(err, "unmarshalling %s content", file.Name())
	return errors.Wrapf(err, "failed to unmarshal parameters file %s", file.Name())

Support new KEP-9 package format in installs #343

Support new KEP-9 package format in installs #343

Conversation

alenkacz commented Jun 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kensipe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

runyontr left a comment

Choose a reason for hiding this comment

zen-dog left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alenkacz commented Jun 14, 2019 •

edited

Loading