[feature] Understanding packages from directory #229

Bowenislandsong · 2020-03-26T15:15:45Z

This commit adds a feature to interpret bundle name and channels based on a given bundle directory and update bundle's annotation YAML. This feature is triggered when opm alpha generate/build is not supplying packageName, channels, and defaultChannels information.
This Bundle directory needs to be inside a Package directory that contains all bundles.

Bowenislandsong · 2020-03-26T15:17:04Z

/cc @kevinrizza @ecordell @gallettilance @exdx @njhale @awgreene

Bowenislandsong · 2020-03-26T15:17:34Z

adding test!

Bowenislandsong · 2020-03-26T15:23:41Z

pkg/sqlite/graph_loader.go

@@ -0,0 +1,159 @@
+package sqlite


DO not review this file. This is @exdx and @gallettilance 's part

kevinrizza

Nice work on this Bowen. I have some comments and questions, mostly about how we parse the input and also how we bubble up errors

kevinrizza · 2020-03-26T15:20:02Z

cmd/opm/alpha/bundle/build.go

@@ -55,14 +55,8 @@ func newBundleBuildCmd() *cobra.Command {
 	}

 	bundleBuildCmd.Flags().StringVarP(&packageNameArgs, "package", "p", "", "The name of the package that bundle image belongs to")


IMO we need to update this help message to let users know that it's required if they're not pointing to a directory in the nested bundle format

nice chatch !!

kevinrizza · 2020-03-26T15:20:46Z

pkg/lib/bundle/generate.go

@@ -76,6 +76,24 @@ func GenerateFunc(directory, outputDir, packageName, channels, channelDefault st
 		return err
 	}

+	if channels == "" || packageName == "" || channelDefault == "" {
+		I, err := NewBundleDirInterperter(filepath.Join(directory, ".."))


nit: use lowercase i, err :=

lol, right!

kevinrizza · 2020-03-26T15:27:40Z

pkg/registry/csv.go

@@ -53,6 +60,49 @@ type ClusterServiceVersion struct {
 	Spec json.RawMessage `json:"spec"`
 }

+func NewFromBundleDirectory(bundleDir string) (*ClusterServiceVersion, error) {


Can we change this function name to be more descriptive? NewFromBundleDirectory describes the mechanics of what the function is doing, not the purpose or the output. Maybe ReadCSVFromBundleDirectory?

Right, I thought about saying read or load, went with New because I want to say new ... somewhere... I'll change it since it makes more sense.

kevinrizza · 2020-03-26T15:29:20Z

pkg/registry/csv.go

+	}
+
+	csvFile, err := func() (string, error) {
+		for _, f := range files {


I think there are more clever ways we could find this file. For example we can check all the files in the directory to see if they're yaml files, and then for each yaml file attempt to serialize it into a CSV.

Otherwise this will fail if they don't follow the naming convention described here

I thought it is mandatory to be named that way? ... Could there be a case where there are more than one CSV file in a bundle?

there is no hard requirement on the name of the file...

kevinrizza · 2020-03-26T15:33:30Z

pkg/lib/bundle/generate.go

+	if channels == "" || packageName == "" || channelDefault == "" {
+		I, err := NewBundleDirInterperter(filepath.Join(directory, ".."))
+		if err != nil {
+			return fmt.Errorf("error interpretating bundle from directory %s, %v", directory, err)


spelling nit: interpreting

Also, it feels like the error messaging here is going to be unclear to users. Can we be more descriptive? Right now it seems like we don't differentiate between "this bundle isn't in a directory with our special helpful case" and "you didn't give us enough information to handle this request". For example if they just didn't specify a list of channels in a bundle that's in its own directory, we shouldn't surface "unable to interpret bundle from directory". we should tell them "missing input parameters".

There's also the case where a user just doesn't give us a default channel for example that we were already handling and I think this removes that ability as well

kevinrizza · 2020-03-26T15:38:52Z

pkg/registry/directoryGraphLoader.go

+		return nil, fmt.Errorf("error geting CSVs from all bundles in the package directory, %v", err)
+	}
+
+	loader.loadReplaces()


Why isn't this function just part of the loader.Generate()? It seems like this will make it harder to mock if some of the implementation is inside the constructor.

kevinrizza · 2020-03-26T15:41:48Z

pkg/registry/directoryGraphLoader.go

+
+	ymlFile, err := func() (string, error) {
+		for _, f := range files {
+			if matched, err := path.Match("*.package.yaml", f.Name()); err != nil {


same comment as the parser for the csv, i think we can serialize this rather than rely on a naming convention

kevinrizza · 2020-03-26T15:43:56Z

pkg/sqlite/graph_loader.go

@@ -0,0 +1,159 @@
+package sqlite


this is already merged into master with a new file name fyi: https://github.com/operator-framework/operator-registry/blob/master/pkg/sqlite/graphloader.go#L11

Yeah, i am not sure why it shows here, I will remove it. in the next push

It's because you merged their commits into your repo and are pulling them along, but i squashed their commits and then renamed this file to get rid of the underscore in the name. you need to remove their commits and rebase.

kevinrizza · 2020-03-26T20:13:56Z

cmd/opm/alpha/bundle/build.go


-	bundleBuildCmd.Flags().StringVarP(&channelDefaultArgs, "default", "e", "", "The default channel for the bundle image")
+	bundleBuildCmd.Flags().BoolVarP(&noChannelDefaultArgs, "no-default", "e", false,


This seems like a not great user experience. We shouldn't add complexity here if it's not necessary. IMO we should just attempt to set the default channel like we were doing before if we can't find a packages.yaml file instead of asking the user if they want an error or not.

? if we can not find the package.yaml then we have run the command. By then we would either successfully get the channel or that we didn't. If we got it we are writing it to the annotation file. If we didn't we would have already returned something.

what do you mean by then we ask if users want error or not? There isn't a second step command.

I can only think of returning a warning if the default channel is not found.
Besize, we can tno interpret anything without the package.yaml file.

Because before we didn't attempt to look for it and tried to guess it. Why not retain that implicitly without adding an extra parameter?

kevinrizza

Also, can you include some unit tests?

kevinrizza · 2020-03-26T20:16:22Z

cmd/opm/alpha/bundle/build.go

-	}
+	bundleBuildCmd.Flags().StringVarP(&channelsArgs, "channels", "c", "",
+		"The list of channels that bundle image belongs to" +
+		"(Required if `directory` is not pointing to a bundel in the nested bundel format)")


spelling nit: bundle

kevinrizza · 2020-03-26T20:16:35Z

cmd/opm/alpha/bundle/build.go

-	}
+	bundleBuildCmd.Flags().StringVarP(&packageNameArgs, "package", "p", "",
+		"The name of the package that bundle image belongs to " +
+		"(Required if `directory` is not pointing to a bundel in the nested bundel format)")


spelling nit: bundle

kevinrizza · 2020-03-26T20:17:03Z

cmd/opm/alpha/bundle/build.go

-	bundleBuildCmd.Flags().StringVarP(&dirBuildArgs, "directory", "d", "", "The directory where bundle manifests and metadata for a specific version are located")
+	bundleBuildCmd.Flags().StringVarP(&dirBuildArgs, "directory", "d", "",
+		"The directory where bundle manifests and metadata for a specific version are located " +
+		"(Can be used for interpreting bundle information if in the nested bundle format.)")


I think this last line is not necessary for this specific parameter.

kevinrizza · 2020-03-26T20:17:38Z

cmd/opm/alpha/bundle/generate.go

-	}
+	bundleGenerateCmd.Flags().StringVarP(&packageNameArgs, "package", "p", "",
+		"The name of the package that bundle image belongs to "+
+			"(Required if `directory` is not pointing to a bundel in the nested bundel format)")


Same spelling nits as the build cmd

kevinrizza · 2020-03-26T20:22:46Z

pkg/registry/directoryGraphLoader.go

+
+	ymlFiles := []string{}
+	for _, f := range files {
+		if matched, err := path.Match("*.yaml", f.Name()); err != nil {


can we just check to see if it's yaml? we do something similar when we unpack bundles: https://github.com/operator-framework/operator-registry/blob/master/pkg/registry/conversion.go#L67

Bowenislandsong · 2020-03-26T20:27:58Z

Also, can you include some unit tests?

yep working on that now... thats why it is still wip.

kevinrizza · 2020-03-27T12:58:08Z

pkg/lib/bundle/generate.go

+	if channels == "" || packageName == "" {
+		i, err := NewBundleDirInterperter(filepath.Join(directory, ".."))
+		if err != nil {
+			return fmt.Errorf("please manually input channels and packageName, "+


kevinrizza · 2020-03-27T13:01:35Z

pkg/registry/csv.go

+
+		decoder := yaml.NewYAMLOrJSONDecoder(csvReader, 30)
+		if err = decoder.Decode(&unstructuredCSV); err != nil {
+			return nil, fmt.Errorf("error unmarshalling CSV, %v", err)


but won't this return an error if the directory contains yaml files that aren't csvs?

I will name this better. This just unmarshals to unstructured. It then tries to determin if it is CSV next
if unstructuredCSV.GetKind() != operators.ClusterServiceVersionKind {

kevinrizza · 2020-03-27T13:01:50Z

pkg/registry/csv.go

+	ymlFiles := []string{}
+	for _, f := range files {
+		if !f.IsDir() {
+			ymlFiles = append(ymlFiles, f.Name())


shouldn't we use the yaml decoder here to see if the file is actually a yaml file?

This just grabs all files. I will rename the variables. This gets all files, then it unmarshals to yaml. then it tries to see if it is csv.

This commit adds a feature to interpret bundle name and channels based on a given bundle directory and update bundle's annotation YAML. This feature is triggered when opm alpha generate/build is not supplying packageName, channels, and defaultChannels information. This Bundle directory needs to be inside a Package directory that contains all bundles.

openshift-ci-robot · 2020-03-27T13:34:10Z

@Bowenislandsong: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/images	`38fdca0`	link	`/test images`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kevinrizza · 2020-03-27T15:43:59Z

/lgtm
/approve

openshift-ci-robot · 2020-03-27T15:45:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bowenislandsong, kevinrizza

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kevinrizza]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot requested review from anik120 and kevinrizza March 26, 2020 15:15

openshift-ci-robot requested review from ecordell, exdx, gallettilance and njhale March 26, 2020 15:17

Bowenislandsong changed the title ~~[feature] Understanding packages from directory~~ [wip][feature] Understanding packages from directory Mar 26, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 26, 2020

Bowenislandsong commented Mar 26, 2020

View reviewed changes

kevinrizza requested changes Mar 26, 2020

View reviewed changes

Bowenislandsong force-pushed the understand-packages branch from 7e2e7ab to 8b6549f Compare March 26, 2020 19:44

kevinrizza reviewed Mar 26, 2020

View reviewed changes

kevinrizza requested changes Mar 26, 2020

View reviewed changes

Bowenislandsong force-pushed the understand-packages branch from 54dbef6 to 51d205d Compare March 26, 2020 23:40

Bowenislandsong changed the title ~~[wip][feature] Understanding packages from directory~~ [feature] Understanding packages from directory Mar 27, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 27, 2020

Bowenislandsong force-pushed the understand-packages branch 2 times, most recently from dc7bb29 to 46afb12 Compare March 27, 2020 10:55

kevinrizza reviewed Mar 27, 2020

View reviewed changes

Bowenislandsong force-pushed the understand-packages branch from 46afb12 to 38fdca0 Compare March 27, 2020 13:21

openshift-ci-robot assigned kevinrizza Mar 27, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 27, 2020

Bowenislandsong merged commit 13373cb into operator-framework:master Mar 27, 2020

Bowenislandsong mentioned this pull request Mar 29, 2020

Revert "[feature] Understanding packages from directory" #239

Merged

		@@ -55,14 +55,8 @@ func newBundleBuildCmd() *cobra.Command {
		}

		bundleBuildCmd.Flags().StringVarP(&packageNameArgs, "package", "p", "", "The name of the package that bundle image belongs to")


		bundleBuildCmd.Flags().StringVarP(&channelDefaultArgs, "default", "e", "", "The default channel for the bundle image")
		bundleBuildCmd.Flags().BoolVarP(&noChannelDefaultArgs, "no-default", "e", false,

[feature] Understanding packages from directory #229

[feature] Understanding packages from directory #229

Conversation

Bowenislandsong commented Mar 26, 2020 • edited

Bowenislandsong commented Mar 26, 2020 • edited

Bowenislandsong commented Mar 26, 2020

Choose a reason for hiding this comment

kevinrizza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bowenislandsong Mar 26, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinrizza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bowenislandsong commented Mar 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci-robot commented Mar 27, 2020 • edited

kevinrizza commented Mar 27, 2020

openshift-ci-robot commented Mar 27, 2020

Bowenislandsong commented Mar 26, 2020 •

edited

Bowenislandsong commented Mar 26, 2020 •

edited

Bowenislandsong Mar 26, 2020 •

edited

openshift-ci-robot commented Mar 27, 2020 •

edited