Add validation and uploading of jobs configuration #127

hudymi · 2018-11-15T15:46:04Z

Description

Changes proposed in this pull request:

Added a simple application that uploads configuration to Prow
Enabled validation of job configs

Related issue(s)

aszecowka · 2018-11-16T07:19:04Z

development/tools/cmd/configuploader/README.md

+
+## Overview
+
+This command is responsible for uploading Prow configuration.


be more specific: which configuration

aszecowka · 2018-11-16T07:21:16Z

development/tools/cmd/configuploader/README.md

+## Overview
+
+This command is responsible for uploading Prow configuration.
+


If I add a new job, should I run this command? Probably not, because we have config updater. We should mention that you update jobs using this tool when a new cluster is provisioned (for production cluster or for "forked" clusters)

Added more information.

My proposal: Use it for a newly created Prow cluster or to update changes in the configuration on a cluster from a forked repository that does not use config-updater?

aszecowka · 2018-11-16T08:16:33Z

development/tools/cmd/configuploader/main.go

+	return err
+}
+
+func uploadFromFiles(name, path string, client configMapSetter) error {


consider renaming: updateConfigMapFromDirecotry and uploadFromFile --> updateConfigMapFromFile.
Additionally, it is updating or replacing?

aszecowka · 2018-11-16T08:22:22Z

development/tools/cmd/configuploader/main_test.go

+
+func TestUploadFromFile(t *testing.T) {
+	//given
+	tmpDir, err := ioutil.TempDir("", "TestFindAllRec")


minor: prefix should be the same as a test name

aszecowka · 2018-11-16T08:26:54Z

development/tools/pkg/file/file.go

+	"path/filepath"
+)
+
+func FindAllRec(rootPath, extension string) ([]string, error) {


name of this function is cryptic for me

aszecowka · 2018-11-16T08:28:24Z

development/tools/cmd/configuploader/main.go

+	return &result, nil
+}
+
+func configMapFromFiles(name, rootPath string) (*v1.ConfigMap, error) {


minor: maybe better name will be: configMapFromYamlsInDir

aszecowka · 2018-11-16T08:48:35Z

development/tools/cmd/configuploader/README.md

+
+## Usage
+
+To run, use:


Also, add information that we make an assumption that configmaps already exist

aszecowka · 2018-11-16T08:49:18Z

development/tools/cmd/configuploader/README.md

+
+This command is responsible for uploading Prow configuration.
+
+## Usage


If this tool has to be used for production cluster, please update documentation how to configure production cluster

suleymanakbas91 · 2018-11-16T08:52:34Z

development/tools/cmd/configuploader/README.md

+| --kubeconfig         |   Yes    | The path to the kubeconfig file, needed for connecting to the cluster.                              |
+| --config-path        |    No    | The path to the `config.yaml` file. If set, the configuration will be uploaded.                     |
+| --jobs-config-path   |    No    | The path to the directory with job configurations. If set, the job configurations will be uploaded. |
+| --plugin-config-path |    No    | The path to the `plugins.yaml` file. If set, the plugins configuration will be uploaded.            |


I think --plugins-config-path would be better.

suleymanakbas91 · 2018-11-16T08:58:13Z

development/tools/cmd/configuploader/main.go

+	return err
+}
+
+func configMapFromFile(name, path string) (*v1.ConfigMap, error) {


IMO we should either merge these two functions or name this one as configMapFromSingleFile. Otherwise, they can be mixed up.

aszecowka · 2018-11-16T09:09:47Z

prow/jobs/test-infra/validation.jobs.yaml

+            command:
+              - "/home/prow/go/src/github.com/kyma-project/test-infra/development/validate-scripts.sh"
+    - name: prow/test-infra/validate-configs


we should have a job to check if config uploader compiles - please fix it here or create a follow up task

Follow up created: #134

kazydek · 2018-11-16T08:41:42Z

development/README.md

 <!-- Update the project structure each time you modify it. -->

 ```
-  ├── checker                  # This folder contains code sources of a simple Go application that verifies the configuration of the "config.yaml" and "plugins.yaml" files          
-  ├── check.sh                 # This script runs the "Checker" application.  .                                            
+  ├── checker                  # This folder contains code sources of a simple Go application that verifies the configuration of the "config.yaml", "plugins.yaml" files and job configurations.


Suggested change

├── checker # This folder contains code sources of a simple Go application that verifies the configuration of the "config.yaml", "plugins.yaml" files and job configurations.

├── checker # This folder contains code sources of a simple Go application that verifies the configuration of jobs, "config.yaml", and "plugins.yaml".

kazydek · 2018-11-16T08:56:23Z

development/README.md

-  ├── checker                  # This folder contains code sources of a simple Go application that verifies the configuration of the "config.yaml" and "plugins.yaml" files          
-  ├── check.sh                 # This script runs the "Checker" application.  .                                            
+  ├── checker                  # This folder contains code sources of a simple Go application that verifies the configuration of the "config.yaml", "plugins.yaml" files and job configurations.
+  ├── tools                    # This folder contains code sources of a Go applications for test-infra repository.


Suggested change

├── tools # This folder contains code sources of a Go applications for test-infra repository.

├── tools # This folder contains code sources of Go applications for the "test-infra" repository.

kazydek · 2018-11-16T08:59:04Z

development/README.md

  ├── create-gcp-secrets.sh    # This script downloads Secrets from the GCP storage bucket to your Prow installation.
  ├── install-prow.sh          # This script installs Prow on your cluster.
  ├── provision-cluster.sh     # This script creates a Kubernetes cluster.
  ├── remove-prow.sh           # This script removes Prow from your Kubernetes cluster.
  ├── update-config.sh         # This script updates the new configuration of the "config.yaml" file on a cluster.
+  ├── update-jobs.sh           # This script updates the new configuration of the job configurations on a cluster.


Suggested change

├── update-jobs.sh # This script updates the new configuration of the job configurations on a cluster.

├── update-jobs.sh # This script updates the new configuration of the jobs on a cluster.

kazydek · 2018-11-16T09:05:02Z

development/tools/README.md

+    go run cmd/configuploader/main.go --kubeconfig $HOME/.kube/config --plugin-config-path {pathToPluginsYaml}
+    ```
+
+    For details how to run commands, go to `cmd` directory.


Suggested change

For details how to run commands, go to `cmd` directory.

For details on flags to use with the command, go to the `cmd` directory.

kazydek · 2018-11-16T09:06:41Z

development/tools/README.md

+
+## Overview
+
+This project contains Go applications for test-infra repository.


Suggested change

This project contains Go applications for test-infra repository.

This project contains Go applications for the `test-infra` repository.

kazydek · 2018-11-16T09:43:30Z

development/tools/cmd/configuploader/README.md

+| --kubeconfig         |   Yes    | The path to the kubeconfig file, needed for connecting to the cluster.                              |
+| --config-path        |    No    | The path to the `config.yaml` file. If set, the configuration will be uploaded.                     |
+| --jobs-config-path   |    No    | The path to the directory with job configurations. If set, the job configurations will be uploaded. |
+| --plugin-config-path |    No    | The path to the `plugins.yaml` file. If set, the plugins configuration will be uploaded.            |


kazydek · 2018-11-16T09:49:14Z

development/tools/cmd/configuploader/README.md

+go run cmd/configuploader/main.go --kubeconfig $HOME/.kube/config --plugin-config-path {pathToPluginsYaml}
+```
+
+## Parameters


Suggested change

## Parameters

### Flags

kazydek · 2018-11-16T09:53:06Z

docs/prow/prow-installation-on-forks.md

@@ -2,7 +2,7 @@

 This instruction provides the steps required to deploy your own Prow on a forked repository for test and development purposes.

-> **NOTE:** The following instructions assume that you are signed in to the Google Cloud project with administrative rights and that you have the $GOPATH already set.
+> **NOTE:** The following instructions assume that you are signed in to the Google Cloud project with administrative rights and that you have the \$GOPATH already set.


Suggested change

> **NOTE:** The following instructions assume that you are signed in to the Google Cloud project with administrative rights and that you have the \$GOPATH already set.

> **NOTE:** The following instructions assume that you are signed in to the Google Cloud project with administrative rights and that you have the `$GOPATH` already set.

kazydek · 2018-11-16T09:53:56Z

docs/prow/prow-installation-on-forks.md

@@ -162,12 +161,18 @@ If the files are configured correctly, upload the files on a cluster.
 ./update-plugins.sh ../prow/plugins.yaml
 ```

-2. Use the `update-config.sh {file_path}` script to apply jobs configuration on a cluster.
+2. Use the `update-config.sh {file_path}` script to apply prow configuration on a cluster.


Suggested change

2. Use the `update-config.sh {file_path}` script to apply prow configuration on a cluster.

2. Use the `update-config.sh {file_path}` script to apply Prow configuration on a cluster.

kazydek · 2018-11-16T10:00:55Z

development/tools/cmd/configuploader/README.md

+
+## Overview
+
+This command is responsible for uploading Prow configuration.


Suggested change

This command is responsible for uploading Prow configuration.

This command is responsible for uploading Prow configuration to a Prow cluster.

kyma-bot · 2018-11-16T11:57:27Z

@michal-hudy: Updated the job-config configmap using the following files:

key validation.jobs.yaml using file prow/jobs/test-infra/validation.jobs.yaml

In response to this:

Description

Changes proposed in this pull request:

Added a simple application that uploads configuration to Prow

Enabled validation of job configs

Related issue(s)

See #87

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hudymi added the WIP label Nov 15, 2018

hudymi requested review from a-thaler, bszwarc, clebs, kazydek, klaudiagrz, lilitgh, tomekpapiernik and a team as code owners November 15, 2018 15:46

hudymi added area/ci Issues or PRs related to CI related topics wg/prow and removed WIP labels Nov 15, 2018

Add validation and uploading of jobs configuration

5fe37b6

aszecowka suggested changes Nov 16, 2018

View reviewed changes

suleymanakbas91 reviewed Nov 16, 2018

View reviewed changes

aszecowka reviewed Nov 16, 2018

View reviewed changes

kazydek reviewed Nov 16, 2018

View reviewed changes

Review changes

b6207c0

hudymi mentioned this pull request Nov 16, 2018

Create job for validating tools project #134

Closed

4 tasks

aszecowka approved these changes Nov 16, 2018

View reviewed changes

kazydek approved these changes Nov 16, 2018

View reviewed changes

Changes

359fc23

hudymi merged commit e5521b6 into kyma-project:master Nov 16, 2018

hudymi deleted the config-updater branch November 21, 2018 17:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add validation and uploading of jobs configuration #127

Add validation and uploading of jobs configuration #127

hudymi commented Nov 15, 2018 •

edited

Loading

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

hudymi Nov 16, 2018

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

aszecowka Nov 16, 2018

hudymi Nov 16, 2018

suleymanakbas91 Nov 16, 2018

suleymanakbas91 Nov 16, 2018

hudymi Nov 16, 2018

aszecowka Nov 16, 2018

hudymi Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kazydek Nov 16, 2018

kyma-bot commented Nov 16, 2018


		## Overview

		This command is responsible for uploading Prow configuration.


		This command is responsible for uploading Prow configuration.

		## Usage

	├── checker # This folder contains code sources of a simple Go application that verifies the configuration of the "config.yaml", "plugins.yaml" files and job configurations.
	├── checker # This folder contains code sources of a simple Go application that verifies the configuration of jobs, "config.yaml", and "plugins.yaml".

	├── tools # This folder contains code sources of a Go applications for test-infra repository.
	├── tools # This folder contains code sources of Go applications for the "test-infra" repository.

	├── update-jobs.sh # This script updates the new configuration of the job configurations on a cluster.
	├── update-jobs.sh # This script updates the new configuration of the jobs on a cluster.

	For details how to run commands, go to `cmd` directory.
	For details on flags to use with the command, go to the `cmd` directory.


		## Overview

		This project contains Go applications for test-infra repository.

	\| --plugin-config-path \| No \| The path to the `plugins.yaml` file. If set, the plugins configuration will be uploaded. \|
	\| --plugin-config-path \| No \| The path to the `plugins.yaml` file. Set it to upload plugin configurations to a cluster. \|

	> NOTE: The following instructions assume that you are signed in to the Google Cloud project with administrative rights and that you have the \$GOPATH already set.
	> NOTE: The following instructions assume that you are signed in to the Google Cloud project with administrative rights and that you have the `$GOPATH` already set.

	2. Use the `update-config.sh {file_path}` script to apply prow configuration on a cluster.
	2. Use the `update-config.sh {file_path}` script to apply Prow configuration on a cluster.

Add validation and uploading of jobs configuration #127

Add validation and uploading of jobs configuration #127

Conversation

hudymi commented Nov 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyma-bot commented Nov 16, 2018

hudymi commented Nov 15, 2018 •

edited

Loading