feat(api,worker): migrate all old plugins to new ones #3321

bnjjj · 2018-09-13T10:31:35Z

Description
DO NOT MERGE NOW, need to test on preprod.
Only check changes in engine/worker
Related issues
About tests
Mentions

@ovh/cds

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

* fix(ui): better label modals close #3286 Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

close #fix_3311 Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

Signed-off-by: Yvonnick Esnault <yvonnick.esnault@corp.ovh.com>

…variables (#3316)

There is too many collisions with namegenerator. This explains the non-detecting worker timeout and the running jobs since many days in workflow node run jobs.

Signed-off-by: Yvonnick Esnault <yvonnick.esnault@corp.ovh.com>

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

golangcibot · 2018-09-13T10:32:13Z

engine/worker/builtin.go

+	select {
+	case <-ctx.Done():
+		log.Error("CDS Worker execution canceled: %v", ctx.Err())
+		w.sendLog(buildID, "CDS Worker execution canceled\n", stepOrder, false)


Error return value of w.sendLog is not checked

golangcibot · 2018-09-13T10:32:13Z

contrib/grpcplugins/action/npm-audit-parser/main.go

+}
+
+// parseTemplateParameters parses a list of key value pairs separated by new lines
+func parseTemplateParameters(s string) (map[string]interface{}, error) {


parseTemplateParameters is unused

fsamin

It is possible to share the same Makefile with all plugins ?

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

golangcibot · 2018-09-13T14:33:41Z

contrib/grpcplugins/action/kafka-publish/cli_actions.go

+		return cli.ShowCommandHelp(c, args.First())
+	}
+
+	cli.ShowAppHelp(c)


Error return value of cli.ShowAppHelp is not checked

golangcibot · 2018-09-13T14:33:41Z

contrib/grpcplugins/action/kafka-publish/cli_actions.go

+	args = append(args, c.Args().Tail()...)
+
+	if len(args) != 4 {
+		cli.ShowCommandHelp(c, "listen")


Error return value of cli.ShowCommandHelp is not checked

golangcibot · 2018-09-13T14:33:42Z

contrib/grpcplugins/action/kafka-publish/cli_actions.go

+	}
+
+	if !c.IsSet("kafka-password") && c.String("kafka-password") == "" {
+		cli.ShowCommandHelp(c, "listen")


Error return value of cli.ShowCommandHelp is not checked

golangcibot · 2018-09-13T14:33:42Z

contrib/grpcplugins/action/kafka-publish/cli_actions.go

+	args := []string{c.Args().First()}
+	args = append(args, c.Args().Tail()...)
+	if len(args) != 5 {
+		cli.ShowCommandHelp(c, "ack")


Error return value of cli.ShowCommandHelp is not checked

golangcibot · 2018-09-13T14:33:42Z

contrib/grpcplugins/action/kafka-publish/main.go

+			panic(err)
+		}
+	})
+	app.Run(os.Args)


Error return value of app.Run is not checked

golangcibot · 2018-09-13T14:33:42Z

contrib/grpcplugins/action/marathon/helpers.go

+import "strings"
+
+//Escape characters
+func Escape(s string) string {


Escape is unused

golangcibot · 2018-09-13T14:33:42Z

contrib/grpcplugins/action/kafka-publish/kafka.go

+}
+
+//Send a plan file to kafka
+func sendFileOnKafka(producer sarama.SyncProducer, topic string, r io.ReadCloser) (int, int, error) {


sendFileOnKafka is unused

golangcibot · 2018-09-13T14:33:42Z

contrib/grpcplugins/action/clair/clairctl/docker/dockercli/dockercli.go

+	if err != nil {
+		return schema1.SignedManifest{}, err
+	}
+	histories, err := client.ImageHistory(context.Background(), imageName)


ineffectual assignment to err

golangcibot · 2018-09-13T14:33:43Z

contrib/grpcplugins/action/kafka-publish/kafka_ack.go

+					fmt.Printf("Unable to read bytes: %s\n", err)
+					continue
+				}
+				fmt.Println("Chunk received - action %d", actionID)


Println call has possible formatting directive %d

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

sguiheux · 2018-09-14T13:17:09Z

engine/worker/requirement.go

 	if _, err := os.Stat(pluginBinary); os.IsNotExist(err) {
+		log.Info("Downloading the plugin %s", binary.Name)


sguiheux · 2018-09-14T13:17:16Z

engine/worker/requirement.go

 			return false, err
 		}
-		if err := os.Chmod(pluginBinary, 0700); err != nil {
+
+		log.Info("Get the binary plugin %s", r.Name)


sguiheux · 2018-09-14T13:17:25Z

engine/worker/run.go

@@ -179,8 +179,9 @@ func (w *currentWorker) runJob(ctx context.Context, a *sdk.Action, buildID int64
 	case sdk.PluginAction:
 		//Define a loggin function
 		sendLog := getLogger(w, buildID, stepOrder)
+		log.Info("w.runGRPCPlugin")


sguiheux · 2018-09-14T13:18:23Z

engine/worker/builtin.go

@@ -191,3 +194,100 @@ func (w *currentWorker) runPlugin(ctx context.Context, a *sdk.Action, buildID in
 		return res
 	}
 }
+
+func (w *currentWorker) runGRPCPlugin(ctx context.Context, a *sdk.Action, buildID int64, params *[]sdk.Parameter, stepOrder int, sendLog LoggerFunc) sdk.Result {
+	log.Info("runPlugin> Begin buildID:%d stepOrder:%d", buildID, stepOrder)


sguiheux · 2018-09-14T13:18:30Z

engine/worker/builtin.go

+func (w *currentWorker) runGRPCPlugin(ctx context.Context, a *sdk.Action, buildID int64, params *[]sdk.Parameter, stepOrder int, sendLog LoggerFunc) sdk.Result {
+	log.Info("runPlugin> Begin buildID:%d stepOrder:%d", buildID, stepOrder)
+	defer func() {
+		log.Info("runPlugin> End buildID:%d stepOrder:%d", buildID, stepOrder)


sguiheux · 2018-09-14T13:18:44Z

engine/worker/builtin.go

+			pluginFail(chanRes, sendLog, fmt.Sprintf("Unable to call grpc plugin manifest... Aborting (%v)", err))
+			return
+		}
+		log.Info("plugin successfully initialized: %#v", m)


Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

bnjjj and others added 18 commits September 5, 2018 10:23

feat(api,worker): migrate all old plugins to new ones

19df537

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

163f65e

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

ba4133d

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

31eb509

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

Merge branch 'master' of github.com:ovh/cds into feat_plugin_os_arch

52acf8f

feat(api,worker): migrate all old plugins to new ones

128113f

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api): add trace on workflow queue (#3306)

2f02db5

fix(ui): better label modals (#3309)

40c6a5b

* fix(ui): better label modals close #3286 Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

fix(ui): let user put empty value in run condition (#3312)

50d828a

close #fix_3311 Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(action/gitTag): advanced parameter prefix (#3301)

a25a018

fix(ui): remove old entry admin about migration (#3313)

4d482ca

Signed-off-by: Yvonnick Esnault <yvonnick.esnault@corp.ovh.com>

fix(hatchery/swarm): remove network and trim quotes in services env …

06cdb6d

…variables (#3316)

feat(api): add grpc plugin action handlers (#3308)

673e772

fix(sdk): name generator - add random (#3317)

57b8a60

There is too many collisions with namegenerator. This explains the non-detecting worker timeout and the running jobs since many days in workflow node run jobs.

fix(hatchery/swarm): count only containers with hatchery label (#3300)

22c65b1

Signed-off-by: Yvonnick Esnault <yvonnick.esnault@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

ce4f681

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

f6d3666

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

7f76fad

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

bnjjj added the feature label Sep 13, 2018

bnjjj requested review from fsamin, richardlt, sguiheux and yesnault as code owners September 13, 2018 10:31

golangcibot reviewed Sep 13, 2018

View reviewed changes

fsamin requested changes Sep 13, 2018

View reviewed changes

feat(api,worker): migrate all old plugins to new ones

06706d1

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

golangcibot reviewed Sep 13, 2018

View reviewed changes

bnjjj added 3 commits September 14, 2018 10:00

feat(api,worker): migrate all old plugins to new ones

77671f7

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

80feefa

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

7097370

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

sguiheux requested changes Sep 14, 2018

View reviewed changes

bnjjj added 2 commits September 14, 2018 16:22

feat(api,worker): migrate all old plugins to new ones

debbef5

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

feat(api,worker): migrate all old plugins to new ones

bf22c14

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>

yesnault approved these changes Sep 14, 2018

View reviewed changes

fsamin approved these changes Sep 17, 2018

View reviewed changes

sguiheux approved these changes Sep 17, 2018

View reviewed changes

sguiheux merged commit 2ae4672 into master Sep 17, 2018

bnjjj deleted the feat_plugin_os_arch_exec branch October 30, 2018 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(api,worker): migrate all old plugins to new ones #3321

feat(api,worker): migrate all old plugins to new ones #3321

bnjjj commented Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

fsamin left a comment

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

golangcibot Sep 13, 2018

bnjjj Sep 14, 2018

golangcibot Sep 13, 2018

sguiheux Sep 14, 2018

sguiheux Sep 14, 2018

sguiheux Sep 14, 2018

sguiheux Sep 14, 2018

sguiheux Sep 14, 2018

sguiheux Sep 14, 2018

		if _, err := os.Stat(pluginBinary); os.IsNotExist(err) {
		log.Info("Downloading the plugin %s", binary.Name)

feat(api,worker): migrate all old plugins to new ones #3321

feat(api,worker): migrate all old plugins to new ones #3321

Conversation

bnjjj commented Sep 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fsamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment