Skip to content

Commit

Permalink
feat(api,ui): add more options about vcs events (#4566)
Browse files Browse the repository at this point in the history
* feat(api,ui): add more options about vcs events

Signed-off-by: Benjamin Coenen <benjamin.coenen@corp.ovh.com>
  • Loading branch information
bnjjj authored and yesnault committed Sep 30, 2019
1 parent ed18a99 commit d534d1c
Show file tree
Hide file tree
Showing 23 changed files with 415 additions and 126 deletions.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
58 changes: 58 additions & 0 deletions docs/content/docs/concepts/workflow/notifications.md
Expand Up @@ -12,6 +12,64 @@ On a workflow you can have 2 kinds of notifications:

You can configure user notifications to send email or a message on jabber with different parameters. Inside the body of the notification you can customise the message thanks to the CDS variable templating with syntax like `{{.cds.myvar}}`. You can also use `HTML` to customise the message, then in order to let CDS interpret your message as an `HTML` one you just need to wrap all your message inside html tag like this `<html>MyContentHere</html>`.

## VCS Notifications

You can configure for which node in your workflow CDS have to send a status on your repository service provider (Github, Bitbucket, ...). You can configure if you want to have a comment on your pull-request when your workflow fails or you can just disable pull-request comment to only have status of your pipelines. By default you already have a default template for your pull-request comment but you can customize it with different kinds of templating. To have access about the `node run` data and write some loops and conditions you can use the standard syntax as the [go templating](https://golang.org/pkg/text/template/#hdr-Actions) but with `[[` `]]` delimitters. You can also use the CDS interpolation engine with the same syntax you already know and use inside pipelines, for example: `{{.cds.workflow}}` to get the name of the workflow.

For the go templating you have few variables you can use/iterate over.

- `.Stages`: an array of stages with `.RunJobs`inside which are the array of runned jobs with their `.Name`
- `.RunJobs`: Inside a stage object which are the array of runned jobs
- `.Job.Action.Name`: The name of the runned job
- `.Job.Status`: The status of runned job
- `.Tests`: array of tests results
- `.Total`: total number of tests
- `.TotalOK`: total number of OK tests
- `.TotalKO`: total number of KO tests
- `.TotalSkipped`: total number of skipped tests

If you need to know about other variable you can check data structure [here](https://github.com/ovh/cds/blob/master/sdk/workflow_run.go#L40).

For example by default the template of pull-request comment is:

```
[[- if .Stages ]]
CDS Report [[.WorkflowNodeName]]#[[.Number]].[[.SubNumber]] [[ if eq .Status "Success" -]] ✔ [[ else ]][[ if eq .Status "Fail" -]] ✘ [[ else ]][[ if eq .Status "Stopped" -]] ■ [[ else ]]- [[ end ]] [[ end ]] [[ end ]]
[[- range $s := .Stages]]
[[- if $s.RunJobs ]]
* [[$s.Name]]
[[- range $j := $s.RunJobs]]
* [[$j.Job.Action.Name]] [[ if eq $j.Status "Success" -]] ✔ [[ else ]][[ if eq $j.Status "Fail" -]] ✘ [[ else ]][[ if eq $j.Status "Stopped" -]] ■ [[ else ]]- [[ end ]] [[ end ]] [[ end ]]
[[- end]]
[[- end]]
[[- end]]
[[- end]]
[[- if .Tests ]]
[[- if gt .Tests.TotalKO 0]]
Unit Tests Report
[[- range $ts := .Tests.TestSuites]]
* [[ $ts.Name ]]
[[range $tc := $ts.TestCases]]
[[- if or ($tc.Errors) ($tc.Failures) ]] * [[ $tc.Name ]] ✘ [[- end]]
[[end]]
[[- end]]
[[- end]]
[[- end]]
```

Which, for a pipeline with 1 stage and a job in failure, is displayed like this:

```
CDS Report build#11.0 ✘
* Stage 1
* my job ✘
```

And displayed on GitHub:

![example_pr_comment.png](../images/example_pr_comment.png?height=200px)
## Events

If you need to trigger some specific actions on the technical side, like for example use a microservice which listens to all events in your workflow (updates, launch, stop, etc.), you can add an event integration like, for example, [Kafka]({{< relref "/docs/integrations/kafka/kafka_events.md">}}) and listen to the kafka topic to trigger some actions on your side. Events are more like sending notifications to machines instead of user notifications which are made for users. The see structure of sent events, you can look [here](https://github.com/ovh/cds/blob/master/sdk/event.go) and [here](https://github.com/ovh/cds/blob/master/sdk/event_workflow.go).
4 changes: 2 additions & 2 deletions docs/content/docs/integrations/bitbucket.md
Expand Up @@ -14,8 +14,8 @@ This integration enables some features:

- [Git Repository Webhook]({{<relref "/docs/concepts/workflow/hooks/git-repo-webhook.md" >}})
- Easy to use action [CheckoutApplication]({{<relref "/docs/actions/builtin-checkoutapplication.md" >}}) and [GitClone]({{<relref "/docs/actions/builtin-gitclone.md">}}) for advanced usage
- Send [build notifications](https://developer.atlassian.com/server/bitbucket/how-tos/updating-build-status-for-commits/) on your Pull-Requests and Commits on Bitbucket
- Send comments on your Pull-Requests when a workflow is failed
- Send [build notifications](https://developer.atlassian.com/server/bitbucket/how-tos/updating-build-status-for-commits/) on your Pull-Requests and Commits on Bitbucket. [More informations]({{<relref "/docs/concepts/workflow/notifications.md#vcs-notifications" >}})
- [Send comments on your Pull-Requests when a workflow is failed]({{<relref "/docs/concepts/workflow/notifications.md#vcs-notifications" >}})

## How to configure Bitbucket Server integration

Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/integrations/bitbucketcloud.md
Expand Up @@ -14,7 +14,7 @@ This integration enables some features:

- [Git Repository Webhook]({{<relref "/docs/concepts/workflow/hooks/git-repo-webhook.md" >}})
- Easy to use action [CheckoutApplication]({{<relref "/docs/actions/builtin-checkoutapplication.md" >}}) and [GitClone]({{<relref "/docs/actions/builtin-gitclone.md">}}) for advanced usage
- Send [build notifications](https://confluence.atlassian.com/bitbucket/check-build-status-in-a-pull-request-945541505.html) on your Pull-Requests and Commits on Bitbucket Cloud
- Send [build notifications](https://confluence.atlassian.com/bitbucket/check-build-status-in-a-pull-request-945541505.html) on your Pull-Requests and Commits on Bitbucket Cloud. [More informations]({{<relref "/docs/concepts/workflow/notifications.md#vcs-notifications" >}})

## How to configure Bitbucket Cloud integration

Expand Down
4 changes: 2 additions & 2 deletions docs/content/docs/integrations/github.md
Expand Up @@ -14,8 +14,8 @@ This integration enables some features:
- [Git Repository Webhook]({{<relref "/docs/concepts/workflow/hooks/git-repo-webhook.md" >}})
- [Git Repository Poller]({{<relref "/docs/concepts/workflow/hooks/git-repo-poller.md" >}})
- Easy to use action [CheckoutApplication]({{<relref "/docs/actions/builtin-checkoutapplication.md" >}}) and [GitClone]({{<relref "/docs/actions/builtin-gitclone.md">}}) for advanced usage
- Send build notifications on your Pull-Requests and Commits on GitHub
- Send comments on your Pull-Requests when a workflow is failed
- Send build notifications on your Pull-Requests and Commits on GitHub. [More informations]({{<relref "/docs/concepts/workflow/notifications.md#vcs-notifications" >}})
- Send comments on your Pull-Requests when a workflow is failed. [More informations]({{<relref "/docs/concepts/workflow/notifications.md#vcs-notifications" >}})

## Resume on what you have to do before using the GitHub Integration

Expand Down
2 changes: 1 addition & 1 deletion docs/content/docs/integrations/gitlab.md
Expand Up @@ -14,7 +14,7 @@ This integration enables some features:

- [Git Repository Webhook]({{<relref "/docs/concepts/workflow/hooks/git-repo-webhook.md" >}})
- Easy to use action [CheckoutApplication]({{<relref "/docs/actions/builtin-checkoutapplication.md" >}}) and [GitClone]({{<relref "/docs/actions/builtin-gitclone.md">}}) for advanced usage
- Send build notifications on your Pull-Requests and Commits on GitLab
- Send build notifications on your Pull-Requests and Commits on GitLab. [More informations]({{<relref "/docs/concepts/workflow/notifications.md#vcs-notifications" >}})


## How to configure GitLab integration
Expand Down
4 changes: 4 additions & 0 deletions engine/api/api.go
Expand Up @@ -757,6 +757,10 @@ func (a *API) Serve(ctx context.Context) error {
a.serviceAPIHeartbeat(ctx)
}, a.PanicDump())

migrate.Add(sdk.Migration{Name: "AddDefaultVCSNotifications", Release: "0.41.0", Mandatory: true, ExecFunc: func(ctx context.Context) error {
return migrate.AddDefaultVCSNotifications(ctx, a.Cache, a.DBConnectionFactory.GetDBMap)
}})

isFreshInstall, errF := version.IsFreshInstall(a.mustDB())
if errF != nil {
return sdk.WrapError(errF, "Unable to check if it's a fresh installation of CDS")
Expand Down
92 changes: 92 additions & 0 deletions engine/api/migrate/vcs_notifications.go
@@ -0,0 +1,92 @@
package migrate

import (
"context"

"github.com/go-gorp/gorp"

"github.com/ovh/cds/engine/api/cache"
"github.com/ovh/cds/engine/api/workflow"
"github.com/ovh/cds/sdk"
"github.com/ovh/cds/sdk/log"
)

func AddDefaultVCSNotifications(ctx context.Context, store cache.Store, DBFunc func() *gorp.DbMap) error {
db := DBFunc()

res := []struct {
WorkflowID int64 `db:"workflow_id"`
NodeID int64 `db:"id"`
}{}

query := `SELECT w_node.workflow_id, w_node.id
FROM w_node
JOIN w_node_context ON w_node_context.node_id = w_node.id
JOIN application ON application.id = w_node_context.application_id
WHERE application.repo_fullname <> ''`

if _, err := db.Select(&res, query); err != nil {
return sdk.WrapError(err, "cannot get workflow id and node id to update")
}

// switch to a hashmap of workflow id with key
nodesByWorkflowID := map[int64][]int64{}
for _, resp := range res {
nodesByWorkflowID[resp.WorkflowID] = append(nodesByWorkflowID[resp.WorkflowID], resp.NodeID)
}

for workflowID, nodes := range nodesByWorkflowID {

count, err := db.SelectInt("SELECT COUNT(id) FROM workflow_notification WHERE workflow_id = $1 and type = $2", workflowID, sdk.VCSUserNotification)
if err != nil {
log.Error("migrate.AddDefaultVCSNotifications> cannot count workflow_notification for workflow id %d", workflowID)
continue
}
if count != 0 {
continue
}

notif := sdk.WorkflowNotification{
Settings: sdk.UserNotificationSettings{
Template: &sdk.UserNotificationTemplate{
Body: sdk.DefaultWorkflowNodeRunReport,
},
},
WorkflowID: workflowID,
Type: sdk.VCSUserNotification,
NodeIDs: nodes,
}
dbNotif := workflow.Notification(notif)

tx, err := db.Begin()
if err != nil {
log.Error("migrate.AddDefaultVCSNotifications> cannot begin transaction for workflow id %d : %v", workflowID, err)
continue
}

//Insert the notification
if err := tx.Insert(&dbNotif); err != nil {
_ = tx.Rollback()
log.Error("migrate.AddDefaultVCSNotifications> Unable to insert workflow notification : %v", err)
continue
}
notif.ID = dbNotif.ID

//Insert associations with sources
query := "INSERT INTO workflow_notification_source(workflow_notification_id, node_id) VALUES ($1, $2) ON CONFLICT DO NOTHING"
for i := range notif.NodeIDs {
if _, err := tx.Exec(query, notif.ID, notif.NodeIDs[i]); err != nil {
_ = tx.Rollback()
log.Error("migrate.AddDefaultVCSNotifications> Unable to insert associations between node %d and notification %d : %v", notif.NodeIDs[i], notif.ID, err)
continue
}
}

if err := tx.Commit(); err != nil {
_ = tx.Rollback()
log.Error("migrate.AddDefaultVCSNotifications> cannot commit transaction for workflow id %d : %v", workflowID, err)
}
}

return nil
}
5 changes: 5 additions & 0 deletions engine/api/notification.go
Expand Up @@ -27,6 +27,11 @@ func (api *API) getUserNotificationTypeHandler() service.Handler {
SendToGroups: &sdk.False,
Template: &sdk.UserNotificationTemplateJabber,
},
sdk.VCSUserNotification: {
Template: &sdk.UserNotificationTemplate{
Body: sdk.DefaultWorkflowNodeRunReport,
},
},
}, http.StatusOK)
}
}
Expand Down
29 changes: 29 additions & 0 deletions engine/api/workflow/dao.go
Expand Up @@ -715,14 +715,43 @@ func Insert(db gorp.SqlExecutor, store cache.Store, w *sdk.Workflow, p *sdk.Proj
return sdk.WrapError(err, "Insert> Unable to insert Workflow Data")
}

customVcsNotif := false
// Insert notifications
for i := range w.Notifications {
n := &w.Notifications[i]
if n.Type == sdk.VCSUserNotification {
customVcsNotif = true
}
if err := InsertNotification(db, w, n); err != nil {
return sdk.WrapError(err, "Unable to insert update workflow(%d) notification (%#v)", w.ID, n)
}
}

if !customVcsNotif {
notif := sdk.WorkflowNotification{
Settings: sdk.UserNotificationSettings{
Template: &sdk.UserNotificationTemplate{
Body: sdk.DefaultWorkflowNodeRunReport,
},
},
WorkflowID: w.ID,
Type: sdk.VCSUserNotification,
}
for _, node := range w.WorkflowData.Array() {
if node.IsLinkedToRepo(w) {
if node.Ref == "" {
node.Ref = node.Name
}
notif.SourceNodeRefs = append(notif.SourceNodeRefs, node.Ref)
}
}
if len(notif.SourceNodeRefs) > 0 {
if err := InsertNotification(db, w, &notif); err != nil {
return sdk.WrapError(err, "Unable to insert VCS workflow(%d) notification (%#v)", w.ID, notif)
}
}
}

dbWorkflow := Workflow(*w)
if err := dbWorkflow.PostUpdate(db); err != nil {
return sdk.WrapError(err, "Insert> Unable to create workflow data")
Expand Down
21 changes: 20 additions & 1 deletion engine/api/workflow/dao_notification.go
Expand Up @@ -2,6 +2,7 @@ package workflow

import (
"database/sql"

"github.com/go-gorp/gorp"
"github.com/ovh/cds/engine/api/database/gorpmapping"
"github.com/ovh/cds/sdk"
Expand Down Expand Up @@ -58,12 +59,29 @@ func loadNotification(db gorp.SqlExecutor, w *sdk.Workflow, id int64) (sdk.Workf
if notifNode != nil {
n.SourceNodeRefs = append(n.SourceNodeRefs, notifNode.Name)
}

}

return n, nil
}

func loadVCSNotificationWithNodeID(db gorp.SqlExecutor, workflowID, nodeID int64) (sdk.WorkflowNotification, error) {
dbnotif := Notification{}
query := `SELECT workflow_notification.*
FROM workflow_notification
JOIN workflow_notification_source ON workflow_notification.id = workflow_notification_source.workflow_notification_id
WHERE workflow_notification.workflow_id = $1 AND workflow_notification_source.node_id = $2 AND workflow_notification.type = $3`
//Load the notification
if err := db.SelectOne(&dbnotif, query, workflowID, nodeID, sdk.VCSUserNotification); err != nil {
if err == sql.ErrNoRows {
return sdk.WorkflowNotification{}, nil
}
return sdk.WorkflowNotification{}, sdk.WrapError(err, "Unable to load notification for workflow id %d and node id %d", workflowID, nodeID)
}
dbnotif.WorkflowID = workflowID

return sdk.WorkflowNotification(dbnotif), nil
}

func InsertNotification(db gorp.SqlExecutor, w *sdk.Workflow, n *sdk.WorkflowNotification) error {
n.WorkflowID = w.ID
n.ID = 0
Expand Down Expand Up @@ -122,5 +140,6 @@ func (no *Notification) PostGet(db gorp.SqlExecutor) error {
if err := gorpmapping.JSONNullString(res, &no.Settings); err != nil {
return sdk.WrapError(err, "cannot parse user notification")
}

return nil
}
17 changes: 16 additions & 1 deletion engine/api/workflow/workflow_run_event.go
Expand Up @@ -199,6 +199,21 @@ func sendVCSEventStatus(ctx context.Context, db gorp.SqlExecutor, store cache.St
if !node.IsLinkedToRepo(&wr.Workflow) {
return nil
}

notif, errN := loadVCSNotificationWithNodeID(db, wr.WorkflowID, node.ID)
if errN != nil {
return sdk.WrapError(errN, "cannot load notification")
}

// vcs notification not enabled
if notif.ID == 0 {
return nil
}

if nodeRun.VCSReport == "" {
nodeRun.VCSReport = notif.Settings.Template.Body
}

app = wr.Workflow.Applications[node.Context.ApplicationID]
if node.Context.PipelineID > 0 {
pip = wr.Workflow.Pipelines[node.Context.PipelineID]
Expand Down Expand Up @@ -314,7 +329,7 @@ func sendVCSEventStatus(ctx context.Context, db gorp.SqlExecutor, store cache.St
return fmt.Errorf("sendEvent> err:%s", err)
}

if vcsConf.Type != "gerrit" {
if vcsConf.Type != "gerrit" && (notif.Settings.Template.DisableComment == nil || !*notif.Settings.Template.DisableComment) {
//Check if this branch and this commit is a pullrequest
prs, err := client.PullRequests(ctx, app.RepositoryFullname)
if err != nil {
Expand Down
9 changes: 7 additions & 2 deletions sdk/exportentities/workflow_notif.go
Expand Up @@ -67,7 +67,9 @@ func craftNotificationEntry(w sdk.Workflow, notif sdk.WorkflowNotification) ([]s
entry.Settings.Template.Body = ""
}
if entry.Settings.Template.Body == "" && entry.Settings.Template.Subject == "" {
entry.Settings.Template = nil
if entry.Settings.Template.DisableComment == nil || !*entry.Settings.Template.DisableComment {
entry.Settings.Template = nil
}
}
}

Expand Down Expand Up @@ -177,7 +179,10 @@ func ProcessNotificationValues(notif NotificationEntry) (sdk.WorkflowNotificatio
n.Settings.Template.Subject = defaultTemplate.Subject
}
if n.Settings.Template.Body == "" {
n.Settings.Template.Subject = defaultTemplate.Body
n.Settings.Template.Body = defaultTemplate.Body
}
if n.Settings.Template.DisableComment == nil || !*n.Settings.Template.DisableComment {
n.Settings.Template.DisableComment = nil
}
}
return n, nil
Expand Down
4 changes: 4 additions & 0 deletions sdk/exportentities/workflow_notif_test.go
Expand Up @@ -84,6 +84,10 @@ notifications:
Details : {{.cds.buildURL}}
Triggered by : {{.cds.triggered_by.username}}
Branch : {{.git.branch}}
- type: vcs
settings:
template:
disable_comment: true
`,
},
},
Expand Down

0 comments on commit d534d1c

Please sign in to comment.