Flakefinder v2 - implement several reports #168

dhiller · 2019-08-22T16:55:11Z

As said here flakefinder needs some more work, so this PR (based on #146) has been created to work on it.

Note: after #146 has been merged, this PR needs a rebase

Goal: extend the flakefinder so that it produces reports for:

sum of 4 weeks
sum of last week and
in addition e.g. the last 3 days individually

bazel go module support is not yet that great

Write report to kubevirt bucket under /reports/flakefinder/flakefinder-$isodate.html, then create index.html as wrapping page. This page contains the links to the last 50 reports.

NOTE: this is not yet working, `bazel build //robots/flakefinder` yields a compile error that I need support with to understand and fix.

As the job is analyzing kubevirt repo, it belongs to this group, therefore it is moved there. Remove other definitions for cleanup.

- Pin image id - Remove unnecessary build steps - Add reasonable title attribute to html report files - Add details to README

- Fix formatting - remove personal path

- 24h -> daily / 168h -> weekly / 672h -> four-weekly - Convert report index colspan into header to avoid further problems

dhiller · 2019-08-26T13:23:07Z

ATM the report index page looks like this:

So, as you might guess there are three report spans here (24h, 1 week, and 4 weeks). Of course not really well designed, but should still be useful.

Fix test target, add injection of latest image sha id into job yaml

- Export type for experted method - Remove unused type - export MkaeQuery, add test

slintes

This looks great already, some small comments inline.

Not sure if it's worth for such a small application, I won't block on this, but I think from a structural POV this can be improved a bit:

create cmd and pkg packages and move files there
for e.g. downloader.go: create a Downloader struct, which can be initialized with NewDownloader(...) and takes the context, client etc., and make the funcs methods of that struct

But it's fine for me as it is. This can be done in a follow up, or in case the application grows.

slintes · 2019-08-27T11:08:11Z

robots/flakefinder/README.md

@@ -0,0 +1,74 @@
+flakefinder


nice readme 👍

slintes · 2019-08-27T11:08:42Z

robots/flakefinder/README.md

+
+### Build
+
+    bazel run //robots/flakefinder:app


maybe worth making build and push make targets?

Will come in the next commits. :)

slintes · 2019-08-27T11:09:19Z

robots/flakefinder/downloader.go

+	return res
+}
+
+type finishedStatus struct {


seems to be unused?

Good spot, thanks!

Goland spotted it ;)

slintes · 2019-08-27T11:15:22Z

robots/flakefinder/index.go

+	ReportFiles map[ReportFileMergedDuration]string
+}
+
+type reportFile struct {


slintes · 2019-08-27T11:17:55Z

robots/flakefinder/downloader.go

+				return nil, fmt.Errorf("Cannot read started.json (%s) in bucket '%s'", dirOfStartedJSON, bucket)
+			}
+			if !IsLatestCommit(startedJSON, pr) {
+				break


No, as I saw this is correct as is. I'm currently debugging some kind of mapping problem there.

Added log message if it can't find junit xml, cleaned formatting, added tests for report output.

dhiller · 2019-08-27T12:34:50Z

So, the reason for the empty daily report is that the junit xml file can't be found. Interestingly this is the case for all the merged PRs since yesterday.

dhiller · 2019-08-27T14:43:29Z

I think I found it. @slintes you were partially right.

Add modules, update vendor, update image id to latest.

slintes · 2019-08-29T12:09:59Z

robots/flakefinder/main.go

@@ -23,6 +23,9 @@ import (
 	"context"
 	"flag"
 	"fmt"
+	"github.com/google/go-github/v28/github"
+	"github.com/sirupsen/logrus"
+	"golang.org/x/oauth2"


nit: move these new imports down

slintes · 2019-08-29T12:10:17Z

robots/flakefinder/main.go

@@ -42,7 +44,7 @@ func flagOptions() options {
 	flag.DurationVar(&o.merged, "merged", 24*7*time.Hour, "Filter to issues merged in the time window")
 	flag.Var(&o.endpoint, "endpoint", "GitHub's API endpoint")
 	flag.StringVar(&o.token, "token", "", "Path to github token")
-	flag.StringVar(&o.graphqlEndpoint, "graphql-endpoint", github.DefaultGraphQLEndpoint, "GitHub's GraphQL API Endpoint")
+	//flag.StringVar(&o.graphqlEndpoint, "graphql-endpoint", github.DefaultGraphQLEndpoint, "GitHub's GraphQL API Endpoint")


if not needed, please remove

slintes · 2019-08-29T12:16:49Z

robots/flakefinder/main.go

-	for _, issue := range issues {
-		pr, err := c.GetPullRequest("kubevirt", "kubevirt", issue.Number)
+	for nextPage := 1; nextPage > 0; {
+		issues, response, err := c.Search.Issues(ctx, query, &github.SearchOptions{ListOptions: github.ListOptions{Page: nextPage}})


the new client can do c.PullRequests.List(...), any reason to stay with c.Search.Issues(....) and then get every single PR?

We need to search for merged PRs. I did not find out how to do this with PullRequest.List, or did I overlook something?

OK, changed to use PullRequests.List.

Remove unused method, add comment on why starting from midnight

slintes · 2019-08-29T15:35:36Z

/lgtm

slintes · 2019-08-29T15:35:54Z

great work 👍

dhiller · 2019-08-29T15:37:52Z

/approve

kubevirt-bot · 2019-08-29T15:38:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dhiller

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dhiller]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubevirt-bot · 2019-08-29T15:39:41Z

@dhiller: Updated the job-config configmap in namespace kubevirt-prow using the following files:

key kubevirt-periodics.yaml using file github/ci/prow/files/jobs/kubevirt/kubevirt-periodics.yaml
key project-infra-presubmits.yaml using file github/ci/prow/files/jobs/project-infra/project-infra-presubmits.yaml

In response to this:

As said here flakefinder needs some more work, so this PR (based on #146) has been created to work on it.

Note: after #146 has been merged, this PR needs a rebase

Goal: extend the flakefinder so that it produces reports for:

sum of 4 weeks

sum of last week and

in addition e.g. the last 3 days individually

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* cluster: add support for different runtimes with this patch, we open the road for different helper tools in addition to the standard `gocli`, which is still the default choice. We want to make room for the future podman support, which is currently experimented with in `github.com/fromanirh/pack8s`, and this seems the safest and less invasive way. Signed-off-by: Francesco Romani <fromani@redhat.com> * cluster: move container var under conditional We use the _cli_container var iff we want to use the stable `gocli` tool, hence move the variable declaration under the if branch. Signed-off-by: Francesco Romani <fromani@redhat.com>

rmohr and others added 17 commits August 22, 2019 12:34

Switch to vendor mode

73b7c59

bazel go module support is not yet that great

Add vendor dependencies for flakefinder

cf789d1

Simple flakefinder implementation

da8ddb7

Add popup with prowjob list

6abcb12

First version of report and index generation

28410d4

Write report to kubevirt bucket under /reports/flakefinder/flakefinder-$isodate.html, then create index.html as wrapping page. This page contains the links to the last 50 reports.

Refactoring / cleanup

60e5d63

Add bazel rules for docker, update go rules

a5eb303

NOTE: this is not yet working, `bazel build //robots/flakefinder` yields a compile error that I need support with to understand and fix.

Fix grpc naming dependency problem

facf22d

Creation and pushing of the image

cf3342f

Add flakefinder prow job

e7551b5

Add gazelle:resolve for grpc naming

e67215b

Added flakefinder README

2fe66dd

Remove command from job spec

cfbd252

Change go_image to app

ceb8028

Replace go_image.binary with app.binary, fix args

f7e03dd

Move job periodic-publish-flakefinder-reports to kubevirt-periodics.yaml

3f0df9b

As the job is analyzing kubevirt repo, it belongs to this group, therefore it is moved there. Remove other definitions for cleanup.

Cleanup

4479cf1

- Pin image id - Remove unnecessary build steps - Add reasonable title attribute to html report files - Add details to README

kubevirt-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 22, 2019

kubevirt-bot requested review from danielBelenky and davidvossel August 22, 2019 16:55

kubevirt-bot added the size/XXL label Aug 22, 2019

dhiller mentioned this pull request Aug 23, 2019

simple flakefinder implementation and job configuration #146

Closed

dhiller added 3 commits August 23, 2019 12:45

Format report html

1211bd1

Fix file headers

90bbd8a

Add new line

ae8a550

dhiller force-pushed the flakefinder-v2 branch from 92f6b48 to cd51739 Compare August 23, 2019 16:16

dhiller added 4 commits August 26, 2019 12:30

Address review comments

79b0c49

- Fix formatting - remove personal path

Minor improvements, first tests

20042b9

Add tests for filtering report items

997087a

Add hour duration to report filename

8c6b916

Add three jobs for different report data spans

2cf760b

- 24h -> daily / 168h -> weekly / 672h -> four-weekly - Convert report index colspan into header to avoid further problems

dhiller force-pushed the flakefinder-v2 branch from cd51739 to 2cf760b Compare August 26, 2019 11:49

fedepaol mentioned this pull request Aug 27, 2019

Background color in flakefinder overview report #174

Closed

dhiller added 2 commits August 27, 2019 11:23

Makefile cleanup

6d01663

Fix test target, add injection of latest image sha id into job yaml

Further cleanup

8309792

- Export type for experted method - Remove unused type - export MkaeQuery, add test

slintes reviewed Aug 27, 2019

View reviewed changes

Further cleanup

cf9ec3a

Added log message if it can't find junit xml, cleaned formatting, added tests for report output.

Fix skipping reports

3e4404c

dhiller added 5 commits August 27, 2019 16:45

Remove binaries

c83cfc6

Fix report data collation

a9f7299

Change ot google github client api

0be4d23

Fix breaking api changes

4f3beeb

Finalize changes with new github client

5f14db6

Add modules, update vendor, update image id to latest.

dhiller changed the title ~~[WIP] Flakefinder v2 - implement several reports~~ Flakefinder v2 - implement several reports Aug 29, 2019

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 29, 2019

slintes reviewed Aug 29, 2019

View reviewed changes

Use PR.List only

f88229c

Remove unused method, add comment on why starting from midnight

kubevirt-bot assigned slintes Aug 29, 2019

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 29, 2019

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 29, 2019

kubevirt-bot merged commit db3ccd8 into kubevirt:master Aug 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flakefinder v2 - implement several reports #168

Flakefinder v2 - implement several reports #168

dhiller commented Aug 22, 2019

dhiller commented Aug 26, 2019 •

edited

Loading

slintes left a comment

slintes Aug 27, 2019

slintes Aug 27, 2019

dhiller Aug 27, 2019

slintes Aug 27, 2019

dhiller Aug 27, 2019

slintes Aug 27, 2019

slintes Aug 27, 2019

slintes Aug 27, 2019

dhiller Aug 27, 2019

dhiller commented Aug 27, 2019

dhiller commented Aug 27, 2019

slintes Aug 29, 2019

slintes Aug 29, 2019

slintes Aug 29, 2019

dhiller Aug 29, 2019 •

edited

Loading

dhiller Aug 29, 2019

slintes commented Aug 29, 2019

slintes commented Aug 29, 2019

dhiller commented Aug 29, 2019

kubevirt-bot commented Aug 29, 2019

kubevirt-bot commented Aug 29, 2019

Flakefinder v2 - implement several reports #168

Flakefinder v2 - implement several reports #168

Conversation

dhiller commented Aug 22, 2019

dhiller commented Aug 26, 2019 • edited Loading

slintes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhiller commented Aug 27, 2019

dhiller commented Aug 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhiller Aug 29, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slintes commented Aug 29, 2019

slintes commented Aug 29, 2019

dhiller commented Aug 29, 2019

kubevirt-bot commented Aug 29, 2019

kubevirt-bot commented Aug 29, 2019

dhiller commented Aug 26, 2019 •

edited

Loading

dhiller Aug 29, 2019 •

edited

Loading