Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Carry over labels from previous jobs in same scenario if still failing #564

Closed
wants to merge 4 commits into from

Conversation

Projects
None yet
4 participants
@okurz
Copy link
Member

commented Feb 22, 2016

It is possible to label all failing tests but tedious to do by a human user
as many failures are just having the same issue until it gets fixed.
It helps if a label is preserved for a build that is still failing. This
idea is inspired by
https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin

The scheduler might not be a good location for "_carry_over_labels" so this
method could also be moved elsewhere.

Related issue: https://progress.opensuse.org/issues/10212

@okurz

This comment has been minimized.

Copy link
Member Author

commented Feb 22, 2016

@aaannz @coolo do you suggest to move the "_carry_over_labels" somewhere else than scheduler? If yes, where?

@@ -669,6 +669,44 @@ sub _job_stop_children {
}
}

sub _carry_over_labels {
my ($job) = @_;
if ($job->comments > 0) {

This comment has been minimized.

Copy link
@aaannz

aaannz Feb 22, 2016

Member

Is it possible to hit this condition? When job is set as !PASSED then comments are carried over, right? So without some good chunk of timing luck, you don't even have chance to add any new comments. Or did I miss something?

This comment has been minimized.

Copy link
@okurz

okurz Feb 22, 2016

Author Member

you are right, this would only hit under special circumstances reg. timing. You could consider this defensive programming. It won't harm and I could easily remove this or try to come up with the cornercase in a test if possible it all ;-) … ok will remove it for now.

# TODO can we also use another user id to tell that
# this comment was created automatically and not by a
# human user?
user_id => $comment->user_id,

This comment has been minimized.

Copy link
@aaannz

aaannz Feb 22, 2016

Member

I think using original comment creator is ok, or create something like comment link. For user-less events in audit log I use undef as NULL user id, but that to work you would need to change Schema/Result/Comments:L53 to

__PACKAGE__->belongs_to(owner => 'OpenQA::Schema::Result::Users', 'user_id', {join_type => 'left'});

This comment has been minimized.

Copy link
@okurz

okurz Feb 22, 2016

Author Member

I am thinking about extending this further down the road to keep a bit of history, e.g. encode the original author but create the new comment by "the machine". So let's stick the the original author for now and extend this later.

Makefile Outdated
@@ -1,4 +1,4 @@
COVERAGE_THRESHOLD ?= 59.8
COVERAGE_THRESHOLD ?= 60.3

This comment has been minimized.

Copy link
@aaannz

aaannz Feb 22, 2016

Member

Oh sh*t, you are serious with this! :)

my %js_settings = map { $_ => $job->settings_hash->{$_} } @scenario_keys;
my $subquery = schema->resultset("JobSettings")->query_for_settings(\%js_settings);
my @conds = [{'me.state' => 'done'}, {id => {'<', $job->id}}, {'me.id' => {-in => $subquery->get_column('job_id')->as_query}}];
my $limit_previous = 200; # arbitrary limit not to search all jobs

This comment has been minimized.

Copy link
@aaannz

aaannz Feb 22, 2016

Member

I'm confused by 200 limit. Do you want to carry comments from older jobs when i.e. the previous one didn't have any comment?

This comment has been minimized.

Copy link
@okurz

okurz Feb 22, 2016

Author Member

I guess the infile comment that I had got lost. The 200 is arbitrary and deserves an explanation. So up to 200 previous incarnations of the same scenario test results are searched. If no label is found within 200, no label is carried over, assuming a different problem to be present.

This comment has been minimized.

Copy link
@okurz

okurz Feb 23, 2016

Author Member

oh, I did find the comment, scroll to the right … funny tidy ;-)

This comment has been minimized.

Copy link
@coolo

coolo Feb 24, 2016

Contributor

that's why you should put comments in front of the code :)

@okurz okurz force-pushed the okurz:feature/comment_carry_over branch from 5c73838 to 0e64e5b Feb 23, 2016

@okurz

This comment has been minimized.

Copy link
Member Author

commented Feb 23, 2016

updated:

  • Move comment for "limit_previous" to its own line to prevent getting lost in right-scrolling
  • Delete statement never hit for "comments already present"
  • Do not carry over labels crossing a passed result
  • Add more tests for different state changes
@@ -0,0 +1,110 @@
# Copyright (C) 2016 SUSE Linux GmbH

This comment has been minimized.

Copy link
@aaannz

aaannz Feb 24, 2016

Member

Its SUSE LLC now.

This comment has been minimized.

Copy link
@okurz

okurz Feb 24, 2016

Author Member

are you sure:

$ git grep -h Copyright | sort | uniq --count | sort -n | tail -n 10
      2 # Copyright (C) 2016 Red Hat
      2 # Copyright (C) 2016 SUSE Linux GmbH
      3 # Copyright (C) 2015 Red Hat
      3 # Copyright (C) 2016 SUSE LLC
      6 # Copyright (c) 2015 SUSE LINUX GmbH, Nuernberg, Germany.
      8  * Copyright jQuery Foundation and other contributors
     10 # Copyright (C) 2015 SUSE LLC
     11 # Copyright (C) 2015 SUSE Linux Products GmbH
     35 # Copyright (C) 2015 SUSE Linux GmbH
     92 # Copyright (C) 2014 SUSE Linux Products GmbH

;-P

I would not mind a copyright checker script, until then …

This comment has been minimized.

Copy link
@sysrich

sysrich Feb 24, 2016

Member

SUSE Linux GmbH and SUSE LLC are both valid

SUSE Linux Products GmbH is no longer valid
On 24 Feb 2016 7:18 pm, "Oliver Kurz" notifications@github.com wrote:

In t/17-labels_carry_over.t
#564 (comment):

@@ -0,0 +1,110 @@
+# Copyright (C) 2016 SUSE Linux GmbH

are you sure:

$ git grep -h Copyright | sort | uniq --count | sort -n | tail -n 10
2 # Copyright (C) 2016 Red Hat
2 # Copyright (C) 2016 SUSE Linux GmbH
3 # Copyright (C) 2015 Red Hat
3 # Copyright (C) 2016 SUSE LLC
6 # Copyright (c) 2015 SUSE LINUX GmbH, Nuernberg, Germany.
8 * Copyright jQuery Foundation and other contributors
10 # Copyright (C) 2015 SUSE LLC
11 # Copyright (C) 2015 SUSE Linux Products GmbH
35 # Copyright (C) 2015 SUSE Linux GmbH
92 # Copyright (C) 2014 SUSE Linux Products GmbH

;-P

I would not mind a copyright checker script, until then …


Reply to this email directly or view it on GitHub
https://github.com/os-autoinst/openQA/pull/564/files#r53981489.

This comment has been minimized.

Copy link
@aaannz

aaannz Feb 25, 2016

Member

Well, @lnussel initiated discussion with legal about that and response was:

SUSE LLC is the owner of SUSE IP. You should use it everywhere. But
don't delete but rather append.

Then there was this #441 (see outdated diff) and since then we started replacing all to SUSE LLC. Not all at once, of course, only when file is changed.

This comment has been minimized.

Copy link
@coolo

coolo Feb 25, 2016

Contributor

SUSE LLC is the one we should use for new code. SUSE Linux GmbH only exists to pay our bills

This comment has been minimized.

Copy link
@okurz

okurz Feb 25, 2016

Author Member

we agreed @aaannz will take a look into checking this automatically.

@coolo

  1. wanna keep as is and merge
  2. I change this single line with a very cost-inefficient commit fixup when other files are still wrong
  3. you have more suggestions how to improve, e.g. regarding the code in Scheduler
    ?

@okurz okurz force-pushed the okurz:feature/comment_carry_over branch from d5e8959 to c47fce9 Feb 24, 2016

@okurz

This comment has been minimized.

Copy link
Member Author

commented Feb 24, 2016

updated:

  • rebased

okurz added some commits Feb 19, 2016

Carry over labels from previous jobs in same scenario if still failing
It is possible to label all failing tests but tedious to do by a human user
as many failures are just having the same issue until it gets fixed.
It helps if a label is preserved for a build that is still failing. This
idea is inspired by
https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin

The scheduler might not be a good location for "_carry_over_labels" so this
method could also be moved elsewhere.

Related issue: https://progress.opensuse.org/issues/10212
Do not carry over labels over passes
After a job passed a new issue in a subsequent fail is assumed to be failed
for a different reason.

The test is restructured into subtests for the individual scenarios.

For restarting the jobs in tests the test detail templates showed an error
because of undefined job start time so we just set it to zero duration in
these cases.

@okurz okurz force-pushed the okurz:feature/comment_carry_over branch from c47fce9 to e110f86 Feb 24, 2016

@coolo

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2016

rebased this as #641

@coolo coolo closed this Apr 13, 2016

@okurz okurz deleted the okurz:feature/comment_carry_over branch Apr 25, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.