Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export flaky tests #211

Merged
merged 1 commit into from
May 17, 2023
Merged

Export flaky tests #211

merged 1 commit into from
May 17, 2023

Conversation

nikita8
Copy link
Contributor

@nikita8 nikita8 commented May 12, 2023

Export list of flaky tests.

Re-queued jobs are not marked as failed. So, to determine the flakiness I am looking into the re-queue count for the test or check if it failed earlier and passed on job retry.

@nikita8 nikita8 force-pushed the nikita8/export-flaky-tests branch 3 times, most recently from 7e40e9d to 349d6ff Compare May 15, 2023 22:08
redis.pipelined do |pipeline|
pipeline.sadd(
key('flaky-reports'),
id.dup.force_encoding(Encoding::BINARY)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
id.dup.force_encoding(Encoding::BINARY)
id.b

ruby/lib/ci/queue/redis/build_record.rb Outdated Show resolved Hide resolved
@nikita8 nikita8 force-pushed the nikita8/export-flaky-tests branch from 085a794 to 5b7e5ab Compare May 16, 2023 20:13
ruby/lib/ci/queue/redis/build_record.rb Outdated Show resolved Hide resolved
ruby/lib/ci/queue/redis/build_record.rb Outdated Show resolved Hide resolved
ruby/lib/minitest/queue/runner.rb Outdated Show resolved Hide resolved
ruby/test/fixtures/log/flaky_tests.txt Outdated Show resolved Hide resolved
@nikita8 nikita8 force-pushed the nikita8/export-flaky-tests branch 2 times, most recently from ecd87eb to 4de2cbc Compare May 16, 2023 20:29
@nikita8 nikita8 marked this pull request as ready for review May 16, 2023 22:50
@@ -48,10 +48,23 @@ def record_error(id, payload, stats: nil)
end

def record_success(id, stats: nil)
redis.pipelined do |pipeline|
errror_reports_deleted_count, requeued_count, _ = redis.pipelined do |pipeline|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
errror_reports_deleted_count, requeued_count, _ = redis.pipelined do |pipeline|
error_reports_deleted_count, requeued_count, _ = redis.pipelined do |pipeline|

@nikita8 nikita8 force-pushed the nikita8/export-flaky-tests branch from 4de2cbc to 96a754a Compare May 17, 2023 15:21
@nikita8 nikita8 merged commit b32c573 into master May 17, 2023
@shopify-shipit shopify-shipit bot temporarily deployed to rubygems May 17, 2023 16:06 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants