Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-11576] [Workaround in place] Flaky Driver tests #53981

Open
typesanitizer opened this issue Oct 6, 2019 · 2 comments
Open

[SR-11576] [Workaround in place] Flaky Driver tests #53981

typesanitizer opened this issue Oct 6, 2019 · 2 comments

Comments

@typesanitizer
Copy link

@typesanitizer typesanitizer commented Oct 6, 2019

Previous ID SR-11576
Radar rdar://problem/53507844
Original Reporter @typesanitizer
Type Bug
Status Reopened
Resolution
Additional Detail from JIRA
Votes 0
Component/s Compiler
Labels Bug, Driver, Test
Assignee @beccadax
Priority Medium

md5: 030f49a4cea15f1cd5854a4e823825ae

Issue Description:

I've occasionally run into issues with the tests Driver/linker.swift, Driver/linker-clang_rt.swift and Driver/embed-bitcode.swift being flaky.

The following test results are with e052da7 (tag: swift-DEVELOPMENT-SNAPSHOT-2019-09-25-a) but I've seen the same issues at other points as well.

Here's a Python runner to check this:

#!/usr/bin/env python3

import subprocess
import sys

total_runs = 200
num_fails = 0

for _ in range(total_runs):
  devnull = subprocess.DEVNULL
  done = subprocess.run(['../llvm/utils/lit/lit.py',
    '../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/Driver',
    '--filter=(linker|embed-bitcode)'], stdout=devnull, stderr=devnull)
  if done.returncode != 0:
    num_fails += 1
    sys.stdout.write('-')
  else:
    sys.stdout.write('+')
  sys.stdout.flush()

print("\nEncountered {} failures out of {} total runs".format(
    num_fails, total_runs))

Running the script once gave me 4 failures out of 200 runs.

N.B. The script also runs 4 other tests, not sure how those are interacting with the flaky tests, but it seems hard/impossible to get the failure if only the 2 flaky tests are run.

@typesanitizer
Copy link
Author

@typesanitizer typesanitizer commented Oct 24, 2019

Fixed by PR 27780.

@beccadax
Copy link
Contributor

@beccadax beccadax commented Oct 24, 2019

What landed in #27780 is a workaround for the problem, which is actually a bug in macOS. I’ve filed a bug report with the appropriate team at Apple. The workaround makes some tests slower, so I’m leaving this bug open so we remember to remove the workaround once we can do so.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants