Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Many analysis server integration tests time out on Windows bots (marked skipped) #32356

Closed
DanTup opened this Issue Feb 28, 2018 · 11 comments

Comments

Projects
None yet
5 participants
@DanTup
Copy link
Member

commented Feb 28, 2018

I re-enabled all the tests on Windows because they pass locally, however these ones are timing out on the buildbots:

analysis_server/test/integration/analysis/get_navigation_test
analysis_server/test/integration/analysis/reanalyze_concurrent_test

I'm re-suppressing them in pkg/pkg.status for Windows to fix the build.

On my machine they're taking around 13 seconds (on the builbots it's > 30s).

@whesse

This comment has been minimized.

Copy link
Member

commented Mar 1, 2018

The timeouts are happening inside the pkg:test or pkg:unittest framework, so we cannot affect that by changing our test.py timeout. The tests need to be rewritten to not time out, or the framework's timeout needs to be changed for that test if possible.

The status should be changed to RuntimeError (if consistent) or Pass, RuntimeError (if flaky)

@DanTup DanTup changed the title Two analysis server integration tests time out on Windows Three analysis server integration tests time out on Windows Mar 1, 2018

dart-bot pushed a commit that referenced this issue Mar 1, 2018

Update status for analysis server tests on Windows
Various tests are currently flaky so marked with Pass, Slow/RuntimeError.

Bug: #32355
Bug: #32356
Change-Id: I86755a5eeb72ad92eaa22f950306e745c14e67fc
Reviewed-on: https://dart-review.googlesource.com/44260
Reviewed-by: William Hesse <whesse@google.com>

@DanTup DanTup changed the title Three analysis server integration tests time out on Windows Many analysis server integration tests time out on Windows bots Mar 1, 2018

@DanTup DanTup changed the title Many analysis server integration tests time out on Windows bots Many analysis server integration tests time out on Windows bots (marked skipped) Mar 1, 2018

dart-bot pushed a commit that referenced this issue Mar 1, 2018

Mark integration tests as skip on Windows due to timeouts
Bug: #32356
Change-Id: I78613615c2de2869535810536464d3dc04f1f45b
Reviewed-on: https://dart-review.googlesource.com/44542
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
@DanTup

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2018

I haven't dug into this, but I did just test quickly one of these tests on both Windows and my Mac a few times to see if there was a discrepancy between platforms but I see 10-17 seconds for 'test/integration/analysis/reanalyze_concurrent_test.dart` on both platforms.

Possibly some of them would work just setting as slow, but I'd rather we seem them all passing locally on a Windows machine first (they fail for various reasons for me, I haven't investigated yet and might not get chance soon).

@bwilkerson

This comment has been minimized.

Copy link
Member

commented Sep 1, 2018

@devoncarew Is this still an issue?

@DanTup

This comment has been minimized.

Copy link
Member Author

commented Sep 1, 2018

Looks like some of these are still marked as skip:

analysis_server/test/integration/*: Skip # Issue 32356

I'm not sure if there's a way we can run them on the bots to see if they still time out without merging to master? (when I last looked, they were much faster than the timeout locally, but it's not clear if on the bots they were just slower (possible due to less resources/contention) or just hung).

@bwilkerson

This comment has been minimized.

Copy link
Member

commented Sep 2, 2018

We can create a CL that removes the skips, then add the windows bots to the list of bots to run for the dry run.

@DanTup

This comment has been minimized.

Copy link
Member Author

commented Sep 3, 2018

Here's a CL removing the skip:

https://dart-review.googlesource.com/c/sdk/+/72641

I see a "Choose Tryjobs" option and can see a bunch of win ones - is it safe to just tick all of them and run, or might some of them do special things (like build releases)? Seems I didn't include which bots failed in my original message :(

@bwilkerson

This comment has been minimized.

Copy link
Member

commented Sep 3, 2018

It's safe to select all of them, but you only need to select the ones that start with "analyzer-" or "pkg-" because those are the only ones that should be running any of the analyzer tests.

@DanTup

This comment has been minimized.

Copy link
Member Author

commented Sep 4, 2018

I ticked both of those, and they went green... I've kicked off a normal Dry Run and if it passes I guess we should land it and review the full run afterwards.

@bwilkerson

This comment has been minimized.

Copy link
Member

commented Sep 4, 2018

Sounds good to me, thanks!

@DanTup

This comment has been minimized.

Copy link
Member Author

commented Sep 4, 2018

I've landed https://dart-review.googlesource.com/c/sdk/+/72641. I'll check on it in a little while to ensure nothing broke, but it does appear to break anything and you spot it before me, feel free to revert.

dart-bot pushed a commit that referenced this issue Sep 4, 2018

Unskip Windows analysis server integration tests
Bug: #32356
Change-Id: Iec178a071a030864d71fc0413f8a35f7c360b3f1
Reviewed-on: https://dart-review.googlesource.com/72641
Reviewed-by: Brian Wilkerson <brianwilkerson@google.com>
Commit-Queue: Danny Tuppeny <dantup@google.com>
@DanTup

This comment has been minimized.

Copy link
Member Author

commented Sep 4, 2018

As far as I can tell, everything that was previously green remained green, so I believe this is resolved. If they turn out to be flaky and need reverting/investigating, feel free to re-open and/or ping me!

@DanTup DanTup closed this Sep 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.