Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

broken arm ubuntu build from the recent update #23268

Closed
sandreenko opened this Issue Mar 15, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@sandreenko
Copy link
Member

sandreenko commented Mar 15, 2019

#23249 broke arm32 Ubuntu Jenkins testing (at least), see example here.

@AaronRobinsonMSFT could you please fix it and explain why you merged PR with failed jobs?

cc @dotnet/arm32-contrib

@AaronRobinsonMSFT

This comment has been minimized.

Copy link
Member

AaronRobinsonMSFT commented Mar 15, 2019

The logs didn't indicate any build breaks, from what I saw they were related to non-relevant tests under src/tests/Regression. I will look at what is happening here. Sorry.

@AaronRobinsonMSFT

This comment has been minimized.

Copy link
Member

AaronRobinsonMSFT commented Mar 15, 2019

explain why you merged PR with failed jobs?

@sandreenko This was a mistake on my part for sure. The reason I merged was because of the consistently unreliable legs for some runs. See the number of times I needed to restart the legs for the revert #23269. There are so many false negative in these legs and "just rerun it" isn't really a good solution because it breeds a sense of unreliability. I also find it ironic that as soon as I started the PR with the fix, 2 ARM runs failed within 5 minutes which were because of flaky infrastructure. I should have done more to investigate the failing runs, but the vast majority of the time it is because of flakiness in the system - that is why I merged the PR.

cc @RussKeldorph

@AaronRobinsonMSFT

This comment has been minimized.

Copy link
Member

AaronRobinsonMSFT commented Mar 15, 2019

Bah. Both my run as well as @jkoritzinsky's run has failures that appear to be unrelated to either change. I observe in the Azure pipline that both build for ARM on Windows and Linux. Both runs seemed to fail for macOS, but of course pass on rerun. The Jenkins runs seem to have no obvious reason for failure, especially since the Azure piplines pass. All that being said, I don't really know what to trust since both PRs have various levels of "pass". Once I get green from Azure pipline, I am going to check in and ignore the Jenkins runs because if I don't I think I will be here all night.

@sandreenko

This comment has been minimized.

Copy link
Member Author

sandreenko commented Mar 15, 2019

@sandreenko This was a mistake on my part for sure. The reason I merged was because of the consistently unreliable legs for some runs. See the number of times I needed to restart the legs for the revert #23269. There are so many false negative in these legs and "just rerun it" isn't really a good solution because it breeds a sense of unreliability. I also find it ironic that as soon as I started the PR with the fix, 2 ARM runs failed within 5 minutes which were because of flaky infrastructure. I should have done more to investigate the failing runs, but the vast majority of the time it is because of flakiness in the system - that is why I merged the PR.

Yes, the current level of flakiness is awful 😢 . And ADO/Helix do not look like a silver bullet (especially after yesterday outage).

We still are trying to keep both system as green as possible, we ignore issues like:
"Jenkins java closed conenction", but we try to take care of build and test failures with exclusion lists etc. However, for me it looks like it gets harder each day, or maybe the last week was just very unlucky.

Thanks again for the fast fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.