Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make _test_xla_generate less flaky #22996

Merged
merged 3 commits into from
Apr 28, 2023
Merged

Make _test_xla_generate less flaky #22996

merged 3 commits into from
Apr 28, 2023

Conversation

ydshieh
Copy link
Collaborator

@ydshieh ydshieh commented Apr 25, 2023 •
edited

What does this PR do?

Make _test_xla_generate less flaky by relaxing the condition:

  • if number of examples < 10: be strict, no difference is allowed
  • otherwise, only fail the test if there are more than 10% of examples give different outputs between XLA and non-XLA versions.

Since this test is slow (generation), better not to decorate with is_flaky.

For TFPegasusModelTest::test_xla_generate_slow: there were more than 10 failures in 70 runs. With this PR, 0 failure shows.

@ydshieh ydshieh marked this pull request as draft April 25, 2023 19:00
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Apr 25, 2023 •
edited

The documentation is not available anymore as the PR was closed or merged.

@ydshieh ydshieh requested review from gante and sgugger April 25, 2023 21:53
@ydshieh ydshieh marked this pull request as ready for review April 25, 2023 21:53
sgugger
sgugger reviewed Apr 25, 2023
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for me if @gante agrees. Thanks!

gante
gante approved these changes Apr 28, 2023
View reviewed changes
Copy link
Member

@gante gante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for thinking about a solution for this flaky problem 🙏

@ydshieh ydshieh merged commit cf7baf4 into main Apr 28, 2023
4 checks passed
@ydshieh ydshieh deleted the less_flaky branch April 28, 2023 11:27
@ydshieh ydshieh mentioned this pull request May 3, 2023
Merged
gojiteji pushed a commit to gojiteji/transformers that referenced this pull request Jun 5, 2023
@ydshieh @gojiteji
Make _test_xla_generate less flaky (huggingface#22996)
0fce494 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
novice03 pushed a commit to novice03/transformers that referenced this pull request Jun 23, 2023
@ydshieh @novice03
Make _test_xla_generate less flaky (huggingface#22996)
601f6b6 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gante gante gante approved these changes

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@ydshieh @HuggingFaceDocBuilderDev @gante @sgugger