Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] Detect '0 tests' when running emulator tests #11078

Closed
david-allison opened this issue Apr 24, 2022 · 1 comment · Fixed by #11096
Closed

[Enhancement] Detect '0 tests' when running emulator tests #11078

david-allison opened this issue Apr 24, 2022 · 1 comment · Fixed by #11096
Labels
Dev Development, testing & CI
Milestone
2.16 release

Comments

@david-allison
Copy link
Member

david-allison commented Apr 24, 2022 •
edited
Loading

We were testing the API21 emulator, and it didn't run any tests, but the action passed.

We should modify the github action to check the number of tests run for emulator tests and ensure that it's greater than zero.

If it's not, fail the action
@david-allison david-allison added the Dev Development, testing & CI label Apr 24, 2022
@mikehardy
Copy link
Member

mikehardy commented Apr 24, 2022 •
edited
Loading

Suggested implementation is after the emulator test run, inspect the actual test results report to make sure tests executed were > 0

probably determine filename --> cat file --> grep for the results line pattern --> cut out the number of tests executed, test for > 0 or similar

the test results file will need to be parsed like that because the test itself is "passing", it successfully executes 0 tests, the problem is that for some reason (like on API22 emulator in CI I think?) it finds 0 tests instead of the actual number of androidTests we have

mikehardy added a commit to mikehardy/Anki-Android that referenced this issue Apr 25, 2022
@mikehardy
fix(build): assert that more than zero androidTests executed
5f22b76 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
@mikehardy mikehardy mentioned this issue Apr 25, 2022
Merged
mikehardy added a commit to mikehardy/Anki-Android that referenced this issue Apr 25, 2022
@mikehardy
fix(build): assert that more than zero androidTests executed
2d07ad0 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
mikehardy added a commit to mikehardy/Anki-Android that referenced this issue Apr 26, 2022
@mikehardy
fix(build): assert that more than zero androidTests executed
58af2ea 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
mikehardy added a commit to mikehardy/Anki-Android that referenced this issue Apr 26, 2022
@mikehardy
fix(build): assert that more than zero androidTests executed
33aab79 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
mikehardy added a commit to mikehardy/Anki-Android that referenced this issue Apr 26, 2022
@mikehardy
fix(build): assert that more than zero androidTests executed
bc86a19 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
mikehardy added a commit that referenced this issue Apr 26, 2022
@mikehardy
fix(build): assert that more than zero androidTests executed
88b60ef 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes #11078
@github-actions github-actions bot added this to the 2.16 release milestone Apr 26, 2022
oyeraghib pushed a commit to oyeraghib/Anki-Android that referenced this issue Apr 30, 2022
@mikehardy @oyeraghib
fix(build): assert that more than zero androidTests executed
33d7c21 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
oyeraghib pushed a commit to oyeraghib/Anki-Android that referenced this issue May 14, 2022
@mikehardy @oyeraghib
fix(build): assert that more than zero androidTests executed
fd7f262 
Sometimes an emulator "runs" tests, but it actually discovered zero
tests, runs zero tests, and still reports success. Lookin' at you API21.

Fixes ankidroid#11078
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Dev Development, testing & CI
Projects
None yet
Milestone
2.16 release
Development

Successfully merging a pull request may close this issue.

build(deps): fix build issues mikehardy/Anki-Android
2 participants
@mikehardy @david-allison