Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add coverage testing on Jenkins #6339

Merged
merged 2 commits into from
Sep 10, 2023
Merged

Add coverage testing on Jenkins #6339

merged 2 commits into from
Sep 10, 2023

Conversation

Pansysk75
Copy link
Member

Runs coverage tests on Jenkins, and uploads results to Codacy.
Relies on fetching grcov utility.

@hkaiser
Copy link
Member

hkaiser commented Sep 1, 2023

Please ignore the cmake formatting issue, that one has trickled through from master.

@hkaiser hkaiser added this to the 1.10.0 milestone Sep 1, 2023
@Pansysk75
Copy link
Member Author

retest lsu

@Pansysk75 Pansysk75 force-pushed the test-coverage branch 4 times, most recently from 9040f9c to d5db114 Compare September 6, 2023 17:05
@StellarBot
Copy link

Performance test report

HPX Performance

Comparison

BENCHMARKFORK_JOIN_EXECUTORPARALLEL_EXECUTORSCHEDULER_EXECUTOR
For Each(=)??(=)

Info

PropertyBeforeAfter
HPX Datetime2023-05-10T12:07:53+00:002023-09-06T17:05:04+00:00
HPX Commitdcb54150f53ba5
Compiler/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Hostnamemedusa08.rostam.cct.lsu.edumedusa08.rostam.cct.lsu.edu
Clusternamerostamrostam
Envfile
Datetime2023-05-10T14:50:18.616050-05:002023-09-06T12:15:03.169581-05:00

Comparison

BENCHMARKNO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch(=)

Info

PropertyBeforeAfter
HPX Datetime2023-05-10T12:07:53+00:002023-09-06T17:05:04+00:00
HPX Commitdcb54150f53ba5
Compiler/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Hostnamemedusa08.rostam.cct.lsu.edumedusa08.rostam.cct.lsu.edu
Clusternamerostamrostam
Envfile
Datetime2023-05-10T14:52:35.047119-05:002023-09-06T12:17:14.408419-05:00

Comparison

BENCHMARKFORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATORPARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATORSCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add(=)(=)(=)
Stream Benchmark - Scale(=)-=
Stream Benchmark - Triad(=)(=)(=)
Stream Benchmark - Copy(=)-(=)

Info

PropertyBeforeAfter
HPX Datetime2023-05-10T12:07:53+00:002023-09-06T17:05:04+00:00
HPX Commitdcb54150f53ba5
Compiler/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Hostnamemedusa08.rostam.cct.lsu.edumedusa08.rostam.cct.lsu.edu
Clusternamerostamrostam
Envfile
Datetime2023-05-10T14:52:52.237641-05:002023-09-06T12:17:31.433601-05:00

Explanation of Symbols

SymbolMEANING
=No performance change (confidence interval within ±1%)
(=)Probably no performance change (confidence interval within ±2%)
(+)/(-)Very small performance improvement/degradation (≤1%)
+/-Small performance improvement/degradation (≤5%)
++/--Large performance improvement/degradation (≤10%)
+++/---Very large performance improvement/degradation (>10%)
?Probably no change, but quite large uncertainty (confidence interval with ±5%)
??Unclear result, very large uncertainty (±10%)
???Something unexpected…

@Pansysk75 Pansysk75 force-pushed the test-coverage branch 8 times, most recently from 13d6f74 to 31c36e4 Compare September 9, 2023 18:33
@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
+1.45% ∅
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (7849eef) 148709 124399 83.65%
Head commit (2b3fdde) 188085 (+39376) 160070 (+35671) 85.11% (+1.45%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#6339) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

@Pansysk75
Copy link
Member Author

Coverage summary from Codacy

...

Finally this worked :) I will disable this guy, we don't need a comment.

I'll add the ReadMe badge as well, and check whether I forgot to include some part of HPX in the build.

Currently, it runs for every PR, which takes 2 to 3 hours on Rostam (because it builds and runs all tests). I think that's fine for now, but we could get away only running it on master, to save resources.

Also, currently, it reports as failed if some build/run failed, but it will intentionally still attempt to get the coverage results.

hkaiser
hkaiser reviewed Sep 10, 2023
View reviewed changes
README.rst Outdated Show resolved Hide resolved
@Pansysk75
Copy link
Member Author

Pansysk75 commented Sep 10, 2023 •
edited
Loading

@hkaiser This is ready to merge, the only caveat being that things like APEX and CUDA (and probably others) aren't included in the tests that are currently being ran, so those parts of the codebase might not show up as "covered" by tests. Same goes for compile time tests.

I suggest merging this first so that we can a few runs and thus get a better image in the dashboard. We can tweak it later if needed.

Oh, also @hkaiser you'll need to disable the commenting because you are the admin in Codacy. It's in the settings tab, under "Integrations". Thanks!

@Pansysk75 Pansysk75 marked this pull request as ready for review September 10, 2023 13:45
@hkaiser
Copy link
Member

hkaiser commented Sep 10, 2023

@hkaiser This is ready to merge, the only caveat being that things like APEX and CUDA (and probably others) aren't included in the tests that are currently being ran, so those parts of the codebase might not show up as "covered" by tests. Same goes for compile time tests.

I suggest merging this first so that we can a few runs and thus get a better image in the dashboard. We can tweak it later if needed.

Thanks @Pansysk75! This is great.

Oh, also @hkaiser you'll need to disable the commenting because you are the admin in Codacy. It's in the settings tab, under "Integrations". Thanks!

Should we really disable the comments?

hkaiser
hkaiser approved these changes Sep 10, 2023
View reviewed changes
Copy link
Member

@hkaiser hkaiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@hkaiser
Copy link
Member

hkaiser commented Sep 10, 2023

bors merge

bors bot pushed a commit that referenced this pull request Sep 10, 2023
@Pansysk75
Merge #6339
46762f9 
6339: Add coverage testing on Jenkins r=hkaiser a=Pansysk75

Runs coverage tests on Jenkins, and uploads results to Codacy.
Relies on fetching `grcov` utility.  

Co-authored-by: Panos Syskakis <pansysk75@gmail.com>
@Pansysk75
Copy link
Member Author

Thanks @Pansysk75! This is great.

My pleasure :))

Should we really disable the comments?

We can give them a chance, I don't mind at all. You can always toggle them directly from the dashboard.

@bors
Copy link

bors bot commented Sep 10, 2023

This PR was included in a batch that successfully built, but then failed to merge into master (it was a non-fast-forward update). It will be automatically retried.

@bors
Copy link

bors bot commented Sep 10, 2023

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

@Pansysk75 Pansysk75 merged commit 0079615 into master Sep 10, 2023
58 of 65 checks passed
@Pansysk75 Pansysk75 deleted the test-coverage branch September 10, 2023 15:39
@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
+1.40% ∅
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (7849eef) 148709 124399 83.65%
Head commit (610f270) 188085 (+39376) 159969 (+35570) 85.05% (+1.40%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#6339) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

@Pansysk75 Pansysk75 restored the test-coverage branch September 16, 2023 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hkaiser hkaiser hkaiser approved these changes

Assignees
No one assigned
Labels
category: CI category: Jenkins
Projects
None yet
Milestone
1.10.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Pansysk75 @hkaiser @StellarBot