Skip to content

Speed up tests#66

Merged
mmiller-max merged 1 commit intomasterfrom
mm/speed-up-tests
Feb 17, 2021
Merged

Speed up tests#66
mmiller-max merged 1 commit intomasterfrom
mm/speed-up-tests

Conversation

@mmiller-max
Copy link
Copy Markdown
Member

Not sure how much this will affect test speed, so a WIP PR.

Will close #51 eventually.

@codecov
Copy link
Copy Markdown

codecov bot commented Feb 1, 2021

Codecov Report

Merging #66 (30877bc) into master (7cbd4c6) will increase coverage by 1.25%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #66      +/-   ##
==========================================
+ Coverage   90.45%   91.70%   +1.25%     
==========================================
  Files           6        6              
  Lines         398      398              
==========================================
+ Hits          360      365       +5     
+ Misses         38       33       -5     
Impacted Files Coverage Δ
src/runner.jl 81.48% <0.00%> (+0.61%) ⬆️
src/testsets.jl 99.00% <0.00%> (+3.96%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cbd4c6...30877bc. Read the comment docs.

@mmiller-max mmiller-max force-pushed the mm/speed-up-tests branch 5 times, most recently from 2fca078 to 2abcac1 Compare February 17, 2021 10:36
@mmiller-max
Copy link
Copy Markdown
Member Author

OK this is better, but still not great. On Ubuntu and Julia V1 its gone from 8m to 5m30s. Changes were:

  • Don't activate test package environment
  • Stop checking registry as much
  • Remove tests that duplicate testing
  • Simplify all test packages
  • Don't clone a fresh registry - we're not testing it

I think most of our time is in the creation of a new Julia process each time TestReports.test is executed. So reducing the time further might be possible by combining tests. However I think it's fine for now, it's not a huge concern and at least it's better.

Open to any suggestions, but also happy just to close this and move on. Can always revisit it later.

@mmiller-max mmiller-max changed the title WIP: Speed up tests Speed up tests Feb 17, 2021
@mmiller-max mmiller-max requested a review from oxinabox February 17, 2021 10:58
Copy link
Copy Markdown
Member

@oxinabox oxinabox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread test/reportgeneration.jl
@testset "Runner tests" begin
@testset "Installed packages by name in primary environment" begin
# Pkg.add
Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
# for speed avoid updating registry, we have a fresh one anyway
Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true```

end

# Own time_taken field
mutable struct TimeTakenTestSet <: AbstractTestSet
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this have been move to not be in a seperate process, that makes sense

- Don't activate test package environment
- Stop checking registry as much
- Remove tests that duplicate testing
- Simplify all test packages
- Don't clone a fresh registry - we're not testing it
@mmiller-max mmiller-max merged commit e4d33bb into master Feb 17, 2021
@mmiller-max mmiller-max deleted the mm/speed-up-tests branch February 17, 2021 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed up Tests

2 participants