Writing hit counts to a shared memory mapped area instead of to a file. #276

petli · 2018-12-21T16:43:51Z

This might speed up UnloadModule enough that it will reliably execute within the
short time that ProcessUnload allows it, even on CI servers with load.

This is heavily based on work done by @Cyberboss, that unfortunately
showed that memory mapped files were to slow to use directly in RecordHit.

Note: I've only had time to test this code on Windows, and based on #241 there might be some differences in the memory map handling on Linux. That must be tested before this can be merged.

@Cyberboss

This might speed up UnloadModule enough that it will reliably execute within the short time that ProcessUnload allows it, even on CI servers with load. This is heavily based on work done by @Cyberboss, that unfortunately showed that memory mapped files were to slow to use directly in RecordHit.

codecov · 2018-12-28T17:24:28Z

Codecov Report

Merging #276 into master will increase coverage by 0.29%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #276      +/-   ##
==========================================
+ Coverage   89.06%   89.36%   +0.29%     
==========================================
  Files          16       16              
  Lines        1912     1946      +34     
==========================================
+ Hits         1703     1739      +36     
+ Misses        209      207       -2

Impacted Files	Coverage Δ
...overlet.core/Instrumentation/InstrumenterResult.cs	`100% <ø> (ø)`	⬆️
src/coverlet.core/Instrumentation/Instrumenter.cs	`99.06% <100%> (+0.01%)`	⬆️
src/coverlet.core/Coverage.cs	`87.01% <100%> (+1.86%)`	⬆️
src/coverlet.core/Symbols/CecilSymbolHelper.cs	`100% <0%> (+0.46%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 63dff09...9b001ad. Read the comment docs.

petli · 2018-12-28T17:28:44Z

Having to fall back to file-based memory maps on Linux. This means I'm not sure how much this will really help with #210.

I did first do an experiment with having a background thread in ModuleTrackerTemplate that collects batches of hits from the threads and writes them to the memory map to try to do more work while the tests are running instead of all of it when it's done. It was predictably slower than this, but still faster than directly updating the memory maps in RecordHit. Perhaps something like that will end up being necessary. (In this case the work using a memory map instead of the hit file will not be wasted, as Coverage can remain as it is and the fancy stuff can be tried in ModuleTrackerTemplate.)

In that experiment I just collected 10k hits in a dictionary before shipping them to the worker thread through a BlockingQueue. Based on that I've been thinking of this approach:

Instrument each method to do a stackalloc of a hit count array for that method invocation
Instrument the hit counting to increment this array, rather than calling RecordHit
Instrument a try-finally to each method that calls RecordHit passing it as a Span<> using the new fancy ref readonly struct things
RecordHit updates the memory map with the hits. This would need to look into tradeoffs between Interlocked.Add() and locking the memory map or a segment of it from other threads

This would add a lot of overhead on calls to small methods, but that might perhaps be offset by a compensating lower total overhead on mid- to large methods.

codemzs · 2019-01-11T01:44:33Z

@tonerdo why is this PR not merged?

petli · 2019-01-11T09:00:03Z

Sorry @codemzs, I think this needs more discussion before merging as I'm dubious it helps much. I'm going to try another approach, so in the meantime I'll close the PR.

petli · 2019-01-11T12:16:33Z

@codemzs @tonerdo Boy, was my hunch wrong (again!). I added measurements to current coverlet, this branch, and another file -writing solution I wanted to try (details below). Writing hit counts to file, either solution, takes around 2000-3000 ticks, while the memory map writing takes around 100 ticks.

Caveats here is the usual: it is difficult to compare the results on a developer laptop with the situation on a loaded build machine. If that has less memory paging could end up causing the mmap solution to be slow too, perhaps slower than the less memory-intense operation of writing to files, but I retract my close and reopens this.

(
For reference, I thought it might be quicker to write the entire hits array as a buffer using the new spans, but it didn't make any significant difference (and in some cases were slower):

var intBuffer = new ReadOnlyMemory<int>(HitsArray);
unsafe
{
    var pinnedBuffer = intBuffer.Pin();
    var byteSpan = new ReadOnlySpan<byte>(pinnedBuffer.Pointer, HitsArray.Length * sizeof(int));
    bw.Write(byteSpan);
}

)

sharwell · 2019-01-13T02:49:58Z

@petli What about using Marshal.Copy?

codemzs · 2019-01-13T22:48:32Z

@petli @tonerdo @Cyberboss Can we please get some traction on this PR?

sharwell · 2019-01-14T04:14:36Z

src/coverlet.core/Instrumentation/ModuleTrackerTemplate.cs

+
+                                        // No need to use Interlocked here since the mutex ensures only one thread updates 
+                                        // the shared memory map.
+                                        *hitLocationArrayOffset += count;


❔ Is the file guaranteed to be zero-filled when it is first created?

Yes, it's either a pure memory map (on Windows) or a new and thus empty file (on Unixlike), and in both cases zero pages are allocated for the map.

test/coverlet.core.tests/Instrumentation/ModuleTrackerTemplateTests.cs

petli · 2019-01-14T10:10:05Z

@petli What about using Marshal.Copy?

@sharwell Does that help getting the bytes into the file? That seems to be more about copying an array into memory, but the Spans solve that without copying.

codemzs · 2019-01-14T18:50:44Z

@petli @tonerdo Is there an ETA when this fix can go in and a new nuget be released? We are trying to integrate coverlet in Microsoft's ML package and currently our CI builds are failing due to bug this PR fixes. I will highly appreciate an ETA to determine if we should use an alternative solution since we are under time constraint. Also, if there is a temporary workaround you can suggest that also will be appreciated.

CC: @shauheen

…ckerTemplate

test/coverlet.core.tests/CoverageTests.cs

# Conflicts: # coverlet.sln

# Conflicts: # src/coverlet.template/ModuleTrackerTemplate.cs

petli · 2019-01-15T21:04:17Z

src/coverlet.template/ModuleTrackerTemplate.cs

            }
+
+            s_isTracking = false;


Here the no-longer threadstatic field is set to false, in the assumption that UnloadModule is not called in parallel in the same appdomain. This is necessary for the multiple-call test case, but if that assumption is not true (cf #291 which changed it from treadstatic) another solution could be found.

Nevermind, @sharwell removes this field entirely in #297... So that should go in to master first, then that change needs to be merged into this PR before it is (again) ready to merge.

petli · 2019-01-15T21:07:56Z

@tonerdo @codemzs @sharwell Merged with the other template-related changes, so this PR is again ready for merging.

sharwell · 2019-01-15T23:47:23Z

I think we should merge #297 first since that is a substantial correctness change for the dotnet/coreclr team, and then update this pull request to focus on the performance improvements.

@petli Sorry for all the churn I know that can be frustrating.

petli · 2019-01-16T09:43:23Z

I think we should merge #297 first since that is a substantial correctness change for the dotnet/coreclr team, and then update this pull request to focus on the performance improvements.

@petli Sorry for all the churn I know that can be frustrating.

I agree on merging #297 first, that order will be easier to sort out codewise too. cc @tonerdo

# Conflicts: # src/coverlet.core/Instrumentation/Instrumenter.cs # src/coverlet.template/ModuleTrackerTemplate.cs

petli · 2019-01-16T11:36:52Z

@tonerdo This PR is now merged with the latest PRs from @sharwell

tonerdo · 2019-01-16T18:47:00Z

@petli quick sanity check that the merge conflict resolution didn't affect the functionality of both this PR and #297

sharwell · 2019-01-16T19:02:39Z

@tonerdo I verified that the most recent form does not impact the intent of my changes that were recently merged

codemzs · 2019-01-16T20:34:27Z

@tonerdo Do we have an ETA for the new nuget with this change?

tonerdo · 2019-01-16T20:54:26Z

@codemzs you'd have a new NuGet within the next 24 hours

codemzs · 2019-01-16T21:02:22Z

Thanks, @tonerdo ! This is good news, looking forward to getting the new nuget and integrating coverlet with dotnet/machinelearning.

CC: @shauheen

ViktorHofer · 2019-01-17T14:02:24Z

Unfortunately this broke System.Private.CoreLib coverage again as MemoryMappedFile isn't available in it: dotnet/corefx#34641 (comment).

ViktorHofer · 2019-01-24T17:20:16Z

@petli you were looped in in the discussion in corefx where Jan pointed out that the early termination is not triggered by the CLR. I created microsoft/vstest#1900 as the vstest framework is the one who terminates the process before the OnProcessExit (+ ProcessExit events) call is completed. This happens if you use dotnet test or dotnet vstest.

I think we should revert this change and follow Jan's recommendation and just do sequential IO transfer. This will unblock the currently failing System.Private.CoreLib coverage measurements.

codemzs · 2019-01-24T17:27:46Z

@ViktorHofer Would that break dotnet/machinelearning?

sharwell · 2019-01-24T18:26:36Z

@codemzs dotnet/machinelearning is still broken due to the vstest bug. I believe the performance improvements in #291 should avoid regressions until vstest is eventually updated to ensure the problem will not occur again.

petli · 2019-01-24T18:46:38Z

I agree, no point complicating things with memory maps when they won't always help.

…

On 24 Jan 2019, at 19:26, Sam Harwell ***@***.***> wrote: @codemzs dotnet/machinelearning is still broken due to the vstest bug. I believe the performance improvements in #291 should avoid regressions until vstest is eventually updated to ensure the problem will not occur again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

ViktorHofer · 2019-01-24T18:53:02Z

Thanks for understanding. Meanwhile I try to get in touch with the vstest team internally to better understand why they are terminating the application too soon.

…pped-hit-counts" This reverts commit 0d285b1, reversing changes made to 15a4e62.

Revert "Merge pull request #276 from petli/memory-mapped-hit-counts"

petli mentioned this pull request Dec 21, 2018

Change hits file instrumentation to be memory mapped #241

Closed

Use file-backed mmaps on Linux

2fb0d73

petli closed this Jan 11, 2019

petli reopened this Jan 11, 2019

codemzs mentioned this pull request Jan 13, 2019

Record hits without thread local variables #291

Merged

codemzs mentioned this pull request Jan 14, 2019

Spurious 'Unable to read beyond end of stream' in CI while collecting results #210

Closed

sharwell reviewed Jan 14, 2019

View reviewed changes

test/coverlet.core.tests/Instrumentation/ModuleTrackerTemplateTests.cs Outdated Show resolved Hide resolved

Use xUnit Collection attribute to prevent parallel tests of ModuleTra…

3985864

…ckerTemplate

sharwell reviewed Jan 14, 2019

View reviewed changes

test/coverlet.core.tests/CoverageTests.cs Show resolved Hide resolved

Use a CollectionFixture to manage ModuleTrackerTemplate

667b5cc

tonerdo mentioned this pull request Jan 15, 2019

0% coverage is being shown after running tests #110

Closed

petli added 2 commits January 15, 2019 14:49

Merge branch 'main-master' into memory-mapped-hit-counts

2e7d751

# Conflicts: # coverlet.sln

Merge branch 'main-master' into memory-mapped-hit-counts

138c8b0

# Conflicts: # src/coverlet.template/ModuleTrackerTemplate.cs

petli commented Jan 15, 2019

View reviewed changes

Merge branch 'main-master' into memory-mapped-hit-counts

9b001ad

# Conflicts: # src/coverlet.core/Instrumentation/Instrumenter.cs # src/coverlet.template/ModuleTrackerTemplate.cs

sharwell approved these changes Jan 16, 2019

View reviewed changes

ViktorHofer mentioned this pull request Jan 16, 2019

Guard instrumentation exclude Interlocked #302

Merged

tonerdo approved these changes Jan 16, 2019

View reviewed changes

tonerdo merged commit 0d285b1 into coverlet-coverage:master Jan 16, 2019

codemzs mentioned this pull request Jan 17, 2019

Add code coverage build def. dotnet/machinelearning#2166

Closed

ViktorHofer mentioned this pull request Jan 17, 2019

[WIP] Enable CoreLib coverage hits dotnet/corefx#34641

Closed

petli deleted the memory-mapped-hit-counts branch January 17, 2019 14:12

ViktorHofer added a commit to ViktorHofer/coverlet that referenced this pull request Jan 24, 2019

Revert "Merge pull request coverlet-coverage#276 from petli/memory-ma…

49cc889

…pped-hit-counts" This reverts commit 0d285b1, reversing changes made to 15a4e62.

ViktorHofer added a commit to ViktorHofer/coverlet that referenced this pull request Jan 24, 2019

Revert "Merge pull request coverlet-coverage#276 from petli/memory-ma…

8ed8807

…pped-hit-counts" This reverts commit 0d285b1, reversing changes made to 15a4e62.

ViktorHofer added a commit to ViktorHofer/coverlet that referenced this pull request Jan 24, 2019

Revert "Merge pull request coverlet-coverage#276 from petli/memory-ma…

72b212d

…pped-hit-counts" This reverts commit 0d285b1, reversing changes made to 15a4e62.

ViktorHofer mentioned this pull request Jan 24, 2019

Revert "Merge pull request #276 from petli/memory-mapped-hit-counts" #322

Merged

ViktorHofer added a commit to ViktorHofer/coverlet that referenced this pull request Jan 24, 2019

Revert "Merge pull request coverlet-coverage#276 from petli/memory-ma…

c71b7d8

…pped-hit-counts" This reverts commit 0d285b1, reversing changes made to 15a4e62.

tonerdo added a commit that referenced this pull request Jan 27, 2019

Merge pull request #322 from ViktorHofer/RevertMMF

c2ba965

Revert "Merge pull request #276 from petli/memory-mapped-hit-counts"

MarcoRossignoli mentioned this pull request Apr 10, 2020

Rethink hits report strategy #808

Closed

petli mentioned this pull request Apr 14, 2020

Add possibility to run coverlet in standalone mode #781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Writing hit counts to a shared memory mapped area instead of to a file. #276

Writing hit counts to a shared memory mapped area instead of to a file. #276

petli commented Dec 21, 2018 •

edited

Loading

codecov bot commented Dec 28, 2018 •

edited

Loading

petli commented Dec 28, 2018 •

edited

Loading

codemzs commented Jan 11, 2019

petli commented Jan 11, 2019

petli commented Jan 11, 2019

sharwell commented Jan 13, 2019

codemzs commented Jan 13, 2019

sharwell Jan 14, 2019

petli Jan 14, 2019

petli commented Jan 14, 2019

codemzs commented Jan 14, 2019 •

edited

Loading

petli Jan 15, 2019

petli Jan 15, 2019

petli commented Jan 15, 2019

sharwell commented Jan 15, 2019 •

edited

Loading

petli commented Jan 16, 2019

petli commented Jan 16, 2019

tonerdo commented Jan 16, 2019

sharwell commented Jan 16, 2019 •

edited

Loading

codemzs commented Jan 16, 2019

tonerdo commented Jan 16, 2019 •

edited

Loading

codemzs commented Jan 16, 2019

ViktorHofer commented Jan 17, 2019

ViktorHofer commented Jan 24, 2019 •

edited

Loading

codemzs commented Jan 24, 2019

sharwell commented Jan 24, 2019

petli commented Jan 24, 2019 via email

ViktorHofer commented Jan 24, 2019

Writing hit counts to a shared memory mapped area instead of to a file. #276

Writing hit counts to a shared memory mapped area instead of to a file. #276

Conversation

petli commented Dec 21, 2018 • edited Loading

codecov bot commented Dec 28, 2018 • edited Loading

Codecov Report

petli commented Dec 28, 2018 • edited Loading

codemzs commented Jan 11, 2019

petli commented Jan 11, 2019

petli commented Jan 11, 2019

sharwell commented Jan 13, 2019

codemzs commented Jan 13, 2019

sharwell Jan 14, 2019

Choose a reason for hiding this comment

petli Jan 14, 2019

Choose a reason for hiding this comment

petli commented Jan 14, 2019

codemzs commented Jan 14, 2019 • edited Loading

petli Jan 15, 2019

Choose a reason for hiding this comment

petli Jan 15, 2019

Choose a reason for hiding this comment

petli commented Jan 15, 2019

sharwell commented Jan 15, 2019 • edited Loading

petli commented Jan 16, 2019

petli commented Jan 16, 2019

tonerdo commented Jan 16, 2019

sharwell commented Jan 16, 2019 • edited Loading

codemzs commented Jan 16, 2019

tonerdo commented Jan 16, 2019 • edited Loading

codemzs commented Jan 16, 2019

ViktorHofer commented Jan 17, 2019

ViktorHofer commented Jan 24, 2019 • edited Loading

codemzs commented Jan 24, 2019

sharwell commented Jan 24, 2019

petli commented Jan 24, 2019 via email

ViktorHofer commented Jan 24, 2019

petli commented Dec 21, 2018 •

edited

Loading

codecov bot commented Dec 28, 2018 •

edited

Loading

petli commented Dec 28, 2018 •

edited

Loading

codemzs commented Jan 14, 2019 •

edited

Loading

sharwell commented Jan 15, 2019 •

edited

Loading

sharwell commented Jan 16, 2019 •

edited

Loading

tonerdo commented Jan 16, 2019 •

edited

Loading

ViktorHofer commented Jan 24, 2019 •

edited

Loading