Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.

Fix nested spin locks in thread pool etw firing #17677

Merged
merged 1 commit into from Apr 20, 2018
Merged

Fix nested spin locks in thread pool etw firing #17677

merged 1 commit into from Apr 20, 2018

Conversation

kouvel
Copy link
Member

@kouvel kouvel commented Apr 19, 2018

UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest fires an ETW event inside a spin lock and a nested spin lock is taken here:

SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)

Nesting spin locks is not allowed, moved the ETW firing outside the lock.

@kouvel
Fix nested spin locks in thread pool etw firing
07f12a2 
`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
@kouvel kouvel added this to the Future milestone Apr 19, 2018
@kouvel kouvel self-assigned this Apr 19, 2018
@kouvel kouvel requested a review from janvorli April 19, 2018 18:32
@kouvel kouvel requested a review from noahfalk April 19, 2018 18:33
@kouvel
Copy link
Member Author

kouvel commented Apr 19, 2018

This happens I guess when this is the first event that is fired in the process (or very rarely thereafter)

@kouvel
Copy link
Member Author

kouvel commented Apr 19, 2018

And occurred while testing tiering

janvorli
janvorli approved these changes Apr 19, 2018
View reviewed changes
Copy link
Member

@janvorli janvorli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@noahfalk
Copy link
Member

LGTM

Adding @brianrob who probably has the most context on this.

@brianrob
Copy link
Member

@kouvel, this will happen on first event on each thread or when a thread exhausts its buffer and needs to write another event.

@kouvel
Copy link
Member Author

kouvel commented Apr 20, 2018

I see, thanks all

@kouvel kouvel merged commit 83fbfec into dotnet:master Apr 20, 2018
@kouvel kouvel deleted the EtwFix branch April 20, 2018 02:47
lkingsford added a commit to lkingsford/coreclr that referenced this pull request Apr 20, 2018
@lkingsford
Merge pull request #1 from dotnet/master
2f0fdc3 
Fix nested spin locks in thread pool etw firing (dotnet#17677)
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 16, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
08778f5 
`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 16, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
7e2ed84 
Port of dotnet#17677 to 2.2

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 16, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
63905da 
Port of dotnet#17677 to 2.2

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 20, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
c766764 
Port of dotnet#17677 to 2.2

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 22, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
6e182c3 
Port of dotnet#17677 to 2.2

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 22, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
8026d15 
Port of dotnet#17677 to 2.2.

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 24, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
1aebaa6 
Port of dotnet#17677 to 2.2

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
kouvel added a commit to kouvel/coreclr that referenced this pull request Aug 27, 2018
@kouvel
Fix nested spin locks in thread pool etw firing (dotnet#17677)
fce15e5 
Port of dotnet#17677 to 2.2

`UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest` fires an ETW event inside a spin lock and a nested spin lock is taken here:

```
SpinLock::GetLock + 0x8E (0x719da71d)
EventPipeBufferManager::AllocateBufferForThread + 0x7B (0x71afa92e)
EventPipeBufferManager::WriteEvent + 0x250 (0x71afbe0a)
EventPipe::WriteEventInternal + 0x125 (0x71951c93)
EventPipe::WriteEvent + 0x91 (0x71951714)
EventPipeWriteEventThreadPoolEnqueue + 0x9E (0x71d468a6)
FireEtwThreadPoolEnqueue + 0xF (0x71b7a04b)
UnManagedPerAppDomainTPCount::QueueUnmanagedWorkRequest + 0x21D (0x71b7a597)
```

Nesting spin locks is not allowed, moved the ETW firing outside the lock.
@kouvel kouvel mentioned this pull request Aug 27, 2018
Merged
kouvel added a commit that referenced this pull request Aug 30, 2018
@kouvel
Merge pull request #19691 from kouvel/Tier22_4
590166a 
This is a port of several changes that went into master after 2.2 forked, including dependencies for, and enabling tiered compilation by default in 2.2. Quick summary of commits is below, see the commit descriptions and PRs for more info.
- Commit 1 - Fix nested spin locks in thread pool etw firing (#17677)
  - Fixes a lock nesting issue when there is an ETW listener, which can occur without tiering, but is almost deterministic with tiering enabled because the first event that is fired typically hits this code path
- Commit 2 - Don't close the JIT func info file on shutdown (#18060)
  - Fixes a crash during shutdown that only occurs when JIT logging is enabled (typically in the coreclr tests and CI). More frequent with tiering enabled because of different JIT timing and background jitting.
- Commit 3 - Apply tiering's call counting delay more broadly (#18610)
  - Fixes a perf issue when tiering is enabled in server first-request scenarios where there is a significant gap between process startup and first request
- Commit 4 - Changes only affect debug builds - Eliminate arm64 contract asserts (#19015)
  - Fixes some incorrect asserts that trigger more frequently with tiering
- Commit 5 - Use 16 bytes to spill SIMD12 (#19237)
  - Fixes a crash in corefx System.Numerics.Tests.Vector3Tests.Vector3EqualsTest. Occurs with minopt JIT or with tiering.
- Commit 6 - Fix an apartment state issue (partial port of #19384)
  - This is a partial port of this PR (only the portion that addresses issue #17822)
  - This is a breaking change, though a minor one that we have concluded is an acceptable risk to take for 2.2
  - Fixes a behavioral difference that can be seen more easily tiering enabled in APIs on the `Thread` class relevant to apartment state. The issue can also be seen in some cases when tiering is disabled.
- Commit 7 - Enable Tiered Compilation by default (#19525)
  - Enables tiering by default, can be disabled through environment, or through .csproj/.json when using dotnet
  - Removes deprecated config variable (EXPERIMENTAL_TieredCompilation) that was previously exposed in 2.1 along with the current config variable (TieredCompilation), along with miscellaneous test fixes
- Commit 8 - Changes only affect tests - Fix tiered compilation option for case-sensitive systems (#19567)
  - Fixes tiering environment variable casing for non-Windows platforms
- Commit 9 - Disable tiered compilation on arm64
  - There is an open issue that may be partly related to minopts on arm64 (https://github.com/dotnet/coreclr/issues/18895). Disabling tiering by default on arm64 to limit exposing new issues.

This change would be followed up with dotnet/corefx#31822
- Adds tests for Commit 6 - Fix an apartment state issue (partial port of #19384)
  - Changes only affect tests

Closes https://github.com/dotnet/coreclr/issues/18973
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@stephentoub stephentoub stephentoub approved these changes

@brianrob brianrob brianrob approved these changes

@janvorli janvorli janvorli approved these changes

@noahfalk noahfalk Awaiting requested review from noahfalk

Assignees

@kouvel kouvel

Labels
area-System.Threading
Projects
None yet
Milestone
Future
6 participants
@kouvel @noahfalk @brianrob @stephentoub @janvorli @dotnet-bot