You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Main Branch: [](https://devdiv.visualstudio.com/DevDiv/_build/latest?definitionId=11311&branchName=main)
6
6
7
7
## Overview
8
8
@@ -19,6 +19,7 @@ The CLR Instrumentation Engine is a profiler implementation and is enabled and c
19
19
*[Getting Started](docs/getting_started.md) for details on how use the CLR Instrumentation Engine.
MicrosoftInstrumentationEngine_IsPreinstalled|1|The preinstalled site extension for CLRIE sets this to help users know that the applicationHost.xdt file for the preinstalled extension was applied. The Application Insights private site extension won't set this.
46
+
MicrosoftInstrumentationEngine_LatestPath|D:\Program Files (x86)\SiteExtensions\InstrumentationEngine\\[LATEST VERSION]|This environment variable is available in Azure App Service v91+ and allows private site extensions to reference the path to the latest preinstalled InstrumentationEngine.
Copy file name to clipboardExpand all lines: docs/release_process.md
+3-3Lines changed: 3 additions & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Releasing the CLR Instrumentation Engine
2
2
3
-
Release occurs from the Master branch. Please notify clrieowners@microsoft.com to initiate the release process.
3
+
Release occurs from the `main` branch. Please notify clrieowners@microsoft.com to initiate the release process.
4
4
5
5
## Microsoft Internal Process
6
6
@@ -16,8 +16,8 @@ automatically publishes artifacts to the internal NuGet feed with a preview vers
16
16
Based on the changes and targeted platform releases, impacted scenarios and partner teams should be involved in testing for regressions.
17
17
18
18
### Release Phase
19
-
1. Once testing completes, PR to merge develop branch to master branch
20
-
2. Manually run the [Master-Signed](https://devdiv.visualstudio.com/DevDiv/_build/index?context=allDefinitions&path=%5CClrInstrumentationEngine%5CGitHub&definitionId=10055&_a=completed) build to create release artifacts
19
+
1. Once testing completes, PR to merge develop branch to main branch
20
+
2. Manually run the [Signed](https://devdiv.visualstudio.com/DevDiv/_build?definitionId=11311) build to create release artifacts
The below links provide the latest released packages for CLRIE. Some Azure platforms already support CLRIE as a first party extension. (see [here](scenarios/azure.md) for details).
Copy file name to clipboardExpand all lines: docs/scenarios/azure.md
+42-2Lines changed: 42 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -47,6 +47,7 @@ Details for CLR Instrumentation Engine environment variables can be found at [En
47
47
48
48
#### As a Preinstalled Site Extension
49
49
50
+
##### Via App Setting
50
51
In order to enable the CLR Instrumentation Engine preinstalled site extension, go to your Azure App Service resource in the Azure Portal and
51
52
add the below setting to the Application Settings (Currently there's a bug where different components set the key with different casings and
52
53
producing 409 conflict errors):
@@ -60,10 +61,49 @@ Please see [Kudu Site Extensions](https://github.com/projectkudu/kudu/wiki/Azure
60
61
61
62
Please note that applicationHost.xdt files for enabled preinstalled site extension are processed before any private site extension.
62
63
64
+
##### Via ApplicationHost.XDT
65
+
66
+
If you ship a private site extension and want to use Instrumentation Engine cooperatively, the following environment variable in Azure App Service will be enabled in ANT91 (2020, late-Oct/early-Nov):
Where [LATEST VERSION] represents the latest available version of the InstrumentationEngine preinstalled site extension.
71
+
72
+
In the private site extension's applicationHost.xdt, you can leverage this environment variable in the following way, note that each of these are set as `InsertIfMissing` to avoid conflicts if the InstrumentationEngine preinstalled site extension is enabled.
Copy file name to clipboardExpand all lines: docs/troubleshooting.md
+57-7Lines changed: 57 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -1,15 +1,45 @@
1
1
# Troubleshooting
2
2
3
-
## Azure
3
+
## Azure Scenarios
4
4
5
5
See [CLRIE in Azure](scenarios/azure.md).
6
6
7
-
## Diagnosing Issues
7
+
## Mitigation
8
+
9
+
When an issue occurs in production, the first step is to mitigate it so the application continues to run. A common cause of problems are from updates to either CLRIE or to one of the Instrumentation Method client that introduces potential conflicts with another client or incompatiblities with the app.
10
+
11
+
### Remove CLRIE
12
+
13
+
The most direct way to stop CLRIE and all Instrumentation Methods is to remove the ICorProfiler environment variables that are being applied to the process (along with a process restart). This may either require updating applicationHost.xdt files, changing the `InstrumentationEngine_EXTENSION_VERSION` Application Setting, or modifying registry keys. Without the ICorProfiler environment variables, the process effectively removes CLRIE & all IMs that are currently running. If the application without CLRIE is still encountering errors, then the issue is something the app owner will need to address.
14
+
15
+
Alternatively, if the user cannot modify the ICorProfiler environment variable, they may be able to at least remove the dlls which are pointed to by the COR_PROFILER_PATH/CORECLR_PROFILER_PATH environment variables. A process restart will still be required.
16
+
17
+
For Azure VM/VMSS, the Visual Studio Production Diagnostics team ships a CLRIE-hosted MSI in the Microsoft.Insights.VMDiagnosticsSettings extension (Type: Microsoft.Azure.Diagnostics.IaaSDiagnostics). This extension will apply the ICorProfiler variables to the registry for IIS at `HKLM\SYSTEM\CurrentControlSet\Services\W3SVC` and `HKLM\SYSTEM\CurrentControlSet\Services\WAS`. In order to disable CLRIE, you will need to remove the VM extension entirely. You can do so either from the Azure Portal or via Azure PowerShell cmdlet `Remove-AzVMExtension`.
18
+
19
+
### Remove Instrumentation Methods
20
+
21
+
Currently this feature is not supported. We could potentially create a tool that supports an allow/block list of InstrumentationMethods that CLRIE uses to ignore environment variables on startup. Concerns of this implementation involve security implications of file-on-disk or require privileged user access.
22
+
23
+
## Investigations and Diagnosing Issues
24
+
25
+
After mitigation and having the application back up and running, the next step is to figure out who the culprit is that produced the error. We recommend using a test environment rather than testing in production to prvent giving customers a very negative experience.
26
+
27
+
### Enable CLRIE only
28
+
29
+
It is recommended to just enable CLRIE without any Instrumentation Methods to ensure there's no regression in CLRIE's steady-state behavior. This is especially important if CLRIE was recently updated because regressions or incomptibilities might be introduced for the current application. We expect these issues to be rare as CLRIE is tested on a variety of platforms and apps and features are added incrementally with emphasis on supporting backwards compatibility.
30
+
31
+
### Enable Instrumentation Methods (one at a time)
32
+
33
+
Profilers are the main feature providers and so should be the focal point of investigation. Instrumentation Methods are loaded by CLRIE via environment variables that have the prefix `MicrosoftInstrumentationEngine_ConfigPath64_` or `MicrosoftInstrumentationEngine_ConfigPath32_` so app owners can easily determine what Instrumentation Methods are running in their process.
34
+
35
+
The user should enable each Instrumentation Method by itself and see if the application runs into the issue. This can confirm directly whether an Instrumentation Method by itself is causing the issue. This does not confirm if the Instrumentation Method is causing the issue or if it relies on behavior from CLRIE that is broken or has bugs.
36
+
37
+
The more complex scenarios occur when multiple Instrumentation Methods are running as their behaviors may conflict without either parties being aware. For these scenarios it is difficult to find a sole owner. We **recommend** starting a discussion by posting a GitHub issue with tags for each of the involved Instrumentation Methods products. Private discussions or discussions with sensitive information can be done via email or other channels once the involved parties are established.
8
38
9
39
### Enable Instrumentation Engine Logging
10
40
11
41
A quick way to see if any errors are occurring in the CLR Instrumentation Engine or Instrumentation Methods is to enable logging. Please see
12
-
[Environment Variables](environment_variables.md) for detailed information on the allowed values to set.
42
+
[Environment Variables](environment_variables.md) for detailed information on the allowed values to set. For Instrumentation Method authors, we recommend enabling the "Dumps" log level which shows the IL transformations.
13
43
14
44
#### Event Log
15
45
@@ -25,11 +55,31 @@ If `MicrosoftInstrumentationEngine_FileLogPath` is set along with `MicrosoftInst
25
55
occur. If the FilePath is set to a directory, then an automatically generated `%TEMP%\ProfilerLog_[PID].txt` file is used, where [PID]
26
56
represents the process id and the %TEMP% variable is evaluated against the user account running the process.
27
57
58
+
#### Instrumentation Method Logging
59
+
60
+
CLRIE's logging infrastructure is outlined [here](./logging.md). During ICorProfilerCallback::Initialize(), InstrumentationMethods can obtain an InstrumentationMethod-specific logger via `IProfilerManager::GetLoggingInstance()` which will prepend log messages with the `[IM:GUID]`. In addition, IMs can log at a different log level from CLRIE by setting an additional environment variable, allowing IMs to filter out CLRIE or other IM logs. See [ProfilerManagerForInstrumentationMethod](./logging.md#profilermanagerforinstrumentationmethod-hcpp) section for more details.
61
+
62
+
Log parsing or an event collector can utilize these patterns to track InstrumentationMethod behaviors and potentially help diagnose issues.
63
+
28
64
### Debugging the Instrumentation Engine
29
65
30
-
When using a local build of the Instrumentation Engine, you will need to disable signing validations and checks by setting the
66
+
When using a local build of the Instrumentation Engine or testing your InstrumentationMethod, you will need to disable signing validations and checks by setting the
31
67
`MicrosoftInstrumentationEngine_DisableCodeSignatureValidation` to 1.
32
68
33
-
Since the CLR Instrumentation Engine initializes on process start, in order for breakpoints in the initialization logic to hit, you will need
34
-
to set the `MicrosoftInstrumentationEngine_DebugWait` environment variable with the value 1. This will cause the Instrumentation Engine to wait
35
-
on startup for a debugger attach event.
69
+
Since the CLR Instrumentation Engine initializes on process start, in order for breakpoints in the initialization logic to hit, you will need to set the `MicrosoftInstrumentationEngine_DebugWait` environment variable with the value of 1. This will cause the Instrumentation Engine to wait on startup for a debugger attach event.
70
+
71
+
#### Breakpoint Locations
72
+
73
+
The best place to start debugging is in [ProfilerManager.cpp](../src/InstrumentationEngine/ProfilerManager.cpp). ProfilerManager is CLRIE's ICorProfiler implementation which gets the first-chance callback during profiling. It controls setting up InstrumentationMethods and forwarding ICorProfiler callback events to them.
74
+
75
+
Setting a breakpoint inside `CProfilerManager::Initialize()` provides the earliest point where CLRIE is beginning to initialize in the process.
76
+
77
+
`CProfilerManager::AddInstrumentationMethod()` provides the logic for loading each InstrumentationMethod and calling Initialize() on them.
78
+
79
+
CLRIE's main purpose is to coordinate instrumentation, so callbacks such as `CProfilerManager::JITCompilationStarted()` and `CProfilerManager::GetReJITParameters()` are also good places to inspect.
80
+
81
+
#### Loading Symbols for CLR/.NET modules
82
+
83
+
In order to debug issues, it is pertinent to have symbols (ie. pdb files) for the modules/assemblies that the process is using. This will aid in investigating IL issues at the point of failure (eg. ProgramInvalidException).
84
+
85
+
If you are debugging with Visual Studio, make sure you have "Microsoft Symbol Server" enabled for symbol loading in *Tools > Options > Debugging > Symbols > Symbol file locations*. Symbols for both managed modules such as clr.dll and native modules like kernel32.dll are available to be loaded.
0 commit comments