New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profiling API #445

Closed
gregoryyoung opened this Issue Mar 14, 2015 · 79 comments

Comments

Projects
None yet
@gregoryyoung

gregoryyoung commented Mar 14, 2015

Is there any timeframe on a profiling API? Are there any public design docs available at this point?

Greg

@jkotas

This comment has been minimized.

Show comment
Hide comment
@jkotas

jkotas Mar 15, 2015

Member

ICorProfiler profiling APIs are available in CoreCLR (on Windows, on Unix - not yet). They work same as in the full .NET Framework. The main difference is that the environment variables to setup the profiler have CoreClr_ prefix to avoid colliding with the full .NET Framework (ie CORECLR_ENABLE_PROFILING, CORECLR_PROFILER_PATH for CoreCLR vs. COR_ENABLE_PROFILING, COR_PROFILER_PATH for full .NET Framework).

Are you looking for anything particular?

Member

jkotas commented Mar 15, 2015

ICorProfiler profiling APIs are available in CoreCLR (on Windows, on Unix - not yet). They work same as in the full .NET Framework. The main difference is that the environment variables to setup the profiler have CoreClr_ prefix to avoid colliding with the full .NET Framework (ie CORECLR_ENABLE_PROFILING, CORECLR_PROFILER_PATH for CoreCLR vs. COR_ENABLE_PROFILING, COR_PROFILER_PATH for full .NET Framework).

Are you looking for anything particular?

@jkotas jkotas added the question label Mar 15, 2015

@sawilde

This comment has been minimized.

Show comment
Hide comment
@sawilde

sawilde Mar 15, 2015

Do you know why *_PROFILER_PATH doesn't come in 32 and 64 bit versions - it limits this feature as the processes being profiled could be a combination of both but you can only register one.

sawilde commented Mar 15, 2015

Do you know why *_PROFILER_PATH doesn't come in 32 and 64 bit versions - it limits this feature as the processes being profiled could be a combination of both but you can only register one.

@jkotas

This comment has been minimized.

Show comment
Hide comment
@jkotas

jkotas Mar 16, 2015

Member

32-bit vs. 64-bit profiler path: It is a good suggestion. Could you please open a separate issue for it?

Member

jkotas commented Mar 16, 2015

32-bit vs. 64-bit profiler path: It is a good suggestion. Could you please open a separate issue for it?

@MattWhilden

This comment has been minimized.

Show comment
Hide comment
@MattWhilden

MattWhilden Mar 31, 2015

Member

I've opened #601 to track the suggestion by @sawilde.

@gregoryyoung do you have enough information or are there other specific's you'd like?

Member

MattWhilden commented Mar 31, 2015

I've opened #601 to track the suggestion by @sawilde.

@gregoryyoung do you have enough information or are there other specific's you'd like?

@MattWhilden

This comment has been minimized.

Show comment
Hide comment
@MattWhilden

MattWhilden Apr 4, 2015

Member

Alright, I'm closing this down. Let me know if that was in error.

Member

MattWhilden commented Apr 4, 2015

Alright, I'm closing this down. Let me know if that was in error.

@MattWhilden MattWhilden closed this Apr 4, 2015

@schourode

This comment has been minimized.

Show comment
Hide comment
@schourode

schourode Jun 4, 2015

@jkotas You mention that the Profiling API is already available on Windows, but not yet on Unix. Is it the plan to provide this API (or something similar) on all platforms in the future?

If I am not mistaken, this API is what the guys over at NewRelic are missing in order to instrument .NET apps on non-Windows hosts. For reference: https://docs.newrelic.com/docs/agents/net-agent/getting-started/new-relic-net#app-frameworks

schourode commented Jun 4, 2015

@jkotas You mention that the Profiling API is already available on Windows, but not yet on Unix. Is it the plan to provide this API (or something similar) on all platforms in the future?

If I am not mistaken, this API is what the guys over at NewRelic are missing in order to instrument .NET apps on non-Windows hosts. For reference: https://docs.newrelic.com/docs/agents/net-agent/getting-started/new-relic-net#app-frameworks

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Jun 4, 2015

We are basically in the same boat.

On Thu, Jun 4, 2015 at 5:03 PM, Jørn Schou-Rode notifications@github.com
wrote:

@jkotas https://github.com/jkotas You mention that the Profiling API is
already available on Windows, but not yet on Unix. Is it the plan to
provide this API (or something similar) on all platforms in the future?

If I am not mistaken, this API is what the guys over at NewRelic are
missing in order to instrument .NET apps on non-Windows hosts. For
reference:
https://docs.newrelic.com/docs/agents/net-agent/getting-started/new-relic-net#app-frameworks


Reply to this email directly or view it on GitHub
#445 (comment).

Studying for the Turing test

gregoryyoung commented Jun 4, 2015

We are basically in the same boat.

On Thu, Jun 4, 2015 at 5:03 PM, Jørn Schou-Rode notifications@github.com
wrote:

@jkotas https://github.com/jkotas You mention that the Profiling API is
already available on Windows, but not yet on Unix. Is it the plan to
provide this API (or something similar) on all platforms in the future?

If I am not mistaken, this API is what the guys over at NewRelic are
missing in order to instrument .NET apps on non-Windows hosts. For
reference:
https://docs.newrelic.com/docs/agents/net-agent/getting-started/new-relic-net#app-frameworks


Reply to this email directly or view it on GitHub
#445 (comment).

Studying for the Turing test

@jkotas

This comment has been minimized.

Show comment
Hide comment
@jkotas

jkotas Jun 4, 2015

Member

This issue should not have been closed. Yes, we should enable the profiling APIs on Unix to support the ecosystem of tools build on top of them going cross-plat. We are aiming for as much parity as possible for the runtime capabilities across platforms.

cc @sergiy-k @mikem8361 @noahfalk

Member

jkotas commented Jun 4, 2015

This issue should not have been closed. Yes, we should enable the profiling APIs on Unix to support the ecosystem of tools build on top of them going cross-plat. We are aiming for as much parity as possible for the runtime capabilities across platforms.

cc @sergiy-k @mikem8361 @noahfalk

@jkotas jkotas reopened this Jun 4, 2015

@mikem8361 mikem8361 self-assigned this Jun 4, 2015

@david-mitchell

This comment has been minimized.

Show comment
Hide comment
@david-mitchell

david-mitchell Jul 15, 2015

Contributor

I've begun to work on this a bit. You can track my progress (or tell me that I'm doing something terrible) at https://github.com/david-mitchell/coreclr/tree/enable-profiling

Contributor

david-mitchell commented Jul 15, 2015

I've begun to work on this a bit. You can track my progress (or tell me that I'm doing something terrible) at https://github.com/david-mitchell/coreclr/tree/enable-profiling

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Jul 15, 2015

👍

On Wed, Jul 15, 2015 at 5:56 PM, David Mitchell notifications@github.com
wrote:

I've begun to work on this a bit. You can track my progress (or tell me
that I'm doing something terrible) at
https://github.com/david-mitchell/coreclr/tree/enable-profiling


Reply to this email directly or view it on GitHub
#445 (comment).

Studying for the Turing test

gregoryyoung commented Jul 15, 2015

👍

On Wed, Jul 15, 2015 at 5:56 PM, David Mitchell notifications@github.com
wrote:

I've begun to work on this a bit. You can track my progress (or tell me
that I'm doing something terrible) at
https://github.com/david-mitchell/coreclr/tree/enable-profiling


Reply to this email directly or view it on GitHub
#445 (comment).

Studying for the Turing test

@sawilde

This comment has been minimized.

Show comment
Hide comment
@sawilde

sawilde commented Jul 15, 2015

👍

@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Jul 15, 2015

Member

Thanks @david-mitchell ! Last I heard a new coworker here at msft will looking at this area soon (he is still in process of transferring teams). In the meantime feel free to reach out to me for any questions, feedback, PRs, etc.

Member

noahfalk commented Jul 15, 2015

Thanks @david-mitchell ! Last I heard a new coworker here at msft will looking at this area soon (he is still in process of transferring teams). In the meantime feel free to reach out to me for any questions, feedback, PRs, etc.

@david-mitchell

This comment has been minimized.

Show comment
Hide comment
@david-mitchell

david-mitchell Jul 17, 2015

Contributor

I've got the Profiling API building on OS X (haven't tried Linux yet), and I've created a proof-of-concept profiler at https://github.com/david-mitchell/CoreCLRProfiler

Contributor

david-mitchell commented Jul 17, 2015

I've got the Profiling API building on OS X (haven't tried Linux yet), and I've created a proof-of-concept profiler at https://github.com/david-mitchell/CoreCLRProfiler

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Jul 17, 2015

Just my .02 I actually prefer the mono model on this one. It is far easier to comprehend than transporting the windows model.

gregoryyoung commented Jul 17, 2015

Just my .02 I actually prefer the mono model on this one. It is far easier to comprehend than transporting the windows model.

@david-mitchell

This comment has been minimized.

Show comment
Hide comment
@david-mitchell

david-mitchell Jul 18, 2015

Contributor

@gregoryyoung the MS Profiling API has a number of features that are not present in mono's profiler. The New Relic profiler, for example, could not be built on top of Mono's Profiling API as it exists today. (see https://github.com/david-mitchell/NewRelicProfiler for more information on this)

Contributor

david-mitchell commented Jul 18, 2015

@gregoryyoung the MS Profiling API has a number of features that are not present in mono's profiler. The New Relic profiler, for example, could not be built on top of Mono's Profiling API as it exists today. (see https://github.com/david-mitchell/NewRelicProfiler for more information on this)

@sawilde

This comment has been minimized.

Show comment
Hide comment
@sawilde

sawilde Jul 18, 2015

@david-mitchell - all I see is code, no details on the issues - perhaps you could expand more (even if you put it in the wiki).

As @gregoryyoung says the mono approach does seem much cleaner (once you get past the low documentation footprint) than the windows COM way; all that constant querying via COM interfaces (I know I've done my fair share of it) is awfully tedious.

I'd be interested to know where the gaps were when/if I decide to go ahead on the mono version of OpenCover.

sawilde commented Jul 18, 2015

@david-mitchell - all I see is code, no details on the issues - perhaps you could expand more (even if you put it in the wiki).

As @gregoryyoung says the mono approach does seem much cleaner (once you get past the low documentation footprint) than the windows COM way; all that constant querying via COM interfaces (I know I've done my fair share of it) is awfully tedious.

I'd be interested to know where the gaps were when/if I decide to go ahead on the mono version of OpenCover.

@david-mitchell

This comment has been minimized.

Show comment
Hide comment
@david-mitchell

david-mitchell Jul 18, 2015

Contributor

The main issue is that the MS version of the API supplies for method rewriting—along with provisions for allocating memory to be used for this purpose and other related functionality—which mono does not.

In any case, the COM API already exists in CoreCLR, and there are advantages to supporting it—for example, porting existing Windows profilers to OS X/Linux should be much easier if a similar API is preserved.

Contributor

david-mitchell commented Jul 18, 2015

The main issue is that the MS version of the API supplies for method rewriting—along with provisions for allocating memory to be used for this purpose and other related functionality—which mono does not.

In any case, the COM API already exists in CoreCLR, and there are advantages to supporting it—for example, porting existing Windows profilers to OS X/Linux should be much easier if a similar API is preserved.

@kangaroo

This comment has been minimized.

Show comment
Hide comment
@kangaroo

kangaroo Jul 18, 2015

Contributor

Basically this boils down to rejit, which is a big thing for profiler vendors. No?

Contributor

kangaroo commented Jul 18, 2015

Basically this boils down to rejit, which is a big thing for profiler vendors. No?

@sawilde

This comment has been minimized.

Show comment
Hide comment
@sawilde

sawilde Jul 18, 2015

Basically this boils down to rejit, which is a big thing for profiler vendors. No?

Nope, looked into it but in the end never really found a need for it - can't talk for the commercial guys but I've yet to see any code in the open-source community that used it outside just exploratory testing. If however is @david-mitchell says there is no way to rewrite a method (I just assumed the profilers in the Open Source space had never needed to and so no obvious code sample) then that is a bit of a downer.

sawilde commented Jul 18, 2015

Basically this boils down to rejit, which is a big thing for profiler vendors. No?

Nope, looked into it but in the end never really found a need for it - can't talk for the commercial guys but I've yet to see any code in the open-source community that used it outside just exploratory testing. If however is @david-mitchell says there is no way to rewrite a method (I just assumed the profilers in the Open Source space had never needed to and so no obvious code sample) then that is a bit of a downer.

@ayende

This comment has been minimized.

Show comment
Hide comment
@ayende

ayende Jul 18, 2015

TypeMock does rejit, using the profiler API, IIRC.

*Hibernating Rhinos Ltd *

Oren Eini* l CEO l *Mobile: + 972-52-548-6969

Office: +972-4-622-7811 *l *Fax: +972-153-4-622-7811

On Sat, Jul 18, 2015 at 9:34 AM, Shaun Wilde notifications@github.com
wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says there
is no way to rewrite a method (I just assumed the profilers in the Open
Source space had never needed to and so no obvious code sample) then that
is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).

ayende commented Jul 18, 2015

TypeMock does rejit, using the profiler API, IIRC.

*Hibernating Rhinos Ltd *

Oren Eini* l CEO l *Mobile: + 972-52-548-6969

Office: +972-4-622-7811 *l *Fax: +972-153-4-622-7811

On Sat, Jul 18, 2015 at 9:34 AM, Shaun Wilde notifications@github.com
wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says there
is no way to rewrite a method (I just assumed the profilers in the Open
Source space had never needed to and so no obvious code sample) then that
is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).

@mattwarren

This comment has been minimized.

Show comment
Hide comment
@mattwarren

mattwarren Jul 18, 2015

Collaborator

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post
http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde notifications@github.com wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says there
is no way to rewrite a method (I just assumed the profilers in the Open
Source space had never needed to and so no obvious code sample) then that
is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).

Collaborator

mattwarren commented Jul 18, 2015

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post
http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde notifications@github.com wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says there
is no way to rewrite a method (I just assumed the profilers in the Open
Source space had never needed to and so no obvious code sample) then that
is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).

@mattwarren

This comment has been minimized.

Show comment
Hide comment
@mattwarren

mattwarren Jul 18, 2015

Collaborator

Sorry that should've been private eye by @gregoryyoung

On Saturday, 18 July 2015, Matt Warren matt.warren@live.co.uk wrote:

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post
http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says
there is no way to rewrite a method (I just assumed the profilers in the
Open Source space had never needed to and so no obvious code sample) then
that is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).

Collaborator

mattwarren commented Jul 18, 2015

Sorry that should've been private eye by @gregoryyoung

On Saturday, 18 July 2015, Matt Warren matt.warren@live.co.uk wrote:

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post
http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says
there is no way to rewrite a method (I just assumed the profilers in the
Open Source space had never needed to and so no obvious code sample) then
that is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).

@gregoryyoung

This comment has been minimized.

Show comment
Hide comment
@gregoryyoung

gregoryyoung Jul 18, 2015

We are not using it at this time. On mono as an example there is a method
entered callback already

On Sat, Jul 18, 2015 at 10:46 AM, Matt Warren notifications@github.com
wrote:

Sorry that should've been private eye by @gregoryyoung

On Saturday, 18 July 2015, Matt Warren matt.warren@live.co.uk wrote:

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post

http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says
there is no way to rewrite a method (I just assumed the profilers in the
Open Source space had never needed to and so no obvious code sample)
then
that is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).


Reply to this email directly or view it on GitHub
#445 (comment).

Studying for the Turing test

gregoryyoung commented Jul 18, 2015

We are not using it at this time. On mono as an example there is a method
entered callback already

On Sat, Jul 18, 2015 at 10:46 AM, Matt Warren notifications@github.com
wrote:

Sorry that should've been private eye by @gregoryyoung

On Saturday, 18 July 2015, Matt Warren matt.warren@live.co.uk wrote:

ReJIT would be used by production profilers, otherwise they can't
dynamically add/remove instrumentation on methods. I.e in the exact
scenario outlined in this blog post

http://blogs.msdn.com/b/davbr/archive/2011/10/10/rejit-limitations-in-net-4-5.aspx

Also from the demos I've seen, I'm pretty sure that private eye
by @gregyoung is using it.

On Saturday, 18 July 2015, Shaun Wilde <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

Basically this boils down to rejit, which is a big thing for profiler
vendors. No?

Nope, looked into it but in the end never really found a need for it -
can't talk for the commercial guys but I've yet to see any code in the
open-source community that used it outside just exploratory testing. If
however is @david-mitchell https://github.com/david-mitchell says
there is no way to rewrite a method (I just assumed the profilers in the
Open Source space had never needed to and so no obvious code sample)
then
that is a bit of a downer.


Reply to this email directly or view it on GitHub
#445 (comment).


Reply to this email directly or view it on GitHub
#445 (comment).

Studying for the Turing test

@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Jul 20, 2015

Member

@gregoryyoung when you mentioned the Mono model was easier, are you referring specifically to the fact that it has a flat C style convention vs. COM interfaces, or other aspects of the API? Depending on what issues you are trying to address a small adapter might be able to expose a mono-like API on top of the existing COM API? I agree with @david-mitchell that the COM API feels like a good place to start the port.

I can say that we've got some existing non-open source folks already using ReJIT / interested in using ReJIT soon. Short of some significant technical obstacle I've always imagined it on our .Net Core roadmap. The jump-stamp implementation that it relies on might not be trivial to port cross platform so I wouldn't be surprised if it was enabled a little later in the overall porting process.

@david-mitchell I took a quick look at your code changes so far. Looks good, My only comment at this point is that Maoni will probably appreciate if we put the FEATURE_TRACE_EVENT ifdefs inside the implementation of the various ETW methods vs. around the call-site. I'm assuming the various platform linkers are all smart enough to eliminate calls to empty methods. I think @brianrob has been identifying a potential ETW substitute on other platforms and a lot of these methods might start looking like:

void EmitEtwEventXYZ
{
    #ifdef FEATURE_WINDOWS_ETW
    // do the right thing for ETW
    #elif FEATURE_LINUX_EVENTS
    // do the right thing for Linux
    // add more options as needed for other OS tracing technologies
    #endif
}
Member

noahfalk commented Jul 20, 2015

@gregoryyoung when you mentioned the Mono model was easier, are you referring specifically to the fact that it has a flat C style convention vs. COM interfaces, or other aspects of the API? Depending on what issues you are trying to address a small adapter might be able to expose a mono-like API on top of the existing COM API? I agree with @david-mitchell that the COM API feels like a good place to start the port.

I can say that we've got some existing non-open source folks already using ReJIT / interested in using ReJIT soon. Short of some significant technical obstacle I've always imagined it on our .Net Core roadmap. The jump-stamp implementation that it relies on might not be trivial to port cross platform so I wouldn't be surprised if it was enabled a little later in the overall porting process.

@david-mitchell I took a quick look at your code changes so far. Looks good, My only comment at this point is that Maoni will probably appreciate if we put the FEATURE_TRACE_EVENT ifdefs inside the implementation of the various ETW methods vs. around the call-site. I'm assuming the various platform linkers are all smart enough to eliminate calls to empty methods. I think @brianrob has been identifying a potential ETW substitute on other platforms and a lot of these methods might start looking like:

void EmitEtwEventXYZ
{
    #ifdef FEATURE_WINDOWS_ETW
    // do the right thing for ETW
    #elif FEATURE_LINUX_EVENTS
    // do the right thing for Linux
    // add more options as needed for other OS tracing technologies
    #endif
}
@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Feb 6, 2017

Member

PR #7719 has been doing some known work for ELT that is Linux specific. PR #9298 is handling some rejit related issues with the ReadyToRun format that appears OS-agnostic. Other than those two known areas I think the rest is a lack of testing.

Member

noahfalk commented Feb 6, 2017

PR #7719 has been doing some known work for ELT that is Linux specific. PR #9298 is handling some rejit related issues with the ReadyToRun format that appears OS-agnostic. Other than those two known areas I think the rest is a lack of testing.

@lt72 lt72 modified the milestones: 2.0.0, Future Mar 29, 2017

@lt72 lt72 assigned davmason and unassigned tzwlai Mar 29, 2017

@lt72

This comment has been minimized.

Show comment
Hide comment
@lt72

lt72 Mar 29, 2017

Contributor

Mostly just untested, David is working on completing testing already.

Contributor

lt72 commented Mar 29, 2017

Mostly just untested, David is working on completing testing already.

@ViktorHofer

This comment has been minimized.

Show comment
Hide comment
@ViktorHofer

ViktorHofer Apr 20, 2017

Member

We anticipate that .Net Core profiling API will work xplat by mid-Fall. I cannot offer a more precise deadline at this time though.

Sound like we should move it to a future milestone?

Member

ViktorHofer commented Apr 20, 2017

We anticipate that .Net Core profiling API will work xplat by mid-Fall. I cannot offer a more precise deadline at this time though.

Sound like we should move it to a future milestone?

@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Jun 21, 2017

Member

Just a PSA since I know this thread has many people interested in profiler APIs, my recent PR #12193 has some profiling changes in it.

Is there interest in creating an informal group where profiler relevant changes can be announced/discussed in the future? I'm not that wise to ways of github but that seems like something we should be able to do if there was interest?

Member

noahfalk commented Jun 21, 2017

Just a PSA since I know this thread has many people interested in profiler APIs, my recent PR #12193 has some profiling changes in it.

Is there interest in creating an informal group where profiler relevant changes can be announced/discussed in the future? I'm not that wise to ways of github but that seems like something we should be able to do if there was interest?

@rfrancisco

This comment has been minimized.

Show comment
Hide comment
@rfrancisco

rfrancisco Oct 18, 2017

Can you give us a status update of this?

rfrancisco commented Oct 18, 2017

Can you give us a status update of this?

@lt72

This comment has been minimized.

Show comment
Hide comment
@lt72

lt72 Oct 19, 2017

Contributor

Profiler API has now been tested on Linux and Windows for x64/x86. ARM32 testing is in progress and going well, with a few known areas to be completed that are currently being worked on. For tracking ARM32 progress, see #14526, #13992, and #13993.

Contributor

lt72 commented Oct 19, 2017

Profiler API has now been tested on Linux and Windows for x64/x86. ARM32 testing is in progress and going well, with a few known areas to be completed that are currently being worked on. For tracking ARM32 progress, see #14526, #13992, and #13993.

@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Oct 21, 2017

Member

FYI I've got a PR out to update the status page: #14644

Separately, I haven't found a good way to create a group that could be used in @ mentions unless all the people belong to the same organization, but it does seem like there was interest in the general idea above. Does anyone have a good suggestion about how to construct something? Currently my best thought would be to create an issue that is dedicated to profiler related announcements and anyone interested could follow it. On the page we could add links to relevant issues and PRs as they appear. The sole purpose of the page would be to generate a notification to followers, all the actual discussion would occur within the issues that were linked. Is that a weird workaround for not being able to group @ mention? Good?

Member

noahfalk commented Oct 21, 2017

FYI I've got a PR out to update the status page: #14644

Separately, I haven't found a good way to create a group that could be used in @ mentions unless all the people belong to the same organization, but it does seem like there was interest in the general idea above. Does anyone have a good suggestion about how to construct something? Currently my best thought would be to create an issue that is dedicated to profiler related announcements and anyone interested could follow it. On the page we could add links to relevant issues and PRs as they appear. The sole purpose of the page would be to generate a notification to followers, all the actual discussion would occur within the issues that were linked. Is that a weird workaround for not being able to group @ mention? Good?

@jaredcnance

This comment has been minimized.

Show comment
Hide comment
@jaredcnance

jaredcnance Oct 21, 2017

Makes sense. If the goal is an announcements page, I would recommend locking the issue to prevent unofficial conversation. It's similar to the main announcements repo: https://github.com/dotnet/announcements

which may even be a better place for it, I don't know.

jaredcnance commented Oct 21, 2017

Makes sense. If the goal is an announcements page, I would recommend locking the issue to prevent unofficial conversation. It's similar to the main announcements repo: https://github.com/dotnet/announcements

which may even be a better place for it, I don't know.

@lt72 lt72 added the documentation label Nov 7, 2017

@lt72

This comment has been minimized.

Show comment
Hide comment
@lt72

lt72 Nov 8, 2017

Contributor

It seems we need to create a dedicated documentation page in CoreCLR repo.

Contributor

lt72 commented Nov 8, 2017

It seems we need to create a dedicated documentation page in CoreCLR repo.

@sillyousu

This comment has been minimized.

Show comment
Hide comment
@sillyousu

sillyousu Nov 14, 2017

@lt72

Profiler API has now been tested on Linux and Windows for x64/x86. ARM32 testing is in progress and going well, with a few known areas to be completed that are currently being worked on. For tracking ARM32 progress, see #14526, #13992, and #13993.

would you like to point me to a minimal Linux x64 profiler, or point me to your test program (for Linux x64)?

is this profiler still compatible with latest dotnet core CLR?

sillyousu commented Nov 14, 2017

@lt72

Profiler API has now been tested on Linux and Windows for x64/x86. ARM32 testing is in progress and going well, with a few known areas to be completed that are currently being worked on. For tracking ARM32 progress, see #14526, #13992, and #13993.

would you like to point me to a minimal Linux x64 profiler, or point me to your test program (for Linux x64)?

is this profiler still compatible with latest dotnet core CLR?

@sillyousu

This comment has been minimized.

Show comment
Hide comment
@sillyousu

sillyousu Nov 14, 2017

We have this profiler with dotnet daily build 2.1.0-preview1-25907-02, recevied the following random segment fault:

(lldb) register read
General Purpose Registers:
       rax = 0x0000000000000003
       rbx = 0x000000000093db50
       rcx = 0x0000000000000000
       rdx = 0x0000000000940390
       rdi = 0x0000000000000000
       rsi = 0x0000000000940390
       rbp = 0x00007fffef072310
       rsp = 0x00007fffef0722e0
        r8 = 0x0000000000940390
        r9 = 0x00007fff480cc950
       r10 = 0x0000000000000000
       r11 = 0x00007ffff6f5cf10
       r12 = 0x00007fff7d6cf548
       r13 = 0x0000000000674ab0
       r14 = 0x0000000000000001
       r15 = 0x0000000000000000
       rip = 0x00007ffff621f99d  libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93
    rflags = 0x0000000000010206
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000
* thread #8: tid = 6263, 0x00007ffff621f99d libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x00007ffff621f99d libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93
    frame #1: 0x00007ffff63574d6 libcoreclr.so`MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) + 710
    frame #2: 0x00007ffff6356c90 libcoreclr.so`MethodDesc::JitCompileCode(PrepareCodeConfig*) + 416
    frame #3: 0x00007ffff6356a22 libcoreclr.so`MethodDesc::PrepareILBasedCode(PrepareCodeConfig*) + 162
    frame #4: 0x00007ffff6358ed1 libcoreclr.so`MethodDesc::DoPrestub(MethodTable*) + 993
    frame #5: 0x00007ffff635888d libcoreclr.so`PreStubWorker + 445
    frame #6: 0x00007ffff62da3c4 libcoreclr.so`ThePreStub + 92
    frame #7: 0x00007fff8061478b
    frame #8: 0x00007fff8061446f
    frame #9: 0x00007fff7f2ee0b1
    frame #10: 0x00007fff80613141
    frame #11: 0x00007fff80612fd8
    frame #12: 0x00007fff80612ca3
    frame #13: 0x00007fff7ddb097d
    frame #14: 0x00007fff806125ea
    frame #15: 0x00007ffff62d96df libcoreclr.so`CallDescrWorkerInternal + 124
    frame #16: 0x00007ffff61fce92 libcoreclr.so`DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 242
    frame #17: 0x00007ffff637a009 libcoreclr.so`RegisterWaitForSingleObjectCallback_Worker(void*) + 201
    frame #18: 0x00007ffff61cf19d libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 413
    frame #19: 0x00007ffff61cf900 libcoreclr.so`ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) + 64
    frame #20: 0x00007ffff6379e6a libcoreclr.so`RegisterWaitForSingleObjectCallback(void*, unsigned char) + 266
    frame #21: 0x00007ffff61eceb4 libcoreclr.so`ThreadpoolMgr::AsyncCallbackCompletion(void*) + 356
    frame #22: 0x00007ffff635e756 libcoreclr.so`UnManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 470
    frame #23: 0x00007ffff61edcfb libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(void*) + 1211
    frame #24: 0x00007ffff6567652 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 306
    frame #25: 0x00007ffff79c4dc5 libpthread.so.0`start_thread + 197
    frame #26: 0x00007ffff6ed273d libc.so.6`__clone + 109
(lldb) sos ClrStack
OS Thread Id: 0x1877 (8)
        Child SP               IP Call Site
00007FFFEF072550 00007ffff621f99d [PrestubMethodFrame: 00007fffef072550] System.Threading.WaitHandle.Dispose(Boolean)
00007FFFEF0726A0 00007FFF8061478B System.Diagnostics.ProcessWaitHandle.Dispose(Boolean)
00007FFFEF072720 00007FFF8061446F System.Threading.WaitHandle.Dispose()
00007FFFEF0727A0 00007FFF7F2EE0B1 System.Diagnostics.Process.StopWatchingForExit()
00007FFFEF072830 00007FFF80613141 System.Diagnostics.Process.CompletionCallback(System.Object, Boolean)
00007FFFEF0728B0 00007FFF80612FD8 System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context(System.Object, Boolean)
00007FFFEF072930 00007FFF80612CA3 System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context_f(System.Object)
00007FFFEF0729B0 00007FFF7DDB097D System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007FFFEF072A60 00007FFF806125EA System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)
00007FFFEF072BA0 00007ffff62d96df [GCFrame: 00007fffef072ba0]
00007FFFEF072C80 00007ffff62d96df [DebuggerU2MCatchHandlerFrame: 00007fffef072c80]

sillyousu commented Nov 14, 2017

We have this profiler with dotnet daily build 2.1.0-preview1-25907-02, recevied the following random segment fault:

(lldb) register read
General Purpose Registers:
       rax = 0x0000000000000003
       rbx = 0x000000000093db50
       rcx = 0x0000000000000000
       rdx = 0x0000000000940390
       rdi = 0x0000000000000000
       rsi = 0x0000000000940390
       rbp = 0x00007fffef072310
       rsp = 0x00007fffef0722e0
        r8 = 0x0000000000940390
        r9 = 0x00007fff480cc950
       r10 = 0x0000000000000000
       r11 = 0x00007ffff6f5cf10
       r12 = 0x00007fff7d6cf548
       r13 = 0x0000000000674ab0
       r14 = 0x0000000000000001
       r15 = 0x0000000000000000
       rip = 0x00007ffff621f99d  libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93
    rflags = 0x0000000000010206
        cs = 0x0000000000000033
        fs = 0x0000000000000000
        gs = 0x0000000000000000
        ss = 0x000000000000002b
        ds = 0x0000000000000000
        es = 0x0000000000000000
* thread #8: tid = 6263, 0x00007ffff621f99d libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x00007ffff621f99d libcoreclr.so`EEToProfInterfaceImpl::JITCompilationFinished(unsigned long, int, int) + 93
    frame #1: 0x00007ffff63574d6 libcoreclr.so`MethodDesc::JitCompileCodeLockedEventWrapper(PrepareCodeConfig*, ListLockEntryBase<NativeCodeVersion>*) + 710
    frame #2: 0x00007ffff6356c90 libcoreclr.so`MethodDesc::JitCompileCode(PrepareCodeConfig*) + 416
    frame #3: 0x00007ffff6356a22 libcoreclr.so`MethodDesc::PrepareILBasedCode(PrepareCodeConfig*) + 162
    frame #4: 0x00007ffff6358ed1 libcoreclr.so`MethodDesc::DoPrestub(MethodTable*) + 993
    frame #5: 0x00007ffff635888d libcoreclr.so`PreStubWorker + 445
    frame #6: 0x00007ffff62da3c4 libcoreclr.so`ThePreStub + 92
    frame #7: 0x00007fff8061478b
    frame #8: 0x00007fff8061446f
    frame #9: 0x00007fff7f2ee0b1
    frame #10: 0x00007fff80613141
    frame #11: 0x00007fff80612fd8
    frame #12: 0x00007fff80612ca3
    frame #13: 0x00007fff7ddb097d
    frame #14: 0x00007fff806125ea
    frame #15: 0x00007ffff62d96df libcoreclr.so`CallDescrWorkerInternal + 124
    frame #16: 0x00007ffff61fce92 libcoreclr.so`DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 242
    frame #17: 0x00007ffff637a009 libcoreclr.so`RegisterWaitForSingleObjectCallback_Worker(void*) + 201
    frame #18: 0x00007ffff61cf19d libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 413
    frame #19: 0x00007ffff61cf900 libcoreclr.so`ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) + 64
    frame #20: 0x00007ffff6379e6a libcoreclr.so`RegisterWaitForSingleObjectCallback(void*, unsigned char) + 266
    frame #21: 0x00007ffff61eceb4 libcoreclr.so`ThreadpoolMgr::AsyncCallbackCompletion(void*) + 356
    frame #22: 0x00007ffff635e756 libcoreclr.so`UnManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 470
    frame #23: 0x00007ffff61edcfb libcoreclr.so`ThreadpoolMgr::WorkerThreadStart(void*) + 1211
    frame #24: 0x00007ffff6567652 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(void*) + 306
    frame #25: 0x00007ffff79c4dc5 libpthread.so.0`start_thread + 197
    frame #26: 0x00007ffff6ed273d libc.so.6`__clone + 109
(lldb) sos ClrStack
OS Thread Id: 0x1877 (8)
        Child SP               IP Call Site
00007FFFEF072550 00007ffff621f99d [PrestubMethodFrame: 00007fffef072550] System.Threading.WaitHandle.Dispose(Boolean)
00007FFFEF0726A0 00007FFF8061478B System.Diagnostics.ProcessWaitHandle.Dispose(Boolean)
00007FFFEF072720 00007FFF8061446F System.Threading.WaitHandle.Dispose()
00007FFFEF0727A0 00007FFF7F2EE0B1 System.Diagnostics.Process.StopWatchingForExit()
00007FFFEF072830 00007FFF80613141 System.Diagnostics.Process.CompletionCallback(System.Object, Boolean)
00007FFFEF0728B0 00007FFF80612FD8 System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context(System.Object, Boolean)
00007FFFEF072930 00007FFF80612CA3 System.Threading._ThreadPoolWaitOrTimerCallback.WaitOrTimerCallback_Context_f(System.Object)
00007FFFEF0729B0 00007FFF7DDB097D System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00007FFFEF072A60 00007FFF806125EA System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(System.Object, Boolean)
00007FFFEF072BA0 00007ffff62d96df [GCFrame: 00007fffef072ba0]
00007FFFEF072C80 00007ffff62d96df [DebuggerU2MCatchHandlerFrame: 00007fffef072c80]
@sillyousu

This comment has been minimized.

Show comment
Hide comment
@sillyousu

sillyousu Nov 14, 2017

and cc @noahfalk . Maybe you can give me some suggestion.

sillyousu commented Nov 14, 2017

and cc @noahfalk . Maybe you can give me some suggestion.

@sillyousu

This comment has been minimized.

Show comment
Hide comment
@sillyousu

sillyousu Nov 14, 2017

Received segfault with dotnet run --no-build.
But run thousands of rounds dotnet bin/Debug/netcoreapp2.1/Foooo.dll without a problem.

sillyousu commented Nov 14, 2017

Received segfault with dotnet run --no-build.
But run thousands of rounds dotnet bin/Debug/netcoreapp2.1/Foooo.dll without a problem.

@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Nov 15, 2017

Member

@sillyousu - I opened a separate issue so we avoid enlarging this already very meandering thread.

Member

noahfalk commented Nov 15, 2017

@sillyousu - I opened a separate issue so we avoid enlarging this already very meandering thread.

@noahfalk

This comment has been minimized.

Show comment
Hide comment
@noahfalk

noahfalk Nov 21, 2017

Member

I think the original issues that opened this thread have been resolved so I'm going to close it. If that is not the case we should probably open a more specific issue on any outstanding questions given how much this thread has meandered.

Above I mentioned the idea of creating a profiler group, I think a number of you liked that idea, and the best I've been able to find is creating an issue similar to the announcement repo. I spent a while chatting with a few our PMs whether I should actually use the announcement repo but ultimately came to the conclusion that it wasn't the best choice. Expectations there a bit more formal and most of the people watching it probably don't care about the low level stuff that a profiler writer is interested in. Instead I created issue #15136 and anyone who is interested is welcome to follow it. I hope this works out, but if you have a better suggestion I'm always open to hear them. Thanks!

Member

noahfalk commented Nov 21, 2017

I think the original issues that opened this thread have been resolved so I'm going to close it. If that is not the case we should probably open a more specific issue on any outstanding questions given how much this thread has meandered.

Above I mentioned the idea of creating a profiler group, I think a number of you liked that idea, and the best I've been able to find is creating an issue similar to the announcement repo. I spent a while chatting with a few our PMs whether I should actually use the announcement repo but ultimately came to the conclusion that it wasn't the best choice. Expectations there a bit more formal and most of the people watching it probably don't care about the low level stuff that a profiler writer is interested in. Instead I created issue #15136 and anyone who is interested is welcome to follow it. I hope this works out, but if you have a better suggestion I'm always open to hear them. Thanks!

@noahfalk noahfalk closed this Nov 21, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment