Performance of System.Console.CursorLeft and System.Console.CursorTop is really, really bad in .NET Core 2.1.2 on Linux

[dlech]

Test case:

I am trying to use this library as a progress bar for a netcoreapp2.1 console app on Linux. However, I found that since it uses System.Console.CursorLeft and System.Console.CursorTop frequently (in loops), that it was completely unusable. If you click on the link, there are some GIFs in the README that show the demo program completing in about 15-20 seconds. I let the demo program run on .NET Core 2.1.2 on Linux for 12 hours!!! and it was still on 0%. I did some profiling on this and found that 50% of the CPU time was in system calls to the kernel reading and writing stdin.

I've read all of the comments in https://github.com/dotnet/corefx/blob/master/src/System.Console/src/System/ConsolePal.Unix.cs, so I understand that there are always going to be performance issues with this on Unix platforms, but this seems so extreme that surely it can be improved somewhat.

I made this pull request to work around the issue, so hopefully it might help anyone else facing slowdowns because of this.

Steps to reproduce:

• git clone https://github.com/Mpdreamz/shellprogressbar.git
• git checkout 21a516004680 for netcoreapp1.1 or git checkout cd998d37d7f7 for netcoreapp2.1
• dotnet run -p src/ShellProgressBar.Example

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.302
 Commit:    9048955601

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.302/

Host (useful for support):
  Version: 2.1.2
  Commit:  811c3ce6c0

.NET Core SDKs installed:
  2.1.302 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.1.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[dlech]

I made my own GIF so you can see just how slow it is in comparison. 😄

[wtgodbe]

@karelz @stephentoub this seems potentially related to https://github.com/dotnet/corefx/issues/30406 & friends.

@dlech we are considering re-writing our protocol for getting cursor position in vNext of .Net Core, we'll keep this issue in mind when designing/testing our solution.

[stephentoub]

this seems so extreme that surely it can be improved somewhat

Do you have suggestions? We get the cursor position by writing a sequence to the terminal, and then reading and parsing a response from the terminal. That's all inherently slow. The only faster solution I'm aware of is keeping track of what we think the current position is, but that's very fragile.

[dlech]

My memory is getting kind of fuzzy, but I think this was working reasonably well in dotnet 1.x (program took minutes instead of days). So maybe look at the changes between 1.x and 2.x?

I did some profiling in dotnet 2.x. If I recall correctly, there were problems with lock contention that were the biggest slowdown in dotnet 2.0, but those were fixed in dotnet 2.1. I don't remember which ones exactly, but it seems like dotnet 2.1.2 was spending most of its time in Linux system calls. If I get a chance, I might try profiling again with dotnet 2.1.3 and see if it makes any difference.

[richlander]

@dlech - want to test your fuzzy thinking relative to 1.x having better perf?

[bradygaster]

@stephentoub just curious what you mean by the fragility? Issue 489 has been linked to this issue, and we'd planned on releasing the RTM of the tool with .NET Core 2.2, but we're going to ship it as a preview given the degraded experience for Ubuntu users and hope to have a fix in place in time for us to RTM (or fully release) the tools in the 3.0 time frame.

I'd like to understand the fragility the fix would cause - we definitely don't want to ask that a "fragile fix" get shipped, but we do want to improve the experience for our Ubuntu users. I'd also like to understand the solution @wtgodbe mentioned above, in which the cursor position protocol might be tweaked. Is there a time frame for this or is it a non-viable solution to this issue?

Thanks all.

[stephentoub]

just curious what you mean by the fragility?

I meant that such an approach would require the System.Console implementation to always have a perfect mirror of exactly where the cursor is. That means it needs to know and be able to hook / react to everything that could possibly move the cursor, including various key strokes, mouse clicks and drags, window resizing, window scrolling, etc., plus such actions hooked by child processes launched by it, etc. I expect it can be done, but the fragility comes from having confidence that we've done it 100% correct. If we miss something, then the System.Console knowledge of where the cursor is could be very out of sync from where it actually is.

in which the cursor position protocol might be tweaked

I'm not sure what @wtgodbe was referring to. Will?

[dlech]

want to test your fuzzy thinking relative to 1.x having better perf?

I just ran the example code with 1.1.10 and it seems to be as slow as 2.1.x.

[richlander]

That's exactly what I was expecting you'd say @stephentoub.

@bradygaster -- does 'top' do anything similar to what you want?

[bradygaster]

@dlech - thanks for verifying the performance is consistent with 1.1.10. that's helpful.

@richlander - this is an interesting consideration.

off the top of my head just imagining visualizing OAI endpoints like this is interesting, but i'm not entirely sure i grok what you're envisioning. mind a sync with @stephentoub and/or @wtgodbe and myself to noodle some ideas friday or next week?

[richlander]

@bradygaster I wasn't suggesting emulating the exact top UX. Instead, I was making the point that top writes dynamic data to the console at arbitrary x,y coordinates and has a certain performance profile. Is that super general UX close to what you want?

Look for something else that does what you want and figure out if it has a model we can emulate in some way. The challenge might be that top can do what it does because it is a constrained app with constrained user input. @stephentoub is likely making the point that we don't have the luxury of a constrained environment. That said, if your app is more like top, then maybe you can rely on your own console implementation that doesn't have to make the same generic reliability guarantees as System.Console.

[bradygaster]

Got it @richlander. Appreciate the clarification. I'm going to tinker around with top more, and also aggressively experiment with the httprepl to see what I'd want as a user from the experience. I've synced up with @mlorbetske on this briefly, too, to get his ideas on the thread, specifically on this:

maybe you can rely on your own console implementation that doesn't have to make the same generic reliability guarantees as System.Console

[wtgodbe]

The change that we were considering is essentially what @stephentoub described - that is, maintaining our own understanding of where the cursor is at all times. As he said, though, that can be very tricky to get correct (and difficult to know that we've gotten it correct). @bradygaster if you'd like to chat with Stephen & I next week I'd be happy to do so

[KathleenDollard]

Just in case it is relevant...

Ubuntu 18.0.4(*) broke global tools because the CurrentCulture changed to en-US-POSIX. Thus code like the following doesn't do what is expected - this code returns false:

Console.WriteLine(string.Compare("nukeeper", "NuKeeper", StringComparison.CurrentCultureIgnoreCase) == 0);

I could imagine complicating scenarios where something similar raised exceptions, causing exceptions to slow things down and cause general chaos.

This wound up in (the beautifully binary issue number) CLI Issue 10101. It was an easy fix for us, because we should have been using InvariantCulture anyway (oops) - so we just removed all use of CurrentCulture in global tools.

(*) We did not confirm that this is restricted to 18.0.4, it is just where we had it reported and it feels plausible that this is a change in or near 18.0.4

[stephentoub]

Just in case it is relevant

I don't believe it is, but thanks :)

Ubuntu 18.0.4(*) broke global tools because the CurrentCulture changed to en-US-POSIX

Just to be clear, CurrentCulture ends up respecting whatever environment is set via environment variables like LANG, and apparently Ubuntu (or the images of it we're using?) is now defaulting to something far from ideal.
cc: @tarekgh