IDE can't run code on Windows, stuck at "Sending request...."

[ghost]

Environment

• SuperCollider version: 3.10.0 64-bit, also tested 32-bit, Bleeding Edge 64 and 32-bit, and 3.7.2.
• Operating system and version: Windows 10 1803, also tested 1511, 1709, 1809.

Steps to reproduce (for bugs)

1. Install SuperCollider 3.10.0 64-bit on Windows 10 1803 (other versions also have the same issue).
2. Open Edit group policy
3. Browse to, Local Computer Policy\Computer Configuration\Windows Settings\Security Settings\Advanced Audit Policy Configuration\System Audit Policies - Local Group Policy Object\Object Access
4. Open Audit Removable Storage
5. Enable Configuration the following audit events, and success (success or failure both cause the issue)
6. Open SuperCollider and it will get stuck at Sending request.....
7. Disable Audit Removable Storage and it will work fine.

Expected Behavior

SuperCollider will open, stick at Sending request for a second or two, then help will open.

Current Behavior

SuperCollider gets stuck at Sending request.....

Background

I work in an IT department at a University and SuperCollider was requested by one of our departments so they could use it for teaching. During testing the lecturer reported the issue. It took quite a while to track the policy down that was causing the problems, and I never expected it to be an audit policy that would cause it, weirdest bug I've seen so far. The policy is enabled from the Microsoft Security Baseline here (https://blogs.technet.microsoft.com/secguide/2018/04/30/security-baseline-for-windows-10-april-2018-update-v1803-final/), so this might cause problems for other organisations as well. At the moment we've disabled Audit Removable Storage, but if there's as better fix or workaround it would be good to get it out. Hopefully I've provided enough detail to recreate the issue.

[sternsc]

Thanks for this, but - my IT support staff say that in our organization, that policy for Audit Removable Storage is not enabled – and I still have the “stuck at Sending Request…” problem. So the fact that disabling it worked for you is probably just circumstantial. This holds for SC 3.8.x--SC 3.10.0 on a 64-bit Win10 Education edition in a centrally managed university environment. If you can think of other Windows policies that might be relevant, that would be great. Brian thinks this problem may be due to a start-up race condition between the SCIDE and SCLANG processes. /Sten

[sternsc]

Update on this issue: the IT-support staff at my university have now seen that the policy "Audit Removable Storage" is in fact active on our systems. Just as @rjsuni noted above, when this policy is deactivated, the problem of Stuck at "Sending request..." goes away. So now I am back in business, finally! But it would be good to understand how to avoid the problem in the first place.
Best /Sten Ternström

[nhthn]

@sternsc thanks for the info. that will definitely help with debugging this.

@jrsurge i think you have some info on this that might be worth leaving in these issue comments?

[nhthn]

i don't have an answer for this, but i did a dive into how the IDE and sclang talk to each other. here it is:

• Main::scProcess is an instance of ScProcess which inherits from QProcess. it sends the signal started() after the process has started running.
• HelpBrowser listens for the started signal in the slot onInterpreterStart
• which calls HelpBrowser::goHome
• which calls sendRequest("HelpBrowser.goHome")
• which displays the "Sending Request..." message and calls Main::scProcess()->evaluateCode("HelpBrowser.goHome")
• which writes that code to the sclang process's standard input.

the key here is that started() is emitted immediately after sclang has started -- the help browser doesn't wait to check if sclang is finished booting, it just immediately pipes that into stdin and displays the "Sending request..." message.

so if sclang is unresponsive, or for some reason doesn't register that standard input, the help browser won't load. note that "Sending Request..." is programmed to go away if the sclang process finishes, but not if it hangs. it's a bit of bad UX, since it really means "the IDE already sent a request, and either sclang is currently working on it or sclang didn't hear it at all."

A: if you can't run code in the IDE, and the help browser is stuck at "Sending Request", then the problem is not the help browser but some more general breakdown of sclang/scide IPC.

B: if you can run code in IDE but the help browser is stuck at "Sending Request", then sclang somehow missed the initial call to HelpBrowser.goHome, which seems like a race condition. i guess started() is running too soon -- the correct solution would be to have sclang communicate to scide when it's successfully booted, and only then call HelpBrowser.goHome.

edit: reading more carefully the above comments in this thread, it's not clear to me whether the people who have reported this issue are experiencing A or B. most of the Windows reports we've gotten on this (#4569, #4564, #4407, #3302) are A -- total IPC breakdown. i don't know if we've seen a confirmed B since 3.10, but i vaguely recall such issues in the 3.9 era.

[sternsc]

Thanks Nathan for looking into this. I can confirm that the problem is that we can NOT run code in the IDE once the Stuck condition is established. The code in the startup file, however, runs ok. This problem goes away if the Windows policy Audit Removable Storage is deactivated. (!!??) Best of luck with that, Sten <Skickat från min mobil>

…

---- Nathan Ho skrev ----

i don't have an answer for this, but i did a dive into how the IDE and sclang talk to each other. here it is: - `Main::scProcess` is an instance of `ScProcess` which inherits from `QProcess`. it sends the signal `started()` after the process has started running. - `HelpBrowser` listens for the `started` signal in the slot `onInterpreterStart` - which calls `HelpBrowser::goHome` - which calls `sendRequest("HelpBrowser.goHome")` - which displays the "Sending Request..." message and calls `Main::scProcess()->evaluateCode("HelpBrowser.goHome")` - which writes that code to the sclang process's standard input. the key here is that `started()` is emitted immediately after `sclang` has started -- the help browser doesn't wait to check if `sclang` is finished booting, it just immediately pipes that into stdin and displays the "Sending request..." message. so if sclang is unresponsive, or for some reason doesn't register that standard input, the help browser won't load. note that "Sending Request..." is programmed to go away if the sclang process finishes, but not if it hangs. it's a bit of bad UX, since it really means "the IDE already sent a request, and either sclang is currently working on it or sclang didn't hear it at all." if you can't run code in the IDE, and the help browser is stuck at "Sending Request", then the problem is not the help browser but some more general breakdown of sclang/scide IPC. if you *can* run code in IDE but the help browser is stuck at "Sending Request", then sclang somehow missed the initial call to `HelpBrowser.goHome`, which seems like a race condition. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #4214 (comment)

[jrsurge]

I've done some testing on my machine.
I'm on Windows 10 Pro, so I have access to group policies.

Setting this policy does recreate this issue, but it's more strange than that, see below.
This policy does seem to be doing something, but why?

This policy setting allows you to audit user attempts to access file system objects on a removable storage device. A security audit event is generated only for all objects for all types of access requested.

If you configure this policy setting, an audit event is generated each time an account accesses a file system object on a removable storage. Success audits record successful attempts and Failure audits record unsuccessful attempts.

SC doesn't access removable storage on this machine to do anything on startup - everything's local. This is confirmed by the below observations:

1. Enabling the Audit Removable Storage policy replicates this issue - although it may not be the cause, see 2
2. The Audit itself is not actually triggered by starting IDE, sclang or scsynth - nothing shows up in the Windows event log, so I'm not sure how the policy fits in.
3. Compiling with SC_USE_QTWEBENGINE=OFF does not remove the issue

So, it's not the help browser. It's something between the IDE and the language.
Why the policy has an impact, I have no idea, but I'm fairly certain it's not really the cause.

It would be really helpful to know which editions of Windows are experiencing this issue.

Is anyone experiencing this on Windows 10 Home?
(Windows 10 Home doesn't have group policies)

It's 100% repeatable on this machine - I'd expect a different success/failure rate if it was a race condition, but it does look like a deadlock between IDE and sclang?

Some lock held by both for a shared resource that sclang is waiting on at the very end of its boot? The main GUI thread isn't blocking, the IDE still responds, but the language is waiting on something.

EDIT: This post was written last night before Nathan's, but I only just got around to posting it this morning. I'm now more certain that, as Nathan suggests, the IPC isn't working, which only looks like a deadlock.

I'll keep digging..

[johnlinnane]

I'm using a university PC (Windows 10 Enterprise), but Audit Removable Storage is not enabled. The checkboxes for success and failure are not ticked. Still I can't run code in the IDE, and the help browser is stuck at "Sending Request".

[nhthn]

@jrsurge great thanks for the info. i'm trying to envision the next steps for debugging, what are your ideas?

@panisterfatheroy interesting -- given the distant relationship between scide/sclang and the Audit Removable Storage option, it seems possible to me that there are other security settings on IT-restricted machines that break our IPC.

for the sake of getting SC running, i believe this problem is specific to the IDE, and alternate editors like scvim and scel will work. you can also run sclang scripts from the command line.

[johnlinnane]

I installed the SuperCollider package in Atom editor. It was looking for sclang.exe here:

C:\Program Files (x86)\SuperCollider\sclang.exe

In the package settings I changed it to here (where it was installed) and it worked in Atom.

C:\Program Files\SuperCollider-3.10.3\sclang.exe

I then tried in SuperCollider IDE and it worked there also. But unfortunately I didn't try SC before I changed atom so I don't know if my changes fixed it or not.

[titogarcia]

I am having the same issue on one Windows 10 laptop. I have checked the Group Policy setting described above and it is disabled. At first I thought that maybe my firewall was blocking connections, but it doesn't work with firewall disabled.

On another Windows 10 Surface tablet it is working.

Interestingly enough, in none of both computers I get sclang.exe working from the command line. It doesn't respond to any statements.

Any suggestions on how to troubleshoot? I have done a procmon but I haven't found anything significant.

[jrsurge]

Hi @titogarcia, thanks for the report. It may be that there are other settings that have the same result, but if we set the Audit Removable Storage policy, it is a guaranteed way for us to reproduce the issue - so it gives us a way to test.
In terms of the command line on Windows, that's a separate issue - currently the way sclang grabs command line input on Windows doesn't work (see README_WINDOWS.md, Known Issues).

I'll hopefully have some time this week to look into the IPC breakdown.

[claremacrae]

Hi, James, these might help to auto-attach to a process of a given exe:

• https://stackoverflow.com/questions/1951124/visual-studio-auto-attach-to-a-process-when-the-process-is-spawned
• https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/a329t4ed(v=vs.100)?redirectedfrom=MSDN
• https://marketplace.visualstudio.com/items?itemName=EntrianSolutions-RichieHindle.EntrianAttachAuto-Attachthedebuggeratprocessstart

[jrsurge]

Hi, James, these might help to auto-attach to a process of a given exe:

• https://stackoverflow.com/questions/1951124/visual-studio-auto-attach-to-a-process-when-the-process-is-spawned
• https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/a329t4ed(v=vs.100)?redirectedfrom=MSDN
• https://marketplace.visualstudio.com/items?itemName=EntrianSolutions-RichieHindle.EntrianAttachAuto-Attachthedebuggeratprocessstart

Thanks Clare!

---

EDIT: Files to have open

• editors/sc-ide/primitives/sc_ipc_client.cpp
• editors/sc-ide/core/sc_process.cpp
• editors/sc-ide/core/main.cpp

Investigation still underway, but as an update, the steps involved in the IPC are:

• scide launces sclang (ScProcess)

• ScProcess is a QProcess - owning the IPC Server

• ScProcess has pointer to IPC client (ScIpcClient) on sclang (init nullptr)

• ScProcess::onStart calls ScIDE_Connect primitive

• ScIDE_Connect initialises a socket: gIpcClient:
  

  supercollider/editors/sc-ide/primitives/sc_ipc_client.cpp

  Lines 183 to 202 in cee86b5

  	static SCIpcClient* gIpcClient = NULL;
  	int ScIDE_Connect(struct VMGlobals* g, int numArgsPushed) {
  	if (gIpcClient) {
  	error("ScIDE already connected\n");
  	return errFailed;
  	}
  	PyrSlot* ideNameSlot = g->sp;
  	char ideName[1024];
  	int status = slotStrVal(ideNameSlot, ideName, 1024);
  	if (status != errNone)
  	return errWrongType;
  	gIpcClient = new SCIpcClient(ideName);
  	return errNone;
  	}

• When gIpcClient is created, it connects to the ScProcess::mIpcServer using the name.

• On receiving the connection request, ScProcess::mIpcServer fires the QLocalServer::newConnection signal, triggering the ScProcess::onNewIpcConnection slot.

• ScProcess::onNewIpcConnection sets the ScProcess::mIpcSocket to whatever the incoming request is (i.e. the socket in sclang).

• On disconnect, ScProcess registers the IPC client pointer as deleteLater, which should release it once the ScProcess finishes.

In cases where the breakdown occurs, the ScProcess::onNewConnection never runs - so the ScProcess::mIpcSocket is always nullptr.

Ongoing questions:
Check if..

• Make sure QLocalServer exists (ScProcess::mIpcServer)
• ScIDE_Connect isn't called
• ScIDE_Connect doesn't run
• gIpcClient is never created
• gIpcClient fails to connect to the ScProcess::mIpcServer.
• gIpcClient connects but the signal/slot between QLocalServer::newConnection and ScProcess::onNewIpcConnection isn't working

The ScProcess::onNewIpcConnection isn't being triggered, so it's something before then.

[nhthn]

miscellaneous thoughts from today's dev meeting:

• as seen in above comment, issue is in IDE and not sclang
• https://doc.qt.io/qt-5/qprocess.html#notes-for-windows-users
• attempt to rewrite QProcess with a lower-level subprocess API like popen, since this may be a bug in QProcess
• maybe there is some way to trace system calls in Windows?

[sternsc]

OK, thanks for your efforts, and for letting me know. I’m afraid I don’t have the know-how (nor time) to pursue this myself. For the time being, the workaround of disabling the Windows system policy “Audit Removable Storage” seems to work for us . Sten Ternström KTH Royal Institute of Technology, Stockholm From: Nathan Ho <notifications@github.com> Sent: Sunday, 17 November 2019 19:18 To: supercollider/supercollider <supercollider@noreply.github.com> Cc: sternsc <sten.ternstrom@tele2.se>; Mention <mention@noreply.github.com> Subject: Re: [supercollider/supercollider] IDE can't run code on Windows, stuck at "Sending request...." (#4214) miscellaneous thoughts from today's dev meeting: * as seen in above comment, issue is in IDE and not sclang * https://doc.qt.io/qt-5/qprocess.html#notes-for-windows-users * attempt to rewrite QProcess with a lower-level subprocess API like popen, since this may be a bug in QProcess * maybe there is some way to trace system calls in Windows? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4214?email_source=notifications&email_token=AAXQZNBNNKYUSOUZBOHMYDTQUGDD7A5CNFSM4GLCSZY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEISKJI#issuecomment-554771749> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXQZNAHTCCLW5J7JC2TA63QUGDD7ANCNFSM4GLCSZYQ> .

[nhthn]

@sternsc we're using this github issue as a thread to discuss continued investigation of this bug -- if you don't want to receive notifications in your inbox, github allows you to unsubscribe.

[jrsurge]

After a lot of staring, I think I found it.

QProcess is all fine - it's writing to stdin on sclang ok.
Confirmed this by writing a logging thread in sclang that continuously dumped stdin.

sclang wasn't reading from stdin properly, so it missed the initial commands from scide to setup the IPC, and it just goes downhill from there.

The fix I have isn't elegant, but appears to work on my machines at least - it'll need testing on other machines.

[jamshark70]

Add another user who is dead in the water: https://scsynth.org/t/complete-n00b-question/1547/4 -- Audit Removable Storage didn't help.

What do we need to do to make @jrsurge 's fix testable on other machines?

[mossheim]

huge thanks to @jrsurge for tracking this one down!