scide: Document.current nulled on Class Library Recompile

[jrsurge]

Environment

• SuperCollider version: 3.10.2
• Operating system: Windows 10, 1809, 17763.503
• Other details (Qt version, audio driver, etc.): Qt 5.11.2

Steps to reproduce

1. Launch scide.
2. Wait for startup to complete.
3. Ctrl/Cmd + Shift + L (or Language > Recompile Class Library)
4. Close the default initial document

Expected vs. actual behavior

Expected: Document to close.

Actual: Document closes and error thrown.

Document.current not successfully reinitialised, so on closing the document, or anything else that relies on it, errors are triggered:

ERROR: Message 'closed' not understood.
RECEIVER:
   nil
ARGS:
CALL STACK:
	DoesNotUnderstandError:reportError
		arg this = <instance of DoesNotUnderstandError>
	Nil:handleError
		arg this = nil
		arg error = <instance of DoesNotUnderstandError>
	Thread:handleError
		arg this = <instance of Thread>
		arg error = <instance of DoesNotUnderstandError>
	Object:throw
		arg this = <instance of DoesNotUnderstandError>
	Object:doesNotUnderstand
		arg this = nil
		arg selector = 'closed'
		arg args = [*0]
	Process:interpretCmdLine
		arg this = <instance of Main>
^^ The preceding error dump is for ERROR: Message 'closed' not understood.
RECEIVER: nil

A vast array of errors get thrown over the course of using SC, some of which are huge. The only way to stop them is to reboot the interpreter.

As far as I can tell, it doesn't render the IDE unusable, but it's not ideal - and particularly troublesome when developing classes and having to recompile the class library often.

A very quick look at the code shows a signal being emitted (line 379):

supercollider/editors/sc-ide/core/sc_process.cpp

Lines 353 to 384 in f65a1ef

	void ScProcess::onResponse( const QString & selector, const QString & data )
	{
	if (selector == QStringLiteral("introspection")) {
	using ScLanguage::Introspection;
	auto watcher = new QFutureWatcher<Introspection>(this);
	connect( watcher, &QFutureWatcher<Introspection>::finished, [=]{
	try {
	Introspection newIntrospection = watcher->result();
	mIntrospection = std::move(newIntrospection);
	emit introspectionChanged();
	} catch (std::exception & e) {
	MainWindow::instance()->showStatusMessage(e.what());
	}
	watcher->deleteLater();
	});
	// Start the computation.
	QFuture<Introspection> future = QtConcurrent::run( [] (QString data) {
	return ScLanguage::Introspection(data);
	}, data );
	watcher->setFuture(future);
	}
	else if (selector == QStringLiteral("classLibraryRecompiled")){
	mCompiled = true;
	emit classLibraryRecompiled();
	}
	else if (selector == QStringLiteral("requestCurrentPath"))
	Main::documentManager()->sendActiveDocument();
	}

But I can't find it being listened to anywhere, apart from the HelpBrowser.
If someone can point me in the right direction, I'm happy to look into making a PR.

[jamshark70]

I've also seen errors with didResignKey and didBecomeKey failing when the language-side idea of the current document becomes nil. I don't remember if the library was always recompiled before that.

There are several bugs with document-status sync between the editor and the language. (The document text mirror gets broken, and I know for sure that recompile isn't involved in that bug.) Somehow the communication gets dropped in Windows, but not in other platforms.

[jrsurge]

Thanks @jamshark70

There are several bugs with document-status sync between the editor and the language. (The document text mirror gets broken, and I know for sure that recompile isn't involved in that bug.) Somehow the communication gets dropped in Windows, but not in other platforms.

I've now checked this on my Linux machine (Ubuntu 18.04, SC 3.10.2, source build), and can confirm it doesn't reproduce. Seems like a Windows-specific issue.

I wasn't able to reproduce on my Windows laptop, until I updated the OS to 1809 (from 1806), now it's 100% reproducible, so it looks like something in a recent update is related?

For now, I've discovered that running:

ScIDE.handshake;

Manages to resync everything and temporarily stop the errors.
I'm assuming this isn't something to advertise, however, just in case it gives a better idea of where it's going wrong?

On further tests, this is the specific line from the handshake method:

supercollider/SCClassLibrary/scide_scqt/ScIDE.sc

Line 28 in 63783ae

this.send(\requestDocumentList);

[jrsurge]

As an update:

On more experiments, this appears to be related to the Server shutdown during the recompile.

• Without booting the server, recompile == no errors
• Booting the server, recompile == errors
• Booting server, killing server, waiting for server to stop, recompile == no errors

A lot changes when the Server gets involved..trying to narrow it down

[geoffroymontel]

Can reproduce it too on my Windows 10 machine with SC 3.11-beta

[eleses]

Happens to me too with 3.10.4. Yes it does seem happen only after recompling the class library, it seems.

It's especially annoying when using the test suite with scripts as in

(
UnitTest.reset;

u = UnitTestScript("MyTest", Document.current.dir ++ "/myUnitTest_unittest3.scd");
u.runScript;
)

since those I run after recompile... getting hit with

ERROR: Message 'dir' not understood.
RECEIVER: nil

is a bit of a "huh, what?" every time it happens. Sure enough every time

Document.current // -> nil

even though it's from a saved document.

Also, after it happens, for me the bug is permanent-until-SCide-restart (similar to the "DDE" #4845 bug in this respect), meaning that closing and reopening that unittest3.scd file (from within SC, as to avoid any DDE failures) still results in

Document.current // -> nil

right after re-opening the file.

Furthermore, after the bug hits, switching to another buffer/file also results in that Document.current // -> nil! So it seems not to matter which document precisely is being accessed. Even modifying then saving a document still keeps that Document.current == nil.

Basically, I need to restart SCide to make that nuisance go away. Simply recompiling the class library again doesn't seem to help with getting out of the Document.current == nil situation.

Also this bug seems unrelated to the DDE bug (beside/despite the fact that they both need SCide restart to clear.) I could get SCide in a state where Document.current == nil, but opening files via DDE (from Windows Explorer) still works. And after an intentional hang of the DDE (by batch-opening files), Document.current is not nil (in those that did open). So these are separate bugs.

[mossheim]

From @totalgee (#4812):

---

I also get this all the time (on Windows 10), especially now with 3.11.0 it seems even more frequent, but I also had it often with 3.10.3 and 3.10.4 for sure.

It happens, e.g. if I have a class file (.sc) open, save it, then press Ctrl-Shift-L to recompile the class library, and then make an edit of the open (saved) file (e.g. press 'space' or hit enter) somewhere in the document. It is really annoying (seems more likely to happen than before, I think).

I also get this other message sometimes, which I think is related to the above problem...(?)

ERROR: Message 'didBecomeKey' not understood.
RECEIVER:
   nil
ARGS:
CALL STACK:
	DoesNotUnderstandError:reportError
		arg this = <instance of DoesNotUnderstandError>
	Nil:handleError
		arg this = nil
		arg error = <instance of DoesNotUnderstandError>
	Thread:handleError
		arg this = <instance of Thread>
		arg error = <instance of DoesNotUnderstandError>
	Object:throw
		arg this = <instance of DoesNotUnderstandError>
	Object:doesNotUnderstand
		arg this = nil
		arg selector = 'didBecomeKey'
		arg args = [*0]
	Meta_Document:prCurrent_
		arg this = <instance of Meta_Document>
		arg newCurrent = nil
	Meta_Document:setActiveDocByQUuid
		arg this = <instance of Meta_Document>
		arg quuid = '{ce76b6c0-4c11-4791-b459-4b92a99c8045}'
		var newCurrent = nil
	< closed FunctionDef >  (no arguments or variables)
	Process:interpretCmdLine
		arg this = <instance of Main>
^^ The preceding error dump is for ERROR: Message 'didBecomeKey' not understood.
RECEIVER: nil

[Spacechild1]

I get those errors all the time on Windows (SC 3.11.1)

[jamshark70]

FWIW: Still an issue in 3.11.2. Here is a screenshot showing three open document panels, but Document.allDocuments contains only two objects 😕 -- this demonstrates for sure that Document references are getting lost.

I think I had done: 1/ open a document; 2/ recompile the class library; 3/ open the other doc from disk and create a new (untitled) doc. The doc from step 1 is the missing reference.

[avdrd]

It think it might be related to some quarks or extensions. Doesn't happed to me at all if I disable all extensions that seem to run extensive code in their initClass, particularly code that compiles SynthDefs. But it does happen occasionally otherwise. Basically with core SC and just JITLibExtensions, it never happens, on any number of classlib recompiles. But throw in some crucial, ddw-stuff, and wslib and then it does happen pretty quickly. I should probably check it some more just with AudacityLabels installed, because that one turned out to be a canary for some other classInit issues. I tested with 3.12.1 on Win 10, latest. Edit: confirmed: just with AudacityLabels activated, it also happens! So initClass related 99% in my opinion!

I had to recompile like 3 times though to make it happen just with AudacityLabels activated, so there is probably some race condition with classlib compilation. It seems the bigger quarks are more likely to make it happen. But once it happened, meaning I got a 'prSetEdited' not understood after trying to change tab right after recompile then:

Document.allDocuments  // -> nil

So in my case it was even worse than what @jamshark70 reports above. After trying to switch tabs again ERROR: Message 'didBecomeKey' not understood. So yeah, all these are related somehow. But what seems to happen after that is that the Document.allDocuments gets gradually repopulated. After touching one tab, although I have like 20 opened, now:

Document.allDocuments //-> [ a Document ]

If I then force:

ScIDE.handshake;

it fixes itself:

Document.allDocuments //-> -> [ a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document ]

But here is the strange part: then I disable the only quark I had (AudacityLabels), recompile class lib and

Document.allDocuments // -> nil

But, I don't get any errors thrown anymore in the post window! And the more interesting part is that this time, while showing no obvious error, tab switching no longer causes Document.allDocuments to be populated on tab switching, meaning after switching between 10 tabs or so,

Document.allDocuments // still nil

All this is pretty confusing since I don't know what the intended behavior is supposed to be. Probably it should be re-populated because if I do another classlib recompile, washout changing quarks at all this time, now right after it's compiled

Document.allDocuments -> [ a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document, a Document ]

So it seems that the bug that causes Document.allDocuments to become nil probably happens from time to time regardless of quarks installed, but oddly enough only the presence of some quarks makes those didBecomeKey and prSetEdited errors to pop up in the aftermath.

@Spacechild1 What quarks do you have installed and activated?

[totalgee]

@avdrd I think you're on the right track... I have lots of quarks in use (similar ones to you, though not AudacityLabels). Several use classInit stuff and I have sometimes seen ordering issues between them. Also, I normally have many document tabs open at once. (And am using a workspace to remember my layout on restart.)

I did notice that on another machine I didn't seem to get this error on recompilation (very often, or at all).

[avdrd]

I tried to fix it like this, but it didn't' really work because the update happens on a deferred thread. I'll have to add a polling loop...

DocuNotFixed {

	*initClass {

		Class.initClassTree(StartUp);
		StartUp.add {
			//Document.allDocuments.postln;

			if(Document.allDocuments.isNil) {
				warn("Document.allDocuments is nil. Redoing handshake.");
				ScIDE.handshake;
			};

			if(Document.allDocuments.isNil) {
				warn("That still didn't change Document.allDocuments.");
			} {
				inform("Document.allDocuments refreshed.")
			}

		}
	}
}

Doesn't fix it::

Class tree inited in 0.02 seconds
WARNING: Document.allDocuments is nil. Redoing handshake.
WARNING: That still didn't change Document.allDocuments.

The interesting bit is that ScIDE itself does this in its initClass:

		StartUp.add {
			if (this.connected) {
				this.handshake
			}
		}

I do wonder if that fails at the this.connected test condition or later.

[avdrd]

So it seems that it doesn't fail at connected, beause the following still doesn't fix it, i.e. connected is true, but the allDoc doesn't get updated immediately after handshake is called. So, I'll have to wait for that instead

DocStillNotFix {

	classvar <>pollTime = 0.2, <>timeout = 2;

	*initClass {

		Class.initClassTree(StartUp);
		StartUp.add {

			//Document.allDocuments.postln;

			var r = Routine {

				var t = 0;

				inform("ScIDE.connected: " + ScIDE.connected);

				// Even with timeout, probably not OK if using non-ScIDE REPL,
				// Unless you somehow manage to set the timeout to 0 for those.
				while { ScIDE.connected.not and: {t < timeout} } {
					t = t + pollTime;
					inform("Waiting for ScIDE connection...");
					pollTime.wait;

				};
				if(ScIDE.connected.not) {

					warn("ScIDE connection timed out after" + timeout + "seconds!");
					^this;

				};
				if(Document.allDocuments.isNil) {

					warn("Document.allDocuments is nil. Redoing handshake.");
					ScIDE.handshake;
				};

				if(Document.allDocuments.isNil) {
					warn("That still didn't change Document.allDocuments.");
				} {
					inform("Document.allDocuments refreshed.")
				};
			};
			r.run;
		}
	}
}

Posts:

Class tree inited in 0.02 seconds
ScIDE.connected:  true
WARNING: Document.allDocuments is nil. Redoing handshake.
WARNING: That still didn't change Document.allDocuments.

[avdrd]

This finally fixes it for me.

DocFix {
	classvar <>pollTime = 0.5, <>timeout = 3;
	*initClass {
		Class.initClassTree(StartUp);
		StartUp.add {
			//Document.allDocuments.postln;
			var r = Routine {
				var t = 0;
				inform("ScIDE.connected:" + ScIDE.connected);
				// Even with timeout, probably not OK if using non-ScIDE REPL,
				// Unless you somehow manage to set the timeout to 0 for those.
				while { ScIDE.connected.not and: {t < timeout} } {
					t = t + pollTime;
					inform("Waiting for ScIDE connection...");
					pollTime.wait;
				};
				if(ScIDE.connected.not) {
					warn("ScIDE connection timed out after" + timeout + "seconds!");
					^this;
				};
				while { Document.allDocuments.isNil and: {t < timeout} } {
					t = t + pollTime;
					inform("Waiting for Document.allDocuments.isNil to change on its own");
					pollTime.wait;
				};
				if(Document.allDocuments.isNil) {
					warn("Document.allDocuments is nil after"+ timeout +"seconds. Redoing handshake.");
					ScIDE.handshake;
				};
				t = 0;
				while { Document.allDocuments.isNil and: {t < timeout} } {
					t = t + pollTime;
					inform("Waiting for Document.allDocuments.isNil to change after forced handshake.");
					pollTime.wait;
				};
				if(Document.allDocuments.isNil) {
					warn("That still didn't change Document.allDocuments.");
				} {
					inform("Document.allDocuments refreshed after" + t + "seconds.");
				};
			};
			r.play(AppClock);
		}
	}
}

Posts

*** Welcome to SuperCollider 3.12.1. *** For help press Ctrl-D.
ScIDE.connected: true
Waiting for Document.allDocuments.isNil to change on its own
Waiting for Document.allDocuments.isNil to change on its own
Waiting for Document.allDocuments.isNil to change on its own
Waiting for Document.allDocuments.isNil to change on its own
Waiting for Document.allDocuments.isNil to change on its own
Waiting for Document.allDocuments.isNil to change on its own
WARNING: Document.allDocuments is nil after 3 seconds. Redoing handshake.
Waiting for Document.allDocuments.isNil to change after forced handshake.
Document.allDocuments refreshed after 0.5 seconds.

In hindsight, waiting for connected is probably not necessary. Only waiting to see some effects after the forced handshake is. Perhaps that could be done in a loop too, but seems a bit risky.

Those timeouts can be pretty short. Well, if you comment out some of the postlns first so you don't get post-msg spammed, it works fine for me just with

classvar <>pollTime = 0.1, <>timeout = 0.2;

[jamshark70]

Should we perhaps have these various stages emit signals, rather than loop-polling? We've got CondVar to help with thread blocking/unblocking; seems to me this is a case where we might use it.

[avdrd]

Alas it's not a 100% fix. I went out for a few hours, during which my computer was asleep, with SC running as it was, but when I came back and resumed it, no class recompiles or anything...

ERROR: Message 'didBecomeKey' not understood.
RECEIVER:
   nil

And sure enough

Document.allDocuments
-> [ a Document, a Document, a Document, a Document ]

instead of the 20 or so that I actually have opened.

So the problem seems more complicated that somehow ScIDE periodically seems to push bad allDocuments values to sclang. No idea how that happens.

And I actually had to ScIDE.handshake manually twice this time. After first one, it still only had 4 documents! So it seems that some calls to ScIDE.handshake can return bad, truncated data in allDocuments!

---

However, after a few more days and perhaps 100 classlib recompiles, I've had zero failures right after recompiling with

classvar <>pollTime = 0.1, <>timeout = 0.4;

So while it's not a 100% solution, it is like 99% for me.

[jamshark70]

Deleted my previous comment because some of it wasn't correct.

@avdrd 's DocFix workaround does get around the didResignKey / didBecomeKey errors in my case... cheers for that, thanks!

The (probably related) other thing is that my modular style coding interface based on ProxySpace tries to push a patch's ProxySpace as the currentEnvironment when switching to the document tab. In Linux, .front is sufficient. In Windows, .front works but then the key seems to be immediately resigned (weird)... if I add a proxyspace.push into my patch loader, it complains that the environment is already current (so it got pushed somewhere) but then a moment later, currentEnvironment has reverted to topEnvironment (while the correct document remains active -- the key should not have been resigned but it was). Only in Windows. -- edit: opened a separate bug for this: #6082.

[JordanHendersonMusic]

I've closed a couple of duplicate issues, often this issue arises in conjunction with #4812