before booting, find zombie servers, stop and advise if needed. #2524

adcxyz · 2016-12-02T20:19:56Z

s.boot can end in a complicated hang when there is a zombie server.
This happens e.g. after sclang crashed, or after hard interpreter reboot:
s.boot fails with "Exception in World_OpenUDP: bind: Address already in use ",
and s. will not reboot because it thinks it is already booting.

This PR checks whether there is already a server responding at the desired address and port,
and if so, informs about it. Reproducer:

// test finding zombies first
s.findZombies(0.1, { "found.".postln }, { "none".postln });

// make sure there are no scsynth processed around
Server.killAll;
s.findZombies(0.1, { "found.".postln }, { "none".postln });

// Start a zombie server - reports that server is booted
x = (Server.program + s.options.asOptionsString(s.addr.port)).unixCmd;
s.findZombies(0.1, { "found.".postln }, { "none".postln });

// try booting s - fails ...
s.boot; // 
// --- ... and informs with:
WARNING: Cannot boot server because of a zombie server at NetAddr.new("127.0.0.1", 57110).
// Kill zombies first, then try booting again:
Server.killAll;
s.boot;
// ---

So one knows what to do.

…vise.

adcxyz · 2016-12-02T20:22:21Z

@rukano
glad you like it :-)

… to be alive.

telephon · 2016-12-03T10:43:38Z

does this overlap with the other improvements?

adcxyz · 2016-12-03T11:08:02Z

It sort of does - prPingApp and findZombies essentially do the same thing,
so only one of them is needed. I think checking earlier is better,
bumping non-response timeout of 0.5 is enough (findZombies has 0.1, prPingApp 3).
I'll update the PR.

adcxyz · 2016-12-03T11:10:05Z

in prPingApp, why do the sync command by hand?

telephon · 2016-12-03T11:21:48Z

I think better replace prPingApp by pidRunning. The only thing is that this won't work over the network. Not sure if that matters at all.

adcxyz · 2016-12-03T11:45:17Z

ok:
on local servers, one can preferentially use pidRunning for status watching,
and use status messages to assess server responsiveness by network.
on remote servers, it has to be status messages is any case.

More general rethinking:
A pinged server may either be a zombie (assuming that There Can Be Only One),
or a legitimate other scsynth process started by someone else that uses the same default port.
In case 2, quit and reboot (as boot now does) shoot another app in the foot.
How about having a flag in the Server class:

classvar <>allowAlienServers = true;

if allowAlienServers is true,

server politely finds the next free port number and uses that,
(much like sclang already does when booting), and
else assume they are zombies, and do quit and boot as is.

adcxyz · 2016-12-03T12:12:47Z

// sketch for findFreePort with multiple servers:

(
// make sure there are no scsynth processed around
Server.killAll;
// Start three servers, as if by other programs, or zombies
x = (Server.program + s.options.asOptionsString(57110)).unixCmd;
x = (Server.program + s.options.asOptionsString(57109)).unixCmd;
x = (Server.program + s.options.asOptionsString(57108)).unixCmd;
)

// assume they are intended, and find free port, then boot
~findFreePort = { 
	"trying port % ...\n".postf(s.addr.port );
	s.findZombies({ 
		s.addr.port = s.addr.port - 1; 
		~findFreePort.value;
	}, { 
		s.boot;		
	});
};

~findFreePort.value;

jamshark70 · 2016-12-03T14:23:24Z

It's starting to sound like improvising workarounds for a fairly large set of cases, instead of analyzing the possible cases and deliberately designing the cleanest solution. My gut feeling is, it's time to back off from code and work up a proper spec. Then the code will be easier to write.

crucialfelix · 2016-12-03T14:25:19Z

It is often the case for me that the other server was started by a different process. I am planning to add an option to supercollider.js at some point that would offer different strategies when you encounter a conflict. fail kill steal Um.. maybe with nicer names. The other option is to just find a free port and use that. You can query the port to see if it's in use.

…

On Sat, Dec 3, 2016 at 11:45 AM Alberto de Campo ***@***.***> wrote: ok: on local servers, one can preferentially use pidRunning for status watching, and use status messages to assess server responsiveness by network. on remote servers, it has to be status messages is any case. More general rethinking: A pinged server may either be a zombie (assuming that There Can Be Only One), or a legitimate other scsynth process started by someone else that uses the same default port. In case 2, quit and reboot (as boot now does) shoot another app in the foot. How about having a flag in the Server class: classvar <>allowAlienServers = true; if allowAlienServers is true, - server politely finds the next free port number and uses that, (much like sclang already does when booting), and - else assume they are zombies, and do quit and boot as is. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2524 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AANWckYM0u3PtAQ-3bKj-eXkEyCuTJ5Zks5rEVZOgaJpZM4LC82K> .

telephon · 2016-12-03T17:13:21Z

A question: when I start a server with a unixCmd (as it is done now), the first argument is the action that is called when the server quits. Now I'm not sure – is this action called in your above scenario?

Because now with the new refactored server, we already get a cleaner quit when the server is killed externally.

adcxyz · 2016-12-03T21:16:47Z

@jamshark70

yes, we have gone back from pull request to discussion. so the PR was at least useful for that. I will work on a big picture of scenarios to be handled, and bring it up here.

@crucialfelix

I had 'hijack' for your 'steal' name in my strategies list ;-)
what is the easiest cross-platform way to check within SC whether a UDP port is open locally?

@telephon

yes, the unixCmd doneAction should be active in the scenario, and is a very good fast way to inform sclang about scsynth having quit for whatever reasons.

more later, adc

adcxyz · 2016-12-03T23:03:17Z

// possible server boot scenarios to support

remote servers

server runs on remote machine, with maxLogins > 1
clients cannot boot or quit server
clients assume server is running (and ping to find out)
clients must know exact server options

maxLogins, ranges of nodeIDs, buses, bufnums etc

clients login with a unique clientID which create private ranges
clients access only their private range of nodeIDs, buffer numbers, buses, etc
clients can only free their own nodes (except emergency r.hardFreeAll);

local servers

server is / servers are booted from and used by various apps,
such SuperCollider.app, supercollider.js, SC-based standalones, others

a. Default - single app, single scsynth
A single app (e.g. SC3.8) uses scsynth, and only a single scsynth instance.

will work fine for most non-expert users, including newbies
should be very simple, stable, and 'just work'.

b. Multiple apps use scsynth/supernova processes (SuperCollider.app, supercollider.js, etc)

apps should keep server ports out of each other's way
when in doubt, leave any servers untouched

c. single app uses multiple parallel scsynth/supernova processes

typically advanced users
typically using fixed list of ports
typically handled in existing quarks?

Network Music setups - multiple networked clients / servers

e.g. Republic, Utopia, others
clients manage (boot etc) local server(s)
clients publish their local servers to peers as remotes.

Logic of Server.boot:

if not local, dont boot and inform user
if server booted or booting (pidRunning), dont boot and inform user
-- no running server found
check whether addr.port is free
if true, boot server with given address
if false, server found at intended port, what to do?

in single app/scsynth scenario,
-> assume that server is a zombie (e.g. after an sclang crash),
and quit and boot, or (faster but riskier) hijack, optionally with .freeAll?

in multiple apps case
-> assume it belongs to another app,
find a free port, and boot with that port

in parallel scsynths case
-> assume we know the exact address:port, quit and boot or hijack.

in Network Music setups
-> if single local server, reboot or hijack; else pick or add an appropriate strategy.

// here is a quick sketch of it in Server
// could also be in a separate ServerRecover class?
Server { 
	classvar <>defaultBootRecoverStrategy = \failAndAsk;
	classvar <>bootRecoverStrategies;
	...
	var <>bootRecoverStrategy; 
	
	*initClass { 
		bootRecoverStrategies = (
			\failAndAsk: { |serv, onBoot, onFailure| ... }, 
			\hijack: { |serv, onBoot, onFailure| ... }, 
			\reboot: { |serv, onBoot, onFailure| ... }, 
			\switchToFreePort: { |serv, onBoot, onFailure| ... }
		)
	}
	
	... 
	
	boot { ...
		if (isLocal.not) { 
			if (this.serverRunning) { 
				"remote server % is running.";
			} { 
				"Cannot boot remote server %, please boot it remotely.";
			};
			^this 
		};
		if (server.serverRunning) { inform ...  ^this };
		if (server.booting) { "booting .."; ^this };
		
		if (this.portIsFree) { 
			this.reallyBoot(...);
		};
		
		bootRecoverStrategies[bootRecoverStrategy ? defaultBootRecoverStrategy]
			.value(this, onBoot, onFailure);
	}

jamshark70 · 2016-12-04T09:00:02Z

Tangentially related: I just had a sclang crash with supernova booted. After relaunching sclang, I first hit "Kill all servers" in the menu, then booted the server, and it told me the client ID was taken. Huh? I think if I kill all servers, it ought to reset everything, shouldn't it?

Good workflow:

Restart interpreter.
Kill old servers.
Boot server.

Silly workflow:

jamshark70 · 2016-12-04T09:01:48Z

Damn mobile. Anyway.

Silly workflow:

Restart interpreter.
Kill old servers.
Recompile the class library just because killing servers doesn't work.
Boot server.

adc added 2 commits December 2, 2016 21:00

before booting, look for zombie servers, and when needed, stop and ad…

bb19334

…vise.

better names and info text.

26507bf

adcxyz mentioned this pull request Dec 2, 2016

rebooting interpreter is incompatible after sclang crashed with booting server from startup #2397

Closed

check server status by pidRunning, so an unresponsive server is known…

495379a

… to be alive.

nhthn added the comp: class library SC class library label Dec 2, 2016

nhthn added this to the 3.9 milestone Dec 2, 2016

nhthn added the has-conflicts label Dec 3, 2016

adcxyz added the Work In Progress (WIP) - don't merge yet label Dec 3, 2016

jamshark70 closed this Dec 4, 2016

jamshark70 reopened this Dec 4, 2016

adcxyz mentioned this pull request Dec 4, 2016

implement boot conflict handling strategies #2529

Closed

adcxyz closed this Dec 4, 2016

adcxyz deleted the topic-findZombies branch December 4, 2016 18:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

before booting, find zombie servers, stop and advise if needed. #2524

before booting, find zombie servers, stop and advise if needed. #2524

adcxyz commented Dec 2, 2016

adcxyz commented Dec 2, 2016

telephon commented Dec 3, 2016

adcxyz commented Dec 3, 2016

adcxyz commented Dec 3, 2016

telephon commented Dec 3, 2016

adcxyz commented Dec 3, 2016

adcxyz commented Dec 3, 2016

jamshark70 commented Dec 3, 2016

crucialfelix commented Dec 3, 2016 via email

telephon commented Dec 3, 2016

adcxyz commented Dec 3, 2016

adcxyz commented Dec 3, 2016

jamshark70 commented Dec 4, 2016

jamshark70 commented Dec 4, 2016

before booting, find zombie servers, stop and advise if needed. #2524

before booting, find zombie servers, stop and advise if needed. #2524

Conversation

adcxyz commented Dec 2, 2016

adcxyz commented Dec 2, 2016

telephon commented Dec 3, 2016

adcxyz commented Dec 3, 2016

adcxyz commented Dec 3, 2016

telephon commented Dec 3, 2016

adcxyz commented Dec 3, 2016

adcxyz commented Dec 3, 2016

jamshark70 commented Dec 3, 2016

crucialfelix commented Dec 3, 2016 via email

telephon commented Dec 3, 2016

adcxyz commented Dec 3, 2016

adcxyz commented Dec 3, 2016

jamshark70 commented Dec 4, 2016

jamshark70 commented Dec 4, 2016