sclang: add functionality of scp-ing SynthDef when it's too big to send over osc #1042

Closed
wants to merge 1 commit into from

5 participants

@miguel-negrao
SuperCollider member

When working with a server on a remote machine one can't send big synthdefs over osc. This patch allows to bypass that limitation by scp-ing the synthdef file over to the remote machine. ssh must be properly configured with ssh keys installed so that no password is needed. Note that remote folder paths with spaces must be properly escapped.

Example usage:

Server.default = s = Server("remote", NetAddr("143.117.78.26",57110));

s.options.sshSynthDefs = true;
s.options.sshUser = "mnegrao";
s.options.sshRemoteSynthDefDir = "Library/Application\\ Support/SuperCollider/synthdefs/"

s.makeWindow
s.initTree

SynthDef(\aaabbbccc, { 1000.collect{ Saw.ar(100) }.sum }).add
Synth(\aaabbbccc)
s.freeAll

Note that paths on the remote directory have to be escaped.

@timblechmann

this is really just a workaround, and i fear it will cause more troubles than harm (mainly long-term maintenance and win32 portability). imho the true solution is to use a communication channel which does not have a limitation on size (tcp or ipv6 udp w/ jumbograms).

@muellmusik
@miguel-negrao
SuperCollider member

It's true that it is just a workaround, but at the moment we don't know if anyone will be implementing the more robust option, and it is turned off by default so it's probably going to be used by very few users. If it becomes a long-term maintenance problem then that means that the more reliable approach never got implemented, in which case there isn't any alternative anyway...

@timblechmann

@muellmusik prNetAddr_SendBundle, prNetAddr_SendMsg, prArray_OSCBytes and the like currently use big_scpacket, which has a size limitation of 65516 bytes (ipv4/udp packet limit). one should probably catch the error and use a larger, heap-allocated object instead of failing. something like:

static int prNetAddr_BundleSize(VMGlobals *g, int numArgsPushed)
{
    PyrSlot* args = g->sp;

    int numargs = slotRawObject(args)->size;
    if (numargs < 1) return errFailed;

    big_scpacket packet;
    int status = makeSynthBundle(&packet, slotRawObject(args)->slots, numargs, true);
    if (status == errNone) {
        SetInt(args, packet.size());
        return errNone;
    }

    // big_scpacket overflow

    typedef scpacket<10485760> huge_scpacket; // 10 mb

    std::unique_ptr<huge_scpacket> packet( new huge_scpacket ); // not real-time safe!
    status = makeSynthBundle(packet.get(), slotRawObject(args)->slots, numargs, true);

    if (status == errNone) {
        SetInt(args, packet.size());
        return errNone;
    } else
        return status;
}

then one has to go through makeSynthBundle and convert it to a template functions, which takes an arbitrary type instead of big_scpacket as argument. if 10 mb turn out not to be enough, one could do a second try with a 100mb packet.

in netAddrSend, one probably has to check for the maximum udp packet size to avoid sending incomplete synthdefs.

@telephon
SuperCollider member

IIRC, prNetAddr_BundleSize is used only for checking the size. The same would have to be implemented in other places, where the bundle is actually sent.

I suppose this isn't limited to synth defs but would apply to all kinds of osc bundles?

@timblechmann

@telephon this is exactly the reason, why i wrote "and the like". all functions in this file, which make use of big_scpacket have to be adapted

@muellmusik

if 10 mb turn out not to be enough, one could do a second try with a 100mb packet.

With SynthDef in particular is it not possible to determine to some extent the size in advance? In that case we could bypass the tests and go straight for a packet of sufficient size.

@timblechmann

@muellmusik with synthdef it may be possible to pessimize. but if you really want to avoid wasting memory, it is better to implement a version of scpacket, which does not use a pre-allocated array, but a std::vector which can grow dynamically. but this is probably not worth the effort.

@muellmusik

That would have general issues with RT safety though, or?

@muellmusik

Actually for prNetAddr_BundleSize and prNetAddr_MsgSize wouldn't it make more sense to just write some code which calculates the size without creating a packet that's discarded? This could be a separate function which is useful in netAddrSend etc. as well.

@telephon
SuperCollider member

@muellmusic: if you can calculate it, this would make very much sense. When I wrote that method some years ago, I had no other choice because I had no idea of how to calculate it without actually making the bundle. It was still better than nothing then ...

@timblechmann

@muellmusik real-time safety is a problem: scpacket is fixed-sized to avoid dynamic memory allocation. therefore it should be the default. dynamic memory allocation via new or std::vector will compromise rt-safety, but if you place an scpacket<10*1024*1024> on the stack, it will overflow (therefore my example allocates it on the heap).

and of course it would be possible to compute the bundle/msg size without creating the packet, e.g. by having another implementation of scpacket, which provides the same API, but just increments the wrpos pointer.

@muellmusik

@muellmusik real-time safety is a problem: scpacket is fixed-sized to avoid dynamic memory allocation. therefore it should be the default. dynamic memory allocation via new or std::vector will compromise rt-safety, but if you place an scpacket<10*1024*1024> on the stack, it will overflow (therefore my example allocates it on the heap).

Yes, that's what I meant.

@miguel-negrao
SuperCollider member

Ok, I think this pull request is not going to be a good solution for this issue. Is it okay to move this to an issue and close the pull request ?

@crucialfelix
SuperCollider member
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment