Nyquist: capture global Audacity variables like *track* in seq's closure

[werame]

Presently in a Nyquist fx plugin something like

;version 4
;type process
(seq *track* (cue *track*))

fails rather with a rather befuddling error that *track* is nil

error: In CUE, argument must be a sound or multichannel sound, got NIL

The workaround (as Steve Daulton showed me) isn't too complicated; to illustrate with more than a mere copy:

;version 4
;type process
(let ((local-sound *track*))
  (seq (mult 0.2 local-sound)
       (cue (mult 0.7 local-sound))))

works as expected.

The problem with the original example seems to be that seq (seemingly in seq3) only captures local variables, which I suppose is generally fine in "pure" Nyquist, but in Audacity there's there are global variables to worry about like *track* in a fx etc. So, ideally, these global variables like *track* should be directly available in seq-uenced sounds.

As one might suspect, seqrep has the same issue:

(seqrep (i 5) (mult (/ i 5.0) (cue *track*)))

Errors. But

(let ((local-sound *track*))
  (seqrep (i 5) (mult (/ i 5.0) (cue local-sound))))

works.

N.B. sim is not affected by this, e.g.

(sim (mult 0.2 *track*) (cue (mult 0.7 *track*)))

works ok. Probably because it's not using a closure capture.

Proposed user-level quick fix

As my posts below have become a bit TLDR, the fix (IMHO) is just

(setf *environment-variables* (cons '*TRACK* *environment-variables*))
(seq (s-rest 0.2) (cue *track*)) ; works now

Likewise

(setf *environment-variables* (cons '*TRACK* *environment-variables*))
(seq (mult 0.2 *track*) (cue (mult 0.7 *track*))) ; no complaints

See #2396 (comment) why it works.

I'm pretty sure is a problem with this in that it will accumulate the sound samples in memory though, because there's probably reference to the *track*'s cons now created when the seq stores its nyq%environment.

But all variable-level equivalent solutions discussed above or below (let, setf etc.) have the same problem that they create a reference to the *track* sound cons cell. So, I guess there's no real fix for this memory ballooning issue using gc-tracked variables. But at least this "quick fix" is the least burdensome at the call site as there don't have be any changes in the (cue ...) part, it can just reference *track*. So it has to a user-level quick fix for now.

The real solution, I suspect, wouldn't be just to have a function call (nyq%environment is built by calling one), but to have only a function call that returns something which is not tracked by the gc, but also has the ability of being reset on a cue.

I see the point now that if one had a "purely functional" interface, something like a putative (get-track-sound) would return a new cons of a new sound object, so each such copy would have its own independent lifespan, without needing to worry about accumulating samples, since each copy could be garbage collected as it is naturally consumed.

[rbdannenberg]

The "right" way to deal with this would be to replace the global track with a function, e.g. (get-track). I think the reason the first example fails is certainly befuddling:

• The first reference to track is evaluated immediately to obtain a sound-reader object that can access Audacity audio.
• This gets cloned and stored in a seq object that implements seq by lazily returning samples, initially from that sound-reader object.
• The rest of the seq, the expression (cue track), is turned into an unevaluated closure, so this track will get evaluated later (and I don't mean "logically" later, I mean really later, after the effect code returns a sound to Audacity.
• Before Audacity starts fetching samples from the result of (seq track (cue track)), internal Audacity code sets track to nil. This makes the sound-reader that was assigned to track un-referenced, and garbage collection will free it, at which point the sound-reader reference count goes to zero, the sound-reader is freed, and now there's only one reference to the actual sound representing the Audacity track. (This remaining reference is the "clone" created by SEQ.)
• So far, this is all good, because we are using a sound that was bound to a global variable, yet we are not accumulating samples in memory as the sound is used up. However, when we reach the end of the sound (the Audacity selection), SEQ moves on and evaluates the closure at the current logical time (you can't evaluate earlier because you don't know when the previous sound will end, and you need the end time of that sound to establish the start time of the next sound; also you don't want to eagerly evaluate SEQ expressions -- they might be infinite by recursion).
• Now, of course, track is nil, so (cue track) fails.

Another thing that "works" is:
(setf mytrack track)
(seq mytrack (cue mytrack))

In this case, mytrack will accumulate all the samples of track in memory, but at least the sequence evaluates.

Trying to capture the entire state including referenced globals like track would almost definitely lead to other "befuddlement," and whether a variable is even referenced is (I think) undecidable (think about calling functions that may or may not refer to globals). Nyquist has some real-time interactive capabilities based on the fact that globals can be changed while sounds are being evaluated.

Conclusion: We may be stuck with track as a global due to legacy code. The "right" way is to eliminate track as a global and replace it with a function call. The video I made last year showing some possibilities of a functional interface to access Audacity track data provides even more motivation, but it's a substantial amount of code with tricky exception handling and other issues to work out.

[werame]

Nyquist has some real-time interactive capabilities based on the fact that globals can be changed while sounds are being evaluated.

That makes sense, but some other systems like SuperCollider manage the distinction by having (dynamically scoped) "environment" variables besides the lexically scoped ones. And in SuperCollider a Pseq doesn't nil the currentEnvironment on the 2nd and further parts of the stream/sequence. It's probably the case that SuperCollider 's solution is less efficient memory-wise, but I don't know enough about the internals of their garbage collctor to say for sure. It's also less of an issue in SuperCollider because the Pseq is held in sclang but that has no access to the actual sound samples, only the (scsynth) server has the actual sound samples (and even the actual ugens). Memory management for scsynth is basically explicit, as you need to tell it when to instantiate and release ugens, although they have self-releasing ugens (those with doneAction=2), as you probably know. If a Pseq tries to command a ugen that has released itself already, it's obviously not going to work. It would get the fabled "FAILURE IN SERVER <something-something> not found". In some sense we're seeing that here, in Nyquist, as *track* is treated as self-releasing when entering the seq closure.

whether a variable is even referenced is (I think) undecidable (think about calling functions that may or may not refer to globals).

As a reasonable approximation one would need separate compilation (to some kind of AST) and execution phases. The objects, including global variables, potentially accessed by cue could reasonably be determined at the AST-compilation phase, for most use cases, I think. It's true that at actual execution time cue may not access all of those, e.g. if there's an if-branch not taken.

But that would mean having an approach like Extempore does, with a compilation phase, so it might not fit the Nyquist philosophy.

I agree those that that would only be an approximation still. Because one could e.g. splice together the *track* symbol by concatenating some strings and then eval that. Come to think of it, I actually had to do nearly something like that in my own code, pass the quoted track around, but only because copying *track* through stack variables loses its get-able properties, only the sound is kept. The dictionary lookup that get does is purely based on the variable's name, not its object hash or address. So, to resolve something like the eval-based access in that kind of nested function call, one would need Faust-style whole program compilation. (For another parallel, in SuperCollider there are at least two ways to stick arbitrary properties to objects, both of which work by address/hash, not by name: addUniqueMethod and addHalo--the latter is an extension library.)

The video I made last year showing some possibilities of a functional interface to access Audacity track data provides even more motivation, but it's a substantial amount of code with tricky exception handling and other issues to work out.

Can you please link this video? My google fu is failing me on this.

[rbdannenberg]

Here's a link to the video: https://youtu.be/bZLgnWNu9SE

Nyquist is not Supercollider or Extempore. The semantics are different. When you say "that would be an approximation still," I can help interpret that as saying "if you changed some of Nyquist's semantics ..." -- of course I'm biased and don't think there's anything wrong with Nyquist semantics, but even if you wanted to change them, it would be a major redesign and reimplementation. One big difference, is that while many other languages have unit generators or calculation graphs that are sort of abstractions for sounds, the main abstraction in Nyquist is "real sound" as in samples that you can store, copy, shift in time, stretch, etc. This has advantages and disadvantages. One disadvantage is that, when sounds are assigned to globals, they can use a lot of memory, so the first thing I teach students is don't assign sounds to globals unless you really know what you are doing. But that's exactly what happens with *track*, which is why I say the *track* interface is just a poor choice of interface to Audacity.

loses its get-able properties
Yes, what's really going on here is the get-able properties are properties of the symbol *track*. The symbol *track* also has a global binding which is a Nyquist sound, so when you pass *track* as a parameter, you are evaluating *track* to obtain and pass the sound. (By the way, throwing a bunch of metadata into *track* properties is not the way I would do it.)

Adding hidden properties to values is not something supported in Lisp. I suppose you could build data structures to map values to properties, but if the values are big objects like sounds, then using them as keys in a lookup structure would prevent them from being garbage collected.

[werame]

Relating to the track get issue, since *track* is pre-defined, it could (in theory) be just instantiated as a derived class from a Sound class, so that its properties would more naturally stick with the object, rather than have to use a side dictionary. That would alleviate the need to pass it quoted much. Except that Sound is a type but not a class in Nyquist, having to do with XLisp having objects tacked on, rather than being a pure OO like Smalltalk and its SuperCollider derivative, where even Function is a Class.

(send *track* :show)

error: bad argument type - #<Sound: #0000022AF2CD9180>
Function: #<Subr-SEND: #0000022AF31724F0>
Arguments:
  #<Sound: #0000022AF2CD9180>
  :SHOW
1> 

Here's a link to the video ...

That surely looks very interesting. The MIDI track immediately made me think of how Reaper flexibly handles all sorts of tracks.

[rbdannenberg]

Trying to think about mostly functional programming languages in terms of mostly object-oriented programming languages is probably not a good idea. Values in Nyquist are certainly not XLisp objects.

[werame]

Yeah, and having both subtyping and subclassing in a language (as separate notions) leads to all kinds of headaches, as in OCaml.

---

Somewhat off-topic: In the video, with aud-get-notes and aud-put-audio you're basically using Nyquist as a VSTi of sorts, albeit based its own API. Based on quick search in the repo, even your fork here, I'm guessing those functions are available only in your off-line private fork, for now.

---

After seeing that there's a special form (setf (symbol-plist sym) props) to write plists into another symbol and that there's a getter (symbol-plist sym) too, as a quick experiment

(let ((local-sound *track*))
  (seq (mult 0.2 local-sound)
       (cue (prog1 (mult 0.7 local-sound)
	               (print (get 'local-sound 'pan))))))

prints NIL to debug. But

(let ((local-sound *track*))
  (setf (symbol-plist 'local-sound) (symbol-plist '*track*))
  (seq (mult 0.2 local-sound)
       (cue (prog1 (mult 0.7 local-sound)
	               (print (get 'local-sound 'pan))))))

prints e.g. 0 (or some other number) to debug, matching the track's pan.

I guess one could also make an object interface e.g. called aud-track that treats sound like another field/property e.g. accessed like (send *aud-track* :get-sound) and (send *aud-track* :get-pan), so as to treat sound as a property of the *aud-track*, but objects are a bit unwieldy in (X)Lisp in terms of syntax, so they don't buy that much here if one were to be used for mere encapsulation.
But one could probably just pass such on object on the stack and not lose the properties or have to symbol-plist them at every caller that saves a reference in another symbol.

Actually, the properties of the *track* don't get lost in the seq closure, only the sound does, so just

(let ((local-sound *track*))
  (seq (mult 0.2 local-sound)
       (cue (prog1 (mult 0.7 local-sound)
	               (print (get '*track* 'pan))))))

works if one doesn't need to pass *track* around in differently named variables.

Thus, a fairly cheesy OOP-like solution to the original issue that basically imitates your (setf mytrack *track*) approach is to put the sound as a property of its symbol...

(putprop '*track* *track* 'sound)
(seq (mult 0.2 *track*)
     (cue (mult 0.7 (get '*track* 'sound))))

And for passing it around on the stack instead of the *track* one can pass its
symbol-plist.

(defun plist-get (plist key) ; doesn't seem to exist in XLisp
  (second (member key plist)))

(defun callee (track-plist)
  (print (plist-get track-plist 'pan)); ; does something with it
  (print (plist-get track-plist 'sound))) ; does something with it


(putprop '*track* *track* 'sound)  ; creating the plist "struct"
(seq (s-rest 0.2) (cue (callee (symbol-plist '*track*))))

I suppose one can ease some of the kludge at the caller site with a defmacro like

 ; ok, a shorter name would be better 
(defmacro track-plist-with-sound (track) 
  `(progn
    (putprop (quote ,track) ,track 'sound)
    (symbol-plist (quote ,track))))

But while that allows one to write

(setq track-plist (track-plist-with-sound *track*))
(seq (s-rest 0.2) (cue (callee track-plist)))

It still doesn't solve the problem that one has to call it early enough, i.e.

(seq (s-rest 0.2) (cue (callee (track-plist-with-sound *track*))))

would still fail.

A version without side-effect on *track*'s own plist

(defun plist-get (plist key)
  (second (member key plist)))

(defun callee (track-plist)
  (print (plist-get track-plist 'pan)); ; does something with it
  (print (plist-get track-plist 'sound))) ; does something with it


(defmacro track-plist-with-sound (track)
  `(cons 'sound (cons ,track (symbol-plist (quote ,track)))))

(let ((track-plist (track-plist-with-sound *track*)))
  (seq (s-rest 0.2) (cue (callee track-plist)))) ; 

Still this isn't an improvement in terms of having to remember to build it before cue.

[werame]

I don't really understand why this doesn't work though

(defmacro cue-with-track (beh)
   `(progn
      (setf saved-track *track*)
      (cue (let ((*track* saved-track)) ,beh))))

(seq (s-rest 0.2) (cue-with-track *track*)); cue nil err
(seq (s-rest 0.2) (cue-with-track saved-track)) ; ibid

Something to do with macro expansion order, perhaps. I also tried
it with (macroexpand (cue, but it makes no difference.

---

Ultimately, the problem might be that

(print (nyq:the-environment))

(which is defined in https://github.com/audacity/audacity/blob/master/nyquist/nyquist.lsp#L183 ) doesn't show any sound objects, only

((0 1.66667 NIL) 1 -1e+21 0 0 1e+21 2205 44100)

so *track* is definitely not in it. And it's only (nyq:the-environment) that is updated in seq's complex recursive macro expansion; saved at

audacity/nyquist/seq.lsp

Line 83 in 91a557d

(nyq%environment (nyq:the-environment)))

and used at

audacity/nyquist/seq.lsp

Line 54 in 91a557d

(defun with%environment (env expr)

---

If I do:

(setf *environment-variables* (cons '*TRACK* *environment-variables*))
(print (nyq:the-environment))

It looks promising

(#<Sound: #0000022AF2CD9260> (0 1.66667 NIL) 1 -1e+21 0 0 1e+21 2205 44100)

So, now I boldy try

(setf *environment-variables* (cons '*TRACK* *environment-variables*))
(seq (s-rest 0.2) (cue *track*))

And it worked. I guess I can call myself a Nyquist hacker now. 🍿

---

I still don't quite understand how something like

(setf my-sound *track*)
(print (nyq:the-environment)) ;; doesn't show any sound objects
(seq (s-rest 0.2) (cue (progn (print (nyq:the-environment)) my-sound)))
;; ^^ doesn't show any sound objects either

avoids that issue and works though, because the setf-d variables aren't
showing up in (nyq:the-environment) in that case, only

((0 1.66667 NIL) 1 -1e+21 0 0 1e+21 2205 44100)
((0.333333 1.66667 NIL) 1 -1e+21 0 0 1e+21 2205 44100)

But I guess the latter doesn't matter for the purpose of this enhancement request.

It's definitely mysterious because they seem to be in the same *obarray*.
Using the trick from the XLisp manual to look up the hash buckets:

(defun lookin (sym)                  ; create a function to
  (aref *obarray*                    ;   look inside *OBARRAY*
    (hash sym (length *obarray*))))  ;   and look for a specific
                                     ;   symbol - returns a list
									 
(setf mysound *track*)
(print (lookin "MYSOUND"))
(print (lookin "*TRACK*"))

prints something like

(MYSOUND AUTONORM)
(*TRACK* :@@ ATOM)

It seems there's a special hack in

audacity/lib-src/libnyquist/nyx.c

Line 511 in 1657601

// Make sure the sound nodes can be garbage-collected. Sounds are EXTERN

that makes the *track* not get copied to new obarrays, but I'm not sure when that gets triggered.

There's also special treatment for parallel returns (vector *track* *track*) in the C code

audacity/lib-src/libnyquist/nyx.c

Line 1024 in 1657601

// will fail if nyx_result is (VECTOR S S) or has two references to

, but nothing for the seq-equivalent of that.

---

Footnote: The bigger bretherin Common Lisp has some more elaborate functions for manipulating environments like augment-environment and also has enclose for binding a lambda to a specific environment, but those functions don't exist in XLisp. So, basically seq is rolling its own equivalent, of sorts.

[werame]

@rbdannenberg I guess you're sick and tired of this topic by now, but does snd-copy help in any way with the GC issue? The manual is rather unclear on this, just says:

(snd-copy sound) [LISP]
Makes a copy of sound. Since operators always make (logical) copies of their sound parameters, this function should never be needed. This function is here for debugging.

So if one writes

(setf mytrack2 (snd-copy *track*))
(seq *track* (cue mytrack2))

Does that allow the samples of *track* to be garbage collected as they are consumed in seq? Or is there a pointer somewhere in what snd-copy returns that prevents that?

A hint how to test what the gc does in situations like this would also be greatly appreciated.

For instance, there seems to be a difference between these:

(setq *gc-flag* T)
(setf *environment-variables* (cons '*TRACK* *environment-variables*))
(seqrep (i 9) (cue (mult (/ i 10.0) *track*)))

[ gc: total 70640, [ny:gc-hook allocating 20000 more cells] 29727 free; samples 1412KB, 1381KB free ]
[ gc: total 90640, 29783 free; samples 1412KB, 1293KB free ]
[ gc: total 90640, 29785 free; samples 1412KB, 1381KB free ]

and

(setq *gc-flag* T)
(setf mtrk *track*)
(seqrep (i 9) (cue (mult (/ i 10.0) mtrk)))

[ gc: total 70640, [ny:gc-hook allocating 20000 more cells] 29726 free; samples 1412KB, 1381KB free ]
[ gc: total 90640, 29780 free; samples 1412KB, 1218KB free ]
[ gc: total 90640, 29782 free; samples 1412KB, 1218KB free ]

The former seems to free memory sooner, but maybe I'm misinterpreting it.

[rbdannenberg]

Wow, so many questions! I'll give some high-level answers, but if that leaves some things unanswered, ask again. You are one of the few who digs deep enough to even ask these kinds of questions.

Beginning with nyq:the-environment, Lisp is not good at information hiding -- you can always poke around and see things maybe you shouldn't. Some of Nyquist is written in C and extends XLisp at a very low level, e.g. Nyquist has types that XLisp does not -- you could not implement Nyquist in "pure" XLisp; it had to be extended with the SOUND type. Other parts of Nyquist, in particular the "environment" and "temporal semantics" are built in XLisp. XLisp does not have STRETCH or AT, but these are fundamental to Nyquist. To make them work, i.e. to obtain Nyquist semantics, you have to assume the implementation will not be tampered with, even though it is unprotected. So while nyq:the-environment is a variable and you can change it, doing so changes Nyquist at the implementation level, and all bets are off in terms of semantics or program behavior.

That said, what you did is very interesting and might provide a path for some future extension of Nyquist having to do with sequences -- it never even occurred to me before -- but it's sort of analogous to saying I could jump into another stack frame in C++ by overwriting the return address. It might even work and be useful, but the C++ reference says that the entire program is undefined when you do that.

The cue-with-track macro trick doesn't work because when you write (seq (s-rest 0.2) (cue-with-track track)), the 2nd part of the SEQ is wrapped in a closure to be evaluated when the first part reaches its logical stop time (0.2). In Lisp, macros get expanded at run time, so the SETF to save *track* in the macro does not occur immediately when SEQ is evaluated.

You could save *track* outside of the SEQ like this:

(let ((saved-track *track*)) (seq (s-rest 0.2) (cue saved-track)))

You could imagine a special form of SEQ that would capture local values; something like

(let-seq ((saved-track *track*)) (s-rest 0.2) (cue saved-track))

and you could write LET-SEQ using DEFMACRO.

Garbage collection

I modified XLisp to support references to external objects. An external object is represented, like everything else, by a CONS-like cell (the size of 2 pointers), a type tag, and some GC bits. If the type-tag is EXTERN, the cell contains

1. a descriptor with a type name and function pointers that implement free() [called by the sweep phase of GC if the cell is unmarked], print(), save() [not used], restore() [not used], and mark() [called by the recursive mark phase of the mark-sweep GC.]
2. a pointer to the object, e.g. a SOUND (sound_type defined in nyqsrc/sound.h). There are other external types as well, e.g., there's a MIDI-like sequence structure adapted from the very old and obsolete CMU MIDI Toolkit, supporting standard midi file IO (independent of Adagio note tracks).

SOUNDS have a 1-to-1 correspondence with XLisp EXERN nodes. When GC deletes the node [literally, see "struct node {...}" declaration in xlisp/xldmem.h], it also frees the SOUND.

But, a SOUND is mostly just metadata telling you: start time, stop time, sample rate, current sample position, logical stop sample count, scale factor and a few other obscure things PLUS a pointer to a shared, reference-counted, list of sample blocks. That's where the samples are actually stored. At the end of the list is a "suspension" which is basically a unit generator that can generate a new block of samples and extend the list. Sometimes the suspension contains a closure that can create a new sound when more samples are needed. That's what SEQ does.

Here are some different cases:

1. delete the last reference to an EXTERN node, e.g. (set foo (osc c4)) (set foo nil), the EXTERN returned by (osc c4) was referenced by foo, but now the reference is gone. The EXTERN and it's SOUND are still around, but when GC runs, the EXTERN node will not be marked, so the sweep phase will free it and pass the SOUND to the corresponding free() function, so the SOUND will be freed as well.
2. copy a Lisp value, e.g. (setf saved *track*) -- this is a simple copy of a pointer to the EXTERN node. There's no reference counting or type checking. It generates a second reference to the EXTERN node, so now if you set *track* to NIL, the EXTERN is still referenced and GC will still mark it, and it will not be freed.
3. snd-copy: This creates a new sound that shares the underlying list of sample blocks and suspension. Now we have two sounds sharing the samples. You can read from one SOUND and it will traverse the list, forcing the suspension to extend the list as needed. The other SOUND is still referencing the head of the list, so all the samples are retained in an ever-growing list. When that SOUND is read, and it traverses the list, it will decrement reference counts on the list elements (which are big blocks, typically about 1000 samples per block) freeing them as it moves along.
4. In the typical case where there's only one SOUND referencing a list of samples, the suspension computes about 1000 samples, the SOUND reader consumes it, decrements it's reference count from 1 to 0, the block is immediately freed, and quite likely it gets reused immediately by the suspension to compute the next block. The list of sample blocks does not grow and consume memory.
5. snd-copy, globals, and shared sounds: There's a tricky case when you use a sound twice, e.g. (let ((s (osc c4))) (sum s s)) -- the same happens with (setf s (osc c4)) and then you use s multiple times. Since s references one SOUND, if you start reading from SOUND, you traverse a list of sample blocks and lose track of the beginning of the sound. So if you use s again, you could get two "readers" interfering with one another. It's like opening a file once and having two threads read from it instead of opening the file separately for each thread. To avoid this, Nyquist functions almost always use snd-copy to make a copy, e.g. (sum s s) operates on copies of s. So now you have THREE SOUNDS -- the original sound bound to the variable s, and two copies held inside the suspension created by SUM. When you return this expression, since s is a local variable, GC will free s and free the EXTERN and SOUND associated with it. This in turn decrements the reference count on the sample block list head from 3 to 2. When SUM is computed, both SOUND copies will traverse the list decrementing reference counts as they go, allowing the blocks to be freed.
6. globals The same happens with sounds assigned to globals, but the global is not GC'd, so there's always a SOUND referencing the samples, and the samples are not freed. This allows the global to be used again and again.

[SteveDaulton]

Another possible approach could be to create a wrapper. For example:

(defmacro wrapper (cmd)
  (let ((*track* *track*))
    (eval cmd)))

(wrapper (seq (s-rest 0.2) (cue *track*)))

[rbdannenberg]

I think that would not work because eval does not evaluate in the scope of the let. This might work:

(defmacro wrapper (cmd)
  `(let ((*track* *track*))
      ,cmd))

but then you are making *track* a local variable -- confusing because *track* is defined as a global and by convention *var* indicates a global. In the case of (seq (cue *track*) (cue *track*)) this would also read the entire track into memory, but I think that can only be solved by changing the interface from *track* to something like (get-selected-track), analogous to letting you open and read a file twice.

[SteveDaulton]

I think that would not work

and yet it does :-)
It also works with:
(wrapper (seq *track* (cue *track*)))

(defmacro wrapper (cmd)
`(let ((track track))
,cmd))

and that works too.

think that can only be solved by changing the interface from *track* to something like (get-selected-track)

Yes, I think that's the real solution.

[rbdannenberg]

and yet it does :-)

Wow, I'm stumped. I think SEQ jumps through some hoops to save the current bindings and use them to evaluate expressions that form the sequence (i.e. (s-rest 0.2) and (cue track)), but I thought EVAL would shield the LET bindings when it evaluates CMD. Maybe it's a bug!

[SteveDaulton]

I don't think it's due to SEQ that this unconventional form works. It also works with other functions. For example:

(defmacro wrapper (cmd)
  (print cmd)
  (let ((a a))
    (print (eval cmd))))

(setf a 42)
(wrapper (setf a (+ a 2)))
(print a)

prints:

(SETF A (+ A 2))
44
42

In the first PRINT statement, CMD is printed as a LIST.
Maybe it's an unexpected implementation detail in Nyquist / XLISP, but it seems that EVAL prevents CMD from being evaluated until reached in the execution of the code in the macro.

If we miss out EVAL, then the result is:

(SETF A (+ A 2))
(SETF A (+ A 2))
44

where we can see that CMD has been evaluated with the global "A" (before the local binding has been created).

[werame]

(SETF A (+ A 2))
44
42

It'd doubtful that's a bug, since Elisp does it exactly the same way... unless you change eval to eval-when-compile, meaning

(defmacro wrapper (cmd)
  (print cmd)
  (let ((a a))
    (print (eval-when-compile cmd))))

(setf a 42)
(wrapper (setq a (+ a 2)))
(print a)

it prints

(setq a (+ a 2))

(setq a (+ a 2))

44

That kinda explains why/how it works with plain eval.

The original form doesn't compile with SBCL though, seemingly because the macro is accessing an unbound global, even if you defvar a. So it's probably not CL compliant, not that it matters much. Actually, checking out some SO Q&A on this, I'm 100% sure it's not CL compliant: "macro[s] are expanded at unspecified time".

Also what the working wrapper does is essentially just quoting the expression, as the return value isn't used, execution isn't delayed, so with the minor inconvenience of having to quote the arg, the following is equivalent

(defun wrapper-f (cmd)
  (print cmd)
  (let ((a a))
    (print (eval cmd))))
	
(setf a 42)
(wrapper-f '(setf a (+ a 2)))
(print a)

prints

(SETF A (+ A 2))
44
42

as well.

And this one is CL compliant (for what's that worth.)

N.B.: In SBCL you can get a working macro wrapper (on global vars) by trivially wrapping the function wrapper, as in:

(defmacro wrapper (cmd)
	`(wrapper-f ',cmd))

(wrapper (setf a (+ a 2)))
(print a)

That also works in Elisp and even in XLisp (modulo the defvar).

[werame]

With respect to the garbage collection issue: First, I want to thank prof. Dannenberg for taking the time to explain all that in detail.

Here are my 2cents on that, as I understand the matter. SOUND objects in Nyquist provide a unifying interface for what would be different kinds of objects in other systems. For instance in SuperCollider there are different notions of Buffers vs Ugens.

Now, the unification (or lack of user-visible subtyping) that Sound objects have in Nyquist has its obvious advantages in terms of a simple(r) API, but there's also this efficiency issue, which comes from them wearing two hats:

• For Sounds that were e.g. procedurally generated, say, even using a random number generator, being able to refer to copies of the exact same sound, down to every sample is obvious important. And that in turn requires that samples be kept (in memory) once produced so that copies whose (snd-copy) playhead is further behind can (later) refer to the same samples.

• When Nyquist is hosted in Audacity, for sounds that are backed by Audacity's tracks, this preservation is actually overkill because even if Nyquist released its internal samples for a *track*, if it were to read them back from Audacity in the same (plugin) run, it would get the same sample values. Audacity has an undo function, which I'm pretty sure requires something like copy-on-write so that no Audacity buffers are actually overwritten from Nyquist, but what Nyquist returns is almost certainly saved in new Audacity buffers, albeit just a copy-on-write partial copy if only a range of a track is selected. I'm not actually familiar with the details on that, so please correct me if I got something wrong, but I think conceptually the semantics are as I described them.

So, it would make sense (to me) to consider less strong GC semantics in Nyquist for sounds that are backed by Audacity *track*s, since if they were to be released but later (e.g. in a seq) needed to be read back again, they'd be guaranteed (by Audacity) to be the same samples. Thus perhaps it's worth considering in Nyquist a notion of weak references for the Sound samples that are backed by Audacity tracks.

Simply having a functional interface (say (get-selected-track)) without this weak references feature doesn't appear (to me) to solve the GC issue, because merely creating and returning a new playhead on (get-selected-track) is essentially what snd-copy is doing already.

[SteveDaulton]

Some years ago there was an experimental version of Nyquist in Audacity that included (something like):
(aud-fetch-samples track channel t0 t1)
(aud-put-samples track channel t0 t1)

A major benefit being that multiple passes could be applied to long tracks without storing the entire track in memory (an actual solution to the old "normalize long sound" problem).

[rbdannenberg]

I think that would not work because eval does not evaluate in the scope of the let.

Sorry for getting this wrong and launching a long discussion -- EVAL in XLISP evaluates with the current lexical bindings. What EVAL does seems to be very dependent on the Lisp implementation. For context, XLISP is pre-common-lisp, and I think it was Common Lisp that started a large effort to (re)consider Lisp semantics carefully, leading to many changes and additions relative to a handful of previous implementations.

[rbdannenberg]

it would make sense (to me) to consider less strong GC semantics in Nyquist for sounds that are backed by Audacity *track*s, since if they were to be released but later (e.g. in a seq) needed to be read back again, they'd be guaranteed (by Audacity) to be the same samples.

That's a very interesting idea: Nyquist could have "deterministic" sounds that recompute on demand to avoid saving samples in memory, and "non-deterministic" sounds that only compute once. SOUNDs as they are were difficult to implement, so introducing another implementation of SOUND would not be simple, and you'd want to think about trade-offs between recomputation and storage (e.g. if it took an hour to compute a SOUND, you'd want to treat it as "non-deterministic" to avoid recomputing a copy. I can also imagine disk-based sounds -- this could probably be done in current Nyquist and Audacity without too much trouble. Just push the (finite only please) sounds to disk (eagerly) and then re-read copies to minimize the number of samples in memory.

Simply having a functional interface (say (get-selected-track)) without this weak references feature doesn't appear (to me) to solve the GC issue.

As Steve points out, we don't have a solid implementation, but there's an "existence proof" that it works. Briefly, for the (seq (get-selected-track) (get-selected-track)) case, there's no track and no global variables ever holding references to any SOUND. Calling get-selected-track creates a SOUND with a suspension that is set up to call into Audacity and retrieve track samples. The SEQ would read from this SOUND to the end of the track. [In a little more detail, SEQ would snd-copy the SOUND first, leaving 2 references, but one would be GC'd early on.]. When we're half-way done, SEQ drops its reference to the first SOUND and calls the second get-selected-track. (It almost doesn't matter that the first SOUND will be garbage collected because reference counting has already freed all the sample buffers that it read in, and it never needed more than one sample buffer at any time.) This creates a wholly new interface object to access samples from Audacity that begins returning samples from the beginning of the track.

Returning to my comments about "deterministic" vs "non-deteriministic" sounds, good style in Nyquist is to represent sounds as functions, e.g. write (defun mytone () (note c4)) rather than (setf mytone (note c4)), only using variables to represent sounds when you really want to avoid recomputation. So the (get-selected-track) kind of interface follows this style, which is just a way of allowing Nyquist to recompute [or re-copy in the case of Audacity tracks] samples rather than store them.

[werame]

EVAL in XLISP evaluates with the current lexical bindings. What EVAL does seems to be very dependent on the Lisp implementation.

After spec-reading on this, the "hyperspec", which I think reproduces the CL 2.0 spec, says this about eval:

Evaluates form in the current dynamic environment and the null lexical environment.

The spec even has this example

(setq form '(1+ a) a 999) ;; =>  999
(eval form) ;;=>  1000
(eval 'form) ;; =>  (1+ A)
(let ((a '(this would break if eval used local value)))
   (eval form)) ;; =>  1000

Unless I'm misinterpreting, it seems to me that XLisp, Elisp, and SBCL are all non-compliant in that "null lexical environment" regard. But that also makes the spec a bit of the odd man out...

And, of course, the spec example does break in all of the above implementations, e.g. Elisp says:

Wrong type argument: number-or-marker-p, (this would break if eval used local value)

SBCL:

Unhandled TYPE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                  {10005D05B3}>:
  The value
    (THIS WOULD BREAK IF EVAL USED LOCAL VALUE)
  is not of type
    NUMBER
  when binding SB-KERNEL::X

Actually, I am missing something else from the spec. To make the SBCL example work, I added defvars, but, as I'm informed on this SO Q&A when I did that I special-changed the binding of the symbol to dynamic! That can be cross-checked with the spec entry on defvar.

defparameter and defvar establish name as a dynamic variable.

Since there's no defvar in XLisp (and it's optional to use it in Elisp), all unbound symbols are treated in these implementations as having been defvar'd, i.e. (globally) given dynamic binding.

---

So, what actually happens with Nyquist seq in Audacity is that except for *track* which is deliberately nil-ed after first use (of its samples), all other (dynamically bound) globals actually survive a seq by themselves, **even those not actually saved in seq's *environment-variables* **

Here's a demo of that happening. Nyquist only tries to save these (this is defined in nyquist.lsp):

(setf *environment-variables*
      '(*WARP* *SUSTAIN* *START* *LOUD* *TRANSPOSE* 
    *STOP* *CONTROL-SRATE* *SOUND-SRATE*))

But Audacity actually defines more, and by this I mean enters them into the obarray, just as it does with *track*, including e.g. *DEFAULT-SOUND-SRATE*. The latter is not on the Nyquist "specials" to be saved given above. So, does *DEFAULT-SOUND-SRATE* survive in seq despite not being explicitly saved? The answer is yes:

(seq *track* (cue (progn (print *DEFAULT-SOUND-SRATE*) (s-rest 0.2))))

Prints the actual value of 44100, even though Nyquist didn't save it in the *environment-variables*.

[rbdannenberg]

"hyperspec" refers to "Common Lisp HyperSpec". Nyquist and XLisp are not Common Lisp and the spec does not apply. (Sure, it's good to know what many serious Lisp people decided what Lisp should do, but compliance with that is not a goal of XLisp or Nyquist.)

Don't assume environment-variables refers to the general XLisp environment or is trying to define all the variable bindings available to SEQ expressions -- it's specifically capturing dynamic variables because Nyquist semantics say that the SEQ behaviors are evaluated in the Nyquist environment (meaning those specific variables: WARP, SUSTAIN, etc.) of the SEQ (modulo adjusting for a new start time). Those variables are dynamic so they get some special treatment.

[werame]

I agree that *environment-variables* is more special because those values change with Nyquist specifics, but to backtrack a little to the Lisp dynamic vs lexical issue, for practical purposes, what happens is that when you eval that wrapper macro proposed by Steve (and which I've modified a bit to make work in CL too):

• Because the a symbol is in the obarray, it's treated as dynamic, so its name gets "copied" into what the eval'd form "sees". In fact, this "copy" is a no-op since the obarray is just shared.

• The value lookup is done (nonetheless) dynamically by finding the let-shadowed "chained cell", not the global, top-level one. In XLisp in particular, it works because the stack frame actually stores symbols names, not just values.

• the setf-writing inside the eval also changes the latter "chained cell". After eval finishes and the let-ed "chained cell" goes out of scope, the symbol points back to the "global" cell.

So, the seemingly lexical variable declared with let was actually a dynamic one. Lisp (not just XLisp) has this confusing notion that it's the symbol that is dynamic (or lexical) at a global level. That's not a local property of each of its appearances in a given lexical context, so you can have a seemingly lexical variable (i.e. let-ed) that is actually dynamic, because its symbol was declared so. In XLisp and Elisp it's a bit more confusing than in CL because that can happen simply because one previously assigned to the unbound symbol, which enters it into the obarray, as if you'd used defvar in CL.

To illustrate the middle bullet, here's a modification of the warapper macro (I'm using the CL-compliant version that uses a function intermediary, so I can test in SBCL too)

(defvar a 42)
 
(defun wrapper-f (cmd)
  (print cmd)
  (let ((a (+ 100 a))) ;; changed so that it's a diff value
    (print (eval cmd))))
 
(defmacro wrapper (cmd)
	`(wrapper-f ',cmd))

(wrapper (setf a (+ a 2)))
(print a)

Prints

(SETF A (+ A 2)) 
144 
42 

Works the same in XLisp and Elisp with the defvar replaced with a setq.

[rbdannenberg]

I'm not confused by the examples, now that I know eval keeps the local scope or environment.
"dynamic" is overloaded. XLisp has a shallow binding scheme for dynamic variables AND a list of contexts, each being an association between symbols and values, to implement it's static scoping rules (and closures).

[werame]

To make something more immediately practical from this discussion, one could have a track-seq (which is the right place to put the macro as prof. Dannenberg indicated earlier) defined like so:

;; just using Steve's shorter, non-CL-compliant approach here,
;; as there's no SBCL-based Nyquist :-)

(defmacro track-seq (&rest args)
  (let ((*track* *track*))
    (eval `(seq ,@args)))) ; eval is immediate on purpose, not a bug

(track-seq *track* (cue *track*))

On the other hand, besides the syntax, I think there's no real functional difference between that and what I suggested earlier,

(setf *environment-variables* (cons '*TRACK* *environment-variables*))

(seq *track* (cue *track*))

which basically just makes seq to that let-work for *track* too.

[rbdannenberg]

I like

(setf save-track *track*)
(seq (cue save-track) (cue save-track))

better -- it makes it clearer that the "solution" is to preserve track as a global. Also, note that this does not work (I think):

(seq (s-rest 0.1) (track-seq (cue *track*) (cue *track*)))

[werame]

note that this does not work (I think):

(seq (s-rest 0.1) (track-seq (cue track) (cue track)))

Of course it doesn't, one has write it like

(track-seq (s-rest 0.1) (seq (cue *track*) (cue *track*)))

which is why my approach to just trick seq to do the let-ing is more convenient for the user.

The only slight perf. advantage of an explicit track-seq that I see is in something like

(seq (track-seq (cue *track*) (cue *track*)) (s-rest 0.1))

Here you release the letted *track* a bit earlier by using the track-seq only for the inner one, as opposed to what happens if a seq that always saves *track*, in which case *track* will be (uselessly) available during the s-rest too.

Also, nitpicking: *track* is a global, it gets set to nil after first playthrough.

[werame]

"dynamic" is overloaded

I apologize if I muddied the water with the "dynamic scoping" terminology but it is what it is. I
didn't come up with it. 🌵

Basically, in Lisp in general, eval isn't exactly the culprit there, the same behavior exists in most Lisps even with simple function calls when a local shadows a global on the stack, even when the shadowing was done outside current lexical context.

I can see now why you're surprised by what XLisp does on eval, because this program

(setq xx 12) ; needs defvar in SBCL

(defun f1 ()
   (print xx))
   
(defun f2 ()
   (let ((xx (+ 33 xx)))
      (f1)))

(f2) ;; prints 45 in Elisp & SBCL, but 12 in XLisp
(f1) ;; prints 12 always :)

So I guess XLisp is a bit less standard in that regard in that it tries to force pure lexical scoping (like Scheme or C does), but fails to do so when eval is used, in which case XLisp reverts to "standard" dynamic scoping Lisp behavior.

Also just

(defun f2e ()
   (let ((xx (+ 33 xx)))
      (eval '(f1))))

doesn't "fool" XLisp, (f2e) is still 12. You have to have the eval inline the expression:

(defun f2ed ()
   (let ((xx (+ 33 xx)))
      (eval '(print xx)))) ; "inline" version

now (f2ed) is 45, even in XLisp.

So, Steve's find was quite a find, in hindsight.

Unfortunately, what my (f2e) discovery in this post entails is that track-seq as proposed earlier will fail to resolve *track* if there are any nested function calls that try to use *track*! 😭 Example

(defun callee ()
   *track*)
   
(defmacro track-seq (&rest args)
  (let ((*track* *track*))
    (eval `(seq ,@args)))) ; eval is immediate on purpose, not a bug

(track-seq (callee) (cue (callee)))
;; error: In CUE, argument must be a sound or multichannel sound, got NIL

So, only my earlier hack of adding to *environment-variables* (or using an actual global) works for that:

(defun callee ()
   *track*)

(setf *environment-variables* (cons '*TRACK* *environment-variables*))

(seq (callee) (cue (callee))) ; still works

I guess one could try a better emulation of seq using a progv like it does in a putative track-seq-v, instead of relying on the eval that doesn't quite "see" across function calls, but I'm not seeing a lot of point in duplicating that much of seq right now in another macro.

[rbdannenberg]

I would sum up by saying if Audacity is going to clobber *track* as soon as you return, then you really shouldn't expect to use *track* except as a special one-time-only access at the start (not in SEQ). I had forgotten that CL supports dynamic scoping -- generally that's frowned upon because it's hard for compilers to analyze, and most would say CL is statically scoped.

[werame]

Well, one thing we have touched on is: if the C side nils *track* after setup, why doesn't it restore it before executing the closure that was setup by seq? It should probably happen around xleval(cons(ms->closure which re-enters the interpreter. Because if there is a seq closure to execute, the user code is clearly not done, so it might need *track* again.

I mean the hack I have above with (setf *environment-variables* effectively forces the closure code to do that restauration itself, from a copy saved within the interpreter. But it seems that a better place for that to happen is in the C code, because it was the one responsible for setting track nil in the first place, and because it knows exactly when to do it: if there's another closure to execute, before it's truly done (with the current plugin).

It seems to me that if the restoration were done from the C side, that would prevent the need to accumulate samples. If I understand this correctly, the *track* is set to nil after the generator code for the 1st behavior in a seq is setup, but before (actually) running it to produce samples. So, if there's another closure to execute, for which the interpreter needs to be re-entered (in that xleval(... closure) there's a symmetrical situation as a when the plugin is first entered, i.e. that 2nd behavior in the seq might need *track* to set up its own generator. By doing this setup and teardown only from the C side, the track could be made nil before this 2nd generator actually runs, just it happened before the first one actually ran.

I guess I can try my idea on my checked-out audacity fork 🥨

As I looked wee bit more at this, one downside of this C-approach compared to (setf *environment-variables* (cons '*TRACK* ... (which catches everything, including e.g. trigger) is that there's not a single interface for these closure callbacks at the C level, e.g. trigger directly calls xleval(... closure. So one would need a bit of a single entry API, like a (C) function for all those closure evals, so the there's just one place to patch with the restore track business. The complete list is probably these 3:

$ git grep -G 'xleval.*clos'
lib-src/libnyquist/nyquist/nyqsrc/multiseq.c:    result = xleval(cons(ms->closure, consa(cvflonum(now))));
lib-src/libnyquist/nyquist/nyqsrc/sndseq.c:        result = xleval(cons(susp->closure, consa(cvflonum(now))));
lib-src/libnyquist/nyquist/nyqsrc/trigger.c:                result = xleval(cons(susp->closure, consa(cvflonum(now))));

It turns out this is a bit complicated because nyx_set_input_audio is not called by anything in nyx.c but directly from effects/nyquist/Nyquist.cpp (which then calls nyx_eval_expression
some 100 lines later).

So nyx doesn't know the actual arguments with which nyx_set_input_audio was called if it (or the GC) discards that first object it built (with sound_create). So nyx needs to save those nyx_set_input_audio call args in some nyx.c-module variables on the call to nyx_set_input_audio and then in the *track*-restore business before calling any of those 3 kinds of seq-type closures it needs to do a (to be written) nyx_restore_input_audio in which in needs to patch those track things back into an object by calling another sound_create. (The nyx_set_audio_params don't actually get swept by the gc, because they are not set to nil, so they need not be restored.)

I also realized that if I do the stuff in previous para, we're like 80% done with making a "pure functional" interface, since the data to build (via sound_create) the object that pulls samples from Audacity can then be (re)built whenever we like/need at nyx level. The only difference between preemptively building it again before entering every seq-type closure vs building it totally on demand when some XLisp function is called is very slight, since the sound object doesn't take up any sizeable GC space until samples are pulled from it. So if a seq-closure is just some s-rest, it's probably no big deal that we pre-emptively create another sound object to represent the *track*, even if the s-rest doesn't use it.

There are a couple of additional points:

1. The restoration has to be conditional on *track* being nil. That's because one can also consume a seq immediately and not from an Audacity pull, when e.g. one builds a wavetable with maketable from sound that's produced by a seq.

(maketable (seq (print *track*) (cue (print *track*))))
*track*

prints

#<Sound: #000002D3C216EBF8>
#<Sound: #000002D3C216EBF8>

meaning it didn't get set to nil in this case.

2. Similar to that, we'd have to consider what if the user wants to e.g. output the first minute the *track*, followed by the last minute. In this case, it would be natural to try and pass the same, partially consumed *track* to the seq closure. So, it would make more sense to actually give back the same object rather than a new one, in some cases, but I'm not entirely sure if it's possible to decide them from the C side. I'm still mulling this last issue.

[rbdannenberg]

why doesn't it restore it before executing the closure that was setup by seq?

If SEQ uses a copy of SOUND in*track* and *track* retains the SOUND, then you have 2 references to the samples, and samples are retained. If you set *track* to nil, the reference count drops to 1 and as samples are consumed, their reference count goes to zero and they are freed, so no accumulation. If you restored *track* to a new copy of the SOUND, the reference count would go back to 2, and samples would accumulate.

I think your ideas might work, but hooking Audacity into the XLisp evaluator and making a special case out of one designated global variable *track* is really extreme. The hacks are compounding:

1. Hack 1: passing parameters through globals: *track* is really a parameter to a user-written function, but rather than implementing it as a parameter in keeping with Nyquist's functional style, a different approach was taken (reminds me of very early BASIC where you could "gosub" to a "subroutine" but you had to pass any parameters by assigning globals.)
2. Hack 2: since assigning a SOUND to a global was a bad idea, let's "fix it" but unassigning the variable before too much damage is done.
3. Hack 3: (proposed) Since unassigning *track* behind the programmer's back seems odd, let's "fix it" by detecting when the programmer might notice and restoring a value.

Actually, if you want to hack the lowest level XLisp functions, why not make a special value you can assign to *track* and change the XLisp code that accesses global variables to check for the special value. If you see it, call into Audacity to construct a new SOUND that reads from the selection. So every time you read *track* you get a fresh new SOUND with reference count 1.

But to do that (or to take your approach) I think you need an extended API, which by the way I've already written. The problem is that the Nyquist API is tangled up with exception handling and track management -- it was too complex for me to have any hope of getting it right, so I got enough working to make the video, hoping to motivate the new API.

[werame]

Fair enough. I'm also thinking right now that maybe just getting this (get-track) implemented would actually be a little easier than also finessing the auto-restauration of *track*.

I do have on design point I'd like some feedback one (as I writing it as you replied...): a question similar to (2.) applies to the proposed functional interface (get-track). Should it always return the same singleton object, and leave it up to the user to cue it to restart it, or should it do that cue-ing itself?

After looking at snd_xform, which is the underlying implementation of cue, and which always seems to return a copy (albeit more thinly constructed one in some cases it seems to me that the right answer here is to always return a new sound from this (get-track), pointing to the beginning of the track (as supplied by Audacity to nyx_set_input_audio) so to do the equivalent of (cue *track*).

If the user needs to have a partially consumed track resumed later (as in the example at 2.) then they would not quite be able to use this always cue-ing (get-track) directly for that effect, alas, unless we made it more complex in that it would also accept a time parameter to seek to, so that the (new) object returned could also start a non-zero position in the Audacity buffer that backs it.

snd_xform when implementing the time shifting function does a SND_flush, which seems pretty efficient enough for skipping samples, albeit it does that in a loop. I guess we can think about a more efficient (get-track position) later, since (shift-time (get-track) position) looks like a useable approach.

[rbdannenberg]

For a functional interface, this come from src/effects/nyquist/NyquistAPI.h. As for "pointing to the beginning of the track", Nyquist sounds have absolute start times, and this API defines time to be Audacity time (tracks start at 0, but if you extract sound from 10s to 20s, then the start time of the sound would be 10.0). CUE returns a new sound constructed by shifting its argument in time to align the argument's original start time with the current start time. This is not necessarily setting things up to immediately return the first sample because the first generated or stored sample can be at a positive offset from the start time, in which case Nyquist pads the sound with zeros (sometimes re-using a pre-initialized reusable zero block, sometimes the zeros can be optimized out and never looked at) until the first "real" sample is reached.

//     track index starts at 0, incremented by 1 for each multi-channel track
//     E.g. if there are two stereo tracks, they are tracks 0 and 1.
//     Properties are:
//         NAME - the name (STRING)
//         TYPE - WAVE, NOTE, or LABEL (SYMBOL)
//         For type WAVE:
//             CHANNELS - 1 or 2 (FIXNUM)
//             CLIPS - ((start end) (start end) ...) for mono (LIST)
//                     or #( ((s e) (s e) ...) ((s e) (s e) ...) ) for 
//                        stereo (ARRAY)
//         For type NOTE and LABEL:
//             no other properties, although if notes and labels were organized
//             into clips, it would be appropriate to add a CLIPS property. 
//             Notes and labels are accessed using AUD-GET-LABELS and 
//             AUD-GET-NOTES
//
// (AUD-GET-AUDIO id start dur) -- return sound or array of sounds.
//     id is name or number
//     start is project time for start of sound (FLONUM)
//     dur is duration in seconds (FLONUM)
//     zeros are inserted as necessary to achieve the requested duration
// 
// (AUD-PUT-AUDIO id expr start maxdur) -- write sound to a track
//     id is name or number
//     expr is a lisp expression to be evaluated to get a sound
//          the expr must evaluate to a mono or stereo sound matching the track
//     start is an offset time applied to the result of expr
//     maxdur is the maximum duration (relative to start) for the sound
//
// (AUD-GET-LABELS id start dur) -- return labels.
//     id is name or number
//     start is project time for start of label region (FLONUM)
//     dur is duration in seconds (FLONUM)
//     returns list of labels: ((T0 T1 "label") (T0 T1 "label") ...)
//         such that label T0 >= start and T0 < start + dur. 
//         Note that labels can have durations > 0 such that T1 > start + dur.
//
// (AUD-PUT-LABELS id labels merge) -- add/replace labels to track
//     id is name or number
//     labels is a label list in the format returned by AUD-GET-LABELS
//     if merge is NULL, replace all labels in the track. Otherwise, merge.
//
// (AUD-GET-NOTES id start dur inbeats) -- return notes and updates from track
//     If inbeats is NULL, all time units are in beats. Otherwise all units are 
//         seconds.
//     id is name or number
//     start is the project time for the start of a region (beats or seconds)
//     dur is the duration for a region (beats or seconds)
//     returns a list of events with time >= start and time < start + dur
//         each event in the returned list has one of two formats:
//         (NOTE key time channel pitch loudness duration 
//          [parameter parameter ...])
//              key is an event identifier, normally MIDI key number (FIXNUM)
//              time is the start time (beats or seconds) (FLONUM)
//              channel is the (usually MIDI) channel number (FIXNUM)
//              pitch is the exact pitch in (fractional) half-steps (FLONUM)
//              loudness is in MIDI velocity units (FLONUM)
//              duration is event duration in beats or seconds (FLONUM)
//              each parameter (optional) is a keyword followed by a value. 
//                  The keyword prepends a colon (":") to the Allegro attribute
//                  name and retains the type code suffix:
//                  s[tring], r[eal], i[nteger], or l[ogical].
//              Example: (NOTE 60 4.0 3 100.0 1.0 :colors "blue")
//         (UPDATE key time channel parameter)
//              key is an event identifier, normally MIDI key number (FIXNUM)
//              time is the start time (beats or seconds) (FLONUM)
//              channel is the (usually MIDI) channel number (FIXNUM)
//              parameter is an attribute keyword followed by value as in NOTE.
// 
// (AUD-PUT-NOTES id notes merge)
//     id is name or number
//     notes is an event list in the format returned by AUD-GET-NOTES
//     if merge is NULL, replace all events in the track. Otherwise, merge.
// 
// (AUD-GET-TIMES id start dur)
//     id is name or number
//     start is project time for start of region (FLONUM)
//     dur is duration in seconds (FLONUM)
//     returns list of breakpoints: ((x y) (x y) ...)
//         such that label x >= start and x < start + dur. 
//
// (AUD-PUT-TIMES id breakpoints) -- add/replace labels to track
//     id is name or number
//     breakpoints is a list in the format returned by AUD-GET-TIMES
//     The entire track content is always replaced by the new breakpoints.
//

[werame]

Ok, a functional interface for all of the aud-get-info equivalents (audio, lables, etc.) is indeed a big job! Thanks for the link to the patch.

I think a bigger showstopper there is how useable that would be given that Audacity tries to impose a strict firewall between ;type tool (="Nyquist macro") and Nyquist fx (;type process) plugins. The firewall exists probably for a good reason, namely that changing audio directly from Nyquist while also issuing Audacity commands would make it hard to keep a consistent state between Audacity and Nyquist. It's worth noting in that regard that for fx plugins, the C++ side meaning Nyquist.cpp sets up its receiving buffer before calling nyx_get_audio. The pure C part in nyx.c has no direct control over where the fx plugin audio ends up in Audacity's C++ side. (To be more accurate, the C++ side, upon seeing that Nyquist returned a sound object, sets up its StaticPutCallback before pulling the actual audio samples from Nyquist. That's how it can accommodate Nyquist returning more less audio than the *track* contained. And the "copy on write" I guessed earlier is just a mCurTrack[i]->ClearAndPaste(mT0, mT1, out ... where out came from Nyquist but mT0 and mT1 are decided only by the C++ side. By the way, the C++ side is quite "paranoid" about a Nyquist fx changing the input audio data, i.e. what Nyquist sees as *track*. It makes a copy of those samples before passing a pointer to them to Nyquist, even though the StaticPutCallback doesn't go the same buffer, as I understand that issue. Probably the reason why it's done like that is that if Nyquist had a bug that corrupted its input data, then if Audacity passed its original data buffer instead of a copy, then if the user hit Undo in Audacity, they would get the data that Nyquist had corrupted, rather than be guaranteed "clean undo" functionality. Even for big corpos this can be quite a challenge, e.g. Premiere doesn't bother with undos for CEP-scripted commands.)

I'm just trying to do a baby step here for getting just the closest function-based equivalent of *track* with its current time semantics (meaning whatever the C++ side set up in nyx_set_input_audio). So, indeed, this current-selection based functional equivalent should be named something like aud-get-selected-audio from an absolute track-time perspective. Even that is bit of a misnomer since when calling fx plugins Audacity will iterate over all selected tracks and call the fx plug-in repeatedly with the selection for each, something I also had to deal with when trying to make my plugins work for "split stereo" tracks.

As for relative vs. absolute time in a track, I did realize previously there's bit of an issue with what (get '*track* 'clips) returns vs *track*'s perspective, e.g. when trying to use clips in an fx plugin, because the aud-get-info clip results are relative to the Audacity track, not relative to *track*. My quick solution to that was to just force the user to select the whole track. I haven't made public any of those plugins yet, but it was while dabbling with those that I needed to e.g. just return to Audacity only a subset of the clips as a seq, which led me to this current issue of *track* being nil in seq closures.

[werame]

@rbdannenberg , by the way, I haven't given up on implementing this (in the functional form), but I thought I'd get my feet we first with something seemingly simpler: removing the
fairly annoying (to me) limitation on what Audacity allows Nyquist to return, e.g.
why can't a Nyquist plug-in (other than tool) insert tracks (except label ones) as part of
the way its return result is processed in Nuquist.cpp? But even that turned out more difficult than I thought it was going to be...

Another issue is that the Audacity undo/rollback model is pretty unclear to me at the moment.
How much actually needs to be kept "on the side" in terms of WaveTracks vs what SQLite3 transactions handle themselves. I suspect that some of the code that does manual rollbacks e.g. of track insertions after plug-in run failure might be unnecessary since SQLite3 stuff was added, but I'm not sure.

[rbdannenberg]

Thanks for all the updates. IMO, this is a design that calls out for a complete redo from the ground up. I think there could be an understandable model of state changes, undo, commits, Audacity-to-Nyquist communication that's both simpler and more powerful than what we have. When I worked on the "NyquistAPI" above, I decided I'd have to abandon existing effects and generator commands and make yet another command. One critical way to look at this is that Nyquist effects were intentionally narrow and limiting in terms of what you could do with Nyquist, but it was never explicitly spelled out what you were expected to do (and not do) with Nyquist. On the Nyquist side, people creatively violated the original intention left and right, so while the original interface applied pressure to limit functionality, the opportunities in Nyquist applied pressure to open up the functionality. In some sense, Nyquist "won" the first round, with various hacks being made to expand what can be implemented in Nyquist. In the long run, though, we're still dealing with the original limited interface, which will cause more and more grief as we continue to hit limitations. But "getting everything right", whatever that means, is a really big job, so I completely understand taking smaller steps.

[werame]

@rbdannenberg I got the other stuff working to a good extent. The good news is that the SQLite transactions do their magic. If a Nyquist plugin run inserts multiple tracks in Audacity (which it can now), they all go away with one undo (Ctrl+Z). And I didn't really have to write any special code for that undo to happen. This is actually is quite useful towards what you want Nyquist to be able to do, i.e. control more of Audacity. Because it can be done in a way that doesn't break the undo system and it's also not overbearing to code.

Somewhat off-topic: while I was doing the other stuff I realized one could have "visual matlab" mode for Audacity now. What I mean is that a simple checkbox at the Nyquist prompt could send the fx result to a new track, instead of replacing the existing selection. Right now my code does that "bouncing" only when there's overflow in terms of the number of channels returned, but conceivably it could be helpful sometimes to do that deliberately. The alternative right now is to duplicate a track before applying an fx to it, which requires slightly more command overhead. Of course, having only one track available to process as "type process" is limited to right now is fairly limiting if one wants to the "visual matlab" idea seriously, meaning one should be able to name (several) tracks as variables for expressions with more than one input. Someone actually pointed out to me in an off-line conversation that GoldWave has this feature in their "expression evaluator", i.e. one can name any track or a combination thereof. Looking now at the online docs for GW that seems to be true, although I haven't tested it my self in practice.