Add variable interpolation to mini notation

[yaxu]

A continuation of discussion on this PR by @bgold-cosmos : #528

[yaxu]

I thought Template Haskell could help with this.

Reading this (in particular "Writing ${alphaNum} interpolates the regex referred to by alphaNum into the larger regex validDotComMail'"):
https://wiki.haskell.org/A_practical_Template_Haskell_Tutorial

I did this:
643a977#diff-409e74a1fde3d6f274c8aabb2c425d70

However the interpolation seems to be ignored:

:set -XTemplateHaskell -XQuasiQuotes
import Sound.Tidal.TH

x = [bp|a b|] :: Pattern Int

[bp|1 2 ${x}|] :: Pattern Int

results in:

(0>½)|1
(½>1)|2

[bgold-cosmos]

Every time I've tried Template Haskell I've been stumped at how to use it properly, but that tutorial looks good so I'll take another look...

[yaxu]

Yes I don't find the documentation too clear! But looking a bit more closely, it seems the interpolation doesn't work until you make a Lift instance:
http://hackage.haskell.org/package/template-haskell-2.14.0.0/docs/Language-Haskell-TH-Syntax.html#t:Lift

I'd say it's not possible to turn a pattern into a q-expression, so this isn't going to work right now.

I think this would be possible if #217 was done.

I should also link in #375.

[bgold-cosmos]

One way to do it is with parser state (#217 may be related). I tried making a "toy" parser for Tidal that supported this, but the difficulty I run into is combining the stateful aspect with the generic type aspect. For example, if you're parsing a Pattern Double then the state (for interpolation) should also be Pattern Double, but combining this with the Euclidean parts (which are always Pattern Int) gets messy. I think maybe the Euclidean stuff needs to be isolated somehow.

[yaxu]

I read that regex tutorial a bit more carefully and managed to get the following proof of concept working on the topic-ast branch:

alex@stanage:~/src/ast/src$ ghci Sound/Tidal/AST.hs
GHCi, version 8.6.3: http://www.haskell.org/ghc/  :? for help
[1 of 9] Compiling Sound.Tidal.Bjorklund ( Sound/Tidal/Bjorklund.hs, interpreted )
[2 of 9] Compiling Sound.Tidal.Pattern ( Sound/Tidal/Pattern.hs, interpreted )
[3 of 9] Compiling Sound.Tidal.Core ( Sound/Tidal/Core.hs, interpreted )
[4 of 9] Compiling Sound.Tidal.Chords ( Sound/Tidal/Chords.hs, interpreted )
[5 of 9] Compiling Sound.Tidal.Utils ( Sound/Tidal/Utils.hs, interpreted )
[6 of 9] Compiling Sound.Tidal.Params ( Sound/Tidal/Params.hs, interpreted )
[7 of 9] Compiling Sound.Tidal.UI   ( Sound/Tidal/UI.hs, interpreted )
[8 of 9] Compiling Sound.Tidal.ParseBP ( Sound/Tidal/ParseBP.hs, interpreted )

Sound/Tidal/ParseBP.hs:68:9: warning: [-Wincomplete-patterns]
    Pattern match(es) are non-exhaustive
    In a case alternative:
        Patterns not matched:
            (TPat_Elongate _)
            (TPat_Var _)
   |
68 | toPat = \case
   |         ^^^^^...
[9 of 9] Compiling Sound.Tidal.AST  ( Sound/Tidal/AST.hs, interpreted )
Ok, 9 modules loaded.
*Sound.Tidal.AST> :set -XTemplateHaskell -XQuasiQuotes -XOverloadedStrings
*Sound.Tidal.AST> let x = fast "2 3" [T.bp2|a b c|]
*Sound.Tidal.AST> let y = [T.bp2|d e ${x} f|]
*Sound.Tidal.AST> T.toPat y
(0>¼)|"d"
(¼>½)|"e"
(½> 13/24)|"a"
( 13/24> 7/12)|"b"
( 7/12>⅝)|"c"
 11/18-(⅝> 23/36)|"b"
( 23/36>⅔)|"c"
(⅔> 25/36)|"a"
( 25/36> 13/18)|"b"
( 13/18>¾)|"c"
(¾>1)|"f"

[yaxu]

Here's the commit:
19ba2e5

It's only working with string patterns so far, I need to understand what's going on more.. But I think this is a really interesting proof of concept.

[dktr0]

Haven't been following this very closely - if this is using TH "on the fly", ie. during interpretation/performance, I would flag that it likely doesn't transfer to MiniTidal/GHCJS/the browser and could thus strongly impede the ability to give people a zero installation Tidal experience. It's worth considering the implications of particular implementation strategies for who ultimately gets to use it, I think.

Apart from that particular (rather huge, IMO) implication, I would observe that - as something which is not type safe and which gives very cryptic feedback - TH is likely most suitable for managing the complexity of large compiled systems rather than for interpretation...

[yaxu]

If that's the case, we simply don't support interpolation in the mini notation in any interpreters without access to template Haskell,.. Potentially offer the same feature a different way (I guess this wouldn't be too tricky in MiniTidal).

Using TH in this way adds type safety at compile time though, and I don't think it gives cryptic feedback - it just returns any parsec errors.

In any case, TH aside, there are wider implications in expanding parseBP's AST to the whole of tidal, which I think will have some really interesting benefits including for systems like minitidal.

[dktr0]

It'd be nice to keep things "together" as much as possible. "Interpolation" (a variable system, basically) is such a significant feature that, once introduced, it could be hard to see a version of Tidal without as really connected to one with. I think you're right that the same feature could be offered a different way - I think my intuition is that there is a non-TH right might make it easier to offer this feature across different software delivery environments.

By the way, have recently reworked MiniTidal using haskell-src-exts by the way, so it is now much easier to extend than previously. In terms of matching "classic Tidal" it is now mostly a matter of filling in missing operators and catching specific patterns of use. More about this elsewhere soon...

That (MiniTidal haskell-src-exts rework) being the case, though, I have been thinking about the possibility of making MiniTidal match not a single ControlPattern but rather a whole Tidal "program" with the d1 d2 hush etc. This is somewhat connected to the original goal ("interpolation") of this issue in the sense that an example like this:

x = "arpy*8"
d1 = s x
d2 = s "$x"

...becomes something that will be possible (without on-the-fly TH) because we will be keeping a dictionary of things that have been defined anyway. I could well be missing something, but I have the sense that TH is being used as a way of getting a dictionary lookup without being responsible for a dictionary, which would beg the question, why not just have a dictionary? Thinking out loud...

set :: String -> Pattern a -> IO () -- defines reusable patterns in MVar dictionaries that are available and statefully updated during parsing

set "x" $ "arpy*8"
set "d1" $ s
set "d2" $ s "$x"

let d1 = set "d1"
(etc)

[yaxu]

Yes, there's already the control state dictionary for this sort of thing. I haven't quite worked out the interface for it but there's a working example in the #tidal-innards channel on talk.lurk.org

As I said, the interesting part isn't really the TH bit but the AST bit of this change. It brings the mini notation together with the rest of tidal, so that

set "x" $ iter 3 $ fast "2 3" "arpy*8"
set "d2" $ s "kick $x"

becomes possible

[dktr0]

Right... but from MiniTidal everything there is already possible! (edit: except the new $x obviously...)

[bgold-cosmos]

@yaxu what happens to the currently playing d2 when x is changed? Or is that part of the interface that hasn't been worked out?

[yaxu]

@bgold-cosmos here's how things actually look at the moment. It works really well..

-- add to BootTidal.hs
let setI = streamSetI tidal
    setF = streamSetF tidal
    setS = streamSetS tidal
    setR = streamSetI tidal
    setB = streamSetB tidal
    metre = cB True "metre"
    setMetre = setB "metre"

-- Use like this
setMetre "t(3,8)"
d1 $ struct metre $ sound "bd"
d2 $ struct (inv metre) $ sound "cp"
setMetre "t(3,8,<0 2 4>)"

This sets state via the same mechanism that external controllers set state, so running patterns will reflect changes.

I think @dktr0 is right that we don't need TH, once everything goes via an AST we can use this state dictionary to embed patterns in each other.

[dktr0]

You can use dictionaries to embed patterns in each other now - doesn't have to wait for an AST refactor.

[yaxu]

I got something working using the state dictionary: e4034b7

setS "foo" (iter 4 "bd sn [cp clap:4] bd*8")

d1 $ sound "bd*2 clap:3 ^foo"

setS "foo" "bd sn"

Currently variables are prefixed with ^.

I'm resisting using the $ a little bit, as Tidal learners have enough trouble with $ without it meaning different things in different places. Thoughts?

Otherwise, we're running out of keyboard characters fast..

No template haskell required.. and we already have an OSC interface for updating this state remotely, for free. (Hopefully that won't be used for griefing live performers..)

[bgold-cosmos]

Oh, that's an interesting way of doing it, I'll have some fun testing this out. Does this also mean that using the MIDI-OSC bridge, whatever's sent on CC# 11 could be used in the mininotation like this?

d1 $ n "c e g ^11" # s "superzow"

[yaxu]

In theory yes, in practice that might need a tweak to the parser, as I think it won't like identifiers that begin with a number.

[yaxu]

I think I like the ^.. looks like an arrow, indicating it's from outside the pattern..

[yaxu]

@bgold-cosmos it supports numeric ids now so midi input should be fine