Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Declarative DSL #2

Closed
nwmcsween opened this issue Jan 22, 2016 · 49 comments
Closed

Declarative DSL #2

nwmcsween opened this issue Jan 22, 2016 · 49 comments
Labels

Comments

@nwmcsween
Copy link

Consider a DSL that is similar to make with explicitly set defaults to map to fqdn's (via import/whatever).
This way targets could be handled by an external program of some sort that 'schedules' hosts (e.g host with least latency to $someplace start http server).

@josephholsten
Copy link

I've been thinking about cfg mgmt in golang, and this DSL is where puppet & chef really shine.

One of their most useful accidents (not present in ansible, terraform) is the ability to create resource-providers entirely in the DSL out of other resources. I'm of the opinion that this lack is causing a lot of reuse pain in the Ansible Galaxy.

I'd love to help spike a plugin/DSL syntax using lua like the https://github.com/mozilla-services/heka project is doing.

@purpleidea
Copy link
Owner

On Mon, Feb 1, 2016 at 1:49 AM, Joseph Anthony Pasquale Holsten <
notifications@github.com> wrote:

I've been thinking about cfg mgmt in golang, and this DSL is where puppet
& chef really shine.

One of their most useful accidents (not present in ansible, terraform) is
the ability to create resource-providers entirely in the DSL out of other
resources. I'm of the opinion that this lack is causing a lot of reuse pain
in the Ansible Galaxy.

Can you be a bit more specific about what you mean here? Are you talking
about create_resources() ?

I'd love to help spike a plugin/DSL syntax using lua like the
https://github.com/mozilla-services/heka project is doing.

Great, I'd love to have you involved. I've actually just started looking
at lua as a possibility because of it's ease of embed-ability. Nothing
certain yet, but a few links that were discussed the other day include:

https://github.com/Shopify/go-lua
https://github.com/yuin/gopher-lua

and examples:
https://github.com/kabukky/journey
https://github.com/xyproto/algernon

More discussion to come!

@nwmcsween
Copy link
Author

Puppet and chef both have bad DSL's in my opinion. A less constricting way to do this is write a DSL that allows you to shell out (such as make) and just focuses on the 'resolving' graphs, this has many benefits.

@purpleidea
Copy link
Owner

On Mon, Feb 1, 2016 at 3:17 AM, Nathan notifications@github.com wrote:

Puppet and chef both have bad DSL's in my opinion. A less constricting way
to do this is write a DSL that allows you to shell out (such as make) and
just focuses on the 'resolving' graphs, this has many benefits.

Could you elaborate on this comment please? I really agree about the DSL
part, which I do think needs improving, but your comments about "resolving
graphs" isn't clear.

Thanks!

@dalen
Copy link

dalen commented Feb 1, 2016

One of their most useful accidents (not present in ansible, terraform) is the ability to create resource-providers entirely in the DSL out of other resources. I'm of the opinion that this lack is causing a lot of reuse pain in the Ansible Galaxy.

Can you be a bit more specific about what you mean here? Are you talking about create_resources() ?

I think this was in reference to the define keyword in Puppet which lets you create a new resource type without actually writing a type & provider.

@purpleidea
Copy link
Owner

On Mon, Feb 1, 2016 at 8:47 AM, Erik Dalén notifications@github.com wrote:

I think this was in reference to the define keyword in Puppet which lets
you create a new resource type without actually writing a type & provider.

If that is the correct interpretation, then it is absolutely a design
requirement for the DSL. The idea of composed "resources" is fundamental,
and while the user should be able to do it in modules, the only thing I
haven't completely figured out is how much of the useful secondary
resources and stdlib I will write in native DSL vs. golang. Eg, if I have a
"hosts" (/etc/hosts) resource, should it be native golang or a combination
of the inotify file type and a template, etc... That conversation will have
to be postponed till we have a DSL.

@purpleidea
Copy link
Owner

Also, I realized that I have mentioned this in discussion, but maybe it's not clear in the blog post, but the current "yaml" definition files are merely placeholders for testing the engine before there is a proper DSL. It is absolutely temporary, and I can't wait to remove all of that parsing code.

@josephholsten
Copy link

the define keyword in Puppet which lets you create a new resource type without actually writing a type & provider.

Yep, this.

I can't wait to remove all of that parsing code.

<3

@josephholsten
Copy link

TL;DR: external process resource-providers require a trade-off between implementation complexity and smart dependency things

@nwmcsween I'd love to do everything by having external tools make the changes. But most unix tools aren't designed to act idempotently. More so, a single tool often doen't do everything required. For example, a mere mkdir or cat will not create files with a specified owner, group or mode. So mere make-style file existence isn't enough to actually verify a file resource has been created correctly. At which point, our dependency resolver needs to know how to check if a thing is in the desired state. Which brings us back to needing resource implementation code.

Ansible's external-plugin model gives the separation between the execution-coordinator and the resource-provider implementation. But it runs each resource a single time, which internally does a separate check-phase and apply-phase. Compare this to puppet, &c which do a complete check-phase in all resources before doing an apply-phase. I don't have a concrete example right now, but this causes the dependency system in ansible to be very weak.

Terraform's external plugin model uses full-duplex communication between the coordinator and the resource provider. This allows a complete check-phase across all resources separate from the apply-phase. It also means the provider doesn't need to be run a separate check-process from the apply-process, even lets you have a single process for all resources using the same provider. The pain here is that you don't get to write plugins as standard unix tools naively reading from ARGV & STDIN, writing to STDOUT.

And I don't even want to think about parallel execution implications for resource providers. But that sounds especially unpleasant with ansible-style plugins.

@nwmcsween
Copy link
Author

This is exactly the same as make, the onus is on the developer to make sure
there are no hidden dependencies

On Tue, Feb 2, 2016, 12:08 PM Joseph Anthony Pasquale Holsten <
notifications@github.com> wrote:

TL;DR: external process resource-providers require a trade-off between
implementation complexity and smart dependency things

@nwmcsween https://github.com/nwmcsween I'd love to do everything by
having external tools make the changes. But most unix tools aren't designed
to act idempotently. More so, a single tool often doen't do everything
required. For example, a mere mkdir or cat will not create files with a
specified owner, group or mode. So mere make-style file existence isn't
enough to actually verify a file resource has been created correctly. At
which point, our dependency resolver needs to know how to check if a thing
is in the desired state. Which brings us back to needing resource
implementation code.

Ansible's external-plugin model gives the separation between the
execution-coordinator and the resource-provider implementation. But it runs
each resource a single time, which internally does a separate check-phase
and apply-phase. Compare this to puppet, &c which do a complete check-phase
in all resources before doing an apply-phase. I don't have a concrete
example right now, but this causes the dependency system in ansible to be
very weak.

Terraform's external plugin model uses full-duplex communication between
the coordinator and the resource provider. This allows a complete
check-phase across all resources separate from the apply-phase. It also
means the provider doesn't need to be run a separate check-process from the
apply-process, even lets you have a single process for all resources using
the same provider. The pain here is that you don't get to write plugins as
standard unix tools naively reading from ARGV & STDIN, writing to STDOUT.

And I don't even want to think about parallel execution implications for
resource providers. But that sounds especially unpleasant with
ansible-style plugins.


Reply to this email directly or view it on GitHub
#2 (comment).

@cavaliercoder
Copy link

I'm a long time user of Puppet DSL and I've come to really hate the constraints it imposes. It is improving, but everything has to be reengineered (we finally have 'for' loops!). I prefer the idea of leveraging an existing language and providing an interface for declaring resources. Basically, like Chef does with Ruby.

Consider the proof that most Puppet modules implement Ruby code to augment the shortcomings of the DSL anyway.

There's so much to start building in terms of language internals, but also writing a parser, testing framework, etc. Would it be worth it?

I'd like to see a design where the user is writing Go/other code to declare resources, dependencies, etc. while taking advantage of Go's conditionals, operators, testing framework, quality tools, etc. But using an API interface which eases/enforces idempotency, extensibility, etc. Admittedly it creates opportunity for users to start breaking things with an unconstrained API... You'd also have to include the Go compiler in your toolset. Go is also not the most declarative DSL like language...

Method chaining/LINQy stuff would be ideal IMHO. Ignoring broken syntax, something like:

AddResource(YumRepo{
  Name: "centos-base",
  URL: "http://...",
})

AddResource(Package{
  Name: "vim-enhanced",
  Version: ">=7.3",
}).DependsOn(Resources.Where({
  Type: "YumRepo",
}))

That's my thoughts though :)

@purpleidea
Copy link
Owner

@cavaliercoder Good comments, here are some thoughts to help continue the discussion...

  1. No matter what decision we end up with, someone will be unhappy. Since this is the case, I've instead tried to focus on what the correct thing is, and go from there.

  2. There is a chance, that there will eventually be some sort of golang and/or c compatible API that you'll be able to use to build the graphs through the API however you want. This might facilitate doing the sort of thing you mentioned above. Doing this in golang will probably be easiest, since the API will be written in golang. There will be a discussion about if this will be something that just splits the community and/or if it's something we want to encourage or not.

  3. Being declarative is very important, in particular because we want to enable the possibility of being able to write analysis tools that would integrate and figure out or detect certain conditions in the multi machine space. Eg: warning, this might cause an infinite loop across N hosts sort of thing. Do you think this sort of analysis would be possible if we used a functional, non-declarative language?

  4. Lastly, if you do want to reuse an existing language (and I agree, I would love to do this instead of having to create one) then we need a suitable candidate! I've casually added Lua to the NACK list and obviously the imperative languages like python/ruby/etc are on there too. I've been trying to learn more about functional reactive programming. I don't suppose anyone knows of a declarative functional programming language or wants to build one. In any case, do you have some suggestions of what language we could use?

Cheers!

@vincentbernat
Copy link

LISP could be a candidate. It's easy to implement, it's easy to parse, it's supported out of the box by any editor. It's easy to make it look like a custom DSL while still being extensible. It's easy to keep it functional. The only downside is that some people don't like parentheses position.

For example, if I take on of your example:

(resources
  (exec
    ((name "exec1")
     (cmd "echo hello from exec1")
     (timeout 0)
     (watchcmd "sleep 10s")
     (state 'present))
    ((name "exec2")
     (cmd "echo hello from exec2")
     (watchcmd "sleep 10s")
     (state 'present))))
(edges
  ((name "e1")
   from (exec "exec1")
   to (exec "exec2")))

Some parentheses could be removed for example:

(exec (("exec1"
            timeout 0
            state 'present)))

So, it looks like the YAML. No need to learn anything. If you want to use a loop, it's easy (but you have to learn the backquote):

(resources
 (exec
   `(loop for i upto 10 collect
     (list (concat "exec" (str i))
            cmd (concat "echo hello " (str i))
            timeout 0
            state 'present))))

If you a user want to introduce a new type of resource, they can write a function. If they want to extend the DSL, they can write a macro.

@purpleidea
Copy link
Owner

purpleidea commented Jul 7, 2016

@vincentbernat Thanks for your input!

One probable requirement of the DSL will include the ability to make it reactive, and also to have the compiler/lexer written in golang. Do you know if either of these already exist in lisp?

Having said that, I did consider lisp, but I'm not sure if it would be a good choice for mainstream users. Readability is an important property. Having said that, there's nothing to stop someone from writing a lisp frontend.

@vincentbernat
Copy link

For a more readable variant, you can look at Clojure syntax which have lists [] and maps {}. Not using parentheses everywhere make it more readable to some users. Note that the main advantage of LISP is that you get both the programming language stuff and the DSL feel. It's difficult to have that in non-homocoinicitic languages.

What you don't want is to transform YAML into a programming language (lke Ansible).

For Go, unfortunately, I know there are many LISP implemented in Go but I have no real experience to point to a good one. What's great with LISP is that you can take a minimal implementation and works from here with macros.

@purpleidea
Copy link
Owner

On Thu, Jul 7, 2016 at 6:05 AM, Vincent Bernat notifications@github.com
wrote:

For a more readable variant, you can look at Clojure syntax which have
lists [] and maps {}. Not using parentheses everywhere make it more
readable to some users. Note that the main advantage of LISP is that you
get both the programming language stuff and the DSL feel. It's difficult to
have that in non-homocoinicitic languages.

Same uncertainty on how reactive aspects would fit in.

What you don't want is to transform YAML into a programming language (lke
Ansible).

Agreed!

For Go, unfortunately, I know there are many LISP implemented in Go but I
have no real experience to point to a good one. What's great with LISP is
that you can take a minimal implementation and works from here with macros.

Thanks again for the idea! If you'd like to get more involved with patch
writing but need some help getting going and learning about some internals,
LMK :)

@vincentbernat
Copy link

Unfortunately, I don't have time at the moment. Writing random comments in GitHub issue is easier than proposing patches. :)

BTW, as an example of LISP as DSL, there is Riemann (http://riemann.io/, but it's written in Clojure, so, more natural choice).

@nwmcsween
Copy link
Author

A full blown language would kill the idea of build system like DAG, it
would also bloat and complicate it immensely. My opinion is if you want
extras you simply shell out.

On Thu, Jul 7, 2016, 5:21 AM Vincent Bernat notifications@github.com
wrote:

Unfortunately, I don't have time at the moment. Writing random comments in
GitHub issue is easier than proposing patches. :)

BTW, as an example of LISP as DSL, there is Riemann (http://riemann.io/,
but it's written in Clojure, so, more natural choice).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAmBbwr1R6cZSB8WaJpeqwKZB8w7rFC2ks5qTO9SgaJpZM4HKge_
.

@dnaeon
Copy link

dnaeon commented Jul 12, 2016

Hi,

Just found your project, looks nice! :)

I'm also working on a similar project, where I've just recently implemented a DSL based on Lua. You might want to look at it in case you decide to go with Lua.

The project I'm working on is located at https://github.com/dnaeon/gru

@purpleidea
Copy link
Owner

@dnaeon Funny enough someone send me a pointer to your project just the other day, although I haven't had a chance to dig into the code yet, sorry.

I did look into using Lua, but for some design reasons unfortunately I don't think it would be a good choice, mainly because of a lack of safety constraints available in the language that we would require. The fact that you mention using it as a DSL, makes me think it's possibly a safe subset of the language that you're using?

In any case, I'd love to chat about it more, and maybe you want to join forces and help out in mgmt? Can I persuade you to have a look at my intro article or video?

https://ttboj.wordpress.com/2016/01/18/next-generation-configuration-mgmt/
https://www.youtube.com/watch?v=GVhpPF0j-iE&html5=1

I'm hanging out in the #mgmtconfig IRC channel on Freenode which you're more than welcome to join.

Cheers and thanks for the message,
James
@purpleidea

@dnaeon
Copy link

dnaeon commented Jul 12, 2016

Hey James,

I'll check out the video and article, thanks for the links!

This week I'm traveling most of the time, but once I get back from my vacation I'll ping you in IRC.

Best regards,
Marin

@purpleidea
Copy link
Owner

@dnaeon Sounds good. - Enjoy your vacation!

Cheers,
James

@neilhwatson
Copy link

Jumping in here and I hope I get the context right. If you're going to invent a DSL for mgmt please consider publishing it as an open standard. There are so many OS config management tools and each has its own language, leading to lock-in. An open standard for future tools to use would be a great thing.

@purpleidea
Copy link
Owner

@neilhwatson I personally have no objections to this, there's nothing proprietary about what we're building. The first step though is to get the code written so we have a minimally viable project. Can you help with some patches ?

@shawncatz
Copy link

Hashicorp Config Language?
I'm guessing it's been considered, but I didn't see it in this conversation (some did mention terraform, but not HCL)... Already implemented in go. Becoming well known. Much of the graph/dependency resolution might be borrowed from existing tools like Terraform.

I'm really interested in being involved in next-gen config management. As much as I love Chef, and others, they are all pretty terrible ecosystems.

@josephholsten
Copy link

@shawncatz I'd be interested in what you find lacking in the chef or puppet ecosystems. I've noticed a much higher amount of modular code and code reuse in those ecosystems than in ansible, terraform and salt.

I think one of the reasons is first class resource types. It requires a complete redesign and rewrite to convert an implementation from ansible task into ansible module, or terraform module into terraform provider. This makes it much harder to get a broad community of resource implementations, and makes it more expensive to prototype new approaches, and creates a hierarchy of contributors: "real" resource devs vs mere users.

I say this as a contributor to HCL and a committer on terraform. I <3 it, but its architecture has serious drawbacks for long term growth.

@purpleidea
Copy link
Owner

@shawncatz How can we help you get more involved? (Feel free to discuss this on IRC with us in #mgmtconfig on Freenode)

@josephholsten and @shawncatz I looked briefly at HCL, and unfortunately I think it lacks some of the features we need, since it's much closer to a yaml than a full DSL. Thanks for the idea though!

@josephholsten
Copy link

If we were looking for a fancy JSON/YAML/TOML alternative, https://github.com/vstakhov/libucl was the inspiration for HCL, and much nicer IM(NS)HO.

I'm mostly in favor of Lua as a golang encore nation language, but if we want a LISP, guile is quite nice.

@purpleidea
Copy link
Owner

As was suggested, I figured I'd like to the HCL lib here: https://github.com/hashicorp/hcl
I had a quick look and it might turn out to be a reasonable base to modify for use in mgmt. hackers wanted.

@dnaeon
Copy link

dnaeon commented Oct 11, 2016

Hey @purpleidea,

Just a thought about HCL - I've used HCL in the previous releases of Gru, and overall was pretty happy with it.

The syntax is fine and configuration expressed in HCL looks nice.

At some point you would want to introduce variable interpolations, add custom functions in the HCL configuration, etc. and would most likely end up with HIL as I did already.

At the end I gave up on HCL and HIL, mainly because of the reasons listed in this issue.

With that said, I'm not saying that HCL/HIL are bad, they just didn't cut it for me and for what I wanted to achieve.

Anyways, thought I'd share this with you, in case you decice to go that way.

P.S. Seems we can't get a hold on IRC because of the time difference :) I'll be joining in there, maybe I'll get a hold of you next time :)

purpleidea pushed a commit that referenced this issue Jun 29, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jun 29, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jun 29, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jun 30, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jul 3, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jul 15, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jul 19, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jul 27, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Jul 30, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 8, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 9, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 9, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 13, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 20, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 21, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 22, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 22, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 27, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 28, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 28, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 28, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 29, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 30, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Aug 31, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Sep 4, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Sep 4, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Sep 19, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Sep 19, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Sep 19, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
purpleidea pushed a commit that referenced this issue Sep 19, 2023
This is the 1st commit message:

lang: Split FuncValue into FuncValue and SimpleFn

Representing an MCL function value as a golang function from Value to
Value was a mistake, it should be a function from Vertex to Vertex.

Here is why this is a mistake:

    The output of a function like

        $f = fn(x) {
          Shell(Sprintf("seq %d", x))
        }

    varies over time, while a single Value does not. Thus, code like

        Map($f, list(1, 2))

    would first produce the value list("1", "1"), but then it would
    _not_ update to list("1", "2") when "seq 2" produces its second
    line. That's because with the mistaken design, when Map receives a
    new FuncValue or a new input list of N elements, Map calls the
    function from Value to Value N times and produces a single output
    list of N elements.

Here is why the corrected design is better:

    Here's what happens with this new design when Map receives a new
    FuncValue or a new input list of N elements.

    First, Map constructs N item-input nodes, each of which extracts a
    different entry from the list. Then, Map calls the function from
    Vertex to Vertex N times, once for each item-input node, and thus
    obtain N item-output nodes. Finally, Map constructs an
    item-collecting node which constructs a list out of all of its
    inputs, and Map connects the N item-output nodes to the
    item-collecting node. This item-collecting node is the output of
    Map.

    The Vertex to Vertex function constructs and connects its own nodes;
    in this case, it constructs an Sprintf node and connects the
    item-input node to it, and then constructs a Shell node and connects
    the Sprintf node to it, and then returns the Shell node as the
    item-output node.

    The two Shell node in this sub-graph emit a first value "1", which
    propagates to the item-collecting node and causes it to output a
    first value list("1", "1"). Then, the second Shell node emits a
    second value "2", which propagates to the item-collecting node and
    causes it to output a second value list("1", "2"), as desired.

Here is how this commit brings us closer to the above plan:

    Changing FuncValue throughout the codebase is a big change. One of
    the difficulties is that it is not just nodes which are emitting
    FuncValues, there are also many other places in the code where
    FuncValue is used to hold a golang function from Value to Value.
    Some of those places now need to hold a golang function from Vertex
    to Vertex, but other places still need to hold a golang function
    from Vertex to Vertex.

    Thus, as a first step, we need to split FuncValue into two types.
    This commit splits the old FuncValue into two types:

    1. The new FuncValue will hold a function from Vertex to Vertex.
       FuncValue is a Value.
    2. A new type named "SimpleFn" will hold a function from Value to
       Value. SimpleFn is not a Value.

    This commit replaces occurrences of the old FuncValue with one of
    those two new types, as appropriate. This commit does not yet adapt
    the surrounding code to make use of the new representation; that
    will be done in future commits. I have annotated the missing parts
    with the following panic message in order to make it easy to find
    which parts still need to be implemented. The "..." part explains
    what needs to be implemented.

        panic("TODO [SimpleFn]: ...");

Here's where I need help:

    One part of the code which is not clear to me are the parts which
    use reflection. I don't understand the purpose of that code well
    enough to explain what needs to be implemented. I have annotated
    those "known unknown" parts of the remaining work with the following
    panic message in order to make it easy to find which parts still
    need more thinking and planning:

        panic("TODO [SimpleFn] [Reflect]: ...");

This is the commit message #2:

lang: Add the core func graph Txn API

This will eventually let functions change the running graph via a
transaction API.

At the moment the core Lock and Unlock primitives aren't implemented.

This is the commit message #3:

lang: A reversible wrapper around Txn

This is useful for the common case in which we call one FuncValue to
construct a bunch of nodes, and later we switch to a different FuncValue
and so we want to remove all the nodes added by the first FuncValue and
replace them by the nodes added by the second FuncValue.

This is the commit message #4:

lang: move FuncValue to its own package

This is the commit message #5:

lang: combine lang/func/structs and ast

Merging those two packages allows us to avoid import cycles when a
Func needs to add an Expr to the graph.

This is the commit message #6:

lang: CallExpr must generate a subgraph

FuncValues are now manipulating the graph instead of manipulating
values, so the logic for calling a FuncValue must now follow suit.

This is the commit message #7:

lang: ExprFunc does not need to store its value

This is the commit message #8:

spelling

This is the commit message #9:

convert between SimpleFn, FuncValue, and Func

This is the commit message #10:

no need for ExprCall.argVertices

it's the exact same thing as ExprCall.Args

This is the commit message #11:

ExprFunc.Func()'s three cases

This is the commit message #12:

FunctionFunc is no longer used

This is the commit message #13:

ExprBool.MergedGraph()

This is the commit message #14:

ExprIf.MergedGraph()

This is the commit message #15:

ExprFunc.MergedGraph()

This is the commit message #16:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Expr interface. The txn type is
interface{} until we have that merged.

This is the commit message #17:

lang: ast, interfaces: Add MergedGraph signature and implementation

This adds a MergedGraph signature to the Stmt interface. The txn type is
interface{} until we have that merged.

This is the commit message #18:

Use MergedGraph signature and implementation

This puts it into play, but doesn't initialize the input args at all.

This is the commit message #19:

Restore ExprBool.Graph() and Func()

This reverts commit 3ea3845.

This is the commit message #20:

Restore ExprIf.Graph()

This reverts commit a62889e.

This is the commit message #21:

restore ExprFunc::Graph() and Func()

This is the commit message #22:

fix type errors

This is the commit message #23:

ExprCall.MergedGraph()

This is the commit message #24:

a more precise type for args

This is the commit message #25:

GraphTxn

This is the commit message #26:

ReversibleTxn.AddGraph

This is the commit message #27:

ExprFunc.MergedGraph()

This is the commit message #28:

ExprVar.MergedGraph()

This is the commit message #29:

sub-graph is spelled subgraph in this codebase

This is the commit message #30:

ExprFunc.mkFunc() was unused

This is the commit message #31:

CallFunc.Stream() should add Funcs to the graph, not Exprs

This is the commit message #32:

move ConstFunc to lang.funcs

This is the commit message #33:

move conversion functions to lang.funcs.simple

I wrote some conversion functions from SimpleFn to FuncValue to Func,
and I want to call them from Func.Stream() implementations, so those
conversion functions should be somewhere in lang.funcs.

This is the commit message #34:

it is the responsibility of the function engine to call Init on the nodes

This is the commit message #35:

FuncValue.Call()

This is the commit message #36:

drop MergedGraph's unused txn parameter

This is the commit message #37:

move CallFunc to funcs.simple

This is the commit message #38:

Func from channel

This is the commit message #39:

ChannelBased{Source,Sink}Func

This is the commit message #40:

MapFunc

This is the commit message #41:

remove unused CallFunc field

This is the commit message #42:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #43:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #44:

extend environment with StmtProg's local variables

This is the commit message #45:

delete unused VarFunc

This is the commit message #46:

appease govet

This is the commit message #47:

add imported variables to the environment

This is the commit message #48:

add builtins to the environment

This is the commit message #49:

accumulate all the imported variables

previously, we were only keeping the imported variables from the last
import statement.

This is the commit message #50:

XXX: wip new function engine

This is the commit message #51:

comments

This is the commit message #52:

remove Engine.{Lock,Unlock} placeholders

This is the commit message #53:

f(...) support for ExprCall

This is the commit message #54:

[REVERT ME] remove unrelated files to make VS Code happy

This is the commit message #55:

don't recreate the subgraphInput

This is the commit message #56:

CallFunc now takes a single input, the FuncValue

Also, by using the map_func logic, CallFunc now detects when it can no
longer emits new values downstream.

This is the commit message #57:

GraphTxn must take a pointer in order to modify the graph

This is the commit message #58:

GraphTxn is unused
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests