cue: Enable builtins to be registered by users

[cueckoo]

Originally opened by @verdverm in cuelang/cue#350

As a user, I would like to register my own builtins to be used in Cue.

• Expose a RegisterBuiltin function which allows adding builtins to the predefined map
• add the builtins to the runtime (or something else)? This would allow users to have multiple runtimes and different builtins per.
• Enable new stdlib functions to be developed, tested, and used prior to modifying or extending the actual list of builtins
• Collect and document how to decide and register as pure vs tool. Alert to potential issues with non-pure builtins in pure cue, can shoot yourself in the foot.

[cueckoo]

Original reply by @shykes in cuelang/cue#350 (comment)

Doesn't this create the risk of fragmenting the language? If I load a file with the extension .cue, I have certain expectations of available builtins and their behavior. Even more so if we allow builtins to not be pure (no real way to avoid it without extreme sandboxing which is another can of worms).

If custom builtins become widely adopted, it kind of changes the role of Cue from a language to a family of similar but incompatible languages. This in turn could hurt adoption of Cue, and the benefits of using it.

What are your thoughts on this?

[cueckoo]

Original reply by @verdverm in cuelang/cue#350 (comment)

Super good question. @mpvl, would love to hear your thoughts

There are definitely a few that people have been asking for. I have a strong desire for the main functions in structural. I'm about to implement them in Go, along the lines of the update / mutation prototype we got working in slack today.

#348 is the other half of this, to have a place to talk about what builtins Cue should have. There are also the annotations for tooling to work with, and a lot should be possible there. Based on your perspective @shykes, what @mpvl thinks, this PR could go to the chopping block pretty quick.

Another thing that came out of the prototype is the difficulty several of us experienced constructing or mutation Cue values in our Golang based tools. If we improve that, so should the experience around building tools around Cue. At the same time, many of us would like to push as much into Cue as we can. It does seem more tedious to switch back and forth, but then again, maybe we can fix that in the Go API area.

---

Yet another line of thought we can add to the mix is how DSLs mix into this. When we write tools that expect certain values and schemas, we are essentially writing in DSLs. There's a lot more to say about this, but I'm pretty tired and just want to capture this thought when I had it. I'm planning to use attributes in custom tooling in conjunction with these DSL ideas.

@mpvl has some ideas: Cue & DSLs ;]

[cueckoo]

Original reply by @ymolists in cuelang/cue#350 (comment)

@shykes I think its safe to expect that builtins (specially the ones that are meant to not be pure) should only come from plugins that live outside of the main cue source/binary. For example kubectl has an extension framework where people can install plugin binaries (at their own risk of course) that work with the kubectl binary.

To me the real question is:

1. can we define an easier API that make life easier for people who want to use cue as a library instead of a binary ?
2. can we define a plugin api that lives outside of the main binary but people can install plugins to. So the main binary can communicate via htpp or even running the actual external binary as kubectl does today.

Both are i think valid means of extending cue without breaking the main expectations of people who want to use cue as it is now.

[cueckoo]

Original reply by @verdverm in cuelang/cue#350 (comment)

@ymolists if a project adds a new built-in, then there is an unknown token as seen by the CUE implementation. How would it infer that the missing token is a built-in vs other unknown token, like a dev forgot to write something. This would mean that one could not run the CUE cli against this tools "CUE" code.

What do you think CUE should do in the case that an unknown built-in is parsed? Does adding builtins in a project mean that their code is no longer CUE, but rather a dialect of CUE?

@shykes point is that with custom builtins, CUE is not reusable or readable across the ecosystem. That would cause significant support time because people will not be aware of this if everyone calls their dialect "CUE"

[cueckoo]

Original reply by @ymolists in cuelang/cue#350 (comment)

@verdverm if the extention is prefixed with a package name i think the distinction will be clear no ? just as in go everyone knows how to expect a package is from the standard library or from an external package.

[cueckoo]

Original reply by @verdverm in cuelang/cue#350 (comment)

Maybe some code examples of what you are thinking would help the discussion. I wouldn't expect CUE to be able to differentiate regardless of where the built-in comes from, based on my understanding of how a built-in is implemented in the CUE source.

Without the Go source which adds and implements, how would CUE differentiate between attempted use of an unknown built-in and an unknown definition (in example). How would an import statement make this differentiation?

[cueckoo]

Original reply by @ymolists in cuelang/cue#350 (comment)

i am refferring to something like dagger. One could be making a call similar to Line 7 that is backed by a builtin ? at that point the user who is writing the cue file knows they are using an extension and not the standard packages that come with cue.

[cueckoo]

Original reply by @verdverm in cuelang/cue#350 (comment)

CUE attributes were designed for this purpose, dagger is a good example of a tool built around CUE. I built https://github.com/hofstadter-io/hof as a tool for code generation.

The problem is less about who's writing the original code and more with others who come along, see ”cue" code, but the cue cli can't read it. If the tool changes the language, is it still CUE code? Essentially we would end up with numerous CUE dialects that cannot be read by the standard tool.

Having read your other open question, I think you'd want to go the route that dagger and hof use, attributes on schemas the tool understands, yet are still able to be run through the cue reference implementation

[steeling]

I don't think we should be worried about fragmenting the language here. Any use of a cue file, is a custom library, and will make expectations about what is imported.

How is that any different than me writing a go binary that imports things from private repos? Or even writing a cue file that imports a schema that I defined? I think the best way is to make the go code feel a part of the cue module itself. Viewing this as "changing the language" vs importing a library is a strange notion to me. Especially if the imports are declared at the top, based on my .mod path, ie:

import (
...
     github.com/steeling/my-cue-pkg/go/myfuncs
...
)

It's pretty obvious that anything referencing myfuncs is not part of the language.

That is, we could add a new directory, say cue.mod/go, or simply a new top-level dir go, and have the cue binary parse files in those directories for any exported functions and register those as if they were "builtins".

See my comment here for our use case.

[verdverm]

Enabling user defined builtins would be akin to adding new keywords to the language. CUE has an import notion for importing other CUE. Adding a builtin requires making changes to the implementation (writing Go and compiling a binary).

If you added a new keyword to Go by leveraging the API and compiling a separate binary, would the official go binary be able to parse it?

The fragmentation happens when N projects add different keywords and call them all the same language, when they are incompatible dialects.

Does this help make the distinction and fragmentation clearer?

---

Dynamic loading of functions sounds like a security hole and would require cue to use the go pkg/plugin?

Given CUE is hermetic by design, I doubt that would be changed. The inability to verify this property for user builtins would cause very subtle and difficult to detect bugs.

[steeling]

I disagree that it is akin to adding a new keyword. It is more akin to defining an import of a library... but in this case you're simply extending support for what languages CUE can import. I get that it requires compilation at a lower level, but this can be hidden from the user and done on the fly.

If I call mycustompkg.MyFunc, that's clearly not a keyword. Nobody that looks at that code thinks that. What they think is "oh, if I want to use that package and function I'll need to properly import it".

Agreed that security concerns are definitely something to consider though. Although go has been battling these types of things with its imports its entire lifespan. Is it that much more dangerous than allowing library imports in go?

CUE being hermetic is a good guard rail. But it's also a handicap for people wanting to do more complex things. I think any software library should provide these guard rails, but give the user flexibility to do the things they want.

[myitcv]

It is more akin to defining an import of a library.

@steeling I get the impression you and @verdverm might be talking past each other. An import of CUE is fundamentally different to a new builtin. Are you perhaps referring to a world whereby a syntax as proposed in #69 (comment) is supported to look like a function call? If so, that is fundamentally a different thing to a builtin.

As @verdverm highlights above, custom builtins would indeed fragment the ecosystem because support for that builtin becomes a function of the evaluator one is using, not the CUE packages that might be imported.

Whilst we absolutely don't close the door to adding more builtins, specifics need to be discussed on a case-by-case basis.

[talk2drys]

I just want to point something out, maybe it is just me though. when I read suggestions/recommendations or even the tutorial pages out there, I get this vibe that it is going to be "tightly coupled with the go language" instead of "it is implemented in go" and will probably be like every other niche tool that came before.

using Kotlin as an example, it is a successful language in its own way but could have done better, I don't think there are any books, videos, or blogs out there, that does teach Kotlin without teaching it in reference to Java. the Kotlin developers i know are java first, might be the same story for cue where it is going to go users mostly.

I think setting the spec (done) and just having the cue CLI as a CLI implementation and encouraging developers to build packages, modules, crates (rust) etc, that allow direct manipulation/access from their respective languages, with this I see a future where tools can just support cue directly without needing YAML or JSON as a middle layer.

[mpvl]

@talk2drys: even though CUE has its roots in Go, it is indeed intended to be standalone, and not tightly coupled. As an example of this, CUE bare string values are closer to JSON than Go and the overall string model is much more closely tied to Swift.

It is true that there is currently only a Go API (which can be compiled to JS) and that the builtin documentation piggybacks on godoc. The main raison for this is time constraints, though. It is very time-consuming to support multiple versions, especially in a pre-v1 scenario. But we intend CUE to cut its umbilical cord more and more over time.

[ghostsquad]

Extending CUE has many benefits. As previously mentioned, sometimes the automation/tooling that wraps CUE is required, and having CUE natively be able to evaluate something isn't really needed.

A good example of this is Tanka, which extends Go-Jsonnet and adds additional functions that enhance Tanka's specific use cases. (https://tanka.dev/jsonnet/native). In this example, I don't think there's a situation where someone would complain that jsonnet doesn't work with that code and that Tanka must be used.

[ghostsquad]

Additionally, as also mentioned, allowing people extend the language, play around with adding new features (via functions) before eventually contributing them back would be nice. Right now, it's actually a battle of wills to get new features added to CUE, as they must be accepted and who knows how long it would take before a release is seen with the functions added. The only way to actually add native functions today is by creating a hard fork of CUE itself.