pkg/**: stdlib and builtin expansion

[cueckoo]

Originally opened by @verdverm in cuelang/cue#348

Generally, we would like to see the standard library expanded.

At this point I am starting a conversation about the topic. We can change this header as things become better understood.

Thoughts:

• More functions from go-stdlib, these seem like low hanging fruit
• both pure / tools portions, how to separate (document this, where things belong), want to expand both most likely
• builtins for implementing structural func's
• builtins for things discussed around the evaluator rewrite and recursion
• external libs or packages, how do we decide what / when to bring in deps to cue?
• should we continue to generate the builtins as is today? or do we need to consider something that scales with the increased size of the stdlib?

Another thought is...

Can we provided a way for users to register their own builtins?

iirc, the way builtins are happened, this might not be overly difficult. There is the issue (still to be created) that fields named the same as builtin funcs are sorted before other fields in a struct.

---

Functions and Packages:

From Go, for pkg/tool

• archive/*
• compress/*
• database (I can see people wanting to do something like an http call here)
• something like a file lock or concurrent access for commands
• more of / from os/, net/, file / filesystem / walktree
• something for logging
• support for readers / writers to chain other funcs and reduce overhead, something like Node/JS pipes
• better filesystem support, globs, in memory FS

From Go, for pure cue pkg expansion

• strings (not sure if there is a gap for this one and the next)
• strconv
• path.Join
• text/template (expose more of the interface, want to register partials for example)

External, for pure cue pkg expansion

• strings, some additions like Kace or a custom written version of the one or two missing funcs
• semver, as configurations are often dealing with these
• structural, object semantics based [diff,patch,merge,pick,filter]

Moonshot:

• access to cue runtime internals, like if I want to walk cue instances / values as defined in cue.Value rather than the exported / data forms.

I can see writing a cue tool which has a schema with it, and then processing local cue packages with the tool. So there may be a need to load and process other cue files from within the tool portion of a cue "program"

[cueckoo]

Original reply by @mikelnrd in cuelang/cue#348 (comment)

Can we provided a way for users to register their own builtins?

I came here to request this. 

I've had some recent success implementing some existing kubernetes tooling using cue. The existing tooling involves hashing, and I am unable to choose the hash function as it needs to match the existing go implementation for compatibility reasons. 

I've done this twice recently (currently in love with cue, thanks a lot for the hard work!):

In the first case the hash function used was sha256 which is already in the cue stdlib, so I was able to complete the implementation in cue successfully.

But in the second case the hash function is https://golang.org/pkg/hash/fnv/ which sadly isn't in cue's stdlib. It'd be great if you'd accept a pull request to add it (?), but more generally I'd love to able to define my own functions like this (understanding, a bit like with Bazel, that they need to be hermetic & deterministic).

Thought I'd post this here as an end-user experience report in case it's helpful.

..................................................

PS: Here's the kubernetes tooling I'm working with by the way:

hash/fnv is used in kubernetes-sigs/cli-utils (which is used by google's new kpt tool):

https://github.com/kubernetes-sigs/cli-utils/blob/f18d7c260fb3f5d3a963088159332a3081cdd5f5/pkg/apply/prune/inventory.go#L263

and here's my cue code (albeit using the wrong hash function):

_hash: hex.Encode(strings.ByteSlice(sha256.Sum256(strings.Join(list.SortStrings(_list), "")), 0, 4))

[cueckoo]

Original reply by @verdverm in cuelang/cue#348 (comment)

Same comment about user defined builtins: cuelang/cue#423 (comment)

[cueckoo]

Original reply by @apstndb in cuelang/cue#348 (comment)

I have also noticed that to generate JWT in CUE is hard because there are no crypto/rsa package.
It requires exec.Run with sign helper command for workaround.
https://github.com/apstndb/cue-sandbox/blob/7d252af7b9f3665243fa319b54d5cbb3ebc68ebc/jwt/jwt.cue#L21-L33

[cueckoo]

Original reply by @verdverm in cuelang/cue#348 (comment)

CUE is hermetic and I'm not sure secure crypto generation aligns with that goal

[cueckoo]

Original reply by @apstndb in cuelang/cue#348 (comment)

Yeah, generally crypto/rsa is not hermetic but SignPKCS1v15(without RSA blinding) and VerifyPKCS1v15 doesn't rely on RNG so they can be hermetic.
If they are supported, CUE can process JWTs.

[cueckoo]

Original reply by @verdverm in cuelang/cue#348 (comment)

SignPKCS1v15 (without RSA blinding)

• The Go function takes a random reader, how do you propose getting around this?
• Does allowing this create weakened security for the sake of doing something with CUE?

[cueckoo]

Original reply by @verdverm in cuelang/cue#348 (comment)

Of note, I just noticed tou are working in the non-hermetic area of the cue cli which has the tooling layer on top of CUE. So it could be feasible to add the crypto functions there, though I wonder how one might select a different source of randomness from outside of the go stdlib (i.e. something like Cloudflare using lava lamps)

[myitcv]

@apstndb @verdverm in the context of this discussion I note the recent adding to cue help injection:

Tag variables

The injection mechanism allows for the injection of system variables:
when variable injection is enabled, tags of the form

    @tag(dir,var=cwd)

will inject the named variable (here cwd) into the tag. An explicitly
set value for a tag using --inject/-t takes precedence over an
available tag variable.

The following variables are supported:

   now        current time in RFC3339 format.
   os         OS identifier of the current system. Valid values:
                aix       android   darwin    dragonfly
                freebsd   illumos   ios       js (wasm)
                linux     netbsd    openbsd   plan9
                solaris   windows
   cwd        working directory
   username   current username
   hostname   current hostname
   rand       a random 128-bit integer