Custom Resource provider for shell commands? _or...how to get cheat Pkl and get a UUID for something_

[mobileskyfi]

I writing a pkl wrapper over the UTM macOS App to build its virtual machine "packages". UTM is a respectable Mac citizen, so it uses a document resource bundle with a .plist config and disk image(s). So I have some pkl that's creating UTM virtual machines from simpler pkl templates. I'll post it on GitHub at some point, but had some questions and a feature request for some rand().

Unfortunately, UTM config.plist use UUIDs, and generating the .plist is mainly what my pkl code does. But given the commentary in #490,

It's important that all Pkl APIs are deterministic, and referentially transparent. Therefore, we're not likely to introduce a UUID API. and the lack of any random() function, this problem was a big tricker than rest of pkl --> plist.

WO I tried using a "Custom Resource Reader" and wiring it up to a bash script. I tried this since I want the pkl to run in some CI like GitHub Action eventually, and some complied Swift/etc just to be able to get a UUID seemed overkill.

My basic thought was some "interpreted-language custom resource" be useful. Thought was to use bash read() to get the msgpack (and pull out an the needed id), get UUID from uuidgen called in bash, then package up in msgpack built in a bash string. This, obviously, didn't "just work". I figured the msgpack be annoying to parse, in bash, but issue was I got nothing from a read().

The no data... made me question whether I'd read the spec right. So also tried some bun and nodejs basic IO code (no libraries) to read the MsgPack provided by pkl pipe via custom resource invocation, that also just "hang" on read. I then realized it must be using non-buffered IO, so char-by-char reading in node or bash would work. While I got to in bash, read -n 1 input_char could read it, that where I stopped that approach on getting UUID.

Instead, I just used read("https://random.org...) for now. But even that has issues. A read(URL) is correctly cache'd, but that logic means you actually get same now "non-random" results when trying to generate a new UUID. Now UTM cares only that there are no dups, so the actual code has to use a sequence number provided else where to make them "locally unique".

function RndStr(length : UInt(this < 512)) = read("https://www.random.org/sequences/?min=10&max=99&col=1000&format=plain&rnd=new").text.trim().take((length+(length/2-1)).toInt()).replaceAll("\t","")

function RndUUID(altsalt : Number(this <= 9999)) = "\(RndStr(8))-\(altsalt.toString().padStart(4,"0"))-\(RndStr(4))-\(RndStr(4))-\(RndStr(12))"

But what I wanted to do was just run "/usr/bin/uuidgen" but Pkl does not seem to have an API to invoke the shell/process either...

I guess my main questions is:

1. Is there actually some way to invoke the shell from pkl script? The docs and DAG Extension #490 phrase's "deterministic and referentially transparent" would indicate the answer, but that been useful here.

2. With a custom resource provider, is it expected to have to use unbuffered I/O to read? I think the concept of having some "compute" logic shoved in some non-compiled script be useful, even outside potential UUID [ab]use ;). Now I do think pkl-lang's read(UR**I**) as nice abstraction to the outside world, especially with the potential custom providers. But having to use the moral equivalence of a gets() is rather difficult in most languages. So here I'm not sure if it a case "that's the reader's problem", or bug/limitation of the spec.

My "feature request" here be some rand() function. In fact, I wrote up the use case, since I see how that goes against the grain of pkl-lang. But some wrapped around rand() would have been useful. In my small test of pkl with UTM, I also need a random MAC address, in addition to UUID. Anyway, it comes up, randomly.

[bioball]

Pkl's read() is an extension point that you can use to define your own underlying behavior. For example, your code can look like:

function uuid(seed: Int) = read("uuid:\(seed)")

If you use Pkl as a CLI, you can do this by creating an external reader.

We provide helper libraries for writing external readers in pkl-go and pkl-swift. These help you create an executable, which you can then provide to Pkl with the --external-resource-reader flag.

If you use Pkl as a library (with Java, Swift, Go, and many of the community language bindings), you can define a resource reader in the corresponding language binding.

However, Pkl will never have a random(). In order to provide a random seed, one solution is to pass it in as an external property. For example, you can do:

local seed = read("prop:seed")

Then run Pkl with:

pkl eval --prop seed=$(random) myProgram.pkl

If you just need one UUID, you can always just pass it in as an external property, too.

pkl eval --prop uuid=$(uuidgen) myProgram.pkl

[HT154]

One big obstacle here is determining what seed should when you're generating multiple identified resources! I've found a few ways to do this in different situations that might be viable:

• Hash related data, eg. read("uuid:\(new PcfRenderer {}.renderValue(<value without uuid>).sha1)")
• Use (or hash) the destination file path, which works decently when there's a fixed number/variety of UUIDs per file

If you're hashing you can also directly generate UUIDs from those hashes if you're clever and save all the trouble of an external reader entirely.

[mobileskyfi]

Thanks! I never actually thought to use a hash, to get some level uniqueness. I do have enough unique context to create a .sha1 - so that works. I just got stuck on trying to wire up uuidgen to read(), which still need some salt to avoid cache.

One big obstacle here is determining what seed should when you're generating multiple identified resources!

Yup, that was my issue with using just --prop is when pkl start I don't know exactly how many UUID (or "random" MacAddress also) be needed since that be determined after pkl runs.

However, Pkl will never have a random()

I totally agree with the principle. Just in trying to "port" a .plist to .pkl, the original had a UUID. And, there was not even some "simple hack", that did not involve a fair amount of supporting code.