builtin_flatten_arrays: expected array, got string #156
Comments
|
In jrsonnet, However, the way it works with strings in go-jsonnet isn't good nor intuitive: std.flattenArrays(['a', 'b', 'c', ['d', 'e']]) == "[ ]abc[\"d\", \"e\"]"I don't think that's what you want here anyway? |
|
I see there is std.flattenDeepArray(['a', 'b', 'c', ['d', 'e']]) == ['a', 'b', 'c', 'd', 'e']`)But isn't implemented in jrsonnet/go-jsonnet yet: I will implement it in jrsonnet, but in meanwhile, you can just use it in your code local flattenDeepArray(value) =
if std.isArray(value) then
[y for x in value for y in flattenDeepArray(x)]
else
[value];
flattenDeepArray(['a', 'b', 'c', ['d', 'e']]) == ['a', 'b', 'c', 'd', 'e'] |
|
Thanks! Perhaps it would make sense to have a compat mode and emit warnings to help teams transition? I made the change and it worked! I had to set the max-stack size, but it dropped the runtime from 145s to 12s. I know they run this in VSCode. Is there an ability to swap that to use yours? time jsonnet deployment/heartland.plan.jsonnet -o /tmp/plan-go.json
jsonnet deployment/heartland.plan.jsonnet -o /tmp/plan-go.json 144.82s user 1.11s system 223% cpu 1:05.37 total
time ~/Downloads/jrsonnet deployment/heartland.plan.jsonnet -o /tmp/plan-rust.json --max-stack 400
~/Downloads/jrsonnet deployment/heartland.plan.jsonnet -o /tmp/plan-rust.json 12.19s user 0.58s system 98% cpu 12.985 total
diff -q /tmp/plan-go.json /tmp/plan-rust.json && echo "identical" || echo "different"
identical |
Maybe I can add reasons why functions are typed this way to error messages, but it isn't possible to implement the compat mode in general. In this case, accepting strings isn't even the intended behavior, it is just upstream jsonnet lacking type checks.
You mean, how to use jrsonnet for file preview in VSCode? As I can see, grafana language server uses embedded go-jsonnet for evaluation, and it is not possible right now to use jrsonnet instead. It should be possible to implement this feature, however. |
|
That's all very reasonable. I think having improved warnings is always nice. For VSCode, we already use the Save and Run extension for scripts so writing a shortcut that invokes jrsonnet should be easy. One last question while I'm bothering you, do you have any advice on how to profile / optimize jsonnet? Presumably our 2.5 minute evaluations are abnormal and perhaps trivial to fix if we understood what the root cause was. We tried profiling go-jsonnet (pprof svg) which showed flattenArrays as a bottleneck, but at the interpreter level rather than our own code. |
|
That is complicated. Some years ago I was trying to optimize my jsonnet deployments, and everything was bottlenecked at the interpreter. I did some domain-specific optimizations, but every time I came to conclusion that original implementations are optimized for conditions, that are wastly different from my code (And from code of kube-prometheus and other projects), so instead I just decided to write my own jsonnet implementation, and there it is :D After couple of years, some optimizations were properly implemented in upstream, but jrsonnet/sjsonnet are still just much better. |
|
In latest release I have also included faster implementation of Standard implementation of prune is evaluating everything exponentially more times, it is first calling itself for every element of array/object (recursively), and then doing the same thing to provide result. As none of jsonnet implementations implement memoization, this is one of examples that will work poorly (In jrsonnet, prune is now much more optimized.) prune(a)::
local isContent(b) =
if b == null then
false
else if std.isArray(b) then
std.length(b) > 0
else if std.isObject(b) then
std.length(b) > 0
else
true;
if std.isArray(a) then
[std.prune(x) for x in a if isContent($.prune(x))]
else if std.isObject(a) then {
[x]: $.prune(a[x])
for x in std.objectFields(a)
if isContent(std.prune(a[x]))
} else
a,And for the demo why standard prune is bad... local
prune(a)=std.trace('prune call!',
local isContent(b) =
if b == null then
false
else if std.isArray(b) then
std.length(b) > 0
else if std.isObject(b) then
std.length(b) > 0
else
true;
if std.isArray(a) then
[prune(x) for x in a if isContent(prune(x))]
else if std.isObject(a) then {
[x]: prune(a[x])
for x in std.objectFields(a)
if isContent(prune(a[x]))
} else
a),
tower(depth) = if depth == 0 then 'el' else local v = tower(depth - 1); [v, v, v];
prune(tower(5))
|
|
oh wow, that really knocked the time down! I built master locally and now it runs in less than a second 😮 time ~/Downloads/jrsonnet-r deployment/heartland.plan.jsonnet -o /tmp/plan-rust.json
~/Downloads/jrsonnet-r deployment/heartland.plan.jsonnet -o 0.35s user 0.05s system 96% cpu 0.410 totalWe use jsonnet for developing client-side workflows that we deploy to shipment facilities to streamline the truck turnaround by having the driver, guard, clerk, forklift operator all collaborate on their devices (video). So these create pretty large, customized workflow plans that we deploy as json assemblies, and our solution engineers wrote a framework in jsonnet to standardize our components. That was a pretty huge productivity boost, but then slow build times killed that again so this is pretty amazing. |
|
Shared with the impacted team this morning, lots of happy people thanks to your work 😁
and the founder on Slack after I added it to the Sunday night company update,
I gave them this snippet to integrate into VS Code until we put something nicer together, "saveAndRun": {
"commands": [
{
"match": "\\.(jsonnet|libsonnet)$",
"cmd": "jrsonnet ${file} &> /tmp/results.json ; code --reuse-window /tmp/results.json",
"useShortcut": true,
"silent": true
}
]
}anyway, just wanted you know this project is greatly appreciated! |
I tried evaluating a complex jsonnet file using the latest from jrsonnet-unstable-builds:
jrsonnet 0.0.0-git.7f29d2da65f7623c71566a288af9988797d9691dThe error seems to dislike the
std.flattenArraysusage in the code snippet.The evaluation works fine in go-jsonnet. I was just doing a quick inspection for our team using go-jsonnet, where their evaluation times are taking 1-3 minutes and was curious if this implementation might assist them. Not sure if I can offer more insights, so you are welcome to close this issue if not detailed enough for you.
code
error
The text was updated successfully, but these errors were encountered: