[quesiton] is jsonnet fmt --string-style input??

[etsangsplk]

example use case:
https://marketplace.visualstudio.com/items?itemName=xrc-inc.jsonnet-formatter

[etsangsplk]

I guess you don't.

go-jsonnet/jsonnet/cmd.go

Line 613 in bed2cd8

panic(fmt.Sprintf("Internal error: No jsonnet fmt."))

[sparkprime]

The formatter has not yet been ported from the C++ version of the codebase. So you need to build the jsonnet commandline utility from the google/jsonnet repo.

[metalmatze]

Are there any current plans on bringing the fmt command to this Go version? This is the only reason we're still installing the C++ one.

[sparkprime]

There's a desire to do it but not any concrete plans simply due to it being quite a large project.

[dancompton]

@sbarzowski @sparkprime Is there any documentation around the desired implementation/direction? Could this be implemented as a builtin that operates on snippets and produces formatted output as a string? It would also be great if we could print a complete jsonnet snippet without converting to json.

Here's that example which was written during a hackathon. This is to modify one field with new memory allocation determined by cloudwatch data.
https://github.com/dan-compton/ecs-metrics/blob/master/ast.go
Of course, this could be implemented without walking the AST, but I wanted to try it out. As some abuses of jsonnet constructs within our manifests are heavily sugared and/or rely on complex logic, it becomes increasingly necessary for manual intervention if the language cannot evaluate snippets (e.g. to determine that a field referenced via $ does not exist in some cases but not others based on a std.extVar loaded in via a jsonnet_to_json code-file as determined by some config setting or the like).

Preferably I could do something like:

local codefile = 'code.jsonnet';
local codefilestr = importsrt 'code.jsonnet';


local codefileModified = std.fmt(std.manifestJsonnet(codefile + {memory: std.native('ecsMemoryAlloc')(codefile.name, codefile.cluster, ...);
local codefileOriginal = std.fmt(codefileStr);
local normalize = function(json, func)
                                       func(json)     
local diffStr := std.diffJson(normalize(codefileModified, function(i) i)), normalize(codefileOriginal, function(i) i));
          
std.trace(diffStr, null)
 
// multiple outputs for a prettier diff function executed via bazel run                    
{
    // manifestJsonnet acts on snippets and returns the result of evaluation as a string 
    // with sugar, hidden fields, all preserved (like the output of importstr)
    "codefile-modified.jsonnet": 
    "codefile-original.jsonnet": std.fmt(codefilestr),
}

So, new functionality would be

• std.manifestJsonnet -- dumps raw jsonnet snippet to string
• std.fmt -- operates on a jsonnet string to create formatted string output
• std.diffJson -- diffs the results of std.manifestJson(jsonnet code) with optional normalization function (e.g. like a jq script except implemented in jsonnet) that is applied to original and new json code. _Alternatively this could just operate on text and you could provide a normalization function to manifestJson :shrug).

If this were implemented, it would allow me to eliminate my json_diff bazel rule and it's dependencies, move all ECS-related bazel rules into std.native functions backed by the aws-go-sdk (instead of bash functions invoked by a template the depend on non-parred awscli), and I could automate the diffing of task and service definitions json generated for our manifests across all of our manifests using only bazel build ...

The overall context is that I've had to reimplement our service-manifest back-end libsonnet code and I'm currently doing validation between old and new libsonnet code for service manifests (which manifest ECS Service and Task definition JSON which is later used by bazel run targets to diff, create, update, delete services in ECS. I've got a robust set of bazel rules that automates this for me now (diffing locally the legacy-manifested svc/task json and new-libsonnet-manifested svc and task defintions as well as the diffs against those locally vs what resides in ECS.

However, it would be great if it could all be implemented through jsonnet. There are situations where significant manual labor would be involved due to the variance of developers leveraging jsonnet constructs across manifests. (like referencing $.env.envName from an imported library without knowledge that the field will no longer exist in the next version of that library). That particular example is easy to fix with sed, but fixing more complex abuses at the jsonnet level becomes impossible without manual intervention.

[sbarzowski]

re: desired implementation/direction
For any formatting support in go-jsonnet, the first step would be to add the information about the formatting (newlines, spaces, comments) to the AST structs. The way it is represented in cpp version seems fine (it doesn't allow every formatting, but it's ok). Then dumping code from AST as-is, without explicit reformatting. Then commandline support and integration with tests from cpp-jsonnet would come. Then actual reformatting logic. Then we could perhaps expose it in some new way (if the benefits outweigh the costs).

@dan-compton I'll try to respond in more detail to this and to other issues on github and on groups in the coming days, but that's quite a lot of stuff to process :-).

[dancompton]

@sbarzowski Understood and sorry for the exhaustive post. My (perhaps obvious) goal is to get people on board with the idea of implementing the required built-ins such that jsonnet can read, parse, and evaluate jsonnet, then manifest it as a formatted jsonnet string (like jsonnet fmt does now). For mass refactors or refactors that deal with complex logic implemented in jsonnet, it would be much simpler than working with the AST package in go.

I see moving jsonnet fmt up to the builtin level as std.fmt(<jsonnet string>) and std.manifestJsonnet(<jsonnet code>) as the two main blockers to achieve this goal.

Thank you for your response and I look forward to discussing here or in the google group.

[sparkprime]

I'm not convinced Jsonnet is the best language for code transformation. It's Turing complete and has lots of features but it's still designed for configuration. E.g. yes for some basic stuff like recursive descent over the AST renaming variables, it would probably work OK. But what if you end up needing to do some actual program analysis? Then you'd really want an imperative language.

I think the best way to give you what you want would be to have a really easy to use framework in Go for doing these transformations, and that would share code with the reformatter and perhaps other tooling like test coverage and linting.

[dancompton]

The go implementation is clear and it took only an hour or two two to implement a basic prototype of this feature using a similar approach to string block and fodder collections (I used /~ and ~/ to compile the verbatim code block and achieve nearly the desired output.

Other types already exist with isCode members to change desugaring behavior, and it seems classically functional to provide vm-executed manifest functions which consume the underlying language atructure and produce a custom one (irrespective of language, we do this for ecs service and task defs]

The arguments I have heard against the implementation of a circular meta interpreter thus far have been regarding referential transparency by @sbarzowsky (I did not understand the linkage to the implementation of a built in manifestJsonnet function) and that jsonnet is a poor language for program transformation.

To the latter point, I am curious as to why there is discussion of expanding upon the current manifest formats. Taken further, what is the use case for jsonnet at all if it is not expressively clear enough to hermetically transform programs based on inputs? Why not instead use go itself and avoid the intermediate format?

This is both possible and extensible via the extension of go templates with go plugins and I have done this with mediocre results (dan-compton/exile)

My core argument is that if bazel is loosely homoiconic in the sense that it evaluates to JSON objects as lisp evaluates to arrays, is JSON(1), yet has a use case which requires Turing completeness and std.native functions to realize, then should it not also be able to lazily desugar and evaluate raw snippets or allow for the manifestation of raw jsonnet. (OR evaluate to some sugared standard)

There are cases wherein it is not simple to detect and correct transform erroneous jsonnet via the go AST. For example, if you have a std.extVar with isCode set, but you are generating pipelines on parallel or conditionally and injecting that code into the extVar, then the task of resolving that via the go AST boils down handling all permutations of conditionally injected dependency code, right?

Another might be the implementation of erroneous code in syntacticallly distinct, but value indistinct ways. If ot were possible to eval raw snippets within manifests, there would be no need to deal with the tedium of go-based ASTs.

Simultaneously, we have projects that are implementing jsonschema in jsonnet, implementing additional output formats, as well as extending the AST. These effects are splintering and diffractional effects.

1. (as an aside can those be implemented differently across architectures?)
2. Do you have a better solution for mass refactorings, jsonnet specification validation (e.g. we define specs and perform online validation on the fields), or tips for refactoring via the go ast in general?

By the way, it took nearly 100 LOC just to modify our container memory fields by walking the go AST. That can become complicated quickly when a decision maker has made choices that obfuscate transformation hierarchy.

[sbarzowski]

Other types already exist with isCode members to change desugaring behavior

Hmmm... I don't really see what isCode has to do with anything. It's just that extVars and top-level-arguments provided by user can be either Jsonnet code (to be evaluated) or a string. It's analogous to import vs importstr. It affects the desugaring only in the sense that if the code is passed, then yes, it needs to be desugared.

I will also repeat a general point that "desugaring" is just an implementation detail. It makes some things simpler, but it crossed my mind more than once to eliminate this step completely. And the users wouldn't notice any change in behavior. I think we should focus on the semantics for now.

regarding referential transparency by @sbarzowsky (I did not understand the linkage to the implementation of a built in manifestJsonnet function)

If you have std.manifestJsonnet(2 + 2) it must produce the same output as std.manifestJsonnet(4). My understanding is that the intended results were respectively "2 + 2" and "4".

Of course it applies to more complex cases too, for example the following would be required:

local foo = 42;
std.manifestJsonnet({a: foo}) == std.manifestJsonnet({a: 42})

local bar(x) = x + 2;
std.manifestJsonnet(bar(17)) == std.manifestJsonnet(18 + 1)

std.manifestJsonnet(({x: 1} + {x: 2}).x) == std.manifestJsonnet({x: 2}.x)

local x = 2 + 2;
std.manifestJsonnet(x) == std.manifestJsonnet(2 + 2)

The core of the issue is that with std.manifetJsonnet you don't know if you have a variable in the raw snippet if it should be substituted for its contents or not.

This problem goes away completely when an expression and a representation of an expression are different things.

My core argument is that if bazel is loosely homoiconic in the sense that it evaluates to JSON objects as lisp evaluates to arrays, is JSON(1), yet has a use case which requires Turing completeness and std.native functions to realize, [...]

You meant Jsonnet, not bazel in the first sentence here, right? Below I'm going to assume you meant Jsonnet.

I wouldn't say that "[Jsonnet] evaluates to JSON objects as lisp evaluates to arrays, is JSON(1)". Jsonnet is not homoiconic and it never tried to be. The interesting parts of Jsonnet (functions, variables) are not JSON-ish in any way. Of course they could be represented as JSON, but that's not different from what you have in any language.

[...] then should it not also be able to lazily desugar and evaluate raw snippets or allow for the manifestation of raw jsonnet.

It already is evaluating the subexpressions ("raw snippets") lazily.

Simultaneously, we have projects that are implementing jsonschema in jsonnet, implementing additional output formats, as well as extending the AST. These effects are splintering and diffractional effects.

(as an aside can those be implemented differently across architectures?)

I don't understand what you mean by "across architectures" here.

Do you have a better solution for mass refactorings, jsonnet specification validation (e.g. we define specs and perform online validation on the fields), [...]

Short answer - not something usable right now, rough ideas at best, and right now we have more basic things to implement. These are very interesting topics, but let's start a separate issues if we want to discuss them.

By the way, it took nearly 100 LOC just to modify our container memory fields by walking the go AST. That can become complicated quickly when a decision maker has made choices that obfuscate transformation hierarchy.

I think it could be made quite a bit shorter using parser.Children function (it would reduce the generic boilerplate to almost nothing). I think it wouldn't be that hard to create some more convenience functions which would help with similar use case (e.g. going a specified path in the ast). But in general transforming the AST can be tricky, I agree.

But let's get back of how the transformations like that could work in Jsonnet. In one of the previous posts you provided this example:
std.manifestJsonnet(codefile + {memory: std.native('ecsMemoryAlloc'))
Could you explain in more detail what it does? It's clear to me that manifestJsonnet takes a Jsonnet snippet and produces a string, but what + would do in this case? What is the final output like?

More examples (ideally involving functions and locals), with operations on the snippets explained would be really helpful in understanding your vision.

[dancompton]

More detailed response later, but yes I meant jsonnet (it was 3AM), and second, I'm referring to the implementation of an operator like http://cl-cookbook.sourceforge.net/macros.html#LtohTOCentry-2 to prevent immediate evaluation (if that makes sense), I'm aware that they are already lazily desugared.

[dancompton]

I don't understand what you mean by "across architectures" here.

I just mean that it's possible to define platforms in bazel that have different constraint values, and that the result of the invocation of a native function could vary based on those inputs.

constraint_setting(name = "native_fn_dependency")

constraint_value(
    name = "native_fn_dependency0",
    constraint_setting = ":native_fn_dependency",
)

constraint_value(
    name = "native_fn_dependency_1",
    constraint_setting = ":native_fn_dependency",
)

platform(
    name = "linux_x86",
    constraint_values = [
        "@bazel_tools//platforms:linux",
        "@bazel_tools//platforms:x86_64",
        ":native_fn_dependency0",
)
platform(
    name = "linux_arm",
    constraint_values = [
        "@bazel_tools//platforms:darwin",
        "@bazel_tools//platforms:x86_64",
        ":native_fn_dependency1",
   ],
)

platform(
    name = "osx",
    constraint_values = [
        "@bazel_tools//platforms:linux",
        "@bazel_tools//platforms:x86_64",
        ":native_fn_dependencyA", // invocation of native fn results in different output
    ],
)

Of course, it's possible to accomplish he same outcome more directly via attributes passed to a macro or perhaps via config settingns.

[dancompton]

Also worth linking to comparison of my idea refactoring with jsonnet vs go AST (I can include an example): #195 (comment)

[sbarzowski]

@dan-compton

So in the linked comment there is this line:

toRefactor {string: "New"}

which, just to be clear, is equivalent to:

toRefactor + {string: "New"}

This is where the whole magic happens. What is the meaning of this +. What does it mean to add an object to quoted code (AST)?

[dancompton]

@sbarzowski

Apologies, that was a mistake in my example code. There is no meaning there because that operator cannot on lhs and rhs that differ in std.type AFAIK. The code should read:

local toRefactor = /~ {string: "Old"} ~/;  or local toRefactor = importstr 'input.jsonnet';
local codeVal = std.parseJsonnet(toRefactor);
local refactoredCodeVal = 
   if codeVal.string == "Old" then 
            codeval {string: "New"} // NOTE(dc): we operate on the parsed jsonnet, not the string value.
   else 
         codeVal;
{
   'input.jsonnet':  std.manifestJsonnet(refactoredCodeVa)l,
}```

[sbarzowski]

There is no meaning there because that operator cannot on lhs and rhs that differ in std.type AFAIK.
Well, that's not required actually. It's just the way it currently is. We're talking about defining a new operation anyway...

Hmmm... I understood that it is already parsed, our special quoting could do that without the need for explicit parsing.

Anyway it's still not the same type. The subexpression codeval {string: "New"}, a raw snippet (codeval) is added to an object ({string: "New"}). I assume the expected result is a raw snippet with literal {string: "New"}.

But how would that work in general? For example in the following cases:

std.parseJsonnet(/~ local name = "string"; {[name]: "Old"} ~/) {string: "New"}

I guess error? Topmost thing is not an object literal...

std.parseJsonnet(/~ {["str" + "ing"]: "Old"} ~/).string

I guess error?

std.parseJsonnet(/~ {"a": {"b": "c"}} ~/) {a: {"d": "e"}}

I guess literal {a: {"d": "e"}}?

std.parseJsonnet(/~ "O" + "ld" ~/) == "Old";

I guess false?

std.parseJsonnet(/~ {num: 7} ~/) { num: 2 + 2 };

I guess literal {num: 4}?

std.parseJsonnet(/~ {num: 7} ~/) { num: std.parseJsonnet(/~ 2 + 2 ~/) };

I guess literal {num: 2 + 2}?

[dancompton]

raw snippet (codeval) is added to an object ({string: "New"}). I assume the expected result is a raw snippet with literal {string: "New"}.

Indeed that is the expected result, though I made the assumption that parseJsonnet would not simply parse the jsonnet, but have similar behavior to import (poorly named).

I noticed that in desgugar.go that type LiteralString is not compatible with ast.Node for token import. Is that because it has not been parsed? If so, would it be possible to implement a token eval, similar to import, but that lexes and parses the LiteralString prior to desugaring?

Your examples highlight some flaws in my understanding of the language. I'll take additional time to review the code. Thanks :)

[sbarzowski]

There is some progress on bringing jsonnetfmt to Go (and deprecating C++ one). It's tracked elsewhere, so I'm closing this one.