Figure out an adoption plan

[yannham]

We are approaching the state of an MVP. Before shipping, we must think first about the adoption story. In particular:

• We must determine who will be the early adopters: what users and what use-cases are we targeting ?
• We must set up a migration plan accordingly. It is not meant to be a full-fledged plot on how Nickel will take over the world but rather a feasible plan for reaching a situation where these users can easily try out Nickel in conjunction with their existing tools, in an incremental way.

Candidates

1. Nix/Nixpkgs

Nix is the best candidate, since Nickel started as a Nix expressions spin-off and always had it in mind as one of the main applications. As it tries to overcome many of the limitations of Nix, this is where it should add the most value.

As of now, a couple of propositions have been sketched to integrate Nickel in Nix:

• Interactive: provide Nix with the Nickel interpreter as an artifact, and enable him to execute Nickel code. However this looks complex, as it requires the modification of the Nix evaluator which would need to be intimately aware of Nickel.
• Reversed control: Notwithstanding syntax, Nickel should eventually be a superset of Nix. Instead of integrating Nickel into Nix, we could do it the other way around and make it possible to import existing Nix expressions in a Nickel program. Then the Nickel interpreter would generate JSON, or a Nix expression, that could be fed back to Nix.

2. Nixops/Terraform (or Bazel)

Other natural candidates are the remaining use cases which originally motivated Nickel. The story would be simpler as Nickel could in principle just generate JSON in place of the native language and tooling. On the other hand, the benefits of switching to yet another language may be harder to substantiate than for Nix, and to overcome the adoption barrier.

3. New cloud project

A third possibility is to start a new cloud project (Nixops/Terraform like) from scratch, which would use Nickel as a native language. However this is a whole new project on its own, and is not a realistic first adoption plan.

[thufschmitt]

Instead of integrating Nickel into Nix, we could do it the other way around and make it possible to import existing Nix expressions in a Nickel program

Could something like that be enabled transparently, for example at the flake level? If flakes really take off (and it seems to be the case), then one could assume that it's not useful to mix both languages inside the same flake, but that we'd only need to be able to import a Nix flake from a Nickel flake (or vice versa).

2. Nixops/Terraform (or Bazel)

FWIW the Bazel model makes it pretty much impossible to use another language than starlark unless bazel itself is modified to support it (it has been proposed to allow writing the rules in another language, but afaik that never went far)

Another possible use-case is k8s. I don't really know it, but my understanding of the situation is that its configuration (in JSON) is awfully complicated so you need some form of preprocessing to make it usable. And there doesn't seem to be a definite way to do that. The most common way is helm which is a yaml templating engine, but it's apparently not really satisfying, so there's some competition that seems to be growing like ksonnet based on jsonnet. Maybe we could sneak in too

[Profpatsch]

* _Interactive_: provide Nix with the Nickel interpreter as an artifact, and enable him to execute Nickel code. However this looks complex, as it requires the modification of the Nix evaluator which would need to be intimately aware of Nickel.

Nix already supports importing other languages, via the import-from-derivation feature. If the language can produce nix expressions in a derivation, nix will be able to import them as if they were provided to the interpreter directly (they are loaded from a store path).

For example, dhall provides the dhall-to-nix tool, and since dhall is a full subset of nix, it can erase all types and convert the resulting normalized expression to a nix file.

In https://github.com/openlab-aux/vuizvui/blob/1be63b59d0f9d1739c5fdc75e1f82079f6dd9d31/pkgs/profpatsch/importDhall.nix I define some functions that can be used like plain import, but on dhall files.

Here’s an example of usage: https://github.com/openlab-aux/vuizvui/blob/1be63b59d0f9d1739c5fdc75e1f82079f6dd9d31/pkgs/profpatsch/xdg-open/default.nix#L79-L123
Note that we can pass functions to and from the dhall expression (even higher order), because it will be “native” nix so everything happens in the nix evaluator after the conversion via dhall-to-nix.

The same strategy could be used for integrating nickel, at least until we have a better mode of interaction.

---

Could something like that be enabled transparently, for example at the flake level? If flakes really take off (and it seems to be the case), then one could assume that it's not useful to mix both languages inside the same flake, but that we'd only need to be able to import a Nix flake from a Nickel flake (or vice versa)

I strongly advise against supporting nickel at a flake level. Tweag is already eyed cautiously by the rest of the nix community. If we start giving our projects first-class support in the nix implementation (especially via another project we are sponsoring, that is flakes), the effect on the nix community is not going to be pretty.

[yannham]

Nix already supports importing other languages, via the import-from-derivation feature. If the language can produce nix expressions in a derivation, nix will be able to import them as if they were provided to the interpreter directly (they are loaded from a store path).

For example, dhall provides the dhall-to-nix tool, and since dhall is a full subset of nix, it can erase all types and convert the resulting normalized expression to a nix file.

That's an interesting approach and indeed seems very lightweight. However, Nickel would rather be a superset of Nix. While an evaluated Nickel expression is eventually just JSON in the end and could easily be exported as a Nix expression, a full fledged program may contain constructions which are not straightforward to compile to Nix, at least at first sight: dynamic typecasts, enriched values, merge, etc.

[Profpatsch]

Nickel would rather be a superset of Nix.
a full fledged program may contain constructions which are not straightforward to compile to Nix, at least at first sight: dynamic typecasts, enriched values, merge, etc.

Nix is a pretty straightforward untyped lambda calculus, so in practice it’ll be at least as powerful once you erase all the nickel types. It might not be feasible from a performance standpoint though, but that’s an optimization problem.

While an evaluated Nickel expression is eventually just JSON in the end

I think the real value here is in expressions that are not just a json in the end, but in expressions that evaluate to functions.

[yannham]

Nix is a pretty straightforward untyped lambda calculus, so in practice it’ll be at least as powerful once you erase all the nickel types. It might not be feasible from a performance standpoint though, but that’s an optimization problem.

Sure, I'm not saying that this is not possible, but that this may require some work, especially with respect to contracts, that are dynamic entities performing some book-keeping at runtime, and which cannot be just erased. I haven't thought too much about it yet though, this was just a first impression.

I think the real value here is in expressions that are not just a json in the end, but in expressions that evaluate to functions.

We agree. My point is if we need to interface two languages in a workflow where the desired end result is JSON/a derivation/any static configuration, then if one language is close to being a superset of the other, it's easier to transpile the less expressive language to the other, execute everything there, and finally produce the configuration, rather than the other way around.

The thing is, Nickel is currently not strictly speaking a superset of Nix, so we have to quantify how much is missing to determine which path looks easier.

[yannham]

A quick update: after more discussion with @edolstra, we are focusing on Kubernetes, with a handful precise examples in mind that we will convert to Nickel, and then try Nickel + Nix (ideally all the code in Nickel, but leveraging Nix to build the app/container). Kubernetes requires complex configuration, does not have a native language, and no configuration or templating language seems to have a monopoly. Some are very ugly. Some have been abandoned. I'll write down more comprehensively about this somewhere.

[andir]

A quick update: after more discussion with @edolstra, we are focusing on Kubernetes, with a handful precise examples in mind that we will convert to Nickel, and then try Nickel + Nix (ideally all the code in Nickel, but leveraging Nix to build the app/container). Kubernetes requires complex configuration, does not have a native language, and no configuration or templating language seems to have a monopoly. Some are very ugly. Some have been abandoned. I'll write down more comprehensively about this somewhere.

What is the goal here? Always correct k8s manifests when the nickel code type checks? Have you tried any of the existing solutions that try to achieve the same? (dhall-kubernetes, kubeval, …)

[yannham]

The motivation to use Nickel for Kubernetes is the same as for Dhall I guess: code reuse, documentation, modularity, and early failures (via both types and contracts), but with slightly different trade-offs.

On the Nickel side, the goal is to have a guiding use-case that drives the coming additions to the language, and test actual usage. Doing so, we can see if things actually works out as expected, rather than continuing to develop features in the abstract. We probably won't be competitive with something like dhall-kubernetes soon, if ever. I think the short-term objective is rather to undergo a baptism of fire.

I have looked at dhall-kubernetes but haven't used it seriously. When the JSON backend is out (soon), and we can start to toy around, then it will probably be a good idea to do the same examples using both Nickel and other such tools (dhall-kubernetes, ksonnet or CUE) to see the difference.

[bew]

Hello!
FYI there's the project Tanka from (and used by) Grafana, using Jsonnet as the templating language. It has a pretty decent stdlib for k8s, and good docs.
https://tanka.dev/

You might find it interesting as an inspiration ¯\_(ツ)_/¯

[keithy]

I would like to see a version of https://nixcloud.io/tour/ and with #661 I venture you will have a much greater audience than just k8s.

[oneingan]

A good nickel example for Open Application Model could be great. The main OAM implementation, KubeVela, choose CUE for it.