[RFC]: Resolve defaults recursively #46

arianvp · 2019-01-30T13:08:45Z

For example, for a deployment:

instead of filling each default with the obvious "empty type" of a field, recursively resolve the default values.

so instead of this:

\(_params : {metadata : (../types/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall)}) ->
{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata = _params.metadata
, spec = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentSpec.dhall))
, status = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentStatus.dhall))
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

this:

{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata =  ../default/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall
, spec =  ../default/io.k8s.api.apps.v1.DeploymentSpec.dhall
, status = ../default/io.k8s.api.apps.v1.DeploymentStatus.dhall
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

This makes a "merge" based API where we compose layers of configurations with // way more natural. Similar to https://kustomize.io

The text was updated successfully, but these errors were encountered:

arianvp · 2019-01-30T13:09:47Z

I'm not sure how well this approach pans out in the presence of required arguments, which we currently handle as a lambda...

Perhaps we can have a type for defaults, and a type for full specs, such that

Deployment = DeploymentDefault // { metadata : ObjectMeta.dhall }

one can then write:

mydeployment  = ({ metadata = blah } //  (defaultDeployment : DefaultDeployment)) : Deployment

arianvp · 2019-01-30T13:20:46Z

Oh :( You can only combine records, not types at the moment: dhall-lang/dhall-lang#175 (comment)

arianvp · 2019-01-30T13:23:59Z

Also, this seems related : dhall-lang/dhall-lang#340 and seems to describe the behaviour I want. so doesn't seem possible with dhall currently :(

arianvp · 2019-01-30T14:04:10Z

More complicated example of how this would look, taking into account dhall-lang/dhall-lang#340

It's not great, but it's better than the current raw API in my opinion

let 
  default = ./default/Deployment.dhall 
  mydeployment = default // { 
    metadata =  { name = "hello" },
    spec = let default = default.spec in (default.spec // {
      replicas = Some 3, 
      template = default.template // {
        spec = let default = default.spec in (default.spec // {
          containers = let default = ./default/Container.dhall in [
            default // {
              image = "nginx",
              ports = default.ports // {
                containerPort = 80
              }
            },
          ]
        })
      }
    })
  }

Gabriella439 · 2019-02-25T15:43:57Z

@arianvp: You can merge record types, using ⩓. You just can't override the type of a record type's field.

f-f · 2019-02-28T19:18:18Z

@arianvp I very much like the idea, but after thinking a bit about this I have some implementation questions:

if we recursively resolve the default value instead of having None for optional values, is there is a semantic difference for Kubernetes between "the field is not there" and "the field is there but the object is empty"?
related to above: the yaml files in output would get HUGE. Currently they are tiny because we have mostly Nones, and we run dhall-to-yaml --omitNull. I think this would get pretty unappealing, since readability of the results would likely be lower
where should we draw the line? Should we stop the recursion at the first record that contains a required field?

Gabriella439 · 2019-03-01T03:37:18Z

@f-f: Note that I can extend dhall-to-yaml so that --omitNull also omits records whose transitive fields are all None

This is something that dhall-nethack does, where if a field is a record then it's not wrapped in Optional and it is always present. Then the default outer record initializes the inner record to all default values

f-f · 2019-03-01T22:33:37Z

@Gabriel439 I didn't consider that we could handle this in dhall-to-yaml :)

Cool, then let's go for this 👍

arianvp · 2019-03-04T20:06:25Z

@f-f: Note that I can extend dhall-to-yaml so that --omitNull also omits records whose transitive fields are all None

I would put this behind a flag, just to be sure. I'm pretty sure there's software in the wild that depends on just "existence" of keys for example

Gabriella439 · 2019-03-05T02:06:51Z

@arianvp: Wouldn't that imply that the software wouldn't be using --omitNull anyway?

Or in other words, is there software in the wild that works correctly when you omit null fields, but misbehaves when you omit empty records?

... as proposed in dhall-lang/dhall-kubernetes#46 (comment) `--omitEmpty` is the same as `--omitNull` except it also omits fields that are empty records. To be precise, it omits fields whose transitive fields are all `null` (and an empty record is a special case of a record whose transitive fields are all `null` since it is vacuously true when there are no fields). This allows Dhall configurations that target YAML to avoid having to nest sub-records inside of an `Optional` value. Omitting unnecessary `Optional` layers reduces the number of default values that the configuration format needs to thread around.

Gabriella439 · 2019-03-06T04:44:44Z

@arianvp @f-f: Here is the pull request to add a new --omitEmpty flag for this purpose: dhall-lang/dhall-haskell#842

... as proposed in dhall-lang/dhall-kubernetes#46 (comment) `--omitEmpty` is the same as `--omitNull` except it also omits fields that are empty records. To be precise, it omits fields whose transitive fields are all `null` (and an empty record is a special case of a record whose transitive fields are all `null` since it is vacuously true when there are no fields). This allows Dhall configurations that target YAML to avoid having to nest sub-records inside of an `Optional` value. Omitting unnecessary `Optional` layers reduces the number of default values that the configuration format needs to thread around.

This ports the conversion script from Python to Haskell (fix #7) The main reasons for this port are that: - the Python script was really hard to maintain for reasons like "converting from Swagger to Dhall is interleaved with string formatting" - in Haskell we can use the dhall library to generate always syntactically correct Dhall AST. It's also much easier to keep an eye on correctness, because types and pattern matching. It also forces us to deal with things like cyclic imports from the get go. Things happening here: - remove the `api` folder, removing the difference between "raw api" and "nice api" - move defaults from `default` to `defaults` folder, as it is in `dhall-nethack` - transition to the new syntax for `Optional` (fix #49) - add `types.dhall` and `defaults.dhall`, so that one can now easily "pin a version" by just importing these two records at a specific commit/tag. They also make it really easy to access objects, e.g. `let types = https://raw.githubusercontent... sha256:... in types.Deployment` - also add typesUnion.dhall (fix #54), so one is able to send to Kubernetes different objects in the same stream. This is also documented in the README - defaults are resolved recursively (fix #46): if there's an import of a "nullable" record, then it's not marked as Optional, making merging objects much easier - default objects are not lambdas anymore, and instead they just don't include the required fields (that is, the ones that are not nullable records), as suggested in dhall-lang/dhall-lang#382 - for objects that are simple types we used to generate a simple lambda `\(a : Text) -> a` as a default, now we just don't generate a default (e.g. see `io.k8s.apimachinery.pkg.apis.meta.v1.Time`) - autoformat all generated Dhall code - remove cyclic imports (fix #47) - update to dhall-1.22 and dhall-json-1.2.8

f-f mentioned this issue Feb 25, 2019

Less verbose records with optional values dhall-lang/dhall-lang#382

Closed

arianvp mentioned this issue Feb 28, 2019

Comparison with other solutions #10

Open

Gabriella439 mentioned this issue Mar 6, 2019

Add new --omitEmpty option dhall-lang/dhall-haskell#842

Merged

This was referenced Mar 27, 2019

Port to Haskell? #7

Closed

Nixify the build #55

Merged

Recursively merge records preferring the RHS when they conflict dhall-lang/dhall-lang#448

Closed

f-f added a commit that referenced this issue Mar 30, 2019

Implement #46, #49, #54

3e4ac82

f-f mentioned this issue Mar 30, 2019

API v2 #57

Merged

12 tasks

f-f closed this as completed in #57 May 1, 2019

Gabriella439 mentioned this issue Oct 20, 2019

Go back to --omitNull instead of --omitEmpty #86

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC]: Resolve defaults recursively #46

[RFC]: Resolve defaults recursively #46

arianvp commented Jan 30, 2019 •

edited

arianvp commented Jan 30, 2019 •

edited

arianvp commented Jan 30, 2019

arianvp commented Jan 30, 2019 •

edited

arianvp commented Jan 30, 2019 •

edited

Gabriella439 commented Feb 25, 2019

f-f commented Feb 28, 2019 •

edited

Gabriella439 commented Mar 1, 2019

f-f commented Mar 1, 2019

arianvp commented Mar 4, 2019

Gabriella439 commented Mar 5, 2019

Gabriella439 commented Mar 6, 2019

[RFC]: Resolve defaults recursively #46

[RFC]: Resolve defaults recursively #46

Comments

arianvp commented Jan 30, 2019 • edited

arianvp commented Jan 30, 2019 • edited

arianvp commented Jan 30, 2019

arianvp commented Jan 30, 2019 • edited

arianvp commented Jan 30, 2019 • edited

Gabriella439 commented Feb 25, 2019

f-f commented Feb 28, 2019 • edited

Gabriella439 commented Mar 1, 2019

f-f commented Mar 1, 2019

arianvp commented Mar 4, 2019

Gabriella439 commented Mar 5, 2019

Gabriella439 commented Mar 6, 2019

arianvp commented Jan 30, 2019 •

edited

arianvp commented Jan 30, 2019 •

edited

arianvp commented Jan 30, 2019 •

edited

arianvp commented Jan 30, 2019 •

edited

f-f commented Feb 28, 2019 •

edited