Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC]: Resolve defaults recursively #46

Open
arianvp opened this Issue Jan 30, 2019 · 11 comments

Comments

Projects
None yet
3 participants
@arianvp
Copy link
Collaborator

arianvp commented Jan 30, 2019

For example, for a deployment:

instead of filling each default with the obvious "empty type" of a field, recursively resolve the default values.

so instead of this:

\(_params : {metadata : (../types/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall)}) ->
{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata = _params.metadata
, spec = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentSpec.dhall))
, status = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentStatus.dhall))
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

this:

{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata =  ../default/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall
, spec =  ../default/io.k8s.api.apps.v1.DeploymentSpec.dhall
, status = ../default/io.k8s.api.apps.v1.DeploymentStatus.dhall
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

This makes a "merge" based API where we compose layers of configurations with // way more natural. Similar to https://kustomize.io

@arianvp

This comment has been minimized.

Copy link
Collaborator Author

arianvp commented Jan 30, 2019

I'm not sure how well this approach pans out in the presence of required arguments, which we currently handle as a lambda...

Perhaps we can have a type for defaults, and a type for full specs, such that

Deployment = DeploymentDefault // { metadata : ObjectMeta.dhall }

one can then write:

mydeployment  = ({ metadata = blah } //  (defaultDeployment : DefaultDeployment)) : Deployment
@arianvp

This comment has been minimized.

Copy link
Collaborator Author

arianvp commented Jan 30, 2019

Oh :( You can only combine records, not types at the moment: dhall-lang/dhall-lang#175 (comment)

@arianvp

This comment has been minimized.

Copy link
Collaborator Author

arianvp commented Jan 30, 2019

Also, this seems related : dhall-lang/dhall-lang#340 and seems to describe the behaviour I want. so doesn't seem possible with dhall currently :(

@arianvp

This comment has been minimized.

Copy link
Collaborator Author

arianvp commented Jan 30, 2019

More complicated example of how this would look, taking into account dhall-lang/dhall-lang#340

It's not great, but it's better than the current raw API in my opinion

let 
  default = ./default/Deployment.dhall 
  mydeployment = default // { 
    metadata =  { name = "hello" },
    spec = let default = default.spec in (default.spec // {
      replicas = Some 3, 
      template = default.template // {
        spec = let default = default.spec in (default.spec // {
          containers = let default = ./default/Container.dhall in [
            default // {
              image = "nginx",
              ports = default.ports // {
                containerPort = 80
              }
            },
          ]
        })
      }
    })
  }


@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Feb 25, 2019

@arianvp: You can merge record types, using . You just can't override the type of a record type's field.

@f-f

This comment has been minimized.

Copy link
Member

f-f commented Feb 28, 2019

@arianvp I very much like the idea, but after thinking a bit about this I have some implementation questions:

  1. if we recursively resolve the default value instead of having None for optional values, is there is a semantic difference for Kubernetes between "the field is not there" and "the field is there but the object is empty"?
  2. related to above: the yaml files in output would get HUGE. Currently they are tiny because we have mostly Nones, and we run dhall-to-yaml --omitNull. I think this would get pretty unappealing, since readability of the results would likely be lower
  3. where should we draw the line? Should we stop the recursion at the first record that contains a required field?
@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Mar 1, 2019

@f-f: Note that I can extend dhall-to-yaml so that --omitNull also omits records whose transitive fields are all None

This is something that dhall-nethack does, where if a field is a record then it's not wrapped in Optional and it is always present. Then the default outer record initializes the inner record to all default values

@f-f

This comment has been minimized.

Copy link
Member

f-f commented Mar 1, 2019

@Gabriel439 I didn't consider that we could handle this in dhall-to-yaml :)

Cool, then let's go for this 👍

@arianvp

This comment has been minimized.

Copy link
Collaborator Author

arianvp commented Mar 4, 2019

@f-f: Note that I can extend dhall-to-yaml so that --omitNull also omits records whose transitive fields are all None

I would put this behind a flag, just to be sure. I'm pretty sure there's software in the wild that depends on just "existence" of keys for example

@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Mar 5, 2019

@arianvp: Wouldn't that imply that the software wouldn't be using --omitNull anyway?

Or in other words, is there software in the wild that works correctly when you omit null fields, but misbehaves when you omit empty records?

Gabriel439 added a commit to dhall-lang/dhall-haskell that referenced this issue Mar 6, 2019

Add new `--omitEmpty` option
... as proposed in dhall-lang/dhall-kubernetes#46 (comment)

`--omitEmpty` is the same as `--omitNull` except it also omits fields that
are empty records.  To be precise, it omits fields whose transitive fields
are all `null` (and an empty record is a special case of a record whose
transitive fields are all `null` since it is vacuously true when there are
no fields).

This allows Dhall configurations that target YAML to avoid having to nest
sub-records inside of an `Optional` value.  Omitting unnecessary `Optional`
layers reduces the number of default values that the configuration format
needs to thread around.
@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Mar 6, 2019

@arianvp @f-f: Here is the pull request to add a new --omitEmpty flag for this purpose: dhall-lang/dhall-haskell#842

Gabriel439 added a commit to dhall-lang/dhall-haskell that referenced this issue Mar 6, 2019

Add new `--omitEmpty` option (#842)
... as proposed in dhall-lang/dhall-kubernetes#46 (comment)

`--omitEmpty` is the same as `--omitNull` except it also omits fields that
are empty records.  To be precise, it omits fields whose transitive fields
are all `null` (and an empty record is a special case of a record whose
transitive fields are all `null` since it is vacuously true when there are
no fields).

This allows Dhall configurations that target YAML to avoid having to nest
sub-records inside of an `Optional` value.  Omitting unnecessary `Optional`
layers reduces the number of default values that the configuration format
needs to thread around.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.