Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC]: Resolve defaults recursively #46

Closed
arianvp opened this issue Jan 30, 2019 · 11 comments
Closed

[RFC]: Resolve defaults recursively #46

arianvp opened this issue Jan 30, 2019 · 11 comments

Comments

@arianvp
Copy link
Member

@arianvp arianvp commented Jan 30, 2019

For example, for a deployment:

instead of filling each default with the obvious "empty type" of a field, recursively resolve the default values.

so instead of this:

\(_params : {metadata : (../types/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall)}) ->
{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata = _params.metadata
, spec = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentSpec.dhall))
, status = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentStatus.dhall))
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

this:

{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata =  ../default/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall
, spec =  ../default/io.k8s.api.apps.v1.DeploymentSpec.dhall
, status = ../default/io.k8s.api.apps.v1.DeploymentStatus.dhall
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

This makes a "merge" based API where we compose layers of configurations with // way more natural. Similar to https://kustomize.io

@arianvp
Copy link
Member Author

@arianvp arianvp commented Jan 30, 2019

I'm not sure how well this approach pans out in the presence of required arguments, which we currently handle as a lambda...

Perhaps we can have a type for defaults, and a type for full specs, such that

Deployment = DeploymentDefault // { metadata : ObjectMeta.dhall }

one can then write:

mydeployment  = ({ metadata = blah } //  (defaultDeployment : DefaultDeployment)) : Deployment
@arianvp
Copy link
Member Author

@arianvp arianvp commented Jan 30, 2019

Oh :( You can only combine records, not types at the moment: dhall-lang/dhall-lang#175 (comment)

@arianvp
Copy link
Member Author

@arianvp arianvp commented Jan 30, 2019

Also, this seems related : dhall-lang/dhall-lang#340 and seems to describe the behaviour I want. so doesn't seem possible with dhall currently :(

@arianvp
Copy link
Member Author

@arianvp arianvp commented Jan 30, 2019

More complicated example of how this would look, taking into account dhall-lang/dhall-lang#340

It's not great, but it's better than the current raw API in my opinion

let 
  default = ./default/Deployment.dhall 
  mydeployment = default // { 
    metadata =  { name = "hello" },
    spec = let default = default.spec in (default.spec // {
      replicas = Some 3, 
      template = default.template // {
        spec = let default = default.spec in (default.spec // {
          containers = let default = ./default/Container.dhall in [
            default // {
              image = "nginx",
              ports = default.ports // {
                containerPort = 80
              }
            },
          ]
        })
      }
    })
  }


@Gabriel439
Copy link
Contributor

@Gabriel439 Gabriel439 commented Feb 25, 2019

@arianvp: You can merge record types, using . You just can't override the type of a record type's field.

@f-f
Copy link
Member

@f-f f-f commented Feb 28, 2019

@arianvp I very much like the idea, but after thinking a bit about this I have some implementation questions:

  1. if we recursively resolve the default value instead of having None for optional values, is there is a semantic difference for Kubernetes between "the field is not there" and "the field is there but the object is empty"?
  2. related to above: the yaml files in output would get HUGE. Currently they are tiny because we have mostly Nones, and we run dhall-to-yaml --omitNull. I think this would get pretty unappealing, since readability of the results would likely be lower
  3. where should we draw the line? Should we stop the recursion at the first record that contains a required field?
@Gabriel439
Copy link
Contributor

@Gabriel439 Gabriel439 commented Mar 1, 2019

@f-f: Note that I can extend dhall-to-yaml so that --omitNull also omits records whose transitive fields are all None

This is something that dhall-nethack does, where if a field is a record then it's not wrapped in Optional and it is always present. Then the default outer record initializes the inner record to all default values

@f-f
Copy link
Member

@f-f f-f commented Mar 1, 2019

@Gabriel439 I didn't consider that we could handle this in dhall-to-yaml :)

Cool, then let's go for this 👍

@arianvp
Copy link
Member Author

@arianvp arianvp commented Mar 4, 2019

@f-f: Note that I can extend dhall-to-yaml so that --omitNull also omits records whose transitive fields are all None

I would put this behind a flag, just to be sure. I'm pretty sure there's software in the wild that depends on just "existence" of keys for example

@Gabriel439
Copy link
Contributor

@Gabriel439 Gabriel439 commented Mar 5, 2019

@arianvp: Wouldn't that imply that the software wouldn't be using --omitNull anyway?

Or in other words, is there software in the wild that works correctly when you omit null fields, but misbehaves when you omit empty records?

Gabriel439 added a commit to dhall-lang/dhall-haskell that referenced this issue Mar 6, 2019
... as proposed in dhall-lang/dhall-kubernetes#46 (comment)

`--omitEmpty` is the same as `--omitNull` except it also omits fields that
are empty records.  To be precise, it omits fields whose transitive fields
are all `null` (and an empty record is a special case of a record whose
transitive fields are all `null` since it is vacuously true when there are
no fields).

This allows Dhall configurations that target YAML to avoid having to nest
sub-records inside of an `Optional` value.  Omitting unnecessary `Optional`
layers reduces the number of default values that the configuration format
needs to thread around.
@Gabriel439
Copy link
Contributor

@Gabriel439 Gabriel439 commented Mar 6, 2019

@arianvp @f-f: Here is the pull request to add a new --omitEmpty flag for this purpose: dhall-lang/dhall-haskell#842

Gabriel439 added a commit to dhall-lang/dhall-haskell that referenced this issue Mar 6, 2019
... as proposed in dhall-lang/dhall-kubernetes#46 (comment)

`--omitEmpty` is the same as `--omitNull` except it also omits fields that
are empty records.  To be precise, it omits fields whose transitive fields
are all `null` (and an empty record is a special case of a record whose
transitive fields are all `null` since it is vacuously true when there are
no fields).

This allows Dhall configurations that target YAML to avoid having to nest
sub-records inside of an `Optional` value.  Omitting unnecessary `Optional`
layers reduces the number of default values that the configuration format
needs to thread around.
f-f added a commit that referenced this issue Mar 30, 2019
@f-f f-f mentioned this issue Mar 30, 2019
12 tasks done
@f-f f-f closed this in #57 May 1, 2019
f-f added a commit that referenced this issue May 1, 2019
This ports the conversion script from Python to Haskell (fix #7)

The main reasons for this port are that:
- the Python script was really hard to maintain for reasons like 
  "converting from Swagger to Dhall is interleaved with string formatting"
- in Haskell we can use the dhall library to generate always syntactically 
  correct Dhall AST. It's also much easier to keep an eye on correctness, 
  because types and pattern matching. It also forces us to deal with things
  like cyclic imports from the get go.

Things happening here:
- remove the `api` folder, removing the difference between "raw api" and "nice api"
- move defaults from `default` to `defaults` folder, as it is in `dhall-nethack`
- transition to the new syntax for `Optional` (fix #49)
- add `types.dhall` and `defaults.dhall`, so that one can now easily "pin a version"
  by just importing these two records at a specific commit/tag. They also make it really
  easy to access objects, e.g.
  `let types = https://raw.githubusercontent... sha256:... in types.Deployment`
- also add typesUnion.dhall (fix #54), so one is able to send to Kubernetes different
  objects in the same stream. This is also documented in the README
- defaults are resolved recursively (fix #46): if there's an import of a "nullable" record,
  then it's not marked as Optional, making merging objects much easier
- default objects are not lambdas anymore, and instead they just don't include the required
  fields (that is, the ones that are not nullable records), as suggested in dhall-lang/dhall-lang#382
- for objects that are simple types we used to generate a simple lambda
  `\(a : Text) -> a` as a default, now we just don't generate a default (e.g. see
  `io.k8s.apimachinery.pkg.apis.meta.v1.Time`)
- autoformat all generated Dhall code
- remove cyclic imports (fix #47)
- update to dhall-1.22 and dhall-json-1.2.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants