Deferred Property Type

[rorychatterton]

Exploring the feasibility of utilizing the pkl language to internally represent a dependency graph in our product, I'd like to propose the introduction of a "Deferred Type." This type would indicate to the pkl compiler that the value of a given output is intended to be computed in a subsequent execution rather than immediately.

A Deferred Type is proposed to encapsulate relationships between entities where initial inputs may be unknown or unavailable. This type fundamentally differs from null or optional types by signaling that its value, while not currently present, is expected to be provided and computed by the system in the future. Specifically, a Deferred Type:

• Cannot be directly assigned by users, emphasizing its system-computed nature.
• Instructs the processing engine to pause or defer actions requiring its value.

Illustrative Example

Consider a scenario involving two entities, where one depends on data that is computed externally and not immediately available:

class FirstEntity {
  name: String
  externallyComputed: Deferred<String>  // Deferred type indication
}

class SecondEntity {
  input: String?
  name: String
  computedOutput: List<String>? = input?.split("")
}

Initial processing might yield partial results, highlighting the deferred computation with an indicator like readyToProcess to reflect the incomplete state:

{
  "one": {"name": "One"},
  "two": {
    "input": null,
    "name": "Two",
    "computedOutput": null,
    "readyToProcess": false
  }
}

Subsequent processing, with the deferred value now available, completes the dependency graph:

{
  "one": {
    "name": "One",
    "externallyComputed": "dce"
  },
  "two": {
    "input": "dce",
    "name": "Two",
    "computedOutput": ["d", "c", "e"],
    "readyToProcess": true
  }
}

Advantages and Considerations

The Deferred Type approach aims to allow pkl to represent state graphs more flexibly, accommodating data that becomes known only at runtime. This could enable more dynamic transformations and processing based on late-bound data. However, potential challenges include managing the complexity of partial parsing and the overhead of re-processing as data availability evolves.

Alternatives and Further Discussion

While the Deferred Type offers a solution to managing data dependencies and runtime unknowns, its implementation and the potential for complex partial parsing warrant careful consideration.

Edit

Had another go at re-writing this, because my initial attempt wasn't particularly clear.