Execution Modes, Configuration Modes and Play Buttons

[jdunkerley]

This discussion is to clarify what we should do for the execution mode and the play button support.

Currently, we have three contexts a function can be in default, Input, and Output. If a context is disabled, it will panic with a Forbidden_Operation when the function is called. Higher-level APIs will perform a test evaluation returning a valid result if they use one of these functions.

The design, as it stands now, adds "execution modes" to control the state of these contexts. Two modes are currently proposed "development" (Input and Output disabled) and "production" (everything enabled). It also suggests using the mode to control the configuration of connections.

This is mixing two concepts which should not be joined.

Execution Modes

We need to control the different behavior of contexts when designing a workflow (i.e. interactively within the IDE) and when running in a script mode (i.e. when executing within the CLI or a job in the cloud). The current naming of "development" and "production" combines the concept with the idea of different databases and datasets. More precisely naming "interactive" and "batch" would make the two modes' behaviors obvious. This would also remove users needing to add new modes, as these two modes would be all required. The IDE would run in "interactive" mode and the CLI in "batch" mode - this would mean a behavior difference between the two worlds, but this feels more like the natural behaviors.

Within "batch" mode everything would be enabled. Within the "interactive" mode, we would disable the "Output" context. The use and meaning of the "Input" context is not currently well designed. It is imagined, it might be something like a cached value if disabled and live value if enabled, but this would need significantly more design and is not planned within the "Beta" release schedule.

Individual node control of contexts would still be needed. However, with the simplified behaviors, the Runtime.enable_context and Runtime.disable_context would not need to take a second argument. If a node disables a context, it will not be enabled in "batch" mode. If a node enables a context, it would be executed on all input changes within the "interactive" mode.

## Enable a specific context in the runtime.

   Arguments:
   - context: Name of the context to enable.
Runtime.enable_context : Text -> (Any -> Any) -> Any
Runtime.enable_context context ~action =
    ## Implementation out of scope for design.

## Disable a specific context in the runtime.

   Arguments:
   - context: Name of the context to disable.
Runtime.disable_context : Text -> Any -> Any) -> Any
Runtime.disable_context context ~action =
    ## Implementation out of scope for design.

For the current planned releases, any context other than "Output" for these functions would result in an Illegal_Argument panic.

The CLI and IDE would not need additional arguments to control these modes as they would execute as specified above.

Configuration Modes

The drop-down with modes would still be an essential feature of the IDE and would be needed to drive the configuration of connections. The set of these modes would be heavily driven by the user and integrated into the cloud-based configuration system. We could also enable it within the project.yaml for project-level configurations. This would enable connecting to different servers and files depending on the mode.

The suggested CLI argument of --execution-mode would be replaced with --configuration-mode and the Runtime.get_execution_mode function would be renamed to Runtime.get_configuration_mode.

## Gets the value of the current configuration mode.
Runtime.get_configuration_mode : Text
Runtime.get_configuration_mode =
    ## Implementation out of scope for design.

This would be a useful feature but would be a lower priority than "interactive" / "batch" and play button support (described below).

Workflow Level Play Button (Design Partners)

The idea of the play button is to execute the workflow within the IDE in a "batch" mode. The behavior should be as follows:

1. The IDE locks to read-only mode.
2. Cached values are deleted.
3. The engine is set to "batch" mode - i.e. the contexts are all enabled.
4. All nodes (including inputs) are re-evaluated, and the results are stored in the server cache.
5. The engine is set to "interactive" mode - i.e. the contexts are disabled.
6. The IDE unlocks.

Ideally, the play button would become a stop button while this is running giving the user the ability to abort. Not required for Design Partners but ideally developed soon after.

This would allow the user to run the workflow in a "batch" mode and then interactively edit the workflow without the need to re-run the workflow. The user would be able to see the workflow results in the IDE.

Eventually, we would want to be able to execute with a single context being enabled. This is not expected for the "Design Partners" or "Beta" release. You could, for example, enable the "Input" context and re-evaluate the workflow without running the outputs. This is only mentioned as it might affect the design of the language server protocol for the above process.

An additional similar button (🔃) is needed to re-evaluate the workflow in the "interactive" mode, effectively this refreshes the inputs and recalculates. This would be a simpler process of invalidating the cache and then evaluating the nodes from top to bottom, similar to opening the workflow in the first place.

Node Level Play Button (Beta Release)

Clicking the node behavior should be as follows:

1. The IDE locks to read-only mode.
2. The cached values are preserved for all nodes except the one triggered.
3. The specific node would be re-evaluated with the language server being instructed to enable the "Output" context.
4. Downstream nodes are re-evaluated in the normal way with "Output" disabled, and their cached values updated.
5. The IDE unlocks.

This would allow the user to run a specific node in a "batch" mode and then work with results downstream. The user would be able to see the node results in the IDE. They can also use this button to refresh a single nodes value (for example, re-read an input file) as enabling "Output" would have no effect on other nodes.

The longer-term goal would be to enable it on a connection-by-connection basis, such as updating all the tables in a specific database. This would be a more complex design and need to be designed later. For example, triggering an update to the production database with the current values as cached being written to the target tables.

[jdunkerley]

cc: @sylwiabr @NedHarding ahead of our meeting today

[farmaazon]

One quick question, regarding Workflow Level Play Button section: Why we need to clear the entire cache? Invalidating all "execution mode getters" (and all dependent nodes) could suffice, as running the same code which does not depend on execution mode should give the same result. Am I missing something?

[jdunkerley]

As we don't currently have a decent design for the "Input" nodes (ones who value would change on re-evaluation), it seemed best to refresh the whole workflow and make sure all are up to date.

We will do a design and more thought on these nodes at later point, at which point we might be able to limit to a subset of nodes in the refresh.

[wdanilo]

A few comments on my end. But basically, this drastically changes the way contexts work and I do not like the new idea. I think we should get back on a call to discuss it again. But basically, I believe that what we had designed until now is way better than what's proposed here. (some of the points below describe why I think so). Anyway, we should not change so important things when the development started, as it will cause delays and confusion in our team. So this needs to be sorted out ASAP.

1. Higher-level APIs will perform a test evaluation returning a valid result if they use one of these functions. - a minor thing, but it might be confusing to readers. Not all high-level APIs will do it. Only a very narrow set of them that would like to provide users with an alternative workflow when the context is disabled, right?
2. I don't think we should disable "input" in development. I believe this would render the whole product quite useless then.
3. It also suggests using the mode to control the configuration of connections. - I don't understand this sentence.
4. This is mixing two concepts which should not be joined. - I don't understand this sentence either. What concepts are you referring here to?
5. The current naming of "development" and "production" combines the concept with the idea of different databases and datasets. - What do you mean that the naming combines concepts (another question - which concept?) with idea of databases and datasets? I was reading it so many times and I do not understand the meaning of this at all.
6. More precisely naming "interactive" and "batch" would make the two modes' behaviors obvious, why "batch" will be obvious? I don't see why batch mode is a mode with output context disabled, but with all nodes running interactively.
7. This would also remove users needing to add new modes, as these two modes would be all required - why? I remember I had a call with you, James, when we've been discussing other modes, like e.g. a mode that disables access to camera, or other IO devices, disables networking etc. Our contexts are similar to how android IO permissions work. I don't see how 2 modes would be a solution to everything.
8. The IDE would run in "interactive" mode and the CLI in "batch" mode - this would mean a behavior difference between the two worlds, but this feels more like the natural behaviors. - what do you mean by "worlds"? Also, why do you think it's more natural? For me, that is very counter-intuitive that running the same file from CLI and GUI would produce different results.
9. Within the "interactive" mode, we would disable the "Output" context. – I don't like it. I want to be able to interactively work with output enabled in many cases.
10. The use and meaning of the "Input" context is not currently well designed. - why? I thought it is clear how contexts work. Even this doc describes it on the beginning (see my next point).
11. The use and meaning of the "Input" context is not currently well designed. It is imagined, it might be something like a cached value if disabled and live value if enabled, but this would need significantly more design and is not planned within the "Beta" release schedule. - I don't see why input context has anything to do with a cache. If input context is disabled, then input operations just throw error. That's it, nothing more. Also, I do not see big benefit of disabling input context, maybe only for debugging purposes. This is also why I don't understand why this doc states that the "input" context should be disabled in the "development" mode.
12. Regarding the play button - let's talk about its behavior after we sort out the problem with contexts above. I don't think it's worth discussing it now if we are getting back to basics of how everything works here.

To sum this up, we need to have a call about it ASAP. Until then, the development should either follow the previous specification or should be frozen.

[jdunkerley]

1. Higher-level APIs will perform a test evaluation returning a valid result if they use one of these functions. - a minor thing, but it might be confusing to readers. Not all high-level APIs will do it. Only a very narrow set of them that would like to provide users with an alternative workflow when the context is disabled, right?

This is as per the current design. A clear warning would be attached.

Currently this is expected on about 5 functions (just the write operations), the lower level File stream operation would be under the Output context and would panic if disabled.

[jdunkerley]

2. I don't think we should disable "input" in development. I believe this would render the whole product quite useless then.

Agree in the suggested new design we are not using, nor disabling, the Input context for now.

Ideally, we would think further on what a disabled function within the Input context would do. It can't just panic and return as this would stop the user from being able to build further down stream nodes. The current suggestion but not properly thought out yet was to have a single evaluation and then cache. If then triggered with the same parameters the result would be the cached value. This, however, needs much more thought and design.

[jdunkerley]

3. It also suggests using the mode to control the configuration of connections. - I don't understand this sentence.
4. This is mixing two concepts which should not be joined. - I don't understand this sentence either. What concepts are you referring here to?
5. The current naming of "development" and "production" combines the concept with the idea of different databases and datasets. - What do you mean that the naming combines concepts (another question - which concept?) with idea of databases and datasets? I was reading it so many times and I do not understand the meaning of this at all.

The names of development and production imply a different environment for database and such within a large organization.

For example, one I worked for had:

• development: fake data, small scale
• dev-test: fake data, small scale
• qa-test: fake data, real scale
• ua-test: secured with real data
• rc: secured with real data, ready for release version
• production

The execution behavior and the the configuration are two separate concepts. We want to be able to drive configuration based on an environment but this does not adjust the method of execution within the script.

[wdanilo]

Following our call today, I would like to provide you with a brief summary of the design that we discussed to ensure that we are both in agreement. Please find the summary below.

Runtime Config

There exists an entity named Runtime.Config that holds global settings for the workflow. To modify these settings, you can use the API provided below. Alternatively, in the future, it will be possible to read these settings from JSON files.

## The mode name the workflow should be run in.
type Runtime.ModeName = Text

## Runtime configuration object. Defines a global configuration for the workspace evaluation
## behavior, including:
## - Configuration of runtime modes, such as "production" or "development". Each mode 
##   controls the I/O permissions. For example, the "development" mode can prevent writing
##   to files.
## - I/O connections. For example, predefined, company-wide databases connections.
## - Custom, user-defined key-value store for any other configuration. For example, users can 
##   define an "optimization-testing" key, which can be set to "true" or "false", which will 
##   control whether their custom functions use a new optimization technique they are testing,
##   no matter the runtime mode is chosen.
type Runtime.Config
    current_mode: Runtime.ModeName
    modes: Map Runtime.ModeName Runtime.Mode
    connections: Runtime.ConnectionManager
    metadata: Set Text Text

## Runtime mode defines which permissions are enabled or disabled. For example, it can
## define that reading files is allowed, but writing to them is not. It also contains an arbitrary
## key-value store that can be used by users to set custom metadata for this mode. For example,
## users can define a "optimization-testing" key, which can be set to "true" or "false", which will
## control whether their custom functions use a new optimization technique they are testing.
type Runtime.Mode
    permissions: Map Runtime.Permission Runtime.PermissionState
    metadata: Set Text Text
    ## Defines whether the GUI allows for graph editing in this mode.
    # This is not planned for the upcoming release.
    allow_graph_editing: Bool

    ## Check whether the given permission is enabled or disabled. If not defined, permissions
    ## default to the disabled state.
    check_permission: Runtime.Permission -> Runtime.PermissionState
    check_permission self permission = self.permissions.get permission . or_default

## Runtime permission types, such as permission to read or write files. In the future, these
## permissions will be extended to provide fine-grained access to I/O devices, such as access
## to camera or microphone.
type Runtime.Permission
    Read
    Write
    StdOut
    StdIn
    Camera
    Clipboard
    ...

## Defines whether the permission is disabled or enabled.
type Runtime.PermissionState
    Enabled
    Disabled

## Get the current Runtime Configuration object.
Runtime.config: Runtime.Config
Runtime.config =
    ## Implementation out of scope for design.

## Set a new Runtime Configuration object. This invalidates all caches and causes the 
## workflow to be recomputed.
Runtime.set_config: Runtime.Config -> Nothing
Runtime.set_config new_config =
    ## Implementation out of scope for design.

## Modify the Runtime Configuration object. This invalidates all caches and causes the 
## workflow to be recomputed.
Runtime.modify_config: (Runtime.Config -> Runtime.Config) -> Nothing
Runtime.modify_config f =
    ## Implementation out of scope for design.

Controlling Runtime Modes Permissions in libraries

Furthermore, an API has been established for managing runtime mode permissions, which utilizes the previously mentioned Runtime Config API:

## Checks whether the permission is enabled or disabled in the provided runtime mode.
Runtime.check_permission: Runtime.Permission -> Runtime.ModeName? -> Runtime.PermissionState
Runtime.check_permission permission mode=Runtime.config.current_mode =
    Runtime.config.modes.get mode . or_default . check_permission permission

## Sets the permission for the provided runtime mode. If the runtime mode is not defined,
## results in a dataflow error.
Runtime.set_permission:  
    Runtime.Permission -> 
    Runtime.PermissionState ->
    Runtime.ModeName? -> 
    Nothing ! Runtime.ModeNotDefined
Runtime.set_permission permission permission_state mode=Runtime.config.current_mode =
    ## Implementation out of scope for design.

## Checks whether the permission is enabled. If it is, evaluates the provided function and returns it
## result. If not, panics.
Runtime.if_permission_enabled: Runtime.Permission -> Runtime.ModeName? -> ~_t -> _t
Runtime.if_permission_enabled permission mode=Runtime.config.current_mode ~f =
    if Runtime.check_permission permission mode 
        then f.eval 
        else panic Runtime.PermissionDenied permission

In the event that Runtime.PermissionDenied is triggered, the GUI should present a clear and concise warning message. The low-level APIs must utilize the aforementioned API. For instance, the file read function should be defined in the following manner:

read_file file_name = 
    Runtime.if_permission_enabled Runtime.Permission.Read <| 
        ... code of reading the file ...

Certain high-level APIs may opt to utilize the aforementioned API to offer alternative functionality for a given function if the related permission has been disabled. For example, the database API could generate temporary in-memory databases to supply the user with interactive outcomes, even if database write permissions are disabled. Nevertheless, it is ultimately up to the author of the library to decide whether or not to include such functionality. By default, any high-level APIs that depend on low-level APIs will produce an exception if permission is denied.

For convenience, the following API is defined as well:

## Enables the write permission in the given mode.
Runtime.enable_write_permission: Runtime.ModeName? -> Nothing ! Runtime.ModeNotDefined
Runtime.enable_write_permission mode=Runtime.config.current_mode =
    Runtime.set_permission Runtime.Permission.Write Runtime.PermissionState.Enabled mode

## Enables the write permission in the given mode.
Runtime.disable_write_permission: Runtime.ModeName? -> Nothing ! Runtime.ModeNotDefined
Runtime.disable_write_permission mode=Runtime.config.current_mode =
    Runtime.set_permission Runtime.Permission.Write Runtime.PermissionState.Disabled mode

Controlling current Runtime Mode

The GUI displays a drop-down widget in the top menu allowing setting the runtime mode:

Basically, switching the mode is the same as evaluating the following Enso code:

Runtime.modify_config (.current_mode = "development")

Please note, that calling modify_config invalidates caches and causes the graph to recompute.

Controlling current connections configuration

Although not currently planned on the timeline, the following functionality is closely linked to the concepts outlined in this document and will be introduced in the near future. Another drop-down menu (or similar widget) will be added, giving users the ability to select their desired connection configuration. For instance, users may choose to direct all database write nodes to write to databases specified by the company or to local databases instead. This widget, like the existing Runtime Mode widget, will employ the provided API to alter the Runtime.Config.connections field.

Enabling and disabling node evaluation in a given mode

Next to the nodes, there should be an icon allowing for disabling or enabling permissions on a particular node:

The icon behavior is summed up in the following table:

Mode	Global write permission	Icon state	Icon after clicking	code before clicking	code after clicking
"development"	Enabled			node2 = node1.fn	node2 = Runtime.disable_write_permission "development" <| node1.fn
"development"	Disabled			node2 = Runtime.disable_write_permission "development" <| node1.fn	node2 = node1.fn
"production"	Disabled			node2 = node1.fn	node2 = Runtime.enable_write_permission "production" <| node1.fn

Please be aware that at present, the GUI is only able to recognize the syntax that has been provided. While we intend to expand this functionality in the future to support other types of expressions that manage permissions, for now, this is the only format that the visual environment recognizes.

Additionally, please be aware that the aforementioned feature permits the management of the evaluation mode solely in the presently selected runtime mode. However, we intend to incorporate a context menu to the button in the future, enabling the enablement/disabling of write permissions in modes other than the one currently selected, as well as the control of other permissions.

Play button behavior

Play button should be active in every mode where the write permission is disabled, like the "development" mode. After clicking it, the following pseudo code should be evaluated:

old_caches = get_all_node_caches
clear_all_caches
Runtime.set_permission Runtime.Permission.Write Runtime.PermissionState.Enabled
// Evaluate all nodes. During node evaluation, graph editing is not allowed. User can stop the 
// evaluation by pressing escape or the stop button (which appears in the place of the play button).
Runtime.set_permission Runtime.Permission.Write Runtime.PermissionState.Disabled

For instance, suppose that the only distinction between the "development" and "production" modes is that the "development" mode lacks write permissions while the "production" mode does not. In that case, pressing the play button is equivalent to briefly shifting from "development" to "production" mode. Nevertheless, this description is oversimplified since modes can vary in numerous ways, including customized, user-defined metadata. Therefore, it is advisable to adhere to the above pseudocode when implementing this functionality.

Runtime modes, project opening, and CLI

To prevent users from unintentionally overwriting their data, the runtime mode should always switch to the development mode automatically before the engine evaluation begins when opening a project. When executing the workflow with CLI, the production mode is utilized by default until the --runtime.mode flag overrides it.

A few words about the "development" and "production" names

James mentioned that the names "development" and "production" may not be the most suitable options. However, we currently do not have a definite answer regarding what the new names should be. Even the names "interactive" and "batch" pose an issue, as the GUI is still interactive when switching to the "batch" mode. It is important to note that although this document employs the current names, we should remain open to the possibility that they may change in the coming weeks.

[jdunkerley]

The high level play button functionality seems correct. We also need to consider the design of play for a single node in this new design.

I think this gives us the correct ability to control of write. As we discussed, joining the configuration of connections and the control of context still feels wrong to the needs I think we need to address. In this new design, they are completely entwined in the Runtime.Config structures.

The control of the "execution mode" being a standard Enso function presents some significant challenges. What if a script changes it two lines down - what should that mean for the execution?

Likewise, I think writing the configuration itself should not be part of the Enso language. Can a script reconfigure "development" half way through to enable "Output" as it stands.

With the goal of allowing more context scenario in future we had deliberately avoided hard coding "Output" into the enable/disable commands. This reverses this.

This feels a fairly big redesign from the previous design doc and the suggestions above - once we have a little more clarity we will need to fully rewrite that to make sure we are on the same page.

[farmaazon]

From the GUI perspective, this design brings a lot of questions. And it seems to affect greatly our current plan. Given that the feature has to be delivered within 2 weeks, I just don't see space for another round of questions and clarifications. Can we stick to the previous design, without any "runtime changes by evaluation"?

[farmaazon]

Ok, follow up from call with @wdanilo :

• Changing the execution mode is done by language server API as in previous design versions (no magical evaluations).
• The current mode and permissions will be also provided by LS API, perhaps by delivering serialized Runtime.Config structure.
• The only change is that the re-evaluate icon has two versions: the current crossed out icon and non-crossed out version.
  • If the current mode has write permission globally enabled, the icon on every node is crossed out, and toggling it on disables the permission.
  • If the current mode has write permission globally disabled, the icon on every node is non-crossed out, and toggling it on enables the permission
  • In any case, toggling the icon off remove the "Runtime.enable/disable" permission for current node.

cc @vitvakatu @MichaelMauderer @Procrat

[vitvakatu]

Thank you, @farmaazon, you addressed my fears. I'll update the planned issues to reflect these changes. It shouldn't affect the overall estimate.

[NedHarding]

This is getting more and more complicated and is holding back development of the prodoct. I have so many thoughts. Random thoughts:

1. Development and Production are concepts that related to the use of the product. They are NOT the right terms for changing how it runs in the GUI. Only the user can decide when those terms are used.
2. If we only had a single boolean flag (RunningAsGUI) we could do 90% of what we need in a really simple way. Let's not overcomplicated it.
3. I have seen solutions written down, but no where have I seen a list of the problems we are trying to solve.
4. We have to move faster. Long discussions like this on design can't happen.

Requirements:

1. Disable outputs in GUI
2. Run with all outputs enabled (Play button)
3. Selectively enable (one time) a single output

I think that's it, right? Why so complicated?

[wdanilo]

Hi @NedHarding, we are on the same page with James now I think and it is not really getting more complex. My comment may look so, because it defines a lot of APIs that were not explicitly defined earlier, but under the hood, I believe this is simpler than the previous version. Anyway, James will be assembling that in the final design today I think, so we are on a good path to have it committed! :)

Regarding your points:

1. Absolutely. The names are to be changed, but we are unsure regarding the correct ones yet. But we are totally aware that they need change.
2. I don't think that single boolean flag will be anyhow simpler. Most of the design above is how we handle discovering which functions should be banned from execution and how it should work on the engine side. Instead of a single boolean flag we have a map String -> BoolLike now, which is practically of the same complexity.
3. Good point, this probably should be included as a special section in the design doc, but it will make writing it longer.
4. The above is not quite a discussion - it was a summary from yesterday call, written down, so it can be used to fast update the design doc, so everyone are on the same page. This topic is complex, especially because we need to tell engine which functions should be banned and how. But I think we are on the same page now.

Sorry for super short and not in-depth reply, I wanted to reply before heading off to bed today in order to cover some of the points and give James more space to focus on updating the final design. In no way I'm telling that I don't agree. I do agree. I want to move faster with development and possibly skip design phase as often as possible. I'm unsure if we could do it here though, as this topic caused thousands of questions from all sides - GUI / Libs / Engine, and the answers were not obvious. Again, I'm happy as it seems that we are on the same page now (after todays call with James).

[wdanilo]

The final design was posted here! https://github.com/enso-org/design/pull/55 😄