Run Configuration Variables

[dsanderbi]

Hi,

i have a question regarding run configuration metadata for pipelines and workflows.

I noticed that pipeline run configuration allows you to define variables, whereas workflow run configuration do not.

pipeline run configuration

workflow run configuration

My question is: What is the intended purpose of these variables, and what is the use case for having them only on pipeline run configurations but not on workflow run configurations?

Initially, I expected these variables to be the ones shown in the Run dialog, allowing me to provide values at execution time. However, it seems that these variables are actually stored directly in the pipeline itself. They also take precedence over variables defined in the run configuration if both define the same variable.

Another thing I noticed is that, the variable icon is missing.

Could someone explain the design rationale behind this feature and the intended way to use it?
Thanks!

[mattcasters]

They were implemented to make certain specific Beam pipeline scenarios possible in the early days. It transferred some configuration options (like folder names). These days we have the option for Beam and Spark to use an environment config file to solve that.
I suppose adding the variables to an environment configuration file makes usually more sense.

[dsanderbi]

Thanks for the info!

[mattcasters]

For things like the missing icon please always file a bug report. We're sometimes slow, ... but we do fix issues ;-)