[Bug]: Project variables are not resolved when referenced in Environment file paths

[dsanderbi]

Apache Hop version?

2.19.0

Java version?

21

Operating system

macOS

What happened?

When using Apache Hop project variables to shorten and centralize file paths, the variables are correctly resolved when configuring Environment Settings files through the UI. However, the same variables are not resolved when the workflow or pipeline is executed.

For example, the project configuration defines variables such as:

{
  "metadataBaseFolder" : "${PROJECT_HOME}/metadata",
  "unitTestsBasePath" : "${PROJECT_HOME}",
  "dataSetsCsvFolder" : "${PROJECT_HOME}/datasets",
  "enforcingExecutionInHome" : true,
  "autoExportMetadataFilename" : "metadata.json",
  "parentProjectName" : "default",
  "config" : {
    "variables" : [ {
      "name" : "CONFIG_HOME",
      "value" : "${PROJECT_HOME}/config"
    } ]
  }
}

When selecting a file through the Browse functionality in the Environment Settings, Apache Hop uses the project variable name to reference the selected file instead of ${PROJECT_HOME}/config

When the workflow or pipeline is started, Apache Hop fails to resolve ${CONFIG_HOME} and therefore cannot find the referenced file to load its variables.
You need to use ${PROJECT_HOME}/config to get the variables from json.

Issue Priority

Priority: 2

Issue Component

Component: Hop Gui

[hansva]

It seems our documentation is missing this, but Environment variables are loaded before project variables.
The current hierarchy is:

Everything happens in Project.modifyVariables(), called from ProjectsUtil.enableProject():

L211-230 — parent project's variables (recursive)
L236-257 — PARENT_PROJECT_*, HOP_PROJECT_NAME, HOP_ENVIRONMENT_NAME, PROJECT_HOME
L261-290 — environment configuration files: variables.resolve(configurationFile) then read described variables from it
L292-316 — HOP_METADATA_FOLDER, unit tests folder, datasets folder
L317-321 — the project's own described variables (config.variables in project-config.json, i.e. CONFIG_HOME)

[dsanderbi]

Okay, thanks for the information.

We would need a fix so the file paths selected through the environment settings do not automatically use the project variables.
Instead, the selected files should always be referenced relative to the project home.

I checked for logging metadata, as long as you define a project variable like LOGGING_HOME=${PROJECT_HOME}/logging and you store the pipeline inside, use the browse button to select the file, the path will be set to ${LOGGING_HOME}/logging.hpl and the ETL will fail after startup

so environments and metadata paths are affected!

[hansva]

Honestly, I would like to remove the project-level variables

[dsanderbi]

Honestly, I would like to remove the project-level variables

So the next release is 3.0 😄 ?

I know 😆 , but I know/have Customers that use those variables that now run into issues.

[dsanderbi]

@hansva, i made a mistake when testing the logging metadata, here the path gets resolved in the right order and the pipeline is found!

So only the environment is a problem!

[sramazzina]

Realistically, I personally agree with @hansva by removing project level variables. Environment variables seems more than enough. But what about a very simple and stupid project that for simplicity do not have any environment because it doesn't make any sense in having an environment? This makes me pause for a moment and, as a result, makes me unsure about this decision

[hansva]

You still have:

• engine variables
• pipeline/workflow variables

[sramazzina]

Ah yes that is an option. Didn't thought about that