[Bug]: Variable substitution for file-transform paths still broken on remote Hop Server when "Export linked resources" is enabled

[LPETIT458]

Related issue

Follow-up to #7209 (fixed in 2.19 — variable substitution for transform file paths on remote Hop Server). The general case described in #7209 now works correctly for us on [VOTRE VERSION], but the issue reappears specifically when the "Export linked resources to server" option is enabled on the run configuration.

Apache Hop version?

2.19.0

Java version?

JDK 21

Operating system

Red Hat Enterprise Linux

What happened?

Scenario

• A pipeline extracts data from a source table (e.g. dwh.my_table) and feeds two file-output transforms, each using variables in the filename field:
  • Hop file output — Filename: ${NFS}/${REP}/extraction-10000.cube
  • Text file output — Filename: ${NFS}/${REP}/mon_fichier-10000 (extension txt)
• ${NFS} and ${REP} are defined at the project/environment level.
• The pipeline runs via a Pipeline Run Configuration using:
  • Engine type: Hop remote pipeline engine
  • Hop server: a registered remote Hop Server
  • Run Configuration: local
  • Export linked resources to server: checked

Expected result

The variables ${NFS} and ${REP} should resolve to their configured values, exactly as they do when "Export linked resources to server" is left unchecked.

Actual result

The pipeline executes without error ("OK"), but the output files are written to the wrong path. Inspecting the Hop Server's install directory (HOP_HOME) shows a folder literally named ${DATA_PATH_1} created at the root of the Hop installation folder — i.e. a variable reference is used verbatim as a literal path segment instead of being substituted, and the resolved location ends up nested inside the server's own Hop install directory rather than the intended target path.

With "Export linked resources to server" unchecked, the same pipeline resolves ${NFS}/${REP}/... correctly.

Steps to reproduce

1. Define variables (e.g. ${NFS}, ${REP}) in the project/environment configuration.
2. Build a pipeline with a source input feeding two File-type transforms (reproduced with both a Hop file output .cube and a Text file output .txt), each using ${NFS}/${REP}/<filename> in the Filename field.
3. Create a Pipeline Run Configuration with Engine type "Hop remote pipeline engine", pointing to a remote Hop Server.
4. Check "Export linked resources to server".
5. Run the pipeline.
6. Observe: execution reports success, but files land under a literal ${DATA_PATH_1}-named folder inside the Hop Server's own install directory instead of the resolved ${NFS}/${REP} path.
7. Uncheck "Export linked resources to server" and re-run: paths resolve correctly.

Additional context

• Systematic — reproduces on every File-type transform tested, not intermittent.
• Same incorrect resolution occurs when the path points to an S3 location, not just local/NFS paths — suggesting the substitution breaks upstream of the specific VFS/file-transform implementation, in how variables are resolved for the exported/linked resource before being handed to the transform.
• Affects both Pipeline and Workflow run configurations.
• A minimal reproduction pipeline is attached (see comments) — extraction feeding a Hop file output and a Text file output, both using ${NFS}/${REP} in their Filename field.

Issue Priority

Priority: 1 (systematic regression affecting all File-type transforms in a common remote-execution setup; same severity class as #7209, which was P1)

Issue Component

Component: Hop Server

Complement information

Sama problem with S3 output : name for S3 connexion became a name of directory in path on hop-server....

[hansva]

.take-issue

[LPETIT458]

Additional context

• Systematic — reproduces on every File-type transform tested, not intermittent.
• Same incorrect resolution occurs when the path points to an S3 location, not just local/NFS paths — suggesting the substitution breaks upstream of the specific VFS/file-transform implementation, in how variables are resolved for the exported/linked resource before being handed to the transform.
• Separate but related S3-specific symptom: when the path targets an S3 location, the path is regenerated on the Hop Server with the S3 connection's name substituted in place of the actual connection/bucket details — and this happens regardless of whether "Export linked resources to server" is checked or not. This suggests the S3 path-resolution issue may be a distinct (but related) bug from the "Export linked resources" one described above, since it isn't gated by that option.
• Affects both Pipeline and Workflow run configurations.

[hansva]

The VFS locations were not correctly loaded in 2.19. That issue has been resolved in #8106
Hop Server carries its own file providers loaded with the metadata of the server. It can not load/resolve VFS locations that are being supplied via a resource bundle. This required significant changes, as the file systems were a shared singleton.

[hansva]

I pushed a fix #8240 are you able to build and test this? I have added an integration test, and it seems to be working as expected, but real environments with reproducible paths usually yield better results

[LPETIT458]

While waiting for an architectural fix, is there another workaround we could put in place? The approach we're considering (a dedicated, ephemeral remote Hop Server process per execution, restarted each time to reload the metadata) would work, but it comes with a real cost: JVM startup time and project reload on every single execution, which becomes a real penalty if pipelines are short or frequent. Is there an approach that would allow VFS connections to be isolated per execution within a single running server process, without having to spin up a whole new process each time?

[LPETIT458]

I don't currently have an environment to test fix #8240. If you had a full Hop zip build for Linux, I might be able to give it a try? Since I've never applied a fix myself before, it'll be a bit tricky for me. Sorry about that.