MinIO/S3 file reading issues with Remote Pipeline Execution on Hop Server

[tribyk]

Hi everyone,

I'm facing an issue with the MinIO/S3 integration or configuration when running a pipeline remotely on the Hop Server (Docker container). The same pipeline works perfectly when executed by using the local pipeline engine (via Hop GUI or Hop Web Docker container), but it fails to read files or their content from the bucket when executed via remote pipeline engine on the server (submitted in Hop GUI/Web).

Details

• I had the issue with Hop 2.16, 2.17 and the 2.18 Snapshot
• Goal: I want to use the JSON Input action to read files from a MinIO/S3 bucket (currently mainly using a Docker container, but also tested with a server hosted instance) utilizing the MinIO connection metadata configuration
• Hop local pipeline engine: Successfully reads files from the bucket using the MinIO metadata configuration (so the MinIO and JSON Input action configuration seem correct)
• Hop remote pipeline engine: No files/content read, although MinIO logs show the request arrives (so no network issue)
• Observed difference: When using local pipeline engine, MinIO logs a GetObject call, whereas using the remote pipeline engine it logs a HeadObject call - not sure why this is happening, but it explains, why there is no data read.

MinIO Log - local pipeline engine (shortened)

[REQUEST s3.GetObject]
GET /file-bucket/testdata.json
[RESPONSE]
200 OK
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Amz-Request-Id: 188F79AA1BB5C5E6
X-Content-Type-Options: nosniff
X-Ratelimit-Limit: 4369
X-Ratelimit-Remaining: 4369
X-Xss-Protection: 1; mode=block
Accept-Ranges: bytes
Content-Length: 1308
Content-Type: application/json
ETag: "716439095f871b1b4cc92195baa67076"
Server: MinIO
Vary: Origin,Accept-Encoding
<BLOB>

MinIO Log - remote pipeline engine (shortened)

[REQUEST s3.HeadObject]
HEAD /file-bucket/testdata.json
[RESPONSE]
200 OK
X-Ratelimit-Limit: 4369
Accept-Ranges: bytes
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin,Accept-Encoding
X-Ratelimit-Remaining: 4369
X-Xss-Protection: 1; mode=block
Content-Length: 1308
Content-Type: application/json
ETag: "716439095f871b1b4cc92195baa67076"
Server: MinIO
X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
X-Amz-Request-Id: 188F79C3FBBC7AFD
X-Content-Type-Options: nosniff

Alternative attempt with S3 configuration via environment variables

Made this collapsible, because it's more of a follow-up problem

As I understand, it should also be possible to use Hops S3 implementation to access Minio (please correct me if I'm wrong) or atleast another S3 object store. So I tried configuring the connection for that.

• I tried using AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  • Got the following error:

Error browsing to location: 's3://file-bucket'
FileSystemException: Could not list the contents of folder "s3:///file-bucket".
Root cause: AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId;…

• I'm not sure how to override the S3 endpoint URL - does it always default to Amazon? - is this not possible?
• I tried various environment variables (AWS_S3_ENDPOINT, AWS_S3_ENDPOINT_URL, AWS_ENDPOINT_URL_S3, HOP_S3_ENDPOINT, HOP_S3_ENDPOINT_URL), but none had any effect.

Additional attempts

• tried overriding HOP_OPTIONS with -Daws.endpointUrl.s3, -Daws.endpoint and similiar, but no success
• created a .aws folder with config and credentials files in an attempt to override the S3 endpoint -> still no success
• created an Environment in Hop and set the variables for the S3 connection there as well, but it had no effect

Conclusion

• It works locally to read files from a MinIO bucket, but not remotely - so it's likely a configuration or context-related issue on the server side that I have
• I'm not sure if I'm missing a specific configuration or if this is a bug in Hop

I’ve tried all options I could find in the documentation/code and online or was suggested by AI tools, but haven’t managed to resolve the issue. Any help or pointers would be greatly appreciated!

Thanks in advance.

[tribyk]

Update:
I found the reason, why the remote execution did not read any data. It seems like the MinioConnectionDefinition metadata is not used by the Hop server, when executing the pipeline, although the Minio metadata is part of the XML the Hop UI is sending, when registering the pipeline (I checked that).
When I put the MinioConnectionDefinition in the servers default project directory, the remote pipeline can read the data/file from the bucket.

I would guess, that this behaviour is a bug, because for example metadata for a database connection works as expected. I configure the connection information in the UI and execute the pipeline remotely on the server. In this case the server just uses the metadata, which was sent by the UI, and reads/writes normally from/to the database. I would have expected, that the Minio metadata is used the same way.

So my new question is: is it intended, that the Minio metadata must be provided in the servers file system instead of the UIs Pipeline XML or is this a bug?

[hansva]

It's a bug that seems to be happening with multiple metadata types

[tribyk]

Thank you for clarifying that.
Is there any rough estimate or timeline for when this issue might be resolved? Just asking, so that I can plan accordingly. In the meantime I have the workaround with the server project directory.

[hansva]

We don't give timelines for individual tickets or problems, order of work depends a bit on the developer and their reasons, but usually boils down to. 1) things that pay the bills, 2) things that actively annoy or enjoy the developer, 3) the rest

After thinking for a bit... this is an issue that doens't have a ticket and is part of a decision that was made...

Creating filesystems only happens on startup/project switch because creating them has a bit of a loading kickback (it's a pretty expensive operation). When the server is started, we create the filesystems that have been configured, but they are not being reloaded/refreshed when a pipeline or workflow is started. We might have to change that mechanism for executions via the server.

[mattcasters]

This shouldn't happen. Both the local server metadata and the one from the XML are taken into account when adding a pipeline to the server during remote execution. I would be very much interested in a bug report with reproduction path.