Interoperability with Minio

[cyberduck]

1c67bf1 created the issue

7.1.2 works without issues

7.2.0 onwards keeps getting an error on just simply listing a bucket contents

Request Error: null. Please contact your web hosting service provider for assistance.

server: minio 2019-10-12T01:39:57Z with --compat option

[cyberduck]

1c67bf1 commented

THIS IS THE CONSOLE DEBUG OUTPUT IF I CONNECT TO THE MINIO SERVER IP AND PORT AND THEN ASK TO LIST CONTENTS TO MYBUCKET123

default	11:56:20.321525+0000	Cyberduck	Not supported to read versioning configuration for Path{path='/mybucket123', type=[directory, volume]} Request Error: null

default	11:56:20.323044+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'}

default	11:56:20.323921+0000	Cyberduck	Failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:511)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3041)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3016)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2682)
	at ch.cyberduck.ui.cocoa

default	11:56:21.481405+0000	Cyberduck	Ignore return code -1

default	11:56:21.512844+0000	Cyberduck	Missing result for worker SessionListWorker{directory=Path{path='/mybucket123', type=[directory, volume]}}. Use default value.

[cyberduck]

1c67bf1 commented

THIS IS THE CONSOLE DEBUG OUTPUT IF I USE THE VIRTUAL HOST METHOD IN MINIO e.g http://mybucket123.s3.myserver.com:9000

i have checked the XML output being returned from the minio server using Charles Proxy and its a VALID XML that gets returned with Contents listed for files

default	12:40:18.754559+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'}

default	12:40:18.755397+0000	Cyberduck	Failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$29.run(BrowserController.java:3070)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$32.run(BrowserController.java:3189)
	at ch.cyberduck.ui.cocoa.controller.Bro
default	12:40:20.375029+0000	Cyberduck	Ignore return code -1
default	12:40:20.392688+0000	Cyberduck	Missing result for worker MountWorker{cache=ch.cyberduck.core.ReverseLookupCache@3eee3e2b}. Use default value.

[cyberduck]

@dkocher commented

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

[cyberduck]

1c67bf1 commented

Replying to [comment:3 dkocher]:

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

this is already enabled, as i couldnt access it first time round without setting the MINIO_DOMAIN in docker as it generated a different error

but after enabling it, im now getting this XML error as listed above

[cyberduck]

1c67bf1 commented

ive just tried 7.1.2 with virtual-host-style and that doesn't work either

just keeps getting same error

Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler. Please contact your web hosting service provider for assistance

but the log drawer show it returns HTTP/1.1 200 OK with a valid XML too

[cyberduck]

@dkocher commented

Regression from 9523997.

[cyberduck]

@dkocher commented

Can you provide the captured XML response?

[cyberduck]

1c67bf1 commented

Replying to [comment:9 dkocher]:

Can you provide the captured XML response?

https://pastebin.com/7knB6DcW

[cyberduck]

1c67bf1 commented

ok i think i might have found the problem and maybe a bug but will try explain

1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,
   so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

2. if i use a hostname with minio and have virtual-host-style DISABLED, cyberduck ALWAYS trys accessing the bucket using virtual-host-style regardless e.g: http://wtf123.s3.myserver.com:12345/
   so i think it would be nice to have an option inside the connection settings to specify weather to use virtual-host-style or path-style (maybe per connection?)

3. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/
   i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

[cyberduck]

@dkocher commented

Thanks for summarizing the issues. Accessing via the hostname set with MINIO_DOMAIN this cannot be reproduced.

[cyberduck]

@dkocher commented

The error when accessing using the bucket name in Host configuration setting of the bookmark is that we expect the server to return a bucket list and not an object list.

[cyberduck]

@dkocher commented

We have requested a documentation update for Minio in minio/docs#280.

[cyberduck]

1c67bf1 commented

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

[cyberduck]

@dkocher commented

Replying to [comment:15 si458]:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?
We will see if we can support this via an option in the connection profiles.

[cyberduck]

1c67bf1 commented

Replying to [comment:16 dkocher]:

Replying to [comment:15 si458]:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?
We will see if we can support this via an option in the connection profiles.

thank you that would be helpful!

or even a tick box saying saying 'USE PATH-STYLE' in the edit bookmarks window/open connection window

[cyberduck]

@dkocher commented

Replying to [comment:11 si458]:

ok i think i might have found the problem and maybe a bug but will try explain

1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,
   so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

In e5e2a97.

[cyberduck]

@dkocher commented

Pending change for Failed to parse XML document error to detect if we are connecting to a specific bucket and should list objects instead of all buckets.

[cyberduck]

@dkocher commented

Replying to [comment:11 si458]:

3. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/
   i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Set the bucket name in the Path setting in the bookmark instead.

[cyberduck]

@ylangisc commented

In 586fc49.

[cyberduck]

d9b8c46 commented

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?

[cyberduck]

@dkocher commented

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?
Is this Ceph instance publicly reachable over the Internet?

[cyberduck]

d9b8c46 commented

Replying to [comment:25 dkocher]:

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?
Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

[cyberduck]

@dkocher commented

Replying to [comment:26 dimm00]:

Replying to [comment:25 dkocher]:

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?
Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

Awesome. Please write to [mailto:support@cyberduck.io].

[cyberduck]

@dkocher commented

Duplicate for #10956.

[cyberduck]

83f1b57 commented

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?

Was that solved? I'm seeing the same problem with Ceph here.

[cyberduck]

1c67bf1 commented

i dont have this issue anymore as its been fixed in latest cyberduck version using the S3 (HTTPS) profile from the minio help page

the only issue i found was when using my home broadband 'virgin media' they take over my dns records,

so when cyberduck was looking for 'mybucket.s3.myserver.com' is should of returned blank A record as i only have an A record for 's3.myserver.com' which ment cyberduck would then reverted back to using the path format,

but 'virgin media' kept returning an IP address of theres for there stupid advanced search error page, so cyberduck kept trying to connect to them and just timing out instead of an error

once i disabled the 'advanced search error page' and rebooted our router and cleared my dns cache on laptop,

cyberduck started working without any issues

[cyberduck]

83f1b57 commented

Aha. What worked for me now, was to create a bookmark and add the bucket-url to the PATH there. So, worksforme now. Thanks.

[cyberduck]

@dkocher commented

We have another fix for Minio (#11018) that will be included in version 7.3.1.

[cyberduck]

@dkocher commented

Documentation for Minio in Minio Cloud Storage.