Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interoperability with Minio #10888

Closed
cyberduck opened this issue Nov 27, 2019 · 29 comments
Closed

Interoperability with Minio #10888

cyberduck opened this issue Nov 27, 2019 · 29 comments

Comments

@cyberduck
Copy link
Collaborator

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 created the issue

7.1.2 works without issues

7.2.0 onwards keeps getting an error on just simply listing a bucket contents

Request Error: null. Please contact your web hosting service provider for assistance.

server: minio 2019-10-12T01:39:57Z with --compat option

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

THIS IS THE CONSOLE DEBUG OUTPUT IF I CONNECT TO THE MINIO SERVER IP AND PORT AND THEN ASK TO LIST CONTENTS TO MYBUCKET123

default	11:56:20.321525+0000	Cyberduck	Not supported to read versioning configuration for Path{path='/mybucket123', type=[directory, volume]} Request Error: null

default	11:56:20.323044+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'}

default	11:56:20.323921+0000	Cyberduck	Failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:511)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3041)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3016)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2682)
	at ch.cyberduck.ui.cocoa

default	11:56:21.481405+0000	Cyberduck	Ignore return code -1

default	11:56:21.512844+0000	Cyberduck	Missing result for worker SessionListWorker{directory=Path{path='/mybucket123', type=[directory, volume]}}. Use default value.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

THIS IS THE CONSOLE DEBUG OUTPUT IF I USE THE VIRTUAL HOST METHOD IN MINIO e.g http://mybucket123.s3.myserver.com:9000

i have checked the XML output being returned from the minio server using Charles Proxy and its a VALID XML that gets returned with Contents listed for files

default	12:40:18.754559+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'}

default	12:40:18.755397+0000	Cyberduck	Failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$29.run(BrowserController.java:3070)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$32.run(BrowserController.java:3189)
	at ch.cyberduck.ui.cocoa.controller.Bro
default	12:40:20.375029+0000	Cyberduck	Ignore return code -1
default	12:40:20.392688+0000	Cyberduck	Missing result for worker MountWorker{cache=ch.cyberduck.core.ReverseLookupCache@3eee3e2b}. Use default value.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

Replying to [comment:3 dkocher]:

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

this is already enabled, as i couldnt access it first time round without setting the MINIO_DOMAIN in docker as it generated a different error

but after enabling it, im now getting this XML error as listed above

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

ive just tried 7.1.2 with virtual-host-style and that doesn't work either

just keeps getting same error

Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler. Please contact your web hosting service provider for assistance

but the log drawer show it returns HTTP/1.1 200 OK with a valid XML too

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

Regression from 9523997.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

Can you provide the captured XML response?

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

Replying to [comment:9 dkocher]:

Can you provide the captured XML response?

https://pastebin.com/7knB6DcW

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

ok i think i might have found the problem and maybe a bug but will try explain

  1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,
    so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

  2. if i use a hostname with minio and have virtual-host-style DISABLED, cyberduck ALWAYS trys accessing the bucket using virtual-host-style regardless e.g: http://wtf123.s3.myserver.com:12345/
    so i think it would be nice to have an option inside the connection settings to specify weather to use virtual-host-style or path-style (maybe per connection?)

  3. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/
    i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

Thanks for summarizing the issues. Accessing via the hostname set with MINIO_DOMAIN this cannot be reproduced.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

The error when accessing using the bucket name in Host configuration setting of the bookmark is that we expect the server to return a bucket list and not an object list.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

We have requested a documentation update for Minio in minio/docs#280.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

@dkocher commented

Replying to [comment:15 si458]:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?
We will see if we can support this via an option in the connection profiles.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 27, 2019

1c67bf1 commented

Replying to [comment:16 dkocher]:

Replying to [comment:15 si458]:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?
We will see if we can support this via an option in the connection profiles.

thank you that would be helpful!

or even a tick box saying saying 'USE PATH-STYLE' in the edit bookmarks window/open connection window

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 28, 2019

@dkocher commented

Replying to [comment:11 si458]:

ok i think i might have found the problem and maybe a bug but will try explain

  1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,
    so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

In e5e2a97.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 28, 2019

@dkocher commented

Pending change for Failed to parse XML document error to detect if we are connecting to a specific bucket and should list objects instead of all buckets.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Nov 28, 2019

@dkocher commented

Replying to [comment:11 si458]:

  1. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/
    i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Set the bucket name in the Path setting in the bookmark instead.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Dec 9, 2019

@ylangisc commented

In 586fc49.

Loading

@cyberduck cyberduck closed this Dec 9, 2019
@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 7, 2020

d9b8c46 commented

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 10, 2020

@dkocher commented

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?
Is this Ceph instance publicly reachable over the Internet?

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 11, 2020

d9b8c46 commented

Replying to [comment:25 dkocher]:

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?
Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 22, 2020

@dkocher commented

Replying to [comment:26 dimm00]:

Replying to [comment:25 dkocher]:

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?
Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

Awesome. Please write to [mailto:support@cyberduck.io].

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 22, 2020

@dkocher commented

Duplicate for #10956.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Apr 24, 2020

83f1b57 commented

Replying to [comment:24 dimm00]:

Trying to make cyberduck work with ceph S3, which only supports path style
Tried using IP, which broke the cert and didn't work either
Is there a way to reenable path style S3?

Was that solved? I'm seeing the same problem with Ceph here.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Apr 24, 2020

1c67bf1 commented

i dont have this issue anymore as its been fixed in latest cyberduck version using the S3 (HTTPS) profile from the minio help page

the only issue i found was when using my home broadband 'virgin media' they take over my dns records,

so when cyberduck was looking for 'mybucket.s3.myserver.com' is should of returned blank A record as i only have an A record for 's3.myserver.com' which ment cyberduck would then reverted back to using the path format,

but 'virgin media' kept returning an IP address of theres for there stupid advanced search error page, so cyberduck kept trying to connect to them and just timing out instead of an error

once i disabled the 'advanced search error page' and rebooted our router and cleared my dns cache on laptop,

cyberduck started working without any issues

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Apr 24, 2020

83f1b57 commented

Aha. What worked for me now, was to create a bookmark and add the bucket-url to the PATH there. So, worksforme now. Thanks.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Apr 24, 2020

@dkocher commented

We have another fix for Minio (#11018) that will be included in version 7.3.1.

Loading

@cyberduck
Copy link
Collaborator Author

@cyberduck cyberduck commented Jan 18, 2021

@dkocher commented

Documentation for Minio in Minio Cloud Storage.

Loading

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants