Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Folders gone AWOL in Nextcloud 13.02 #5916

Closed
dragon2611 opened this issue May 10, 2018 · 17 comments

Comments

@dragon2611
Copy link

commented May 10, 2018

Upgraded minio to RELEASE.2018-05-10T00-00-42Z - Since doing so all "folders" in the bucket have vanished from nextcloud as such Nextcloud can only see files directly in the bucket root.

Expected Behavior

Folders should show up.

Current Behavior

Folders are missing.

Possible Solution

Steps to Reproduce (for bugs)

Upgrade to RELEASE.2018-05-10T00-00-42Z, mount Minio as S3 external storage in NC 13.02

Context

Minio is used as a storage backend for my NC server, this makes nextcloud largely unusable.

Your Environment

RELEASE.2018-05-10T00-00-42Z on Ubuntu 18.04-LTS

@vadmeste

This comment has been minimized.

Copy link
Member

commented May 10, 2018

@dragon2611 my first testing didn't show any error, this is how I configured NextCloud 13.02

  'objectstore' => array(
          'class' => 'OC\\Files\\ObjectStore\\S3',
          'arguments' => array(
                  'bucket' => 'nextcloud',
                  'autocreate' => true,
                  'key'    => 'minio',
                  'secret' => 'minio123',
                  'hostname' => 'localhost',
                  'port' => 9000,
                  'use_ssl' => false,
                  'region' => 'us-east-1',
                  // required for some non amazon s3 implementations
                  'use_path_style'=> true,
          ),
  ),

Can you test the previous release and see if the bug disappears or not ? https://dl.minio.io/server/minio/release/linux-amd64/archive/minio.RELEASE.2018-05-04T23-13-12Z

@deekoder

This comment has been minimized.

Copy link
Contributor

commented May 10, 2018

ping @dragon2611 can you confirm that it works now?

@dragon2611

This comment has been minimized.

Copy link
Author

commented May 10, 2018

I'll retest when I can, hopefully this morning/tomorrow depending on people's timezone.

Noticed access denied errors on console if that's any help to the devs, on the underlaying filesystem the files are owned by the user running minio and both the minio webui and MC can see them.

There is a haproxy (opnsense) Infront doing ssl termination with a wildcard letsencrypt cert.

This worked fine on the previous version so I'm pretty sure it's something in my setup has interacted badly with the update.

Nextcloud is 13.02 running as a snap on Ubuntu 18.04.

Note: I'm on a mobile device at present please excuse typos.etc

@dragon2611

This comment has been minimized.

Copy link
Author

commented May 11, 2018

Retested, Problem still persists with latest release

If I run RELEASE.2018-05-04T23-13-12Z or older then NC sees folders, if I run anything newer than that release it goes missing.

It does also spit out a load of file access denied messages

`API: HeadObject(bucket=nextcloud, object=test)
Time: 10:06:33 BST 05/11/2018
RemoteHost: 10.0.232.254:16972
UserAgent: aws-sdk-php/3.35.2 Guzzle/5.3.1 curl/7.47.0 PHP/7.1.17
Error: file access denied
1: cmd/fs-v1-helpers.go:248:cmd.fsStatFile()
2: cmd/fs-v1.go:604:cmd.(*FSObjects).getObjectInfo()
3: cmd/fs-v1.go:633:cmd.(*FSObjects).getObjectInfoWithLock()
4: cmd/fs-v1.go:638:cmd.(*FSObjects).GetObjectInfo()
5: cmd/object-handlers.go:82:cmd.(ObjectLayer).GetObjectInfo-fm()
6: cmd/object-handlers.go:258:cmd.objectAPIHandlers.HeadObjectHandler()
7: cmd/api-router.go:59:cmd.(objectAPIHandlers).HeadObjectHandler-fm()
8: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()

`

I also changed the endpoint URL in nextcloud to use the Minio server directly incase it was HAproxy causing an issue, but same problem occurs.

The working version also spits out the file access denied error to the console (even though I can actually access the files fine)

@harshavardhana

This comment has been minimized.

Copy link
Member

commented May 11, 2018

`API: HeadObject(bucket=nextcloud, object=test)
Time: 10:06:33 BST 05/11/2018
RemoteHost: 10.0.232.254:16972
UserAgent: aws-sdk-php/3.35.2 Guzzle/5.3.1 curl/7.47.0 PHP/7.1.17
Error: file access denied
1: cmd/fs-v1-helpers.go:248:cmd.fsStatFile()
2: cmd/fs-v1.go:604:cmd.(*FSObjects).getObjectInfo()
3: cmd/fs-v1.go:633:cmd.(*FSObjects).getObjectInfoWithLock()
4: cmd/fs-v1.go:638:cmd.(*FSObjects).GetObjectInfo()
5: cmd/object-handlers.go:82:cmd.(ObjectLayer).GetObjectInfo-fm()
6: cmd/object-handlers.go:258:cmd.objectAPIHandlers.HeadObjectHandler()
7: cmd/api-router.go:59:cmd.(objectAPIHandlers).HeadObjectHandler-fm()
8: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()

We will verify and let you know, do you have nextcloud configuration that you can share with us? are you using Minio as primary storage or secondary storage?

@dragon2611

This comment has been minimized.

Copy link
Author

commented May 11, 2018

Minio is mounted as external storage in nextcloud.

The only change I've made to the snap config was to increase the php memory limit.
nextcloud 13.0.2snap1 6916 13/candidate nextcloud -

@vadmeste

This comment has been minimized.

Copy link
Member

commented May 11, 2018

Minio is mounted as external storage in nextcloud.

Ah okay, I just knew about the external storage thing, I will test it right now.

@vadmeste

This comment has been minimized.

Copy link
Member

commented May 11, 2018

This is not a regression per-se, it was more like a luck to have NextCloud external storage working with Minio before since this latter doesn't fully support empty directories yet.. The same thing applies for all applications that are working with S3 as regular file system. I will update the team and discuss this.

@dragon2611

This comment has been minimized.

Copy link
Author

commented May 11, 2018

The directories are not empty however they all contain content.

@dragon2611

This comment has been minimized.

Copy link
Author

commented May 13, 2018

@vadmeste is seems minio.RELEASE.2018-04-19T22-54-58Z is actually the latest one that will work with nextcloud, newer releases upto RELEASE.2018-05-04T23-13-12Z show the folders but I get a request error if I try and actually open any of the files contained within.

The latest release won't show any folders but will allow files in the root of the bucket to be downloaded.

@harshavardhana

This comment has been minimized.

Copy link
Member

commented May 13, 2018

Folders being shown was purely incidental due to an implementation issue as mentioned by @vadmeste we will see how to bring it back, but there are no guarantees right now.

@ThisIsQasim

This comment has been minimized.

Copy link

commented Sep 14, 2018

Tried with the latest build 2018-09-12T18:49:56Z. Can confirm that minio still doesn't support nested objects under Nextcloud.

@ThisIsQasim

This comment has been minimized.

Copy link

commented Oct 12, 2018

I can understand that this is free software and is provided on an as-is basis, that there are no guarantees it'll work as expected and that the maintainers have limited time so they have to prioritize issues but can you at least acknowledge that an issue exists by not putting the working as intended tag on it?

@harshavardhana

This comment has been minimized.

Copy link
Member

commented Oct 12, 2018

I can understand that this is free software and is provided on an as-is basis, that there are no guarantees it'll work as expected and that the maintainers have limited time so they have to prioritize issues but can you at least acknowledge that an issue exists by not putting the working as intended tag on it?

It is a correct tag, it is already explained why it worked in the previous releases which were purely an accident. It is not possible for us to support empty directories in the traditional sense and currently we are not planning on supporting such requirements. "working as intended" means that there are currently no plans on addressing this and software is behaving as it is intended.

@dragon2611

This comment has been minimized.

Copy link
Author

commented Oct 12, 2018

Typing on phone so sorry for typos more so that usual.

I think the question here becomes is how does owncloud behave with amazon s3 and if the problem manifests itself in s3 then its ownclouds problem and if it doesn’t then either amazon have done something to mask the issue or minios s3 api compatibly isn’t on par with real s3.

Sadly I stopped using owncloud a while back for Seafile so I’m not ina position to test at the moment

@harshavardhana

This comment has been minimized.

Copy link
Member

commented Oct 12, 2018

I think the question here becomes is how does owncloud behave with amazon s3 and if the problem manifests itself in s3 then its ownclouds problem and if it doesn’t then either amazon have done something to mask the issue or minios s3 api compatibly isn’t on par with real s3.

We are not behaving in exact terms as AWS S3, we cannot because

  • We use regular filesystem on the backend and it is not possible to address this without requiring
  • Supporting directories on object storage is an esoteric use-case. Most applications don't need we have implemented a particular style of the requirement for Spark and Hadoop which fulfills the overall requirement.

Closing as per last comment from @dragon2611

@ThisIsQasim

This comment has been minimized.

Copy link

commented Oct 17, 2018

It is not possible for us to support empty directories in the traditional sense

It is not just empty directories. Any directories, empty or otherwise, that are placed inside the bucket root fail to show up.

I am sorry if I misunderstood the "working as intended" label. My understanding was that the intention here was to provide an S3 compatible object storage. So anything that works with the real S3 should ideally work with minio but thanks for clearing up that that was never the intention and the goal is to provide compatibility in loose terms not exact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.