New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared files, oC 9.1 adding "(2)" at the end #25718

Closed
Mysterycz opened this Issue Aug 8, 2016 · 180 comments

Comments

Projects
None yet
7 participants
@Mysterycz

Mysterycz commented Aug 8, 2016

Steps to reproduce (I don't know how to reproduce it, I need to investigate more)

  1. Use oC 9.1
  2. Share some files
  3. Wait untill some random magic happens and oC will add to all shared files some "(2)"

I found same issue here, but these steps to reproduce does not work:
#10736

Expected behaviour

oC should not rename shared files ...

Actual behaviour

oC adds "(2)" to all shared files (owner does not see renamed + "(2)" files

Server configuration

Ubuntu 14.04
Apache2
MySQL
PHP 5.6
oC 9.1
Updated from 9.04
Installed from official website manually

Integrity check: No errors have been found.

List of activated apps:

  - activity: 2.3.2
  - comments: 0.3.0
  - dav: 0.2.5
  - encryption: 1.3.0
  - external: 1.2
  - federatedfilesharing: 0.3.0
  - federation: 0.1.0
  - files: 1.5.1
  - files_pdfviewer: 0.8.1
  - files_sharing: 0.10.0
  - files_texteditor: 2.1
  - files_trashbin: 0.9.0
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 15.0.0
  - notifications: 0.3.0
  - ownnote: 1.08
  - provisioning_api: 0.5.0
  - systemtags: 0.3.0
  - templateeditor: 0.1
  - updatenotification: 0.2.1
  - user_ldap: 0.9.0
Disabled:
  - files_antivirus
  - files_external
  - user_external

The content of config/config.php:

{
    "system": {
        "updatechecker": false,
        "instanceid": "***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            ***
        ],
        "datadirectory": "\/home\/owncloud\/owncloud",
        "overwrite.cli.url": "*",
        "dbtype": "mysql",
        "version": "9.1.0.15",
        "default_language": "cz",
        "dbname": "ownCloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "ldapIgnoreNamingRules": false,
        "loglevel": 2,
        "appstore.experimental.enabled": true,
        "theme": "HZS",
        "maintenance": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "mail_from_address": "owncloud",
        "mail_smtpmode": "php",
        "mail_domain": "**"
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: yes

Are you using an external user-backend, if yes which one: LDAP

Client configuration

Browser: Opera 39
Operating system: Windows 10

@elektro-wolle

This comment has been minimized.

Show comment
Hide comment
@elektro-wolle

elektro-wolle Aug 8, 2016

Have you shared the folder to a group? Then it might be a duplicate to #24575

elektro-wolle commented Aug 8, 2016

Have you shared the folder to a group? Then it might be a duplicate to #24575

@Mysterycz

This comment has been minimized.

Show comment
Hide comment
@Mysterycz

Mysterycz Aug 8, 2016

Yes, some of files/folders were shared folders to groups. Some of them were shared files/folders to individual person account. It affects all of them. I don't think it is duplicate to #24575, because shared folders are only renamed not duplicated. It could be related to this issue ofcourse, i don't know.

Mysterycz commented Aug 8, 2016

Yes, some of files/folders were shared folders to groups. Some of them were shared files/folders to individual person account. It affects all of them. I don't think it is duplicate to #24575, because shared folders are only renamed not duplicated. It could be related to this issue ofcourse, i don't know.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Aug 9, 2016

Member

Without the exact sharing scenario it will be impossible to tell.

But I'm pretty confident that the fix for #24575 should cover most cases. The only exception is #25564

For #24575 a fix will be released with the upcoming 9.1.1. The fix is here #25534.

I'm going to close this for now. If you are able to find out the exact sharing scenario, you can first check if it is covered in the scenario list here https://github.com/owncloud/core/pull/25534/files#diff-b4888b15e6a7d80fe706602129494bfbR713

If it isn't, please reopen and I'll add it there for testing.

Member

PVince81 commented Aug 9, 2016

Without the exact sharing scenario it will be impossible to tell.

But I'm pretty confident that the fix for #24575 should cover most cases. The only exception is #25564

For #24575 a fix will be released with the upcoming 9.1.1. The fix is here #25534.

I'm going to close this for now. If you are able to find out the exact sharing scenario, you can first check if it is covered in the scenario list here https://github.com/owncloud/core/pull/25534/files#diff-b4888b15e6a7d80fe706602129494bfbR713

If it isn't, please reopen and I'll add it there for testing.

@PVince81 PVince81 closed this Aug 9, 2016

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 21, 2016

Hi !

I have exactly the same problem since 9.1.0
I wait 9.1.1 to upgrade my cloud, but the result is the same.

This is my oc_share table on phpmyadmin :

oc_share

And an other screen of a user :

capture d ecran 2016-09-21 a 18 27 29

It's strange, the bug is only on ldap account, and not all ...
When i remove share, and share it again the bug back ...
After clean tables and when i stop app files_sharing, the database are correct on 'oc_share' ...

(sorry for my bad english ...)

`Enabled:

  • activity: 2.3.2
  • bookmarks: true
  • calendar: true
  • comments: 0.3.0
  • contacts: true
  • dav: 0.2.6
  • documents: true
  • external: 1.2
  • federatedfilesharing: 0.3.0
  • federation: 0.1.0
  • files: 1.5.1
  • files_pdfviewer: 0.8.1
  • files_sharing: 0.10.0
  • files_texteditor: 2.1
  • files_trashbin: 0.9.0
  • files_versions: 1.3.0
  • files_videoplayer: 0.9.8
  • firstrunwizard: 1.1
  • gallery: 15.0.0
  • notifications: 0.3.0
  • provisioning_api: 0.5.0
  • systemtags: 0.3.0
  • templateeditor: 0.1
  • updatenotification: 0.2.1
  • user_ldap: 0.9.0
    Disabled:
  • encryption
  • files_antivirus
  • files_external
  • user_external`

<?php $CONFIG = array ( 'updatechecker' => false, 'instanceid' => '***REMOVED SENSITIVE VALUE***', 'passwordsalt' => '***REMOVED SENSITIVE VALUE***', 'secret' => '***REMOVED SENSITIVE VALUE***', 'trusted_domains' => array ( 0 => '***REMOVED SENSITIVE VALUE***', ), 'datadirectory' => '/data/profs', 'overwrite.cli.url' => '***REMOVED SENSITIVE VALUE***', 'dbtype' => 'mysql', 'version' => '9.1.1.3', 'dbname' => 'owncloud', 'dbhost' => 'localhost', 'dbtableprefix' => 'oc_', 'dbuser' => 'owncloud', 'dbpassword' => '***REMOVED SENSITIVE VALUE***', 'logtimezone' => 'UTC', 'installed' => true, 'theme' => 'dgm', 'maintenance' => false, 'loglevel' => 0, );

Server configuration
Ubuntu 16.04
Apache 2.4.18
MySQL 5.7.15
PHP 7.0.8
oC 9.1.1
Updated from 9.1.0 <- 8.2.X

ccheveaux commented Sep 21, 2016

Hi !

I have exactly the same problem since 9.1.0
I wait 9.1.1 to upgrade my cloud, but the result is the same.

This is my oc_share table on phpmyadmin :

oc_share

And an other screen of a user :

capture d ecran 2016-09-21 a 18 27 29

It's strange, the bug is only on ldap account, and not all ...
When i remove share, and share it again the bug back ...
After clean tables and when i stop app files_sharing, the database are correct on 'oc_share' ...

(sorry for my bad english ...)

`Enabled:

  • activity: 2.3.2
  • bookmarks: true
  • calendar: true
  • comments: 0.3.0
  • contacts: true
  • dav: 0.2.6
  • documents: true
  • external: 1.2
  • federatedfilesharing: 0.3.0
  • federation: 0.1.0
  • files: 1.5.1
  • files_pdfviewer: 0.8.1
  • files_sharing: 0.10.0
  • files_texteditor: 2.1
  • files_trashbin: 0.9.0
  • files_versions: 1.3.0
  • files_videoplayer: 0.9.8
  • firstrunwizard: 1.1
  • gallery: 15.0.0
  • notifications: 0.3.0
  • provisioning_api: 0.5.0
  • systemtags: 0.3.0
  • templateeditor: 0.1
  • updatenotification: 0.2.1
  • user_ldap: 0.9.0
    Disabled:
  • encryption
  • files_antivirus
  • files_external
  • user_external`

<?php $CONFIG = array ( 'updatechecker' => false, 'instanceid' => '***REMOVED SENSITIVE VALUE***', 'passwordsalt' => '***REMOVED SENSITIVE VALUE***', 'secret' => '***REMOVED SENSITIVE VALUE***', 'trusted_domains' => array ( 0 => '***REMOVED SENSITIVE VALUE***', ), 'datadirectory' => '/data/profs', 'overwrite.cli.url' => '***REMOVED SENSITIVE VALUE***', 'dbtype' => 'mysql', 'version' => '9.1.1.3', 'dbname' => 'owncloud', 'dbhost' => 'localhost', 'dbtableprefix' => 'oc_', 'dbuser' => 'owncloud', 'dbpassword' => '***REMOVED SENSITIVE VALUE***', 'logtimezone' => 'UTC', 'installed' => true, 'theme' => 'dgm', 'maintenance' => false, 'loglevel' => 0, );

Server configuration
Ubuntu 16.04
Apache 2.4.18
MySQL 5.7.15
PHP 7.0.8
oC 9.1.1
Updated from 9.1.0 <- 8.2.X

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

Please describe the sharing scenario, who shares what with who.

Member

PVince81 commented Sep 22, 2016

Please describe the sharing scenario, who shares what with who.

@PVince81 PVince81 reopened this Sep 22, 2016

@PVince81 PVince81 added this to the 9.1.2 milestone Sep 22, 2016

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

Hello,

It's an LDAP user who shares a folder of its own owncloud another LDAP user or a local user.
We have not shared between several different cloud.
Just teachers of an educational department that shares them.

The user sees emits sharing the folder named correctly.
The '(2)' are only displayed for the user who receives the shares.

More use could not be simpler to owncloud sharing system to avoid using public clouds.
I hope you have provided enough information, I was not too used to report bugs.

Thx for your help ;)

ccheveaux commented Sep 22, 2016

Hello,

It's an LDAP user who shares a folder of its own owncloud another LDAP user or a local user.
We have not shared between several different cloud.
Just teachers of an educational department that shares them.

The user sees emits sharing the folder named correctly.
The '(2)' are only displayed for the user who receives the shares.

More use could not be simpler to owncloud sharing system to avoid using public clouds.
I hope you have provided enough information, I was not too used to report bugs.

Thx for your help ;)

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

Is it really just a simple case of "ldap_user1" shared folder with "ldap_user2" and when ldap_user2 views the file list it keeps adding a "(2)" ? I thought there was more complex sharing involved like group shares, reshares, etc.

Member

PVince81 commented Sep 22, 2016

Is it really just a simple case of "ldap_user1" shared folder with "ldap_user2" and when ldap_user2 views the file list it keeps adding a "(2)" ? I thought there was more complex sharing involved like group shares, reshares, etc.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

Exactly !

No teacher uses sharing group because we are not much use this Cloud.
And for the reshares i disabled it in the settings.

ccheveaux commented Sep 22, 2016

Exactly !

No teacher uses sharing group because we are not much use this Cloud.
And for the reshares i disabled it in the settings.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

Okay, thanks for the info. I'll see if I can reproduce it locally with this scenario.

Member

PVince81 commented Sep 22, 2016

Okay, thanks for the info. I'll see if I can reproduce it locally with this scenario.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

My steps:

  1. Setup LDAP in OC (used the docker from https://github.com/owncloud/administration/tree/master/ldap-testing)
  2. Pick two users "user1" and "user2" from the LDAP
  3. Login as "user2" then "user1" to initialize their homes
  4. Create a folder "test"
  5. Share "test" with "user2" with default permissions
  6. Login as "user2"
  7. Look at the file list
  8. Access the file list with Webdav

The folder properly appears as "test" for me.

Member

PVince81 commented Sep 22, 2016

My steps:

  1. Setup LDAP in OC (used the docker from https://github.com/owncloud/administration/tree/master/ldap-testing)
  2. Pick two users "user1" and "user2" from the LDAP
  3. Login as "user2" then "user1" to initialize their homes
  4. Create a folder "test"
  5. Share "test" with "user2" with default permissions
  6. Login as "user2"
  7. Look at the file list
  8. Access the file list with Webdav

The folder properly appears as "test" for me.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

Thank you so much for your help.
In fact I do not understand why it does not affect everyone.
And above all I did not bug on another cloud that I use personal way with 2-3 users and certainly not a lot of sharing, but no LDAP users.

Another thing I noticed is that files_sharing a lot of mistakes when I made the command :
"sudo -u www-data occ php app: check code files_sharing"
This does not surrement prevents its operation, but this is the only app with much error ...

And I have the same errors on my personal ownCloud with this command.

check-code_files_sharing

check-code_files_sharing2

ccheveaux commented Sep 22, 2016

Thank you so much for your help.
In fact I do not understand why it does not affect everyone.
And above all I did not bug on another cloud that I use personal way with 2-3 users and certainly not a lot of sharing, but no LDAP users.

Another thing I noticed is that files_sharing a lot of mistakes when I made the command :
"sudo -u www-data occ php app: check code files_sharing"
This does not surrement prevents its operation, but this is the only app with much error ...

And I have the same errors on my personal ownCloud with this command.

check-code_files_sharing

check-code_files_sharing2

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

@ccheveaux your are using the wrong code checker, this is only to validate whether an app's code is clean (yes, the code of files_sharing isn't clean yet...). Try going to the admin page instead it will verify code integrity. Or on the CLI, the command you wanted is occ integrity:check-app

Member

PVince81 commented Sep 22, 2016

@ccheveaux your are using the wrong code checker, this is only to validate whether an app's code is clean (yes, the code of files_sharing isn't clean yet...). Try going to the admin page instead it will verify code integrity. Or on the CLI, the command you wanted is occ integrity:check-app

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

@ccheveaux I see in your oc_share table that not all folders received a "(2)". Are the ones without "(2)" from non-LDAP users ? Would be good to analyze the difference between the working ones and broken ones.

Member

PVince81 commented Sep 22, 2016

@ccheveaux I see in your oc_share table that not all folders received a "(2)". Are the ones without "(2)" from non-LDAP users ? Would be good to analyze the difference between the working ones and broken ones.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

I also did the test to create shares with other users tests.
And I have no problem renaming.

My main problem is that I would find how to fix shares that have this bug.
Unfortunately I can not find the variable that causes it ...
And as soon as I tried to clean directly in table 'oc_share' them '(2)' back faster or slower, as if there was a cache or a script that was automatically ...

Ca become annoying, and I'm afraid that even with a clean reinstallation, when I will restore the database, the problem will come back ...

ccheveaux commented Sep 22, 2016

I also did the test to create shares with other users tests.
And I have no problem renaming.

My main problem is that I would find how to fix shares that have this bug.
Unfortunately I can not find the variable that causes it ...
And as soon as I tried to clean directly in table 'oc_share' them '(2)' back faster or slower, as if there was a cache or a script that was automatically ...

Ca become annoying, and I'm afraid that even with a clean reinstallation, when I will restore the database, the problem will come back ...

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

Are the users without "(2)" users who haven't logged in since a long time ? Because the logic that triggers duplicate detection is done when the filesystem for the share recipient is setup.

Member

PVince81 commented Sep 22, 2016

Are the users without "(2)" users who haven't logged in since a long time ? Because the logic that triggers duplicate detection is done when the filesystem for the share recipient is setup.

@elektro-wolle

This comment has been minimized.

Show comment
Hide comment
@elektro-wolle

elektro-wolle Sep 22, 2016

Since I've had the same problem as @ccheveaux with LDAP-Users on 9.1:
The folder was shared by an LDAP User and some of the LDAP Users this folder is shared to, see the " (2)". After several reloads in the web-frontend, the " (2)" switches to " (2) (2)" and sometimes (maybe, if a third user accesses the page) to " (3)".
It doesn't matter if the folder is shared via a LDAP-group or directly to another user.

The main problem is the data-corruption on the client side if the owncloud-client is active. The client propagates the rename to the connected computer regardless of any open files, leaving sometime the original folder (without " (2)") due to the open file and the renamed folder.

A rename within the browser is only possible if the name is shorter then 512 chars. Sometimes this is not possible even after a few minutes due to the long " (2) (2) (2) (2) (2) ... (2) (2)" name.

elektro-wolle commented Sep 22, 2016

Since I've had the same problem as @ccheveaux with LDAP-Users on 9.1:
The folder was shared by an LDAP User and some of the LDAP Users this folder is shared to, see the " (2)". After several reloads in the web-frontend, the " (2)" switches to " (2) (2)" and sometimes (maybe, if a third user accesses the page) to " (3)".
It doesn't matter if the folder is shared via a LDAP-group or directly to another user.

The main problem is the data-corruption on the client side if the owncloud-client is active. The client propagates the rename to the connected computer regardless of any open files, leaving sometime the original folder (without " (2)") due to the open file and the renamed folder.

A rename within the browser is only possible if the name is shorter then 512 chars. Sometimes this is not possible even after a few minutes due to the long " (2) (2) (2) (2) (2) ... (2) (2)" name.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

Those with no (2) are on the screenshot is a sharing "user_ldap->public" or "user_ldap-> user_ldap".

I am currently doing a share with a teacher who is not connected for several weeks.

Yesterday, after cleaning (2) 'oc_sharing', this folder is incremented very quickly.
(Maybe because I had freshly to reinstall the owncloud package)
Today I clean and (2) do not come back yet, but they return to users connected at the moment with the owncloud desktop client.

ccheveaux commented Sep 22, 2016

Those with no (2) are on the screenshot is a sharing "user_ldap->public" or "user_ldap-> user_ldap".

I am currently doing a share with a teacher who is not connected for several weeks.

Yesterday, after cleaning (2) 'oc_sharing', this folder is incremented very quickly.
(Maybe because I had freshly to reinstall the owncloud package)
Today I clean and (2) do not come back yet, but they return to users connected at the moment with the owncloud desktop client.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

That could be an explanation indeed @elektro-wolle !
And how to you corrected the problem ?
You have to go on each client ?
And what manipulation you doing ?

ccheveaux commented Sep 22, 2016

That could be an explanation indeed @elektro-wolle !
And how to you corrected the problem ?
You have to go on each client ?
And what manipulation you doing ?

@elektro-wolle

This comment has been minimized.

Show comment
Hide comment
@elektro-wolle

elektro-wolle Sep 22, 2016

I've created a cronjob on the server, replacing the " (2)" in the name of the oc_share table and told the users to not use the desktop-client. This is not really a solution, I'm still waiting for 9.1.1 and currently evaluating other alternatives to owncloud.

elektro-wolle commented Sep 22, 2016

I've created a cronjob on the server, replacing the " (2)" in the name of the oc_share table and told the users to not use the desktop-client. This is not really a solution, I'm still waiting for 9.1.1 and currently evaluating other alternatives to owncloud.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

So if I understand you well cleaned by the cron table.
There is no possibility to stop by each client to make a deletion of the account on the desktop client and then recreate it?
It seems to me that there is no data loss if one like this?
Naturally between deleting and recreating the account on the desktop client, a table cleaning should be done I think.

ccheveaux commented Sep 22, 2016

So if I understand you well cleaned by the cron table.
There is no possibility to stop by each client to make a deletion of the account on the desktop client and then recreate it?
It seems to me that there is no data loss if one like this?
Naturally between deleting and recreating the account on the desktop client, a table cleaning should be done I think.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Sep 22, 2016

Can you share me you're script to replacing the (2) please ?

ccheveaux commented Sep 22, 2016

Can you share me you're script to replacing the (2) please ?

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

I don't think this is related to the desktop client.
When the issue does happen, it might influence the desktop client.
But the cause of the problem is not created by the desktop client, because as @elektro-wolle said even refreshing the web UI already adds a "(2)".

I'm looking at the code that usually adds "(2)" to see if there is something that could make it mis-detect a duplicate.

Member

PVince81 commented Sep 22, 2016

I don't think this is related to the desktop client.
When the issue does happen, it might influence the desktop client.
But the cause of the problem is not created by the desktop client, because as @elektro-wolle said even refreshing the web UI already adds a "(2)".

I'm looking at the code that usually adds "(2)" to see if there is something that could make it mis-detect a duplicate.

@elektro-wolle

This comment has been minimized.

Show comment
Hide comment
@elektro-wolle

elektro-wolle Sep 22, 2016

It's just an cronjob calling mysql with the following statement:
update oc_share set file_target = replace(file_target, ' (2)', '') where file_target like '%(2)%'; if I remember correctly.
But this should only run when no one is connected with the desktop-client.

elektro-wolle commented Sep 22, 2016

It's just an cronjob calling mysql with the following statement:
update oc_share set file_target = replace(file_target, ' (2)', '') where file_target like '%(2)%'; if I remember correctly.
But this should only run when no one is connected with the desktop-client.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Sep 22, 2016

Member

Now trying to recreate a similar setup like @ccheveaux by enabling the same third party apps in case it's one of them messing with the FS setup.

Member

PVince81 commented Sep 22, 2016

Now trying to recreate a similar setup like @ccheveaux by enabling the same third party apps in case it's one of them messing with the FS setup.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 17, 2016

Member

also quite weird: all my test files have a size of 0, but the ghost entry reports a size of "4454" bytes. There is no such file with this size in the data folder.

Member

PVince81 commented Oct 17, 2016

also quite weird: all my test files have a size of 0, but the ghost entry reports a size of "4454" bytes. There is no such file with this size in the data folder.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 17, 2016

Member

I reran cron and also the users script. The breakpoints didn't stop any more, no more "(2)" added.
Re-ran the sync from scratch for "dl01780", "tl0569f", "to1efa9" but the problem didn't happen. Well, I stopped earlier with "to1efa9"'s account as there are more than 12k files.

If the problem doesn't appear any more, then maybe I need to run all this in parallel to have some kind of race condition kick in... Now also trying to delete all the "lastFeatureRefresh" keys in oc_preferences to make sure the LDAP stuff gets resynced, just in case...

The only insert statements I received so far were for avatars, but none approach the size observed above.

Member

PVince81 commented Oct 17, 2016

I reran cron and also the users script. The breakpoints didn't stop any more, no more "(2)" added.
Re-ran the sync from scratch for "dl01780", "tl0569f", "to1efa9" but the problem didn't happen. Well, I stopped earlier with "to1efa9"'s account as there are more than 12k files.

If the problem doesn't appear any more, then maybe I need to run all this in parallel to have some kind of race condition kick in... Now also trying to delete all the "lastFeatureRefresh" keys in oc_preferences to make sure the LDAP stuff gets resynced, just in case...

The only insert statements I received so far were for avatars, but none approach the size observed above.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 17, 2016

Member

No more luck today apparently. Tomorrow hopefully.

@gregoryR please tell me what processes are running against the test system:

  • cron ?
  • sync clients with which accounts ? are they syncing all the time or just pinging ?

also how high is the TTL in the LDAP config of OC ?

Member

PVince81 commented Oct 17, 2016

No more luck today apparently. Tomorrow hopefully.

@gregoryR please tell me what processes are running against the test system:

  • cron ?
  • sync clients with which accounts ? are they syncing all the time or just pinging ?

also how high is the TTL in the LDAP config of OC ?

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 18, 2016

Member

FINALLY, I got something.

What I did:

  1. delete from oc_preferences where configkey='lastFeatureRefresh'; => this will force the LDAP code to refresh the info of all users
  2. update oc_jobs set last_run=0,last_checked=0,reserved_at=0; => reset cron jobs

Then in parallel:

  1. Run curl command to simulate fetching all pages in the users page => this will trigger LDAP user refresh
  2. Run cron.php
  3. I also have an idle sync client for users "dl01780", "tl0569f", "to1efa9"

Shortly after running cron + the users page thing, I got this stack trace from the LDAP refresh:

0  OCA\Files_Sharing\SharedMount->generateUniqueTarget() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/SharedMount.php:149
1  OCA\Files_Sharing\SharedMount->verifyMountPoint() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/SharedMount.php:97
2  OCA\Files_Sharing\SharedMount->__construct() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/SharedMount.php:71
3  OCA\Files_Sharing\MountProvider->getMountsForUser() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/MountProvider.php:92
4  OC\Files\Config\MountProviderCollection->OC\Files\Config\{closure}() /srv/www/htdocs/owncloudtest/lib/private/Files/Config/MountProviderCollection.php:76
5  array_map()     /srv/www/htdocs/owncloudtest/lib/private/Files/Config/MountProviderCollection.php:77
6  OC\Files\Config\MountProviderCollection->getMountsForUser() /srv/www/htdocs/owncloudtest/lib/private/Files/Config/MountProviderCollection.php:77
7  OC\Files\Filesystem::initMountPoints() /srv/www/htdocs/owncloudtest/lib/private/Files/Filesystem.php:422
8  OC\Files\Storage\Shared->init() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/sharedstorage.php:99
9  OC\Files\Storage\Shared->getWrapperStorage() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/sharedstorage.php:448
10 OC\Files\Storage\Wrapper\Wrapper->instanceOfStorage() /srv/www/htdocs/owncloudtest/lib/private/Files/Storage/Wrapper/Wrapper.php:485
11 OC\Files\Storage\Shared->instanceOfStorage() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/sharedstorage.php:122
12 OC_Util::{closure:/srv/www/htdocs/owncloudtest/lib/private/legacy/util.php:147-153}() /srv/www/htdocs/owncloudtest/lib/private/legacy/util.php:148
13 OC\Files\Storage\StorageFactory->wrap() /srv/www/htdocs/owncloudtest/lib/private/Files/Storage/StorageFactory.php:100
14 OC\Files\Storage\StorageFactory->getInstance() /srv/www/htdocs/owncloudtest/lib/private/Files/Storage/StorageFactory.php:82
15 OC\Files\Mount\MountPoint->createStorage() /srv/www/htdocs/owncloudtest/lib/private/Files/Mount/MountPoint.php:137
16 OC\Files\Mount\MountPoint->getStorage() /srv/www/htdocs/owncloudtest/lib/private/Files/Mount/MountPoint.php:160
17 OC\Files\View->getFileInfo() /srv/www/htdocs/owncloudtest/lib/private/Files/View.php:1356
18 OC\Files\Node\Root->get() /srv/www/htdocs/owncloudtest/lib/private/Files/Node/Root.php:180
19 OC\AvatarManager->getAvatar() /srv/www/htdocs/owncloudtest/lib/private/AvatarManager.php:94
20 OCA\User_LDAP\User\User->setOwnCloudAvatar() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User/User.php:523
21 OCA\User_LDAP\User\User->updateAvatar() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User/User.php:499
22 OCA\User_LDAP\User\User->update() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User/User.php:143
23 OCA\User_LDAP\User_LDAP->userExists() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User_LDAP.php:277
24 OC\User\Manager->get() /srv/www/htdocs/owncloudtest/lib/private/User/Manager.php:139
25 OC\User\Session->getUser() /srv/www/htdocs/owncloudtest/lib/private/User/Session.php:192
26 OC\App\AppManager->isEnabledForUser() /srv/www/htdocs/owncloudtest/lib/private/App/AppManager.php:152
27 OC_App::isEnabled() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:313
28 OCP\App::isEnabled() /srv/www/htdocs/owncloudtest/lib/public/App.php:131
29 require_once()  /srv/www/htdocs/owncloudtest/apps/user_ldap/appinfo/app.php:72
30 OC_App::requireAppFile() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:186
31 OC_App::loadApp() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:149
32 OC_App::loadApps() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:119
33 {main}          /srv/www/htdocs/owncloudtest/remote.php:148

It is trying to add "(2)" for "/BDD-Capteurs" for user "tl0569f":

  $this->user                    = (string[7]) 'tl0569f';
  $this->superShare              = (object|OC\Share20\Share[19]);
    $this->superShare->id        = (string[4]) '1148';
    $this->superShare->fileId    = (int) '393224';
    $this->superShare->shareOwner = (string[7]) 'ld019cd';
    $this->superShare->permissions = (int) '31';
    $this->superShare->target    = (string[13]) '/BDD-Capteurs';
    $this->superShare->rootFolder = (object|OC\Files\Node\LazyRoot[2])+;

Now from what I see in the mounts list, this share is already mounted for that user:

$evalResult                      = (object|OCA\Files_Sharing\SharedMount[12]);
  $evalResult->storage           = (object|OCA\Files_Trashbin\Storage[10])+;
  $evalResult->recipientView     = (object|OC\Files\View[5])+;
  $evalResult->user              = (string[7]) 'tl0569f';
  $evalResult->superShare        = (object|OC\Share20\Share[19])+;
  $evalResult->groupedShares     = (array[1])+;
  $evalResult->class             = (string[24]) '\OC\Files\Storage\Shared';
  $evalResult->storageId         = (null);
  $evalResult->arguments         = (array[4])+;
  $evalResult->mountPoint        = (string[28]) '/tl0569f/files/BDD-Capteurs/';
  $evalResult->mountOptions      = (array[0]);
  $evalResult->*OC\Files\Mount\MountPoint*loader = (object|OC\Files\Storage\StorageFactory[1])+;
  $evalResult->*OC\Files\Mount\MountPoint*invalidStorage = (bool) '0';

so next step is to find out why the FS setup for user "tl0569f" is running twice through this code path instead of detecting that we already have it. In initMountPoints there's code that tracks which FS was already setup.

Member

PVince81 commented Oct 18, 2016

FINALLY, I got something.

What I did:

  1. delete from oc_preferences where configkey='lastFeatureRefresh'; => this will force the LDAP code to refresh the info of all users
  2. update oc_jobs set last_run=0,last_checked=0,reserved_at=0; => reset cron jobs

Then in parallel:

  1. Run curl command to simulate fetching all pages in the users page => this will trigger LDAP user refresh
  2. Run cron.php
  3. I also have an idle sync client for users "dl01780", "tl0569f", "to1efa9"

Shortly after running cron + the users page thing, I got this stack trace from the LDAP refresh:

0  OCA\Files_Sharing\SharedMount->generateUniqueTarget() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/SharedMount.php:149
1  OCA\Files_Sharing\SharedMount->verifyMountPoint() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/SharedMount.php:97
2  OCA\Files_Sharing\SharedMount->__construct() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/SharedMount.php:71
3  OCA\Files_Sharing\MountProvider->getMountsForUser() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/MountProvider.php:92
4  OC\Files\Config\MountProviderCollection->OC\Files\Config\{closure}() /srv/www/htdocs/owncloudtest/lib/private/Files/Config/MountProviderCollection.php:76
5  array_map()     /srv/www/htdocs/owncloudtest/lib/private/Files/Config/MountProviderCollection.php:77
6  OC\Files\Config\MountProviderCollection->getMountsForUser() /srv/www/htdocs/owncloudtest/lib/private/Files/Config/MountProviderCollection.php:77
7  OC\Files\Filesystem::initMountPoints() /srv/www/htdocs/owncloudtest/lib/private/Files/Filesystem.php:422
8  OC\Files\Storage\Shared->init() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/sharedstorage.php:99
9  OC\Files\Storage\Shared->getWrapperStorage() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/sharedstorage.php:448
10 OC\Files\Storage\Wrapper\Wrapper->instanceOfStorage() /srv/www/htdocs/owncloudtest/lib/private/Files/Storage/Wrapper/Wrapper.php:485
11 OC\Files\Storage\Shared->instanceOfStorage() /srv/www/htdocs/owncloudtest/apps/files_sharing/lib/sharedstorage.php:122
12 OC_Util::{closure:/srv/www/htdocs/owncloudtest/lib/private/legacy/util.php:147-153}() /srv/www/htdocs/owncloudtest/lib/private/legacy/util.php:148
13 OC\Files\Storage\StorageFactory->wrap() /srv/www/htdocs/owncloudtest/lib/private/Files/Storage/StorageFactory.php:100
14 OC\Files\Storage\StorageFactory->getInstance() /srv/www/htdocs/owncloudtest/lib/private/Files/Storage/StorageFactory.php:82
15 OC\Files\Mount\MountPoint->createStorage() /srv/www/htdocs/owncloudtest/lib/private/Files/Mount/MountPoint.php:137
16 OC\Files\Mount\MountPoint->getStorage() /srv/www/htdocs/owncloudtest/lib/private/Files/Mount/MountPoint.php:160
17 OC\Files\View->getFileInfo() /srv/www/htdocs/owncloudtest/lib/private/Files/View.php:1356
18 OC\Files\Node\Root->get() /srv/www/htdocs/owncloudtest/lib/private/Files/Node/Root.php:180
19 OC\AvatarManager->getAvatar() /srv/www/htdocs/owncloudtest/lib/private/AvatarManager.php:94
20 OCA\User_LDAP\User\User->setOwnCloudAvatar() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User/User.php:523
21 OCA\User_LDAP\User\User->updateAvatar() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User/User.php:499
22 OCA\User_LDAP\User\User->update() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User/User.php:143
23 OCA\User_LDAP\User_LDAP->userExists() /srv/www/htdocs/owncloudtest/apps/user_ldap/lib/User_LDAP.php:277
24 OC\User\Manager->get() /srv/www/htdocs/owncloudtest/lib/private/User/Manager.php:139
25 OC\User\Session->getUser() /srv/www/htdocs/owncloudtest/lib/private/User/Session.php:192
26 OC\App\AppManager->isEnabledForUser() /srv/www/htdocs/owncloudtest/lib/private/App/AppManager.php:152
27 OC_App::isEnabled() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:313
28 OCP\App::isEnabled() /srv/www/htdocs/owncloudtest/lib/public/App.php:131
29 require_once()  /srv/www/htdocs/owncloudtest/apps/user_ldap/appinfo/app.php:72
30 OC_App::requireAppFile() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:186
31 OC_App::loadApp() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:149
32 OC_App::loadApps() /srv/www/htdocs/owncloudtest/lib/private/legacy/app.php:119
33 {main}          /srv/www/htdocs/owncloudtest/remote.php:148

It is trying to add "(2)" for "/BDD-Capteurs" for user "tl0569f":

  $this->user                    = (string[7]) 'tl0569f';
  $this->superShare              = (object|OC\Share20\Share[19]);
    $this->superShare->id        = (string[4]) '1148';
    $this->superShare->fileId    = (int) '393224';
    $this->superShare->shareOwner = (string[7]) 'ld019cd';
    $this->superShare->permissions = (int) '31';
    $this->superShare->target    = (string[13]) '/BDD-Capteurs';
    $this->superShare->rootFolder = (object|OC\Files\Node\LazyRoot[2])+;

Now from what I see in the mounts list, this share is already mounted for that user:

$evalResult                      = (object|OCA\Files_Sharing\SharedMount[12]);
  $evalResult->storage           = (object|OCA\Files_Trashbin\Storage[10])+;
  $evalResult->recipientView     = (object|OC\Files\View[5])+;
  $evalResult->user              = (string[7]) 'tl0569f';
  $evalResult->superShare        = (object|OC\Share20\Share[19])+;
  $evalResult->groupedShares     = (array[1])+;
  $evalResult->class             = (string[24]) '\OC\Files\Storage\Shared';
  $evalResult->storageId         = (null);
  $evalResult->arguments         = (array[4])+;
  $evalResult->mountPoint        = (string[28]) '/tl0569f/files/BDD-Capteurs/';
  $evalResult->mountOptions      = (array[0]);
  $evalResult->*OC\Files\Mount\MountPoint*loader = (object|OC\Files\Storage\StorageFactory[1])+;
  $evalResult->*OC\Files\Mount\MountPoint*invalidStorage = (bool) '0';

so next step is to find out why the FS setup for user "tl0569f" is running twice through this code path instead of detecting that we already have it. In initMountPoints there's code that tracks which FS was already setup.

@gregoryR

This comment has been minimized.

Show comment
Hide comment
@gregoryR

gregoryR Oct 18, 2016

@PVince81 I'll try to answer questions in order

@gregoryR how are you actually triggering the error ?
Do you have a system cron running on the test instance ?
Are you running the sync client for some users ?

Just wait and (2) begins to appear in oc_share.file_target

Yes cron is running for www-data in test instance

crontab -l -u www-data
# m h  dom mon dow   command
10,25,40,55 * * * * php -f /export/home/clouddebug/cron.php

Since #26355 crontab occasionally (one or two time a day, crontab runs every 15 minutes) output a Fatal Error

PHP Fatal error:  Cannot redeclare OCA\Files_Sharing\encodeShare() (previously declared in /export/home/clouddebug/apps/files_sharing/lib/SharedMount.php:161) in /export/home/clouddebug/apps/files_sharing/lib/SharedMount.php on line 161

Sync client is running for user dl01780 and me gr05542. Sync client isn't just pinging. Running continuously

I see that your oc_storages table contains old entries "local::/export/home1/owncloud/data/". I assume that your data dir is not "/export/home1/owncloud/data/" any more.
There seem to be a lot of obsolete storage entries.

Yes /export/home1/owncloud/data/ was the old data directory, on the local filesystem of the webserver. Data is now on an NFS share. Migration have been done by

  • put owncloud in maintenance mode (or stop the webserver I don't remember precisely)
  • rsync data folder from local storage to nfs storage
  • change datadirectory config/config.php
  • restart owncloud

Ohhhh, there's a ghost here.

When I run find in fs I only see the first two entries

find . -name 'Datarmor et web services*'
./files/documents/datarmor/Datarmor et web services.xlsx
./files_versions/documents/datarmor/Datarmor et web services.xlsx.v1468246070

In oc_filecache

SELECT * FROM `oc_filecache` WHERE `path` LIKE '%Datarmor et web services%'
"760660","90","files/documents/datarmor/Datarmor et web services.xlsx","6f91a26d2932c14372f642c70657e759","760647","Datarmor et web services.xlsx","18","3","17798","1474884460","1474884460","0","0","4a2c2fb8d7d99bd6009fa234331f9440","27",
"810626","90","files_versions/documents/datarmor/Datarmor et web services.xlsx.v1468246070","f1efbd2df991f3a1110f599728c18e99","810625","Datarmor et web services.xlsx.v1468246070","13","3","12198","1474884460","1474884460","0","0","7394e06611021da28195e01d73f50c3e","27",

Well, I stopped earlier with "to1efa9"'s account as there are more than 12k files.

I'm seeing just 90 files in tl0569f filesystem

find files/ -type f -ls |wc -l
90

but I remember that this user had loaded a lot of files in his account.

also how high is the TTL in the LDAP config of OC ?

TTL is 600

gregoryR commented Oct 18, 2016

@PVince81 I'll try to answer questions in order

@gregoryR how are you actually triggering the error ?
Do you have a system cron running on the test instance ?
Are you running the sync client for some users ?

Just wait and (2) begins to appear in oc_share.file_target

Yes cron is running for www-data in test instance

crontab -l -u www-data
# m h  dom mon dow   command
10,25,40,55 * * * * php -f /export/home/clouddebug/cron.php

Since #26355 crontab occasionally (one or two time a day, crontab runs every 15 minutes) output a Fatal Error

PHP Fatal error:  Cannot redeclare OCA\Files_Sharing\encodeShare() (previously declared in /export/home/clouddebug/apps/files_sharing/lib/SharedMount.php:161) in /export/home/clouddebug/apps/files_sharing/lib/SharedMount.php on line 161

Sync client is running for user dl01780 and me gr05542. Sync client isn't just pinging. Running continuously

I see that your oc_storages table contains old entries "local::/export/home1/owncloud/data/". I assume that your data dir is not "/export/home1/owncloud/data/" any more.
There seem to be a lot of obsolete storage entries.

Yes /export/home1/owncloud/data/ was the old data directory, on the local filesystem of the webserver. Data is now on an NFS share. Migration have been done by

  • put owncloud in maintenance mode (or stop the webserver I don't remember precisely)
  • rsync data folder from local storage to nfs storage
  • change datadirectory config/config.php
  • restart owncloud

Ohhhh, there's a ghost here.

When I run find in fs I only see the first two entries

find . -name 'Datarmor et web services*'
./files/documents/datarmor/Datarmor et web services.xlsx
./files_versions/documents/datarmor/Datarmor et web services.xlsx.v1468246070

In oc_filecache

SELECT * FROM `oc_filecache` WHERE `path` LIKE '%Datarmor et web services%'
"760660","90","files/documents/datarmor/Datarmor et web services.xlsx","6f91a26d2932c14372f642c70657e759","760647","Datarmor et web services.xlsx","18","3","17798","1474884460","1474884460","0","0","4a2c2fb8d7d99bd6009fa234331f9440","27",
"810626","90","files_versions/documents/datarmor/Datarmor et web services.xlsx.v1468246070","f1efbd2df991f3a1110f599728c18e99","810625","Datarmor et web services.xlsx.v1468246070","13","3","12198","1474884460","1474884460","0","0","7394e06611021da28195e01d73f50c3e","27",

Well, I stopped earlier with "to1efa9"'s account as there are more than 12k files.

I'm seeing just 90 files in tl0569f filesystem

find files/ -type f -ls |wc -l
90

but I remember that this user had loaded a lot of files in his account.

also how high is the TTL in the LDAP config of OC ?

TTL is 600

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 18, 2016

Member

So far all I know is that initMountPoints has already run for a user X, and somehow it runs again for that user. I'm trying to find the proof.

So far, looking at initMountPoints, I see one potential code path that could bypass the usersSetup check. The check is here: https://github.com/owncloud/core/blob/v9.1.1/lib/private/Files/Filesystem.php#L396

and later we set the value to say "we already processed this user": https://github.com/owncloud/core/blob/v9.1.1/lib/private/Files/Filesystem.php#L408

But in-between there are calls to the user manager to get the user: https://github.com/owncloud/core/blob/v9.1.1/lib/private/Files/Filesystem.php#L401

From what I observe in the stack trace posted previously, a call to $userManager->get() can trigger an avatar update when using LDAP with avatars. This is likely to happen only when a feature refresh is due (see lastFeatureRefresh in oc_preferences). Whenever avatar refresh is due, the AvatarManager itself will also trigger an initMountPoints for the user whose avatar will be setup.

Since at this stage we haven't yet set usersSetup, it is likely that initMountPoints will run a second time for the same user.

Still looking for proof of this, but will also look into ways to prevent this so we can test.

Member

PVince81 commented Oct 18, 2016

So far all I know is that initMountPoints has already run for a user X, and somehow it runs again for that user. I'm trying to find the proof.

So far, looking at initMountPoints, I see one potential code path that could bypass the usersSetup check. The check is here: https://github.com/owncloud/core/blob/v9.1.1/lib/private/Files/Filesystem.php#L396

and later we set the value to say "we already processed this user": https://github.com/owncloud/core/blob/v9.1.1/lib/private/Files/Filesystem.php#L408

But in-between there are calls to the user manager to get the user: https://github.com/owncloud/core/blob/v9.1.1/lib/private/Files/Filesystem.php#L401

From what I observe in the stack trace posted previously, a call to $userManager->get() can trigger an avatar update when using LDAP with avatars. This is likely to happen only when a feature refresh is due (see lastFeatureRefresh in oc_preferences). Whenever avatar refresh is due, the AvatarManager itself will also trigger an initMountPoints for the user whose avatar will be setup.

Since at this stage we haven't yet set usersSetup, it is likely that initMountPoints will run a second time for the same user.

Still looking for proof of this, but will also look into ways to prevent this so we can test.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 18, 2016

Member

Side note: starting with 9.2 the avatar manager will be decoupled from FS setup, so this kind of crazy issues shouldn't happen any more: #26124 (not backportable as it changes the FS structure)

Member

PVince81 commented Oct 18, 2016

Side note: starting with 9.2 the avatar manager will be decoupled from FS setup, so this kind of crazy issues shouldn't happen any more: #26124 (not backportable as it changes the FS structure)

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 18, 2016

Member

@ccheveaux @gregoryR could you guys try with this new patch: #26396 (https://github.com/owncloud/core/pull/26396.patch). This should solve the problem by preventing duplicate setup FS of users with LDAP.

In my local test env the problem didn't appear any more after this change after several retries.

Member

PVince81 commented Oct 18, 2016

@ccheveaux @gregoryR could you guys try with this new patch: #26396 (https://github.com/owncloud/core/pull/26396.patch). This should solve the problem by preventing duplicate setup FS of users with LDAP.

In my local test env the problem didn't appear any more after this change after several retries.

@gregoryR

This comment has been minimized.

Show comment
Hide comment
@gregoryR

gregoryR Oct 18, 2016

@PVince81 patch #26396 is applied to my test instance. I've cleaned my oc_share table.
I'm off for a few days, will give feed back on monday

gregoryR commented Oct 18, 2016

@PVince81 patch #26396 is applied to my test instance. I've cleaned my oc_share table.
I'm off for a few days, will give feed back on monday

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 18, 2016

Member

Thanks. Now regardless whether this patch fixes this specific issue or not, I think it should be merged because it would prevent any other potential similar issues.

Member

PVince81 commented Oct 18, 2016

Thanks. Now regardless whether this patch fixes this specific issue or not, I think it should be merged because it would prevent any other potential similar issues.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Oct 19, 2016

Sorry @PVince81 i don't have test instance to try this patch :/
And lot of work atm.
My product instance is "stable" for this moment with this #25718 (comment)
I hope my next upgrade of ownCloud don't break it or fix it correctly :/

ccheveaux commented Oct 19, 2016

Sorry @PVince81 i don't have test instance to try this patch :/
And lot of work atm.
My product instance is "stable" for this moment with this #25718 (comment)
I hope my next upgrade of ownCloud don't break it or fix it correctly :/

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 19, 2016

Member

Based on all the information found so far, I managed to find steps to reproduce this issue:

Steps:

  1. Setup LDAP with at least two users "user1" and "user2"
  2. Setup LDAP in OC
  3. Login as "user1"
  4. Share a folder "test" with "user2"
  5. Login as "user2"
  6. Share a folder "abc" with "user1"
  7. delete from oc_preferences where configkey='lastFeatureRefresh' <= this was the missing bit which made it random!
  8. curl -X PROPFIND -H "Depth: 1" http://user1:test@localhost/owncloud/remote.php/webdav
  9. select * from oc_share

Then repeat the last three steps several times.
You'll see that the file_target of "test" will receive further "(2)" appended.

Member

PVince81 commented Oct 19, 2016

Based on all the information found so far, I managed to find steps to reproduce this issue:

Steps:

  1. Setup LDAP with at least two users "user1" and "user2"
  2. Setup LDAP in OC
  3. Login as "user1"
  4. Share a folder "test" with "user2"
  5. Login as "user2"
  6. Share a folder "abc" with "user1"
  7. delete from oc_preferences where configkey='lastFeatureRefresh' <= this was the missing bit which made it random!
  8. curl -X PROPFIND -H "Depth: 1" http://user1:test@localhost/owncloud/remote.php/webdav
  9. select * from oc_share

Then repeat the last three steps several times.
You'll see that the file_target of "test" will receive further "(2)" appended.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Oct 19, 2016

What I also noticed but may be an isolated problem.
A ten users (using lots of shares) no longer able to connect to ownCloud.
I had to delete their account to recreate itself.
He has mounted the data and i scanned with a "occ file:scan username" for each users.
But unfortunately i lost the shares of these people who have been forced to recreate them. (I have not found anything to save them).

For several weeks my body production works again correctly, but if I remove your patch then (2) back.

ccheveaux commented Oct 19, 2016

What I also noticed but may be an isolated problem.
A ten users (using lots of shares) no longer able to connect to ownCloud.
I had to delete their account to recreate itself.
He has mounted the data and i scanned with a "occ file:scan username" for each users.
But unfortunately i lost the shares of these people who have been forced to recreate them. (I have not found anything to save them).

For several weeks my body production works again correctly, but if I remove your patch then (2) back.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 19, 2016

Member

no longer able to connect to ownCloud.

Any error messages ? Was it web UI or desktop client ?
Not sure if related with this "(2)" issue, except maybe that a recursive replication of these could cause memory errors I guess.

Member

PVince81 commented Oct 19, 2016

no longer able to connect to ownCloud.

Any error messages ? Was it web UI or desktop client ?
Not sure if related with this "(2)" issue, except maybe that a recursive replication of these could cause memory errors I guess.

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Oct 19, 2016

image

And desktop client can't connect.

Following our patches and tests, I had a period or my ownCloud instance was enormously slow in terms of connection and access the files through Web UI.
Ditto for synchronizing the desktop client.

It became extremely chaotic so I completely reinstalled distribution and ownCloud, because just install ownCloud was not enough.

After this and re-import the database, ownCloud was responsive but (2) was still present and active.
I apply your patch again and that stopped these (2).

Finally I delete and recreate the blocked user accounts and I noticed that it was those sharing much.

Since I did not touch anything to await a future last days I would do during the school holidays.
The server has been reinstalled on a virtual machine that will allow me to go back very quickly in case of problems.

ccheveaux commented Oct 19, 2016

image

And desktop client can't connect.

Following our patches and tests, I had a period or my ownCloud instance was enormously slow in terms of connection and access the files through Web UI.
Ditto for synchronizing the desktop client.

It became extremely chaotic so I completely reinstalled distribution and ownCloud, because just install ownCloud was not enough.

After this and re-import the database, ownCloud was responsive but (2) was still present and active.
I apply your patch again and that stopped these (2).

Finally I delete and recreate the blocked user accounts and I noticed that it was those sharing much.

Since I did not touch anything to await a future last days I would do during the school holidays.
The server has been reinstalled on a virtual machine that will allow me to go back very quickly in case of problems.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 19, 2016

Member

@ccheveaux was that on v9.1.0 or v9.1.1 ? Because on v9.1.0 there was a memory issue which could cause users to not be able to log in any more.

Member

PVince81 commented Oct 19, 2016

@ccheveaux was that on v9.1.0 or v9.1.1 ? Because on v9.1.0 there was a memory issue which could cause users to not be able to log in any more.

@PVince81 PVince81 added the regression label Oct 19, 2016

@ccheveaux

This comment has been minimized.

Show comment
Hide comment
@ccheveaux

ccheveaux Oct 19, 2016

image

v9.1.1

But after deleting account and create again the problem disappeared.

ccheveaux commented Oct 19, 2016

image

v9.1.1

But after deleting account and create again the problem disappeared.

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 19, 2016

Member

closing as fixed through #26399 which willl be in 9.1.2.

@ccheveaux if you have old logs about the users who couldn't login we could have a look in a separate issue.

Member

PVince81 commented Oct 19, 2016

closing as fixed through #26399 which willl be in 9.1.2.

@ccheveaux if you have old logs about the users who couldn't login we could have a look in a separate issue.

@PVince81 PVince81 closed this Oct 19, 2016

@karamatos

This comment has been minimized.

Show comment
Hide comment
@karamatos

karamatos Oct 19, 2016

I have applied patch 26396 and did everything as before that caused the (2) (2) ... problem. The problem has not reappeared. It has been about 24 hours and I have been running various sync clients without issue.

I did notice a strange thing with avatars. Even though I had avatars turned off in config.php, the system was still trying to update the database with avatar information. I think it was coming from the LDAP system and the mysql error was: The total blob data length (10067609) is greater than 10% of the total redo log size. It was trying to update a record in oc_cards.

The query was large and it slowed down every ownCloud access for the user with the avatar. The solution was to turn avatars back on and upload a new avatar. That seemed to fix the avatar database problem. I turned avatars off again and applied patch 26396 and the main problem is gone. I am keeping avatars off as I read that the next version will change the way avatars are handled.

Will patch 26396 be in the next release?

karamatos commented Oct 19, 2016

I have applied patch 26396 and did everything as before that caused the (2) (2) ... problem. The problem has not reappeared. It has been about 24 hours and I have been running various sync clients without issue.

I did notice a strange thing with avatars. Even though I had avatars turned off in config.php, the system was still trying to update the database with avatar information. I think it was coming from the LDAP system and the mysql error was: The total blob data length (10067609) is greater than 10% of the total redo log size. It was trying to update a record in oc_cards.

The query was large and it slowed down every ownCloud access for the user with the avatar. The solution was to turn avatars back on and upload a new avatar. That seemed to fix the avatar database problem. I turned avatars off again and applied patch 26396 and the main problem is gone. I am keeping avatars off as I read that the next version will change the way avatars are handled.

Will patch 26396 be in the next release?

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 19, 2016

Member

Will patch 26396 be in the next release?

Yes, in 9.1.2 but it had to be adjusted due to further code changes: #26399
The effect will be the same.

Regarding the avatars + LDAP issue, can you make a new ticket ? I find it suspicious that LDAP tries to sync avatars even when disabled.

Member

PVince81 commented Oct 19, 2016

Will patch 26396 be in the next release?

Yes, in 9.1.2 but it had to be adjusted due to further code changes: #26399
The effect will be the same.

Regarding the avatars + LDAP issue, can you make a new ticket ? I find it suspicious that LDAP tries to sync avatars even when disabled.

@gregoryR

This comment has been minimized.

Show comment
Hide comment
@gregoryR

gregoryR Oct 24, 2016

@PVince81 I'm back
I can confirm that patch #26396 solve the (2) problem on both test and production instance.
Thanks for all your work

gregoryR commented Oct 24, 2016

@PVince81 I'm back
I can confirm that patch #26396 solve the (2) problem on both test and production instance.
Thanks for all your work

@PVince81

This comment has been minimized.

Show comment
Hide comment
@PVince81

PVince81 Oct 24, 2016

Member

Also thanks everyone here who helped debug the issue !

Member

PVince81 commented Oct 24, 2016

Also thanks everyone here who helped debug the issue !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment