Deleting an (external) storage mount leaves thumnails created in the filesystem #25033

mmattel · 2016-06-08T14:21:25Z

Steps to reproduce

Mount an (external) storage, in my case a SMB picture share
Go into some folders via the files app - thumbs are created
Delete the mount point

Expected behaviour

Automatic deletion of the corresponding thumbs which were related to this mountpoint

Actual behaviour

Thumbs from the deleted mountpoint are not deleted and stay in the users thumbnails directory as "orphans". Means, that this directory will grow forever, or you do a manual cleanup by deleting all as you do not know what belongs together.

Server configuration

Operating system: Ubuntu 14.04

Web server: nginx 1.9.14

Database: mysql

PHP version: 5.5.9

ownCloud version: (see ownCloud admin page)
ownCloud 9.1.0 beta 2 (git)

Updated from an older ownCloud or fresh install:
This was a fresh setup from tar and updated once, daily channel

Where did you install ownCloud from:
tar

Signing status (ownCloud 9.0 and above):

Datei nicht gefunden
Das ausgewählte Dokument wurde auf dem Server nicht gefunden.
Sie können zur Rückkehr zu ownCloud hier klicken.

List of activated apps:

Enabled:
  - activity: 2.3.2
  - comments: 0.3.0
  - dav: 0.2.4
  - federatedfilesharing: 0.3.0
  - federation: 0.1.0
  - files: 1.5.1
  - files_external: 0.6.0
  - files_sharing: 0.10.0
  - files_trashbin: 0.9.0
  - files_versions: 1.3.0
  - gallery: 15.0.0
  - logreader: true
  - provisioning_api: 0.5.0
  - systemtags: 0.3.0
  - updatenotification: 0.2.1
Disabled:
  - bookmarks
  - calendar
  - chat
  - contacts
  - documents
  - encryption
  - music
  - news
  - notes
  - testing
  - user_ldap

The content of config/config.php:

{
    "system": {
        "instanceid": "ock982ae3c5s",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "xxxx"
        ],
        "datadirectory": "\/var\/www\/owncloud\/data",
        "overwrite.cli.url": "https:\/\/xxxx",
        "dbtype": "mysql",
        "version": "9.1.0.8",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "Europe\/Vienna",
        "installed": true,
        "filelocking.enabled": "true",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "localhost",
            "port": 6379,
            "timeout": 0,
            "dbindex": 0
        },
        "assetdirectory": "\/var\/www\/owncloud",
        "asset-pipeline.enabled": true,
        "filesystem_check_changes": 1,
        "preview_max_scale_factor": 1,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\Illustrator",
            "OC\\Preview\\Postscript",
            "OC\\Preview\\Photoshop",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\Raw"
        ],
        "loglevel": 2,
        "maintenance": false,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_from_address": "owncloud",
        "mail_domain": "xxxx",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "xxxx",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

Are you using external storage, if yes which one: local/smb/sftp/...
SMB

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
No

Client configuration

Browser: Opera

Operating system: W7x64

Logs

owncloud.log

No corresponden log entry

The text was updated successfully, but these errors were encountered:

PVince81 · 2016-06-08T15:20:53Z

I believe there is no hook at storage deletion, so the previews subsystem doesn't know that it should delete them.

@georgehrke @oparoz

rullzer · 2016-06-09T13:20:55Z

This should most likely be a background job. As I can imagine there being a whole lot of previews generated.

oparoz · 2016-06-09T13:24:33Z

This should most likely be a background job

I agree

mmattel · 2016-06-09T13:27:51Z

Could be. But how to remember / identify those previews that have belonged to a deleted mount point to be covered by the background job? Only those previews must be deleted, not all...

PVince81 · 2016-06-09T13:29:11Z

I believe that the previews have the file id as file name. So one idea would be to go through all of them and check whether there is a matching entry in the oc_filecache table.

When deleting an external storage mount, in most cases the cache will be cleared for that storage.

oparoz · 2016-06-09T13:30:36Z

The preview class also methods to delete previews, so maybe there doesn't need to talk to the DB directly.

mmattel · 2016-06-09T13:33:20Z

Currently, if you delete a mount point, previews become orphants at the moment of deletion.
If you immediately run a files:scan command, these previews turn into "regular" files. So mount point deletion and related preview deletion must go hand in hand in one process.

mmattel · 2016-06-10T07:28:09Z

@oparoz disagree, because we are not managing previews but the deletion of a mount point. This does not touch preview management and will not be triggered therefore.

Some thoughts:

For upcoming releases:

create a new oc_previews table containing path, storage ID of the mountpoint where they belong to (not the ID where they are written to = users home !) and etag equal to oc_filecache

oc_previews would need to be populated while generating previews as additional step, but has the advantage of not touching oc_filecache structure.

When deleting a mount point, you do the following:

query oc_previews for the storage id to be removed (maybe in chunks because of quantity of results)
loop thru the results and delete the files in the filesystem
continue with removing the mount point (= not touching the current procedure)

The good thing is that nothing else changes in owncloud. Even if your system dies while running the deletion job and has not finished.

In general, the driver for preview file management is in this case not the filesystem (current) but owncloud as it has created those files and not the user!

For current releases we would need, if wanted, a fall back scenario. But the problem is distinguishing valid versus orphaned preview files.

PVince81 · 2016-06-10T08:00:27Z

or have the preview file list in "oc_filecache" for easier listing

mmattel · 2016-06-10T08:22:25Z

Sometimes you need a break and new ideas come 😄
What about adding a column ref_storage in oc_filecache?
This would be completely open to other applications which may want/need to use it.
One drawback is, that you need to filter path component (thumbnails) with a text search for getting previews. But to speed up again, prefilter by ref_storage and the users home storage ID (fast) and filter the result by path component.

PVince81 · 2016-06-10T08:29:21Z

@mmattel can you expand your explanation ? What would be the value and use of "ref_storage" ?

mmattel · 2016-06-10T09:18:13Z

Sure:

storage: this is the ID of the mount point where this file is physically located
ref_storage: this the ID of the mount point of the source where the data for the physical file was coming from

Example:

For previews:
target = function {source}

physical file location = target: location reference = storage
referenced location = source: location reference = ref_storage

oc_storages:
user_x: home_storage_location = ID:1
SMB: smb_share = ID:2

Examples for thumbnails for user_x:
Thumb_1: storage = 1 and ref_storage =1
Thumb_2: storage = 1 and ref_storage =2
ect...

If the user has more than one mounts, you will get possibly different ref_storage values for thumbnails. But this is exactly what we want 😄

On a generic view:
storage must be populated, ref_storage can be populated.

PVince81 · 2016-06-10T09:20:11Z

Ah, I see. This would be only for the thumbnail files.
If that is needed then I think I'd rather have the "oc_previews" table instead to avoid cluttering the "oc_filecache" table which is already huge on some systems.

mmattel · 2016-06-10T09:27:26Z

The idea to have a ref_storage column is independent of where we put it to.

It is more a decision of if we do it for preview generation only, or be more generic and open.
Honestly thinking about this. I like the generic version more and more (whatever the future will bring...)

ownclouders · 2017-12-21T01:22:33Z

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.

ownclouders · 2018-01-29T01:09:52Z

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

ownclouders · 2018-03-09T01:10:44Z

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

stale · 2021-09-21T11:16:03Z

This issue has been automatically closed.

mmattel added the Type:Bug label Jun 8, 2016

nickvergessen added the feature:previews label Jun 8, 2016

PVince81 added the app:files_external label Jun 8, 2016

PVince81 added this to the 9.2-next milestone Jun 8, 2016

mmattel mentioned this issue Aug 11, 2016

[Feature Request] adding column "ref_storage" in table oc_filecache #25770

Closed

PVince81 added the sev3-medium label Jan 13, 2017

PVince81 modified the milestones: backlog, 10.0 Jan 13, 2017

ownclouders added the status/STALE label Nov 16, 2017

mmattel mentioned this issue Dec 14, 2017

Decouple home storage entry from user #29502

Closed

ownclouders closed this as completed Dec 21, 2017

ownclouders removed the status/STALE label Dec 21, 2017

PVince81 reopened this Dec 21, 2017

ownclouders added the status/STALE label Jan 21, 2018

ownclouders closed this as completed Jan 29, 2018

ownclouders removed the status/STALE label Jan 29, 2018

mmattel reopened this Jan 29, 2018

ownclouders added the status/STALE label Mar 1, 2018

ownclouders closed this as completed Mar 9, 2018

ownclouders removed the status/STALE label Mar 9, 2018

mmattel reopened this Mar 9, 2018

ownclouders added the status/STALE label Apr 9, 2018

felixboehm removed the sev3-medium label Apr 9, 2018

ownclouders removed the status/STALE label Apr 9, 2018

ownclouders added the status/STALE label May 10, 2018

PVince81 mentioned this issue Oct 20, 2018

occ command to delete orphaned previews #33240

Closed

stale bot closed this as completed Sep 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deleting an (external) storage mount leaves thumnails created in the filesystem #25033

Deleting an (external) storage mount leaves thumnails created in the filesystem #25033

mmattel commented Jun 8, 2016 •

edited by DeepDiver1975

Loading

PVince81 commented Jun 8, 2016

rullzer commented Jun 9, 2016

oparoz commented Jun 9, 2016

mmattel commented Jun 9, 2016

PVince81 commented Jun 9, 2016

oparoz commented Jun 9, 2016

mmattel commented Jun 9, 2016 •

edited

Loading

mmattel commented Jun 10, 2016

PVince81 commented Jun 10, 2016

mmattel commented Jun 10, 2016

PVince81 commented Jun 10, 2016

mmattel commented Jun 10, 2016 •

edited

Loading

PVince81 commented Jun 10, 2016

mmattel commented Jun 10, 2016

ownclouders commented Dec 21, 2017

ownclouders commented Jan 29, 2018

ownclouders commented Mar 9, 2018

stale bot commented Sep 21, 2021

Deleting an (external) storage mount leaves thumnails created in the filesystem #25033

Deleting an (external) storage mount leaves thumnails created in the filesystem #25033

Comments

mmattel commented Jun 8, 2016 • edited by DeepDiver1975 Loading

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

owncloud.log

PVince81 commented Jun 8, 2016

rullzer commented Jun 9, 2016

oparoz commented Jun 9, 2016

mmattel commented Jun 9, 2016

PVince81 commented Jun 9, 2016

oparoz commented Jun 9, 2016

mmattel commented Jun 9, 2016 • edited Loading

mmattel commented Jun 10, 2016

PVince81 commented Jun 10, 2016

mmattel commented Jun 10, 2016

PVince81 commented Jun 10, 2016

mmattel commented Jun 10, 2016 • edited Loading

PVince81 commented Jun 10, 2016

mmattel commented Jun 10, 2016

ownclouders commented Dec 21, 2017

ownclouders commented Jan 29, 2018

ownclouders commented Mar 9, 2018

stale bot commented Sep 21, 2021

mmattel commented Jun 8, 2016 •

edited by DeepDiver1975

Loading

mmattel commented Jun 9, 2016 •

edited

Loading

mmattel commented Jun 10, 2016 •

edited

Loading