external plugin on local goofys (s3) mount, file corruption (?) #16916

username11236 · 2019-08-28T14:35:59Z

in words:
i have an s3 bucked mounted via goofys on local mnt folder, nextcloud external plugin on local mnt/myfs/cloud. worked well for years. now, when i copy jpgs and vids via webiface to external, they get corrupted. with a possibility to uncorrupt themselves at a later time (many days later)
tried the scan function, different versions of goofys(s3fs)

i can reproduce the corrupt, but have no idea how to reverse it. the files are on the bucket, with their size +30% approx and can not be opened after download. (enc seems to work)

i can recover my data, my problem is that i can not use nextcloud anymore :(

Steps to reproduce

mount s3 bucket with goofys to /mnt/fs, user www-data:www-data with dir 750, file 640
configure external storages, local, to /mnt/fs/cdata, enc, preview, checkchanges
move files from data dir to external
after some time size for pictures shows 0kb, preview fails with filenotfound, download succeeds without any error(!), downloaded file has 0kb; videos show real size (encrypted size) and can be downloaded but not opened (likely still encrypted)
files are on the filesystem and can be downloaded externally (encrypted though)
after some time (multiple days?) it MAY be that files become accessible again, previews are generated and videos are streamable and downloadable without error
if files are moved from data to external, they will be encrypted, and synced to s3, back to step 4

this setup has worked well for years without gamebreaking errors. (the native s3 thing did not ever work in my case)

this setup seem to fk up the signatures inside external so i had to disable the check (return True in the crypt.php like suggested elsewhere)
my files got accessible again, thats when i noticed files with 0kb
problems seem to have started with an nc update, dont know which one though
since turning off signature checks does not pose security risks with encryption on, i dont mind. (having to change the source does bug me a bit though)

the server has a 15Mbyte connection, so syncing is not the issue.
i tried variations of occ scan with no success
different versions of goofys, same problem

Expected behaviour

show the real size, generate prieviews, successful download

Actual behaviour

pictures stay at 0kb, being unaccessible, videos can be downloaded but not opened (likely still enc)

Server configuration

Operating system: Debian GNU/Linux 9

Web server: Apache/2.4.25 (Debian)

Database: mysql Ver 15.1 Distrib 10.1.38-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP version: php7.0 7.0.33-0+deb9u3

Nextcloud version: Nextcloud 15.0.11

Updated from an older Nextcloud/ownCloud or fresh install: from oc times

Where did you install Nextcloud from: download from website, nextcloud.zip

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:

accessibility: 1.1.0
activity: 2.8.2
bruteforcesettings: 1.4.0
calendar: 1.6.5
cloud_federation_api: 0.1.0
comments: 1.5.0
contacts: 3.1.3
dav: 1.8.2
encryption: 2.3.0
federatedfilesharing: 1.5.0
federation: 1.5.0
files: 1.10.0
files_external: 1.6.0
files_pdfviewer: 1.4.0
files_sharing: 1.7.0
files_texteditor: 2.7.0
files_trashbin: 1.5.0
files_versions: 1.8.0
files_videoplayer: 1.4.0
gallery: 18.2.0
logreader: 2.0.0
lookup_server_connector: 1.3.0
nextcloud_announcements: 1.4.0
notifications: 2.3.0
oauth2: 1.3.0
password_policy: 1.5.0
provisioning_api: 1.5.0
ransomware_detection: 0.5.2
serverinfo: 1.5.0
sharebymail: 1.5.0
support: 1.0.0
survey_client: 1.3.0
systemtags: 1.5.0
theming: 1.6.0
twofactor_backupcodes: 1.4.1
updatenotification: 1.5.0
user_usage_report: 1.1.2
workflowengine: 1.5.0
Disabled:
admin_audit
files_downloadactivity
firstrunwizard
previewgenerator
ransomware_protection
spreed
user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "domain.at",
            "cloud.domain.at"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/151.236.10.30",
        "dbtype": "mysql",
        "version": "15.0.11.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "appstore.experimental.enabled": false,
        "loglevel": 2,
        "maintenance": false,
        "part_file_in_storage": false,
        "asset-pipeline.enabled": true,
        "theme": "",
        "data-fingerprint": "e7352febc9ef728257a7e8b8ecf42374",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "cache_path": "\/tmp\/nccache",
        "logfile": "\/var\/log\/nextcloud.log",
        "filesystem_check_changes": 1,
        "filelocking.enabled": true
    }
}

Are you using external storage, if yes which one: Local, pointing on a mounted goofys s3 filesystem on /mnt/cfs ; filemode 0640, dirmode 0750 www-data:www-data

Are you using encryption: yes (seems to be part of the problem)

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Firefox 68.0.2 (64-bit)

Operating system: Ubuntu 64 bit

Logs

Web server error log

none related

Nextcloud log (data/nextcloud.log)

Nextcloud log

please let me know if i missed something
thanks

The text was updated successfully, but these errors were encountered:

kesselb · 2019-08-28T14:58:37Z

Mind to try this patch: #15946?

username11236 · 2019-08-30T12:24:52Z

Thanks for the quick response!

Unfortunately this did not solve my problem

username11236 · 2019-09-03T08:33:41Z

I found a workaround for my problem, and seemingly a second bug too

I added a second external local storage, same settings like the other one, but write protected. It showed the same symptoms like the original one. upon entering the settings again to remove it i found two identical entrys for that storage. I deleted one of them.
This seemingly triggered a 'pending' for the filesizes in my original local s3 mount (in the remaining write protected test mount too). my files got accessible again.

Fun side effect, i could not reproduce the corruption (actually no corruption).

if it should happen again: I will add a local storage, exit the menue, enter it again, remove the two(!) entrys I just created, and wait for the 'pending' to go away.
not perfect, but good enough for me ;)

signature check needs to stay deactivated, do not know how to work around that; open for suggestions though

the bugs on my installation might have their origin within the setup i use, not effecting anyone else, maybe ever.

Since my setup works again I will close this, if anybody wants additional information, feel free to contact me.

Thanks for the help!

username11236 added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Aug 28, 2019

kesselb added feature: encryption (server-side) feature: external storage labels Aug 30, 2019

username11236 closed this as completed Sep 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

external plugin on local goofys (s3) mount, file corruption (?) #16916

external plugin on local goofys (s3) mount, file corruption (?) #16916

username11236 commented Aug 28, 2019

kesselb commented Aug 28, 2019

username11236 commented Aug 30, 2019

username11236 commented Sep 3, 2019

external plugin on local goofys (s3) mount, file corruption (?) #16916

external plugin on local goofys (s3) mount, file corruption (?) #16916

Comments

username11236 commented Aug 28, 2019

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

kesselb commented Aug 28, 2019

username11236 commented Aug 30, 2019

username11236 commented Sep 3, 2019