Excessive file upload conflicts after update to 3.26.0 (not before) #11974
Comments
|
Same here on three (Android) phones that just upgraded the client app. |
|
Started receiving this behavior last night (I'm guessing am automated update). I suddenly began to get hundreds of upload conflicts from photos that were backed up months ago. I can't dismiss notifications because as soon as I do, another 24 are generated. Samsung galaxy s23 ultra |
|
I think the app settings got messed up after the update. I had to change this setting It didn't seem to help. I uninstalled and reinstalled the app. Then set up the auto upload settings again. So far it seems to be fixed. |
|
Tried to take a look at the release to figure out what might have went wrong. Unfortunately the release notes appear to be poor/insufficient - even the ones on the app store is only a small fraction of what was put in the release Looking through the closed milestone issues: https://github.com/nextcloud/android/milestone/84?closed=1 It seems there was some kind of change to auto upload to add support for other subfolder/dates or something. This seems to me to be the most likely to have cause an accidental regression in sync functionality. #11642 and #11852, tagging the author in case @dean36963 and approving reviewer @AndyScherzinger Note: I am not a nextcloud developer, contributor, or even an android dev. Just trying to help move this issue along. |
|
Yep I'm stuck with this also, a few thousand files all from my instant upload folder became conflicted. Following this, if theres any info needed then happy to share |
|
Seeing the issue here as well...my solution for now was to turn off notifications w/in NextCloud in the hopes that a future (soon) release will correct the behavior. |
|
I'm getting multiple of these alerts too; but strangely all for one file uploaded over a week ago, one of a batch of 20 or so. All the alerts refer to this one file though. Odd. If I click the alert, I get "Error creating conflict dialogue!" pop up over the files screen. |
|
Same issue here, the Android App floods my push messages with file upload conflicts |
|
Same issue here as well. This has happened before in the past. Unacceptable update, where are the Dev's with communication? |
|
just confirming I've got the same error too: |
|
Same on Pixel7 with nextcloud android app - 3.26.0 and NC 27.0.2 with php8.1 and Apache Webserver. |
|
Same for me, some screenshots below. All seem to be stuck between February(more than 1 day, I've seen 2 so far) 2023 and whatever the original date (to the right- server version) should be.
|
|
Same here. Downgraded to 3.25.0 for now. |
|
Sorry if I'm being annoying, but with no developer acknowledgement that the current "stable" release is broken for a growing number of users, I'm tagging in the remaining approving reviewers of the likely problematic PR: @marcoambrosini @jancborchardt @tobiasKaminsky The bug/regression of repeatedly finding upload conflicts is not only very frustrating, it seems nextcloud has bumped up to my top 3 battery draining apps, whereas before it was negligible. |
|
I've reached the point of removing the Android app from my phone & tablet. I've copied all my files off my server. I'll get back to NextCloud when I hear it's functional again. |
|
I have to confirm that the upload behaviour of 3.26.0 is related to enormous battery drain.
|
|
I noticed that I get also notifications for failed uploads on folder were I adjust upload only with wi-fi when I am outside of wi-fi. |
|
Is anyone actually going to look at this issue? If not I am going to remove this from my infrastructure and go elsewhere. |
|
Seeing the same issue on my phone with the update to 3.26.0 as well. While I don't see an increase of battery drainage on my device (Samsung Galaxy S20 FE 5G), I find it very odd that I can't even resolve these conflicts. If I select to keep the server version, the same file will show up as a conflicting file upon the next check cycle of the Nextcloud app. Even more strange is the date Nextcloud reads from the file. Android itself reports the correct date.
The screenshot of @navennn shows a date / time discrepancy as well between local (android) and the server file, although not as dramatic as on my device. Is Nextcloud comparing last changed and file size to detect changes? |
|
Do you only get support if you pay for NC? Support on the forums is awful if you get anything at all. I hoped github would be better. |
|
So there's probably a few issues at play here.
I've had a look, but can't find anything that would cause more conflicts from this change, maybe a more experienced developer of the project could confirm. |
|
@dean36963 thanks for looking into it. Unfortunately it seems like the only auto upload related work captured in the release milestone... Of course, maybe there were prs not captured there, or something more indirect at play 🙁 do you have any way to contact the devs more directly? Personally I think it's worth reverting the release, but I'm just a random NC user |
|
Same here on Xiaomi Redmi note 10. |
|
Ok, |
|
That is what worked for me: I've deleted all data in the Android app settings, also the cache. Then uninstalled the app. After installing the Nextcloud app fresh from the Google Play store and logged in to my NC instance I've setup the synchronization folders and have no more issues. However and why this works - I don't know. But an update should not change a working app or make it worse, so there must be a bug in the update behaviour or how the app is reading existing data on the Android phone. |
|
I've tried the same and it didn't work. Any ideas how to install 3.25? The apk from GitHub can't install properly |
|
Had issues too and made the effort of manually deciding to keep the server version for hundreds of sync conflicts for old and already-uploaded files. -Now it seems to be working again for newly made photos - without change of config or installation.- Battery usage seems to be pretty high though. Edit: Sync conflicts still appear for newly made photos. :( |
|
I know this software is free, but breaking of such a fundamental functionality should not pass basic QA regression testing. Sorry but this is a big red flag in terms of quality and reliability from an otherwise great solution. My solution for now is to downgrade all devices until this (and perhaps the power drain) issue is resolved. |
|
@JonasMayerDev @joshtrichards - results after testing with 30 GB photo folder:
|
This behavior, mixing up file names, is something that happened in my 'file conflict resolving' efforts before as well. Great catch, I forgot about it. I'll be migrating my nextcloud environment in the upcoming days (gasp) and will test with a fresh install of the environment and will add multiple devices for backup, starting these from scratch on Android 3.27. I'll keep you all posted. |
@Artens Just a note for clarity: v3.27.0 doesn't contain all of the changes brought up in this issue. Some are in soon to be released v3.27.1 but others likely will be in v3.28.0 or thereabouts. #12337 is not currently marked for a backport to v3.27.1. |
At least the improvement for autoupload in #12337 should now be included in v3.27.1 if I understand correctly: #12386 |
|
Hi all, I think we finally found the reason for this issue and have a fix for it. ExplainationThe conflicts that were reproducible for me, were caused by interrupting the worker during a non chunked upload. In my case, by force closing the app completely. The interruption of upload could also happen because of the android operating system background task restrictions. If a background job is allowed to execute at a specific time is dependent on different factors e.g. battery saving mode etc. and depending on the android flavor (e.g. android from different manufacturers) which explains why the issue was so hard to reproduce. When killing the upload process in the right moment, the http upload would finish successfully, but the app would not update the internal database of a successful upload. We tried finding bugs in our code that cause the issue and to fix the issue, but in the end it is just not possible using our current approach of sync, which relies on knowing which files were already uploaded, to fix this issue. SolutionPlease check this PR if conflicts still occur for you, if not please see if in the uploads tab under successfully uploaded are files that state "Same file found ...." (see screenshot below): To fix this issue, we now check for every sync conflict if the size, creation and modification date of the remote and local file are the same. If so, we don't notify the user about a conflict and just silently skip the upload with a note in the uploads tab. This solution is not perfect since in theory we not check if the file content is the same and so it could lead to justified conflicts being skipped silently. The desktop sync client uses the same method to tell if a file is "identical" on the remote, and there were no problems with this, which is why we decided to do it in the android client the same way. It fixes the conflicts I could reproduce. I hope it also fixes your conflicts. If not, then there are other bugs that cause these conflicts.
|
|
Awesome news @JonasMayerDev! |
|
Having a checksum for files on the server would be great and was also my first idea, but as far as I know we only have checksums in the metadata of encrypted files. @tovine Do you know a way to get the checksum of a file from the server? I also already asked @tobiasKaminsky, but he didn't know a way to get checksums either. |
As mentioned and referenced we currently do not have checksums provided by the server. This has been discussed numerous times over the last years but always decided against impllemeting it due to expected issues in terms of server performance and database impact in terms up updating the db structures and also calculating the checksums for existing files which is expected to be a huge issue on very large instances with many files. |
|
Fair point, but I still think it would be very useful to have for the purposes of sync (and potentially for detecting unexpected changes or file corruption, like someone else mentioned). |
|
Yes, I think all the points are helpful in getting to a solution that might work. However the first issue is actually to add a column, this does already cause issues on very large instances given the very large table. It could be an option to add it to the meta data table recently introduces but when requesting this info for each and every file, Db access might be very inefficient or maybe access is then only triggered when actually needing this info because sync couldn't resolve matters based on other data. |
Implement it ad-hoc then? Nextcloud has left me with numerous files that got corrupted to 0 bits in recent times, I've always wondered why my phone kept corrupting files, thinking my phone's storage must be unreliable. But then I looked at some NC conflicts and I found it curious that there's an obvious connection. A sync client that practically destroys files needs to at the very least emergency halt the feature altogether, I'd rather have no sync than corrupted files until the lucky day I happen to clue together what might be the problem. This inspires zero confidence in NC for what I consider to be one of the most core, basic needs one might have for an application like this. I'm flabbergasted this is unhandled for months. |
|
Yes, they where identicalBest regards
|
I don't know if the team realizes or not, but this file corruption issue is poisoning the NextCloud name. Performance issues are tolerable; data loss is not. As soon as I encountered this myself - and at a surprising scale - I disabled the NextCloud container completely... I hope to see a robust solution posted in this thread someday so that I can give it another go. If you're declining checksums, what's the alternative that's being worked on? |
|
I did witj the linux shell
|
|
Please shift further file verification discussion to the issue that already exists for tracking it: |
Issue is 8 years old, easy to see why people don't instinctively post there when the way HOW this is tackled is the least of user's concern. But sure I'll add my feedback there as well, because whether it fixes this issue or not, it's a needed feature anyhow, at the very least as an option. |
|
Thanks for fixing this issue 🥳 |
and others
Steps to reproduce
Opening android gallery to view pictures triggers the alert.
Take a picture triggers the alert.
Opening nextcloud app triggers alert.
Expected behaviour
Nextcloud wouldn't try to upload existing files.
Actual behaviour
Nectcloud is trying to re-upload some if not all of the pictures on my phone. I'll get 25 alerts at a time.
The pictures exist in both phone and nextcloud already.
Android version
13
Device brand and model
Samsung Galaxy S23
Stock or custom OS?
Stock
Nextcloud android app version
3.26.0
Nextcloud server version
27.02
Using a reverse proxy?
Yes
Android logs
No response
Server error logs
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: