Improve backup experience from within Nextcloud

[grote]

This is not strictly a Seedvault issue, but I still open it here for tracking.

Resolve various issues to improve the backup experience when Nextcloud is used as a backup location.

• improve DocumentProvider (SAF) performance
  • improve performance of isChildDocument() when parent is root [PR]
  • reduce number of HTTP requests and sleep times needed for one file operation [PR]
• investigate/fix issues with file locking (probably due to unwanted parallelism) (solved by [PR])
• fix crashes listed below
  • NullPointerException: getStatusCode [link] [PR]
  • FileNotFoundException: Failed to upload document with path [link] [ticket] [PR] [PR]
  • NullPointerException: onTransferProgress [link] [PR]
• uploaded files often 0 bytes aka data loss [ticket] [PR]
• write integration tests to be run as part of Nextcloud CI, so support never breaks unexpected [ticket] [PR]
• suppress/group the various upload/download notifications shown for a single backup [ticket] [PR] [PR] [PR]
• figure out how to address issues when Nextcloud passcode is active [ticket] [nice-to-have ticket] [our PR] [PR]

[grote]

This should get fixed inside Nextcloud:

    java.lang.NullPointerException: Attempt to invoke virtual method 'int org.apache.commons.httpclient.StatusLine.getStatusCode()' on a null object reference
        at org.apache.commons.httpclient.HttpMethodBase.getStatusCode(HttpMethodBase.java:570)
        at com.owncloud.android.lib.common.operations.RemoteOperationResult.<init>(RemoteOperationResult.java:328)
        at com.owncloud.android.lib.resources.files.DownloadFileRemoteOperation.run(DownloadFileRemoteOperation.java:85)
        at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:181)
        at com.owncloud.android.operations.DownloadFileOperation.run(DownloadFileOperation.java:160)
        at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:181)
        at com.owncloud.android.files.services.FileDownloader.downloadFile(FileDownloader.java:471)
        at com.owncloud.android.files.services.FileDownloader.access$500(FileDownloader.java:81)
        at com.owncloud.android.files.services.FileDownloader$ServiceHandler.handleMessage(FileDownloader.java:422)

It causes the backup to hang, because we don't get a response from a hanging Nextcloud. Happened with 3.12.1.

[grote]

This one is strange, because there's no real clue in the logs:

D/OwnCloudClient #0: REQUEST PUT /remote.php/webdav/.SeedVaultAndroidBackup/1597318291048/kv/%40pm%40/bmV0LnRhbGVyLmNhc2hpZXI
java.io.FileNotFoundException: Failed to upload document with path /.SeedVaultAndroidBackup/1597318291048/kv/@pm@/bmV0LnRhbGVyLmNhc2hpZXI
        at com.owncloud.android.providers.DocumentsStorageProvider.createFile(DocumentsStorageProvider.java:540)
        at com.owncloud.android.providers.DocumentsStorageProvider.createDocument(DocumentsStorageProvider.java:460)
        at android.provider.DocumentsProvider.callUnchecked(DocumentsProvider.java:1121)
        at android.provider.DocumentsProvider.call(DocumentsProvider.java:1067)

it throws an exception up the stack and makes the backup fail:

E/DatabaseUtils: Writing exception to parcel
    android.os.ParcelableException: java.io.FileNotFoundException: Failed to upload document with path /.SeedVaultAndroidBackup/1597318291048/kv/@pm@/bmV0LnRhbGVyLmNhc2hpZXI
        at android.provider.DocumentsProvider.call(DocumentsProvider.java:1069)

Potentially related: nextcloud/android#5940

Also logged: org.apache.commons.httpclient.NoHttpResponseException: The server [redacted] failed to respond
Unrecovered transport exception

[grote]

Another Nextcloud NullPointerException:

E/UploadFileOperation: Upload of /storage/emulated/0/Android/media/com.nextcloud.client/nextcloud/[account]/.SeedVaultAndroidBackup/1597327518558/kv/@pm@/ZGUuZ3JvYm94LmxpYnJlYXNzaXN0YW50LmRlYnVn to /.SeedVaultAndroidBackup/1597327518558/kv/@pm@/ZGUuZ3JvYm94LmxpYnJlYXNzaXN0YW50LmRlYnVn: Unexpected exception
    java.lang.NullPointerException: Attempt to invoke virtual method 'void android.app.NotificationManager.notify(int, android.app.Notification)' on a null object reference
        at com.owncloud.android.files.services.FileUploader.onTransferProgress(FileUploader.java:708)
        at com.owncloud.android.lib.common.network.FileRequestEntity.writeRequest(FileRequestEntity.java:124)
        at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499)
        at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114)
        at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
        at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
        at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
        at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
        at com.owncloud.android.lib.common.OwnCloudClient.executeMethod(OwnCloudClient.java:219)
        at com.owncloud.android.lib.resources.files.UploadFileRemoteOperation.uploadFile(UploadFileRemoteOperation.java:156)
        at com.owncloud.android.lib.resources.files.UploadFileRemoteOperation.run(UploadFileRemoteOperation.java:114)
        at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:181)
        at com.owncloud.android.operations.UploadFileOperation.normalUpload(UploadFileOperation.java:850)
        at com.owncloud.android.operations.UploadFileOperation.run(UploadFileOperation.java:431)
        at com.owncloud.android.lib.common.operations.RemoteOperation.execute(RemoteOperation.java:181)
        at com.owncloud.android.operations.common.SyncOperation.execute(SyncOperation.java:85)
        at com.owncloud.android.files.services.FileUploader.uploadFile(FileUploader.java:611)
        at com.owncloud.android.files.services.FileUploader$ServiceHandler.handleMessage(FileUploader.java:1347)

[grote]

All work here is done and the result will be available in the upcoming Nextcloud 3.14 or already in their nightly dev releases.