Acceptance tests are failing nightly

[saw-jan]

Build: https://drone.owncloud.com/owncloud/encryption/2732/

Pipelines

CLI

cliEncryption-master-mysql8.0-php7.4
cliEncryption-master-postgres9.4-php7.4
cliCoreMKey-7-3-master-mysql8.0-php7.4
cliCoreMKey-7-4-master-mysql8.0-php7.4

Scenario: recreate masterkey and upload data                                                                               # /var/www/owncloud/testrunner/apps/encryption/tests/acceptance/features/cliEncryption/recreatemasterkey.feature:9
    Given user "Alice" has been created with default attributes and small skeleton files                                     # FeatureContext::userHasBeenCreatedWithDefaultAttributes()
    And user "Alice" has uploaded file "filesForUpload/textfile.txt" to "/somefile.txt"                                      # FeatureContext::userHasUploadedAFileTo()
    When the administrator successfully recreates the encryption masterkey using the occ command                             # EncryptionContext::recreateMasterKeyUsingOccCommand()
    And user "Alice" uploads chunk file "1" of "1" with "AA" to "/somefile.txt" using the WebDAV API                         # FeatureContext::userUploadsChunkedFile()
    Then the downloaded content when downloading file "/somefile.txt" for user "Alice" with range "bytes=0-3" should be "AA" # FeatureContext::downloadedContentWhenDownloadingForUserWithRangeShouldBe()
      The content was expected to be 'AA', but actually is 'This'. HTTP status was 206
      Failed asserting that two strings are equal.
      --- Expected
      +++ Actual
      @@ @@
      -'AA'
      +'This'

API

apiCoreMKey-40-2-master-mysql8.0-php7.4
apiCoreMKey-40-4-master-mysql8.0-php7.4
apiCoreMKey-40-5-master-mysql8.0-php7.4
apiCoreMKey-40-6-master-mysql8.0-php7.4
apiCoreMKey-40-6-master-mysql8.0-php7.4
apiCoreMKey-40-11-master-mysql8.0-php7.4
apiCoreMKey-40-12-master-mysql8.0-php7.4
apiCoreMKey-40-13-master-mysql8.0-php7.4
apiCoreMKey-40-15-master-mysql8.0-php7.4
apiCoreMKey-40-16-master-mysql8.0-php7.4
apiCoreMKey-40-17-master-mysql8.0-php7.4
apiCoreMKey-40-22-master-mysql8.0-php7.4
apiCoreMKey-40-23-master-mysql8.0-php7.4
apiCoreMKey-40-24-master-mysql8.0-php7.4
apiCoreMKey-40-25-master-mysql8.0-php7.4
apiCoreMKey-40-26-master-mysql8.0-php7.4
apiCoreMKey-40-27-master-mysql8.0-php7.4
apiCoreMKey-40-32-master-mysql8.0-php7.4
apiCoreMKey-40-36-master-mysql8.0-php7.4
apiCoreMKey-40-38-master-mysql8.0-php7.4
apiCoreMKey-40-39-master-mysql8.0-php7.4

WebUI

webUIcoreMKey-20-1-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-2-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-4-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-6-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-7-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-9-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-12-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-14-master-chrome-mysql8.0-php7.4
webUIcoreMKey-20-16-master-chrome-mysql8.0-php7.4

[saw-jan]

Related failures in other apps:
❗ https://github.com/owncloud/files_spaces/issues/193
❗ https://github.com/owncloud/ransomware_protection/issues/266

[grgprarup]

The tests are failing after the PR (owncloud/core#40433) got merged into the core.
CC @jvillafanez

[jvillafanez]

I don't know if owncloud/core#40471 will fix any of the tests.

owncloud/core#40433 has an issue when files with versions are moved to the trashbin. That's what owncloud/core#40471 should fix. I don't know if for all the failing test we can pin the blame to that particular issue.

Maybe we can merge owncloud/core#40471 (assuming it passes all the tests) and check if it helps with all these acceptance tests. Worst case, we might need to revert those 2 PRs.

[grgprarup]

I don't know if owncloud/core#40471 will fix any of the tests.

owncloud/core#40433 has an issue when files with versions are moved to the trashbin. That's what owncloud/core#40471 should fix. I don't know if for all the failing test we can pin the blame to that particular issue.

Maybe we can merge owncloud/core#40471 (assuming it passes all the tests) and check if it helps with all these acceptance tests. Worst case, we might need to revert those 2 PRs.

@jvillafanez With this PR (owncloud/core#40471) still, encryption tests are failing. ❌
See test PR #374 and build https://drone.owncloud.com/owncloud/encryption/2749/9/11

[jvillafanez]

I'd vote to revert the owncloud/core#40433 PR. I've checked a couple of failing tests and I haven't seen anything in common. It seems we'll have to check each case one by one, and it will take a lot of time.

Is it possible to collect all the failing tests somewhere in order to have a test plan for the next time we attempt this?

[saw-jan]

Is it possible to collect all the failing tests somewhere in order to have a test plan for the next time we attempt this?

Yeah sure, we will create a list of all the failing tests so that each different failure can be checked well.

CC @grgprarup

[jvillafanez]

PR reverting the changes in owncloud/core#40475

The secondary PR (owncloud/core#40471) hasn't been merged, so we can close it

[phil-davis]

owncloud/core#40475 has been merged. Let's see the overnight CI results...