Smoke tests are flaky

[phil-davis]

I noticed a few of these earlier in the week, but couldn't reproduce it yesterday. Just now I got this:

https://drone.owncloud.com/owncloud/web/20591/10/18

    Then "Brian" downloads the following files
      │ resource │ from             │
      │ data.zip │ Shares/meal plan │
    ✖ failed
      page.click: Timeout 30000ms exceeded.
      =========================== logs ===========================
      waiting for selector "//span[@data-test-resource-name="data.zip"]/ancestor::tr[contains(@class, "oc-tbody-tr")]//button[contains(@class, "oc-table-files-btn-action-dropdown")]"
      ============================================================
          at Object.open (/var/www/owncloud/web/tests/smoke/support/cta/files/sidebar.ts:5:16)
          at AllFilesPage.downloadFiles (/var/www/owncloud/web/tests/smoke/support/page/files/allFiles.ts:64:43)
          at World.<anonymous> (/var/www/owncloud/web/tests/smoke/features/step_definitions/app.files.ts:75:27)

That was running with oC10 server as the back-end. I think that I have only seen it fail when using the oC10 back-end.

[phil-davis]

https://drone.owncloud.com/owncloud/web/20603/10/16

    Then "Alice" downloads the following files
      │ resource      │ from                                 │
      │ parent.txt    │ groups/Kindergarten Koalas/meal plan │
      │ lorem.txt     │ groups/Kindergarten Koalas/meal plan │
      │ lorem-big.txt │ groups/Kindergarten Koalas/meal plan │
      │ data.zip      │ groups/Pre-Schools Pirates/meal plan │
      │ lorem.txt     │ groups/Pre-Schools Pirates/meal plan │
      │ lorem-big.txt │ groups/Pre-Schools Pirates/meal plan │
      │ data.zip      │ groups/Teddy Bear Daycare/meal plan  │
      │ lorem.txt     │ groups/Teddy Bear Daycare/meal plan  │
      │ lorem-big.txt │ groups/Teddy Bear Daycare/meal plan  │
    ✖ failed
      page.click: Timeout 30000ms exceeded.
      =========================== logs ===========================
      waiting for selector "//span[@data-test-resource-name="data.zip"]/ancestor::tr[contains(@class, "oc-tbody-tr")]//button[contains(@class, "oc-table-files-btn-action-dropdown")]"
      ============================================================
          at Object.open (/var/www/owncloud/web/tests/smoke/support/cta/files/sidebar.ts:5:16)
          at AllFilesPage.downloadFiles (/var/www/owncloud/web/tests/smoke/support/page/files/allFiles.ts:64:43)
          at World.<anonymous> (/var/www/owncloud/web/tests/smoke/features/step_definitions/app.files.ts:75:27)

Another example.

[phil-davis]

https://drone.owncloud.com/owncloud/web/20621/11/13

    Given "Brian" has logged in
    ✖ failed
      page.fill: Timeout 30000ms exceeded.
      =========================== logs ===========================
      waiting for selector "#oc-login-username"
      ============================================================
          at OcisLoginAdapter.login (/var/www/owncloud/web/tests/smoke/support/page/login/ocis.ts:12:20)
          at LoginPage.login (/var/www/owncloud/web/tests/smoke/support/page/login/index.ts:16:28)
          at World.<anonymous> (/var/www/owncloud/web/tests/smoke/features/step_definitions/session.ts:10:21)

That one failed with oCIS backend.

I really don't understand why the smoke-test "user journey" tests have become flaky in the last few days.

[phil-davis]

@ScharfViktor FYI - I wonder what has happened to make the smoke test scenarios flaky?

[ScharfViktor]

A large PR has been added in the last few days. #6005
although some of the errors that you have attached do not apply to the PR.
I will identify flaky places in test into CI and fix it

[ScharfViktor]

https://drone.owncloud.com/owncloud/web/20624/11/12

[ScharfViktor]

https://drone.owncloud.com/owncloud/web/20618/11/12

[ScharfViktor]

https://drone.owncloud.com/owncloud/web/20634/10/17

[phil-davis]

PR #6092 should have improved this.

But the merge CI had a smoke test fail:
https://drone.owncloud.com/owncloud/web/20783/9/12

    When "Alice" creates new version of the following files
      │ resource   │ to               │
      │ simple.pdf │ folder_to_shared │
    Then "Alice" ensures that the resource "folder_to_shared/simple.pdf" has 1 version
    ✖ failed
      Error: expect(received).toEqual(expected) // deep equality

      Expected: 1
      Received: 0
          at World.<anonymous> (/var/www/owncloud/web/tests/smoke/features/step_definitions/app.files.ts:156:80)

That was the pipeline running on oCIS. The pipeline on oC10 passed.

We need to see if this is intermittent on only oCIS or also on oC10.