Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent e2e/cypress test failure. #1644

Closed
afs opened this issue Nov 30, 2022 · 15 comments · Fixed by #1734
Closed

Intermittent e2e/cypress test failure. #1644

afs opened this issue Nov 30, 2022 · 15 comments · Fixed by #1734
Assignees

Comments

@afs
Copy link
Member

afs commented Nov 30, 2022

This issue is to collect information

This test in Fuseki UI has started failing for me again.

It has been an intermittent failure; typically if a codebase works (mavne build passes) it keeps working and if it fails it fails nearly every build.

[INFO] [TESTS]   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
[INFO] [TESTS]   │ ✖  datasets.cy.js                           00:27        9        8        1        -        - │
[INFO] [TESTS]   ├────────────────────────────────────────────────────────────────────────────────────────────────┤
[INFO] [TESTS]   │ ✔  query.cy.js                              00:01        1        1        -        -        - │
[INFO] [TESTS]   └────────────────────────────────────────────────────────────────────────────────────────────────┘
[INFO] [TESTS]     ✖  1 of 2 failed (50%)                      00:28       10        9        1        -        -  

It is not maven dependency - running it outside maven has the same failure mode.

@afs
Copy link
Member Author

afs commented Nov 30, 2022

Linux. Docker Desktop installed.

I now have the situation where if it is failing then stopping docker.service causes the build to work again. Restarting usually (not always!) causes the build to fail again.

Docker reconfigures local networking.

@kinow
Copy link
Member

kinow commented Nov 30, 2022

Hmmm. I better take a look at this test. Linking the other issue where we discussed the same test: #1480 (comment)

@afs
Copy link
Member Author

afs commented Nov 30, 2022

I don't think it is the test per se.

Another data point: when it stop the docker.service, it is currently still usually failing once, then starts working.
And restarting docker.service, usually, passes once, then fails consistently.

@kinow
Copy link
Member

kinow commented Nov 30, 2022

I don't think it is the test per se.

Sorry, I had to read the other thread to remember what tests were failing. But your comment might be correct... the linked issue had this test failing in your logs, I believe:

[INFO] [TESTS]   1) datasets
[INFO] [TESTS]        Visits datasets page, also the application landing-page:
[INFO] [TESTS]      AssertionError: Timed out retrying after 7500ms: Expected to find element: `h2.text-center`, but never found it.
[INFO] [TESTS]       at Context.eval (webpack://apache-jena-fuseki/./tests/e2e/specs/datasets.cy.js:31:7)
[INFO] [TESTS] 

That test does very little. It visits /, waits for the loading spinning element to disappear, and tests that the HTML generated contains certain elements. I guess we could check if the tests that are failing are, perhaps, related to that loading / in progress element.

Maybe that element, or the logic to generate it, suffers when there is a delay to load, or a network issue? Is that test still failing, when you have failures? And do the failures always include the same tests? Or does it vary between runs?

@afs
Copy link
Member Author

afs commented Nov 30, 2022

The 3 test failure in #1480 was because HOST was set which it isn't now.

We should probably undo 242725c which reorders tests which seemed to work at the time.

If necessary, we can turn off the e2e test for the release when built on my machine. We can be fairly sure the tests do what they should do and it is an environment or framework issue.

@afs
Copy link
Member Author

afs commented Dec 28, 2022

Occurred in the Linux github action:

https://github.com/apache/jena/actions/runs/3787982879/jobs/6440304739#step:4:4634

Commit 73462ed changes the POM configuration so that it logs the failure but does not fail the maven build.

@kinow
Copy link
Member

kinow commented Dec 28, 2022

That's really excellent to have the complete output, @afs. To save it for later, here's the complete e2e output:

error
2022-12-27T14:17:44.4338825Z [INFO] $ bash -c 'concurrently --names "SERVER,TESTS" --prefix-colors "yellow,blue" --success "first" --kill-others "yarn run serve:fuseki" "vue-cli-service test:e2e ${0}"' --headless --browser chrome
2022-12-27T14:17:44.8292567Z [INFO] [SERVER] $ nodemon src/services/mock/json-server.js
2022-12-27T14:17:44.9471980Z [INFO] [SERVER] [nodemon] 2.0.20
2022-12-27T14:17:44.9475870Z [INFO] [SERVER] [nodemon] to restart at any time, enter `rs`
2022-12-27T14:17:44.9476464Z [INFO] [SERVER] [nodemon] watching path(s): *.*
2022-12-27T14:17:44.9477209Z [INFO] [SERVER] [nodemon] watching extensions: js,mjs,json
2022-12-27T14:17:44.9477862Z [INFO] [SERVER] [nodemon] starting `node src/services/mock/json-server.js`
2022-12-27T14:17:45.1755395Z [INFO] [SERVER] JSON Server is running
2022-12-27T14:17:45.4135963Z [INFO] [TESTS]  INFO  Starting e2e tests...
2022-12-27T14:17:45.4145594Z [INFO] [TESTS]  INFO  Starting development server...
2022-12-27T14:18:10.8091932Z [INFO] [TESTS]  WARNING  Compiled with 2 warnings2:18:10 PM
2022-12-27T14:18:10.8092533Z [INFO] [TESTS] 
2022-12-27T14:18:10.8092812Z [INFO] [TESTS]  warning  
2022-12-27T14:18:10.8093083Z [INFO] [TESTS] 
2022-12-27T14:18:10.8093477Z [INFO] [TESTS] asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
2022-12-27T14:18:10.8093921Z [INFO] [TESTS] This can impact web performance.
2022-12-27T14:18:10.8094234Z [INFO] [TESTS] Assets: 
2022-12-27T14:18:10.8094585Z [INFO] [TESTS]   static/js/datasetQuery.8f1b50bd.js (1.2 MiB)
2022-12-27T14:18:10.8095220Z [INFO] [TESTS]   static/js/chunk-vendors.a5488600.js (360 KiB)
2022-12-27T14:18:10.8095533Z [INFO] [TESTS] 
2022-12-27T14:18:10.8095803Z [INFO] [TESTS]  warning  
2022-12-27T14:18:10.8096070Z [INFO] [TESTS] 
2022-12-27T14:18:10.8096584Z [INFO] [TESTS] entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
2022-12-27T14:18:10.8097072Z [INFO] [TESTS] Entrypoints:
2022-12-27T14:18:10.8097383Z [INFO] [TESTS]   app (767 KiB)
2022-12-27T14:18:10.8097811Z [INFO] [TESTS]       static/js/chunk-vendors.a5488600.js
2022-12-27T14:18:10.8098199Z [INFO] [TESTS]       static/css/app.92ffc00b.css
2022-12-27T14:18:10.8098555Z [INFO] [TESTS]       static/js/app.4decce5d.js
2022-12-27T14:18:10.8098869Z [INFO] [TESTS] 
2022-12-27T14:18:10.8099122Z [INFO] [TESTS] 
2022-12-27T14:18:10.8099365Z [INFO] [TESTS] 
2022-12-27T14:18:10.8099651Z [INFO] [TESTS]   App running at:
2022-12-27T14:18:10.8100102Z [INFO] [TESTS]   - Local:   http://localhost:38165 
2022-12-27T14:18:10.8100535Z [INFO] [TESTS]   - Network: http://10.1.0.199:38165
2022-12-27T14:18:10.8100830Z [INFO] [TESTS] 
2022-12-27T14:18:10.8101138Z [INFO] [TESTS]   App is served in production mode.
2022-12-27T14:18:10.8101786Z [INFO] [TESTS]   Note this is for preview or E2E testing only.
2022-12-27T14:18:10.8102108Z [INFO] [TESTS] 
2022-12-27T14:18:11.1757574Z [INFO] [TESTS] It looks like this is your first time using Cypress: 10.11.0
2022-12-27T14:18:11.1763880Z [INFO] [TESTS] 
2022-12-27T14:18:11.1797504Z [INFO] [TESTS] [STARTED] Task without title.
2022-12-27T14:18:14.7336246Z [INFO] [TESTS] [SUCCESS] Task without title.
2022-12-27T14:18:14.7340310Z [INFO] [TESTS] 
2022-12-27T14:18:14.7340674Z [INFO] [TESTS] Opening Cypress...
2022-12-27T14:18:15.9257480Z [INFO] [TESTS] [5092:1227/141815.924660:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
2022-12-27T14:18:23.4126104Z [INFO] [TESTS] 
2022-12-27T14:18:23.4617778Z [INFO] [TESTS] ====================================================================================================
2022-12-27T14:18:23.4619343Z [INFO] [TESTS] tput: No value for $TERM and no -T specified
2022-12-27T14:18:23.4620165Z [INFO] [TESTS] 
2022-12-27T14:18:23.4628272Z [INFO] [TESTS]   (Run Starting)
2022-12-27T14:18:23.4628754Z [INFO] [TESTS] 
2022-12-27T14:18:23.4712166Z [INFO] [TESTS]   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
2022-12-27T14:18:23.4712770Z [INFO] [TESTS]   │ Cypress:        10.11.0                                                                        │
2022-12-27T14:18:23.4713267Z [INFO] [TESTS]   │ Browser:        Electron 106 (headless)                                                        │
2022-12-27T14:18:23.4714030Z [INFO] [TESTS]   │ Node Version:   v16.13.1 (/home/runner/work/jena/jena/jena-fuseki2/jena-fuseki-ui/node/no      │
2022-12-27T14:18:23.4714523Z [INFO] [TESTS]   │                 de)                                                                            │
2022-12-27T14:18:23.4714960Z [INFO] [TESTS]   │ Specs:          2 found (datasets.cy.js, query.cy.js)                                          │
2022-12-27T14:18:23.4715419Z [INFO] [TESTS]   │ Searched:       tests/e2e/specs                                                                │
2022-12-27T14:18:23.4752994Z [INFO] [TESTS]   └────────────────────────────────────────────────────────────────────────────────────────────────┘
2022-12-27T14:18:23.4753404Z [INFO] [TESTS] 
2022-12-27T14:18:23.4792674Z [INFO] [TESTS] 
2022-12-27T14:18:23.4793201Z [INFO] [TESTS] ────────────────────────────────────────────────────────────────────────────────────────────────────
2022-12-27T14:18:23.4793524Z [INFO] [TESTS]                                                                                                     
2022-12-27T14:18:23.4793870Z [INFO] [TESTS]   Running:  datasets.cy.js                                                                  (1 of 2)
2022-12-27T14:18:27.8737180Z [INFO] [TESTS]  DONE  Compiled successfully in 2275ms2:18:27 PM
2022-12-27T14:18:27.8746280Z [INFO] [TESTS] 
2022-12-27T14:18:28.6788638Z [INFO] [TESTS]  DONE  Compiled successfully in 2978ms2:18:28 PM
2022-12-27T14:18:28.6789766Z [INFO] [TESTS] 
2022-12-27T14:18:28.8206414Z [INFO] [TESTS] 
2022-12-27T14:18:28.8224917Z [INFO] [TESTS] 
2022-12-27T14:18:28.8515532Z [INFO] [TESTS]   datasets
2022-12-27T14:18:29.2182903Z [INFO] [SERVER] GET /$/ping 200 3.447 ms - 26
2022-12-27T14:18:29.2277247Z [INFO] [SERVER] GET /$/server 200 0.592 ms - 145
2022-12-27T14:18:29.3425046Z [INFO] [TESTS]     ✓ Visits datasets page, also the application landing-page (481ms)
2022-12-27T14:18:29.5372497Z [INFO] [SERVER] GET /$/ping 200 0.494 ms - 26
2022-12-27T14:18:29.5589474Z [INFO] [SERVER] GET /$/server 304 0.534 ms - -
2022-12-27T14:18:29.8912445Z [INFO] [TESTS]     ✓ Filters without any data (514ms)
2022-12-27T14:18:29.8921192Z [INFO] [TESTS]     After creating new datasets
2022-12-27T14:18:29.9117103Z [INFO] [SERVER] GET /tests/reset 200 0.605 ms - 2
2022-12-27T14:18:30.0048644Z [INFO] [SERVER] GET /$/ping 200 0.334 ms - 26
2022-12-27T14:18:30.3778624Z [INFO] [SERVER] POST /$/datasets 200 8.980 ms - 2
2022-12-27T14:18:30.4745590Z [INFO] [SERVER] GET /$/server 200 0.289 ms - -
2022-12-27T14:18:30.7919950Z [INFO] [SERVER] POST /$/datasets 200 0.650 ms - 2
2022-12-27T14:18:30.8144636Z [INFO] [SERVER] GET /$/server 200 0.364 ms - -
2022-12-27T14:18:31.1393437Z [INFO] [SERVER] POST /$/datasets 200 0.681 ms - 2
2022-12-27T14:18:31.1648176Z [INFO] [SERVER] GET /$/server 200 0.331 ms - -
2022-12-27T14:18:31.5269694Z [INFO] [SERVER] POST /$/datasets 200 0.682 ms - 2
2022-12-27T14:18:31.5552387Z [INFO] [SERVER] GET /$/server 200 0.305 ms - -
2022-12-27T14:18:31.8739818Z [INFO] [SERVER] POST /$/datasets 200 0.588 ms - 2
2022-12-27T14:18:31.8957063Z [INFO] [SERVER] GET /$/server 200 0.296 ms - -
2022-12-27T14:18:32.2137650Z [INFO] [SERVER] POST /$/datasets 200 0.487 ms - 2
2022-12-27T14:18:32.2408265Z [INFO] [SERVER] GET /$/server 200 0.306 ms - -
2022-12-27T14:18:32.5628853Z [INFO] [SERVER] POST /$/datasets 200 0.592 ms - 2
2022-12-27T14:18:32.5828651Z [INFO] [SERVER] GET /$/server 200 0.321 ms - -
2022-12-27T14:18:32.9107166Z [INFO] [SERVER] POST /$/datasets 200 0.603 ms - 2
2022-12-27T14:18:32.9332303Z [INFO] [SERVER] GET /$/server 200 0.339 ms - -
2022-12-27T14:18:33.2715307Z [INFO] [SERVER] POST /$/datasets 200 0.663 ms - 2
2022-12-27T14:18:33.2990752Z [INFO] [SERVER] GET /$/server 200 0.322 ms - -
2022-12-27T14:18:33.6546723Z [INFO] [SERVER] POST /$/datasets 200 0.585 ms - 2
2022-12-27T14:18:33.6736025Z [INFO] [SERVER] GET /$/server 200 0.325 ms - -
2022-12-27T14:18:34.0500877Z [INFO] [SERVER] POST /$/datasets 200 0.482 ms - 2
2022-12-27T14:18:34.1291869Z [INFO] [SERVER] GET /$/server 200 0.363 ms - -
2022-12-27T14:18:34.3048379Z [INFO] [SERVER] GET /$/server 200 0.341 ms - -
2022-12-27T14:18:34.4029721Z [INFO] [SERVER] GET /$/stats/a 200 0.656 ms - -
2022-12-27T14:18:35.0616745Z [INFO] [SERVER] GET /$/ping 200 0.223 ms - 26
2022-12-27T14:18:40.0692780Z [INFO] [SERVER] GET /$/ping 200 0.233 ms - 26
2022-12-27T14:18:45.0653188Z [INFO] [SERVER] GET /$/ping 200 0.232 ms - 26
2022-12-27T14:18:50.0653852Z [INFO] [SERVER] GET /$/ping 200 0.233 ms - 26
2022-12-27T14:18:54.5906403Z [INFO] [TESTS]       1) Edits the graph
2022-12-27T14:18:54.9693618Z [INFO] [SERVER] GET /$/ping 200 0.216 ms - 26
2022-12-27T14:18:54.9989046Z [INFO] [SERVER] GET /$/server 200 0.334 ms - -
2022-12-27T14:18:55.1348279Z [INFO] [TESTS]       ✓ Visits datasets page (309ms)
2022-12-27T14:18:55.2700053Z [INFO] [SERVER] GET /$/ping 200 0.227 ms - 26
2022-12-27T14:18:55.2927584Z [INFO] [SERVER] GET /$/server 200 1.177 ms - -
2022-12-27T14:18:55.3450341Z [INFO] [SERVER] GET /$/server 304 0.443 ms - -
2022-12-27T14:18:55.5615883Z [INFO] [TESTS]       ✓ Uses the table pagination (349ms)
2022-12-27T14:18:55.6971000Z [INFO] [SERVER] GET /$/ping 200 0.211 ms - 26
2022-12-27T14:18:55.7119979Z [INFO] [SERVER] GET /$/server 304 0.345 ms - -
2022-12-27T14:18:55.7665372Z [INFO] [SERVER] GET /$/server 304 0.340 ms - -
2022-12-27T14:18:56.1116935Z [INFO] [TESTS]       ✓ Sorts the table (502ms)
2022-12-27T14:18:56.3312522Z [INFO] [SERVER] GET /$/ping 200 0.207 ms - 26
2022-12-27T14:18:56.3381078Z [INFO] [SERVER] GET /$/server 200 0.297 ms - -
2022-12-27T14:18:56.4128177Z [INFO] [SERVER] GET /$/server 304 1.523 ms - -
2022-12-27T14:18:56.7499569Z [INFO] [TESTS]       ✓ Filters the table (592ms)
2022-12-27T14:18:56.8750237Z [INFO] [SERVER] GET /$/ping 200 0.218 ms - 26
2022-12-27T14:18:56.8825359Z [INFO] [SERVER] GET /$/server 304 0.311 ms - -
2022-12-27T14:18:56.9755302Z [INFO] [SERVER] GET /$/server 304 0.335 ms - -
2022-12-27T14:18:57.3095500Z [INFO] [SERVER] POST /$/datasets 409 0.478 ms - 37
2022-12-27T14:18:57.4871156Z [INFO] [TESTS]       ✓ Cannot add two datasets with the same name (610ms)
2022-12-27T14:18:57.6107603Z [INFO] [SERVER] GET /$/ping 200 0.214 ms - 26
2022-12-27T14:18:57.6629978Z [INFO] [SERVER] GET /$/server 200 0.308 ms - -
2022-12-27T14:18:57.6880652Z [INFO] [SERVER] GET /$/server 304 0.332 ms - -
2022-12-27T14:18:57.7209621Z [INFO] [SERVER] GET /$/stats/a 304 0.260 ms - -
2022-12-27T14:18:57.7330716Z [INFO] [SERVER] GET /$/stats/a 304 0.248 ms - -
2022-12-27T14:18:57.9241216Z [INFO] [SERVER] GET /a/sparql?query=select+(count(*)+as+%3Fcount)+%7B%3Fs+%3Fp+%3Fo%7D 200 0.505 ms - 259
2022-12-27T14:18:57.9258824Z [INFO] [SERVER] GET /a/sparql?query=select+%3Fg+(count(*)+as+%3Fcount)+%7Bgraph+%3Fg+%7B%3Fs+%3Fp+%3Fo%7D%7D+group+by+%3Fg 200 0.190 ms - 103
2022-12-27T14:18:57.9426918Z [INFO] [SERVER] GET /$/stats/a 304 0.226 ms - -
2022-12-27T14:18:58.0727950Z [INFO] [SERVER] GET /tests/reset 200 0.383 ms - 2
2022-12-27T14:18:58.0920099Z [INFO] [SERVER] GET /__coverage__ 200 0.264 ms - 257
2022-12-27T14:18:58.2823048Z [INFO] [TESTS] 
2022-12-27T14:18:58.2836413Z [INFO] [TESTS] =============================== Coverage summary ===============================
2022-12-27T14:18:58.2836942Z [INFO] [TESTS] Statements   : 72.78% ( 238/327 )
2022-12-27T14:18:58.2940314Z [INFO] [TESTS] Branches     : 79.78% ( 75/94 )
2022-12-27T14:18:58.2940598Z [INFO] [TESTS] Functions    : 74.79% ( 92/123 )
2022-12-27T14:18:58.2940855Z [INFO] [TESTS] Lines        : 72.61% ( 236/325 )
2022-12-27T14:18:58.2941138Z [INFO] [TESTS] ================================================================================
2022-12-27T14:18:58.2996531Z [INFO] [TESTS]       ✓ Visualizes the dataset information (Info View, tab) (527ms)
2022-12-27T14:18:58.2996852Z [INFO] [TESTS] 
2022-12-27T14:18:58.2997039Z [INFO] [TESTS] 
2022-12-27T14:18:58.2997255Z [INFO] [TESTS]   8 passing (29s)
2022-12-27T14:18:58.2997484Z [INFO] [TESTS]   1 failing
2022-12-27T14:18:58.2997677Z [INFO] [TESTS] 
2022-12-27T14:18:58.3011424Z [INFO] [TESTS]   1) datasets
2022-12-27T14:18:58.3012123Z [INFO] [TESTS]        After creating new datasets
2022-12-27T14:18:58.3012658Z [INFO] [TESTS]          Edits the graph:
2022-12-27T14:18:58.3013277Z [INFO] [TESTS]      AssertionError: Timed out retrying after 20000ms: Expected to find element: `table.jena-table > thead > tr > th`, but never found it.
2022-12-27T14:18:58.3013884Z [INFO] [TESTS]       at Context.eval (webpack://apache-jena-fuseki/./tests/e2e/specs/datasets.cy.js:100:9)
2022-12-27T14:18:58.3014218Z [INFO] [TESTS] 
2022-12-27T14:18:58.3014408Z [INFO] [TESTS] 
2022-12-27T14:18:58.3014579Z [INFO] [TESTS] 
2022-12-27T14:18:58.3031651Z [INFO] [TESTS] 
2022-12-27T14:18:58.3031941Z [INFO] [TESTS]   (Results)
2022-12-27T14:18:58.3066139Z [INFO] [TESTS] 
2022-12-27T14:18:58.3089572Z [INFO] [TESTS]   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
2022-12-27T14:18:58.3090275Z [INFO] [TESTS]   │ Tests:        9                                                                                │
2022-12-27T14:18:58.3091198Z [INFO] [TESTS]   │ Passing:      8                                                                                │
2022-12-27T14:18:58.3092142Z [INFO] [TESTS]   │ Failing:      1                                                                                │
2022-12-27T14:18:58.3092577Z [INFO] [TESTS]   │ Pending:      0                                                                                │
2022-12-27T14:18:58.3092983Z [INFO] [TESTS]   │ Skipped:      0                                                                                │
2022-12-27T14:18:58.3093402Z [INFO] [TESTS]   │ Screenshots:  1                                                                                │
2022-12-27T14:18:58.3093807Z [INFO] [TESTS]   │ Video:        false                                                                            │
2022-12-27T14:18:58.3094221Z [INFO] [TESTS]   │ Duration:     29 seconds                                                                       │
2022-12-27T14:18:58.3094656Z [INFO] [TESTS]   │ Spec Ran:     datasets.cy.js                                                                   │
2022-12-27T14:18:58.3095146Z [INFO] [TESTS]   └────────────────────────────────────────────────────────────────────────────────────────────────┘
2022-12-27T14:18:58.3095392Z [INFO] [TESTS] 
2022-12-27T14:18:58.3095571Z [INFO] [TESTS] 
2022-12-27T14:18:58.3095774Z [INFO] [TESTS]   (Screenshots)
2022-12-27T14:18:58.3095974Z [INFO] [TESTS] 
2022-12-27T14:18:58.3108539Z [INFO] [TESTS]   -  tests/e2e/screenshots/datasets.cy.js/datasets -- After creating new datasets --      (1280x720)
2022-12-27T14:18:58.3116922Z [INFO] [TESTS]      Edits the graph (failed).png                                                                   
2022-12-27T14:18:58.3117233Z [INFO] [TESTS] 
2022-12-27T14:18:58.3219112Z [INFO] [TESTS] 
2022-12-27T14:18:58.3233202Z [INFO] [TESTS] ────────────────────────────────────────────────────────────────────────────────────────────────────
2022-12-27T14:18:58.3233689Z [INFO] [TESTS]                                                                                                     
2022-12-27T14:18:58.3234311Z [INFO] [TESTS]   Running:  query.cy.js                                                                     (2 of 2)
2022-12-27T14:19:00.2508942Z [INFO] [TESTS]  DONE  Compiled successfully in 430ms2:19:00 PM
2022-12-27T14:19:00.2509499Z [INFO] [TESTS] 
2022-12-27T14:19:00.3247449Z [INFO] [TESTS] 
2022-12-27T14:19:00.3294344Z [INFO] [TESTS] 
2022-12-27T14:19:00.3584127Z [INFO] [TESTS]   Query
2022-12-27T14:19:00.3712113Z [INFO] [SERVER] GET /tests/reset 200 0.212 ms - 2
2022-12-27T14:19:00.6949228Z [INFO] [SERVER] GET /$/ping 200 0.213 ms - 26
2022-12-27T14:19:01.1622474Z [INFO] [SERVER] POST /$/datasets 200 0.515 ms - 2
2022-12-27T14:19:01.2665880Z [INFO] [SERVER] GET /$/server 200 0.264 ms - -
2022-12-27T14:19:01.6150568Z [INFO] [SERVER] GET /$/server 304 0.255 ms - -
2022-12-27T14:19:01.6687745Z [INFO] [SERVER] GET /$/stats/skosmos 200 0.275 ms - -
2022-12-27T14:19:02.3654216Z [INFO] [SERVER] GET /tests/reset 200 0.227 ms - 2
2022-12-27T14:19:02.3845387Z [INFO] [SERVER] GET /__coverage__ 200 0.202 ms - 257
2022-12-27T14:19:02.5919748Z [INFO] [TESTS] 
2022-12-27T14:19:02.5920397Z [INFO] [TESTS] =============================== Coverage summary ===============================
2022-12-27T14:19:02.5920907Z [INFO] [TESTS] Statements   : 71.35% ( 269/377 )
2022-12-27T14:19:02.5921540Z [INFO] [TESTS] Branches     : 76.85% ( 83/108 )
2022-12-27T14:19:02.5922002Z [INFO] [TESTS] Functions    : 72.85% ( 102/140 )
2022-12-27T14:19:02.5922394Z [INFO] [TESTS] Lines        : 71.2% ( 267/375 )
2022-12-27T14:19:02.5922815Z [INFO] [TESTS] ================================================================================
2022-12-27T14:19:02.6017667Z [INFO] [TESTS]     ✓ Uses the correct SPARQL Endpoint (1939ms)
2022-12-27T14:19:02.6024189Z [INFO] [TESTS] 
2022-12-27T14:19:02.6029748Z [INFO] [TESTS] 
2022-12-27T14:19:02.6040986Z [INFO] [TESTS]   1 passing (2s)
2022-12-27T14:19:02.6041485Z [INFO] [TESTS] 
2022-12-27T14:19:02.6050139Z [INFO] [TESTS] 
2022-12-27T14:19:02.6056169Z [INFO] [TESTS]   (Results)
2022-12-27T14:19:02.6094536Z [INFO] [TESTS] 
2022-12-27T14:19:02.6117474Z [INFO] [TESTS]   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
2022-12-27T14:19:02.6122703Z [INFO] [TESTS]   │ Tests:        1                                                                                │
2022-12-27T14:19:02.6123431Z [INFO] [TESTS]   │ Passing:      1                                                                                │
2022-12-27T14:19:02.6124391Z [INFO] [TESTS]   │ Failing:      0                                                                                │
2022-12-27T14:19:02.6125094Z [INFO] [TESTS]   │ Pending:      0                                                                                │
2022-12-27T14:19:02.6126091Z [INFO] [TESTS]   │ Skipped:      0                                                                                │
2022-12-27T14:19:02.6126554Z [INFO] [TESTS]   │ Screenshots:  0                                                                                │
2022-12-27T14:19:02.6127010Z [INFO] [TESTS]   │ Video:        false                                                                            │
2022-12-27T14:19:02.6127436Z [INFO] [TESTS]   │ Duration:     2 seconds                                                                        │
2022-12-27T14:19:02.6127877Z [INFO] [TESTS]   │ Spec Ran:     query.cy.js                                                                      │
2022-12-27T14:19:02.6128352Z [INFO] [TESTS]   └────────────────────────────────────────────────────────────────────────────────────────────────┘
2022-12-27T14:19:02.6129812Z [INFO] [TESTS] 
2022-12-27T14:19:02.6300836Z [INFO] [TESTS] 
2022-12-27T14:19:02.6641845Z [INFO] [TESTS] tput: No value for $TERM and no -T specified
2022-12-27T14:19:02.6647773Z [INFO] [TESTS] ====================================================================================================
2022-12-27T14:19:02.6650746Z [INFO] [TESTS] 
2022-12-27T14:19:02.6655049Z [INFO] [TESTS]   (Run Finished)
2022-12-27T14:19:02.6680244Z [INFO] [TESTS] 
2022-12-27T14:19:02.6682231Z [INFO] [TESTS] 
2022-12-27T14:19:02.6707335Z [INFO] [TESTS]        Spec                                              Tests  Passing  Failing  Pending  Skipped  
2022-12-27T14:19:02.6711833Z [INFO] [TESTS]   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
2022-12-27T14:19:02.6715321Z [INFO] [TESTS]   │ ✖  datasets.cy.js                           00:29        9        8        1        -        - │
2022-12-27T14:19:02.6715819Z [INFO] [TESTS]   ├────────────────────────────────────────────────────────────────────────────────────────────────┤
2022-12-27T14:19:02.6716264Z [INFO] [TESTS]   │ ✔  query.cy.js                              00:02        1        1        -        -        - │
2022-12-27T14:19:02.6716961Z [INFO] [TESTS]   └────────────────────────────────────────────────────────────────────────────────────────────────┘
2022-12-27T14:19:02.6717396Z [INFO] [TESTS]     ✖  1 of 2 failed (50%)                      00:31       10        9        1        -        -  
2022-12-27T14:19:02.6717645Z [INFO] [TESTS] 
2022-12-27T14:19:02.8051750Z [INFO] [TESTS]  ERROR  Error: Command failed: /home/runner/work/jena/jena/jena-fuseki2/jena-fuseki-ui/node_modules/cypress/bin/cypress run --config baseUrl=http://localhost:38165 --headless
2022-12-27T14:19:02.8055851Z [INFO] [TESTS] Error: Command failed: /home/runner/work/jena/jena/jena-fuseki2/jena-fuseki-ui/node_modules/cypress/bin/cypress run --config baseUrl=http://localhost:38165 --headless
2022-12-27T14:19:02.8056550Z [INFO] [TESTS]     at makeError (/home/runner/work/jena/jena/jena-fuseki2/jena-fuseki-ui/node_modules/execa/index.js:174:9)
2022-12-27T14:19:02.8057112Z [INFO] [TESTS]     at /home/runner/work/jena/jena/jena-fuseki2/jena-fuseki-ui/node_modules/execa/index.js:278:16
2022-12-27T14:19:02.8057464Z [INFO] [TESTS]     at runMicrotasks (<anonymous>)
2022-12-27T14:19:02.8057842Z [INFO] [TESTS]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-12-27T14:19:02.8409553Z [INFO] [TESTS] vue-cli-service test:e2e --headless exited with code 1
2022-12-27T14:19:02.8412705Z [INFO] --> Sending SIGTERM to other processes..
2022-12-27T14:19:02.9077105Z [INFO] [SERVER] yarn run serve:fuseki exited with code SIGTERM
2022-12-27T14:19:02.9243111Z [INFO] error Command failed with exit code 1.
2022-12-27T14:19:02.9244264Z [INFO] info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@kinow
Copy link
Member

kinow commented Dec 28, 2022

@afs do you have a machine where you can reproduce this issue? If so, I think this could solve it:

diff --git a/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/datasets.cy.js b/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/datasets.cy.js
index f2f281ccf2..26e7914af2 100644
--- a/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/datasets.cy.js
+++ b/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/datasets.cy.js
@@ -55,21 +55,19 @@ describe('datasets', () => {
         //       so we use a ?temp=... in the URL to force distinct requests.
         cy
           .visit(`/#/manage/new?temp=${datasetName}`)
-          .then(() => {
-            cy
-              .get('#dataset-name')
-              .type(datasetName)
-            cy
-              .get('#data-set-type-mem')
-              .click()
-            cy
-              .get('button[type="submit"]')
-              .click()
-            // We are redirected to the Manage datasets view.
-            cy
-              .get('table.jena-table')
-              .should('be.visible')
-          })
+        cy
+          .get('#dataset-name')
+          .type(datasetName)
+        cy
+          .get('#data-set-type-mem')
+          .click()
+        cy
+          .get('button[type="submit"]')
+          .click()
+        // We are redirected to the Manage datasets view.
+        cy
+          .get('table.jena-table')
+          .should('be.visible')
       }
     })
     after(() => {
@@ -257,22 +255,20 @@ describe('datasets', () => {
       cy.route('POST', '/$/datasets').as('post')
       cy
         .visit('/#/manage/new')
-        .then(() => {
-          cy
-            .get('#dataset-name')
-            .type('a')
-          cy
-            .get('#data-set-type-mem')
-            .click()
-          cy
-            .get('button[type="submit"]')
-            .click()
-          // Now the UI must have received an HTTP response with status code 409
-          // from Jena, due to the duplicate dataset name.
-          cy
-            .get('@post')
-            .should('have.property', 'status', 409)
-        })
+      cy
+        .get('#dataset-name')
+        .type('a')
+      cy
+        .get('#data-set-type-mem')
+        .click()
+      cy
+        .get('button[type="submit"]')
+        .click()
+      // Now the UI must have received an HTTP response with status code 409
+      // from Jena, due to the duplicate dataset name.
+      cy
+        .get('@post')
+        .should('have.property', 'status', 409)
     })
     it('Visualizes the dataset information (Info View, tab)', () => {
       cy.visit('/#/dataset/a/info')
diff --git a/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/query.cy.js b/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/query.cy.js
index 0e90435a0c..3da4422923 100644
--- a/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/query.cy.js
+++ b/jena-fuseki2/jena-fuseki-ui/tests/e2e/specs/query.cy.js
@@ -25,21 +25,19 @@ describe('Query', () => {
     // Create a sample dataset.
     cy
       .visit('/#/manage/new')
-      .then(() => {
-        cy
-          .get('#dataset-name')
-          .type('skosmos')
-        cy
-          .get('#data-set-type-mem')
-          .click()
-        cy
-          .get('button[type="submit"]')
-          .click()
-        // We are redirected to the Manage datasets view.
-        cy
-          .get('table.jena-table')
-          .should('be.visible')
-      })
+    cy
+      .get('#dataset-name')
+      .type('skosmos')
+    cy
+      .get('#data-set-type-mem')
+      .click()
+    cy
+      .get('button[type="submit"]')
+      .click()
+    // We are redirected to the Manage datasets view.
+    cy
+      .get('table.jena-table')
+      .should('be.visible')
   })
   after(() => {
     // Special endpoint that clears the datasets data.

I was running things inside the .then() block that are required for the tests. My guess is that in some cases that blocks may be still running when the tests that depend on it have already started, causing some tests to fail 🤞 🥁

@afs
Copy link
Member Author

afs commented Dec 29, 2022

I have a branch with 242725c reverted. It fails more frequently.

branch: https://github.com/afs/jena/tree/revert -- currently with the troublesome tests commented out.

Rewriting the "then" didn't make a difference.

I am noticing that if "mvn clean install" fails, running "mvn test" fails quite reliably but if "mvn clean install" passes, running "mvn test" seems more likely to pass (not enough runs yet to be sure).

@kinow
Copy link
Member

kinow commented Dec 29, 2022

I am noticing that if "mvn clean install" fails, running "mvn test" fails quite reliably but if "mvn clean install" passes, running "mvn test" seems more likely to pass (not enough runs yet to be sure).

Huh, that's really surprising. I thought both would end up with surefire calling the maven plugin that calls yarn/npm. Strange. Thanks for confirming that didn't fix it @afs 👍 back to the whiteboard.

@afs
Copy link
Member Author

afs commented Dec 29, 2022

The plugin has several executions defined. Some are <phase>test</phase>, some are not.

@afs
Copy link
Member Author

afs commented Jan 7, 2023

Currently, the e2e tests are set to not fail the build should an error occur.

@afs
Copy link
Member Author

afs commented Jan 25, 2023

(updating status)

Following on from #1718.

If errors occur, I've only seen this 3-1 pattern, whereas it was 1-0.

[INFO] [TESTS]   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
[INFO] [TESTS]   │ ✖  datasets.cy.js                           00:30        9        -        3        -        6 │
[INFO] [TESTS]   ├────────────────────────────────────────────────────────────────────────────────────────────────┤
[INFO] [TESTS]   │ ✖  query.cy.js                              00:15        2        -        1        -        1 │
[INFO] [TESTS]   └────────────────────────────────────────────────────────────────────────────────────────────────┘
[INFO] [TESTS]     ✖  2 of 2 failed (100%)                     00:45       11        -        4        -        7  
[INFO] [TESTS] 

@kinow
Copy link
Member

kinow commented Jan 25, 2023

(hadn't had time to test the initialization order, vacation time is gone, time to go back to office - will try it again on my spare time over next days/weeks 👍 - thanks for the updates, really helpful for troubleshooting it)

@kinow kinow self-assigned this Jan 29, 2023
kinow added a commit to kinow/jena that referenced this issue Jan 29, 2023
kinow added a commit to kinow/jena that referenced this issue Jan 29, 2023
kinow added a commit that referenced this issue Jan 31, 2023
@afs
Copy link
Member Author

afs commented Jan 31, 2023

🎆 🍾

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants