[Cypress Webkit Ci Run] Test stuck during Ci run #24896
Comments
|
@TomdeHaan can you share the full DEBUG logs please? The tail end shared here doesn't contain much info about what could be causing Cypress to hang. |
flotwig
added
browser: webkit
|
I followed up with @TomdeHaan this AM to see what the status of logs were to help move this forward. |
|
@TomdeHaan could you or one of the team members try your current test suite on the binary below & if this issue isn't resolved also try on 10.7.0. Our team believes that this issue is more than likely related to #23830. Please let us know here any relevant outcomes. |
|
Different setup to OP's, but we have this same issue (after tests complete, the debug logs spam about the profiler until it times out), and for us it started on 10.11.0. If we stay on 10.10.0 we're fine, but any version above this, including 12.0.1, seems to trigger this problem. (To clarify different setup; I mean we're hitting this on Chrome, not WebKit, and are using Node 18. We did have the same issue on Node 16 when we initially tried the 10.11.0 upgrade prior to the Node 18 upgrade) |
|
@modern-sapien. Sorry, only saw this message on and of working day. Small update form our side. We will try your suggestion next working week, but we also did some investigation our selves:. Currently we are running Cypress inside Gitlab, using a docker images that we create ourselves, using this image: https://hub.docker.com/layers/cypress/browsers/latest/images/sha256-44f4157bfb319fccf537b590841315c9f777937f02f930e9d295e5cbdeb02381?context=explore. We also install the webkit browser from playwright as mentioned in the documents https://docs.cypress.io/guides/guides/launching-browsers#Browsers. If you want to have more details, please contact us again. And we will update, after we tried your suggestion. |
|
@modern-sapien After the weekend, the team created an image with the version you provided. Question from the team, the title from your version suggests that this is without Webkit while we want to test with Webkit ofcourse. So that set us a little question mark. We first run the test without using the Cypress dashboard and the tests passed (no --group, --parallel --record). Then we activated all that again, and the same timeout as before. We attached the log file to this comment. |
emilyrohrbough
added
external: cloud
Cypress Cloud
|
@TomdeHaan How many times did you run without recording to the dashboard? Was each run successful? When recording to the dashboard, does it always hang or does it ever pass? Also are you using a custom preprocessor? I am wondering what Recap of the deep dive into the logs... I am noticing your second test is not sending the mocha pass event or the mocha test end event. It seems the test is either 1-finishing and we aren't correctly wrapping up the test (seems unlikely) or 2- the browser tab is hanging or crashing and test runner and the test is hanging waiting on the test results to be returned. Has anyone on the team been able to reproduce this locally when running the the docker image you use in CI? When able to repro, have you been able to run in run mode with the headed browser to try & observe what might be happening? |
emilyrohrbough
added
stage: needs information
@emilyrohrbough because of holiday vacations we can't check it currently. But we will come back on this very shortly. |
|
@TomdeHaan, we're going to look into this from the Cypress Cloud (dashboard) side too. The last run for the project is old enough that it's beyond our retention for the (potentially) helpful logs we might have on the Cloud side. Can you initiate another run of the project? Also, Emily's question above regarding whether this issue occurs when not recording to the dashboard would be helpful to answer too. We're just getting back into the swing of normal schedules following the holiday around here as well, but if you let us know when you've initiated another run, we'll dig into it on the Cloud side. Thanks! |
|
@rockhold, I started a new run connected to the Cypress Cloud. I will do some more testing and running today. I will keep you updated. |
@rockhold, @chrisbreiding, @emilyrohrbough. I did some testing myself today, these are the results until now. It did not matter if I run the test with, or without connection to the Cypress Cloud. The Webkit test did fail all the time with 7 runs total. In between I also update the cypress version 12.3.0, this did not influence to outcome. The test still failed. failed-log-with-cypress-cloud.txt |
|
We update our Cypress version to 12.3.0 and make use use of the cypress docker image: 12.3 based on: Now our Webkit tests could not connect to the browser. Cypress Cloud shows the message: Attached the error log. Pleas contact us if you need more information or extra runs. |
|
Our Dockerfile looks like this: FROM cypress/browsers:latest Mimick the running Jenkins environmentRUN groupadd --gid 36030 cypress install webkit for safariRUN npx playwright install-deps Create base directoryRUN mkdir /cypress && chown cypress /cypress USER cypress RUN npm install cypress@12.3.0 RUN cypress installRUN DISPLAY= xvfb-run -a /cypress/node_modules/.bin/cypress verify |
Chiming in here from the cloud side. Not much I can gather on my end. It does seem to be the first tests in both the webkit groups that never get far enough to report their results/tests. So @chrisbreiding not sure if knowing that it never calls |
|
I have a similar issue (#25431), my tests are hanging in AzureDevops CI, but it's a different scenario. I'm using a WebAssembly app that downloads a bunch of .dll files in the browser and it gets stuck while downloading them. They work just fine locally. Update: I just tested and as @me4502 said I can confirm that it works just fine with |
|
Summing up what we know so far:
@TomdeHaan To further narrow down this issue, the best thing would be if a reproduction could be shared with us, containing all required test code and app code to run these tests. If it's at all possible, sending a reproduction to support@cypress.io or sharing in this thread would be a huge step towards solving this. If it's not possible to share a repro, here's some more avenues you could explore. I'm hoping to discover the specific testing steps that are causing a hang after 10.11.0:
|
|
Hi all, @flotwig is there any news on this? We are seeing it in CircleCI, using the Cypress orb. Webkit hangs in CI but works fine locally (running a single spec file and test) With debugging on, I am seeing a profiler constantly running until it times out. It ran for 2 hrs until I aborted it I am using the following code to install webkit on a cypress docker image: Orb version is 2.2.0 |
We made contact with Brittani within Cypress regarding the sharing of our code. She will probably make contact with you about this. |
nagash77
added
Triaged
|
Hi all, When we run test:
We tried many proposed solutions, but they didn't work: |
|
Please help, I have recorded the behavior of webkit and I am sending the file with the test for analysis |
|
Hi @ggrek1988. Is there anything special about your config? Would you be able to share that? |
|
@AtofStryker , uploading cypress.config.js and package.json |
|
@AtofStryker Hi, Maybe you noticed something that could help me run the tests? |
|
Also have issues with running webkit on CICD, test just hang randomly |
|
We have the same/similar sporadic failures with Webkit on GHA. Run details: Run history: |
|
We just hit (and resolved) an issue with the symptom; we be left with the The source had ended with two I am not sure if Cypress outputs an warning of some kind when it encounters two, or if it should throw a hard failure, but after removing the errant file things started working. Please note: I am not Cypress expert, but I hope this is helpful to someone. Edit: It seems that one of the two |
|
@roadSurfer Can you write more about your solution? |
|
Hi, I solved the problem in my project |
Current behavior
We are currently experimenting with cross browser testing within the CiCd pipeline. Our main goal is to run Cypress with Webkit and Chrome and run it in multiple viewports.
We run Cypress on multiple machines within Gitlab, adding the --parallel to connect to the Cypress Dashboard. We found out that with every run the Webkit tests got stuck and eventually timed out.
Example of one of the files:
Example of what it looks like within Gitlab:
Example of timeout log within gitlab:
We enabled the debugger, and when analysing it, we see that the cypress:server:util:process_profiler keeps repeating itself in the end
Desired behavior
All tests run and finish correctly.
Test code to reproduce
Cypress Version
10.11.0
Node version
v16.14.2
Operating System
cypress-image/browsers:latest-cypress-10.11.0 docker image
Debug Logs
cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '453', status: 200, headers: { 'content-type': 'image/png', date: 'Fri, 11 Nov 2022 09:25:31 GMT', 'access-control-allow-origin': '*', 'access-control-allow-credentials': 'true', 'x-content-type-options': 'nosniff', 'last-modified': 'Fri, 11 Nov 2022 09:25:31 GMT', etag: 'W/"ff8cc89-ea33c-5ed2e7897d023"', 'accept-ranges': 'bytes', 'cache-control': 'max-age=2629746, public', expires: 'Fri, 11 Nov 2022 09:30:30 GMT', 'x-xss-protection': '1; mode=block', via: '1.1 dfd84a17eaa88d79994b6524cab4931e.cloudfront.net (CloudFront)', age: '1639861', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', 'strict-transport-security': 'max-age=15724800; includeSubDomains, max-age=63072000; includeSubdomains; preload', 'permissions-policy': 'geolocation=();midi=();notifications=();push=();sync-xhr=();microphone=();camera=();magnetometer=();gyroscope=();speaker=(self);vibrate=();fullscreen=(self);payment=();', 'x-cache': 'Hit from cloudfront', 'x-amz-cf-pop': 'DUB56-P1', 'x-amz-cf-id': 'EUKPC3vBU19zR_nsT4U8eoutO65EerEzUcAC-jC3NPL9lTDZ0NXajQ==' } } } +0ms cypress:server:browsers:webkit-automation received requestfinished { responseReceived: { requestId: '459', status: 200, headers: { 'access-control-allow-origin': 'https://www.ziggo.nl', date: 'Wed, 30 Nov 2022 08:56:32 GMT', pragma: 'no-cache', expires: 'Fri, 01 Jan 1990 00:00:00 GMT', 'cache-control': 'no-cache, no-store, must-revalidate', 'last-modified': 'Sun, 17 May 1998 03:00:00 GMT', 'x-content-type-options': 'nosniff', 'access-control-allow-credentials': 'true', 'content-type': 'image/gif', 'cross-origin-resource-policy': 'cross-origin', connection: 'keep-alive', 'keep-alive': 'timeout=5', 'transfer-encoding': 'chunked', server: 'Golfe2', 'alt-svc': 'h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="46, 43"' } } } +11ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51776 } +4s cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51824 } +933ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51836 } +82ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51830 } +26ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51846 } +0ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51844 } +18ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51838 } +48ms cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51826 } +16ms cypress:server:browsers:webkit-automation received request { browserPreRequest: { requestId: '463', method: 'POST', url: 'https://region1.google-analytics.com/g/collect?v=2&tid=G-TH972FDWFF>m=2oebs0&_p=776742999&cid=872464754.1669798568&ul=en-us&sr=1440x1024&_eu=AAg&_s=9&sid=1669798568&sct=1&seg=1&dl=https%3A%2F%2Fwww.ziggo.nl%2Finternet%2Finternet-only%3Futm_medium%3Dtesting%26utm_source%3Dtesting%26optimizely_disable%3Dtrue&dt=Alleen%20Internet%20Thuis%20%E2%80%93%20Internet%20Only%20Pakketten%20Vergelijken%20%7C%20Ziggo&en=click_on_button&ep.interaction_event=click.button&ep.element_text=ga%20verder&ep.element_link=none&ep.element_classes=z-basket__footer-btn%20btn%20btn-primary&_et=2', headers: { accept: '*/*', referer: 'https://www.ziggo.nl/', origin: 'https://www.ziggo.nl', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15', 'cache-control': 'max-age=0' }, resourceType: 'other', originalResourceType: 'beacon' } } +5s cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 51850 } +255ms cypress:server:util:process_profiler current & mean memory and CPU usage by process group: cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐ cypress:server:util:process_profiler │ (index) │ group │ processCount │ pids │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │ cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤ cypress:server:util:process_profiler │ 0 │ 'WebKit' │ 4 │ '1075, 1080, 1094, 1098' │ 83.81 │ 68.01 │ 1097.35 │ 887.91 │ 1202.64 │ cypress:server:util:process_profiler │ 1 │ 'cypress' │ 1 │ '299' │ 1.89 │ 14.79 │ 295 │ 248.49 │ 295 │ cypress:server:util:process_profiler │ 2 │ 'electron-shared' │ 4 │ '311, 471, 312, 518' │ 0 │ 0 │ 199.98 │ 188.23 │ 199.98 │ cypress:server:util:process_profiler │ 3 │ 'plugin' │ 1 │ '497' │ 0 │ 0.54 │ 137.66 │ 182.66 │ 198.71 │ cypress:server:util:process_profiler │ 4 │ 'other' │ 2 │ '1497, 1498' │ 0 │ 0 │ 3.48 │ 3.47 │ 3.55 │ cypress:server:util:process_profiler │ 5 │ 'TOTAL' │ 12 │ '-' │ 85.7 │ 75.73 │ 1733.46 │ 1391.79 │ 1886.53 │ cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s cypress:server:util:process_profiler current & mean memory and CPU usage by process group: cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐ cypress:server:util:process_profiler │ (index) │ group │ processCount │ pids │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │ cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤ cypress:server:util:process_profiler │ 0 │ 'WebKit' │ 4 │ '1075, 1080, 1094, 1098' │ 81.64 │ 69.52 │ 1072.17 │ 908.38 │ 1202.64 │ cypress:server:util:process_profiler │ 1 │ 'cypress' │ 1 │ '299' │ 0.3 │ 13.34 │ 295.07 │ 253.14 │ 295.07 │ cypress:server:util:process_profiler │ 2 │ 'electron-shared' │ 4 │ '311, 471, 312, 518' │ 0 │ 0 │ 199.98 │ 189.4 │ 199.98 │ cypress:server:util:process_profiler │ 3 │ 'plugin' │ 1 │ '497' │ 0 │ 0.48 │ 137.66 │ 177.66 │ 198.71 │ cypress:server:util:process_profiler │ 4 │ 'other' │ 2 │ '1520, 1521' │ 0 │ 0 │ 3.55 │ 3.48 │ 3.55 │ cypress:server:util:process_profiler │ 5 │ 'TOTAL' │ 12 │ '-' │ 81.94 │ 76.35 │ 1708.42 │ 1423.45 │ 1886.53 │ cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s cypress:server:util:process_profiler current & mean memory and CPU usage by process group: cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐ cypress:server:util:process_profiler │ (index) │ group │ processCount │ pids │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │ cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤ cypress:server:util:process_profiler │ 0 │ 'WebKit' │ 4 │ '1075, 1080, 1094, 1098' │ 86.33 │ 71.21 │ 1063.43 │ 923.89 │ 1202.64 │ cypress:server:util:process_profiler │ 1 │ 'cypress' │ 1 │ '299' │ 0.3 │ 12.16 │ 295.23 │ 256.97 │ 295.23 │ cypress:server:util:process_profiler │ 2 │ 'electron-shared' │ 4 │ '311, 471, 312, 518' │ 0 │ 0 │ 199.98 │ 190.36 │ 199.98 │ cypress:server:util:process_profiler │ 3 │ 'plugin' │ 1 │ '497' │ 0 │ 0.43 │ 137.66 │ 173.66 │ 198.71 │ cypress:server:util:process_profiler │ 4 │ 'other' │ 2 │ '1544, 1545' │ 0 │ 0 │ 3.35 │ 3.47 │ 3.55 │ cypress:server:util:process_profiler │ 5 │ 'TOTAL' │ 12 │ '-' │ 86.63 │ 77.28 │ 1699.65 │ 1448.56 │ 1886.53 │ cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s cypress:server:socket-base socket-disconnecting ping timeout +37s cypress:server:socket-base socket-disconnect ping timeout +0ms cypress:server:util:process_profiler current & mean memory and CPU usage by process group: cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐ cypress:server:util:process_profiler │ (index) │ group │ processCount │ pids │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │ cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤ cypress:server:util:process_profiler │ 0 │ 'WebKit' │ 4 │ '1075, 1080, 1094, 1098' │ 84.24 │ 72.39 │ 1123.76 │ 942.06 │ 1202.64 │ cypress:server:util:process_profiler │ 1 │ 'cypress' │ 1 │ '299' │ 1.55 │ 11.27 │ 268.29 │ 257.91 │ 295.23 │ cypress:server:util:process_profiler │ 2 │ 'electron-shared' │ 4 │ '311, 471, 312, 518' │ 0 │ 0 │ 199.98 │ 191.16 │ 199.98 │ cypress:server:util:process_profiler │ 3 │ 'plugin' │ 1 │ '497' │ 0 │ 0.39 │ 137.66 │ 170.38 │ 198.71 │ cypress:server:util:process_profiler │ 4 │ 'other' │ 2 │ '1569, 1570' │ 0 │ 0 │ 3.33 │ 3.46 │ 3.55 │ cypress:server:util:process_profiler │ 5 │ 'TOTAL' │ 12 │ '-' │ 85.79 │ 77.99 │ 1733.01 │ 1472.27 │ 1886.53 │ cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s cypress:server:util:process_profiler current & mean memory and CPU usage by process group: cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐ cypress:server:util:process_profiler │ (index) │ group │ processCount │ pids │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │ cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤ cypress:server:util:process_profiler │ 0 │ 'WebKit' │ 4 │ '1075, 1080, 1094, 1098' │ 85.64 │ 73.49 │ 1108.19 │ 955.9 │ 1202.64 │ cypress:server:util:process_profiler │ 1 │ 'cypress' │ 1 │ '299' │ 3.54 │ 10.68 │ 230.77 │ 255.83 │ 295.23 │ cypress:server:util:process_profiler │ 2 │ 'electron-shared' │ 4 │ '311, 471, 312, 518' │ 0 │ 0 │ 199.98 │ 191.84 │ 199.98 │ cypress:server:util:process_profiler │ 3 │ 'plugin' │ 1 │ '497' │ 0 │ 0.36 │ 137.66 │ 167.66 │ 198.71 │ cypress:server:util:process_profiler │ 4 │ 'other' │ 2 │ '1592, 1593' │ 0 │ 0 │ 3.47 │ 3.46 │ 3.55 │ cypress:server:util:process_profiler │ 5 │ 'TOTAL' │ 12 │ '-' │ 89.18 │ 78.85 │ 1680.05 │ 1488.25 │ 1886.53 │ cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10sOther
No response
The text was updated successfully, but these errors were encountered: