-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"enableSoftAssert": true does not allow cy.vrtTrack() function to run in async and allow tests to run independently #170
Comments
@alexniculae thanks for your report! there is no restriction on async test execution using this option could you try changing default retry number like here and check the timing
|
@pashidlos, thank you 🙂 the retries are already set to 0 in this case what I see though is that with due to this matter, I assumed that with the option set to either true or false, the tests are waiting for the VRT response/result before being allowed to finish/proceed i tried looking into the VRT code to see if there is anything that keeps the tests ‘blocked’ until VRT finishes processing, but am not perfectly sure if this logic could be coming from here: Line 51 in c8f4775
so more rather then a bug, I’d see this ticket as a feature, to allow the (Cypress) tests to finish/proceed before VRT finishes its processing and replies with its results so the VRT engine does its work in the background, independent from (in this case) Cypress, as the Cypress test is not dependent on the VRT result/response looking forward for your thoughts later edit: probably the Cypress agent should not wait for the VRT response though, only in the case where both of the following conditions are met:
|
@alexniculae you are right, the test is waiting the response from VRT in order to know the result
another thing is to make image comparison async - not sure if this is currently possible I would first try to review if it's possible to speed up response from VRT |
@pashidlos, I see your point here; but in the following scenario:
there is no longer any reason to expect the answer from the image comparison; no matter what the result of the visual test would be, it would be irrelevant for the e2e test; or is there something that i didn’t see? 🙂 for the rest of the configuration options (e.g. possibly this is a simplistic approach, but a ternary to avoid waiting for the visual test result in the first configuration example above might save a few ms per print-screen what do you think?
I would avoid doing this, for the following scenario(s):
from my point of view, this is only doable in the case where e2e tests don’t need to wait for the response from VRT (as per first point in the current comment) I assume that the comparison engine only needs to receive the screenshot (+details/options) from the e2e test and then (if the e2e test does not need to wait for the visual test response) the engine does the comparison by itself and is in no way interdependent on the e2e test so, again the (probably over-simplistic) solution of adding a decision of waiting or not for the response in some cases (
that is also already great as well, but I see my proposal above as going great lengths to optimizing performance in some cases as well (such as the scenario of monitoring the visual tests independently of the e2e ones) 🙂 long message, but i wanted to share the details, as I am pretty sure we have different ways of using and leveraging the power of VRT, which is great, because good things come when mixing opinions and approaches 🙂 in case you’d like that we discuss this further, I’d even be glad to have a call on the topic |
another way this could then be made easier for the users (but these are different issues that we need to consider and log in their own time) is that the visual tests would then no longer serve a purpose in blocking the e2e tests when running the image comparison, and could then run the comparison in async, and:
|
Tested the below use case with the Cypress-Agent:
Is there any way to allow the tests to run async and independent of the engine of VRT, at least when the enableSoftAssert is set to true?
If we don't want to fail the tests because of the results of the visual test, then at least we could leverage the speed of the tests.
Let me know if this issue should actually be moved to the js-sdk - was not 100% sure, but that might be best 🙂
The text was updated successfully, but these errors were encountered: