Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Robot Framework test runs not flagging steps in QASE test runs. #73

Closed
OrrinGradwell opened this issue Nov 3, 2022 · 2 comments
Closed

Robot Framework test runs not flagging steps in QASE test runs. #73

OrrinGradwell opened this issue Nov 3, 2022 · 2 comments
Assignees
@n3r
Labels
robotframework wontfix This will not be worked on

Comments

@OrrinGradwell
Copy link

When executing tests one by one in RF, hooking into QASE, the tests steps in the test run is flagged as expected, be it passed or failed.

As soon as all the tests are executed in RF in one run, then the steps in QASE under the test run is NOT flagged at all. Only the RF status is set to the QASE test run. (Passed || Failed).

Example Outputs:
Executing single test case: clear && robot -d assets/results/ --listener qaseio.robotframework.Listener -i q-2 code/tests

Results in:
image

Console Output:

==============================================================================
Tests :: This section sets up and tears down all tests.                       
==============================================================================
Tests.Fox                                                                     
==============================================================================
Tests.Fox.Privacy Notice :: Ensure that clients has access to and can view ...
==============================================================================
[ WARN ] Missing step for name Set Selenium Timeout                           
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name '${IN_PIPELINES}' == 'true'                    
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name '${ENV}' not in ${ENVIRONMENTS}                
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name '${ENV}' == 'dev' or '${ENV}' == 'qa'          
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Log To Console                                 
[ WARN ] Missing step for name Pause Execution                                
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name "${USE_AWS_PROFILE}" == "not_set"              
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name ${exit_code} > 0                               
[ WARN ] Missing step for name Convert String To Json                         
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Retrieve AWS Secrets                           
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Start Virtual Display                          
[ WARN ] Missing step for name '${SCREEN}' == 'headless'                      
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name '${BROWSER_TYPE}' == 'local'                   
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Maximize Browser Window                        
[ WARN ] Missing step for name Wait Until Location Contains                   
[ WARN ] Missing step for name Begin Session                                  
[ WARN ] Missing step for name Click Element                                  
[ WARN ] Missing step for name Switch Window                                  
[ WARN ] Missing step for name Wait Until Element Is Enabled                  
[ WARN ] Missing step for name Location Should Contain                        
[ WARN ] Missing step for name Page Should Contain                            
[ WARN ] Missing step for name Close All Browsers                             
Privacy Notice :: Validate that clients can view the privacy policy.  | PASS |
------------------------------------------------------------------------------
Tests.Fox.Privacy Notice :: Ensure that clients has access to and ... | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Tests.Fox                                                             | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Tests :: This section sets up and tears down all tests.               | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Output:  /media/veracrypt1/sandbox/tx/rox/assets/results/output.xml
Log:     /media/veracrypt1/sandbox/tx/rox/assets/results/log.html
Report:  /media/veracrypt1/sandbox/tx/rox/assets/results/report.html

Executing Multiple test cases in RF: clear && robot -d assets/results/ --listener qaseio.robotframework.Listener code/tests

Results In:
image

Console Output:

==============================================================================
Tests :: This section sets up and tears down all tests.                       
==============================================================================
Tests.Fox                                                                     
==============================================================================
Tests.Fox.Forgot Password Request :: Validate that users can request a new ...
==============================================================================
[ WARN ] Missing step for name Set Selenium Timeout                           
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name '${IN_PIPELINES}' == 'true'                    
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name '${ENV}' not in ${ENVIRONMENTS}                
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name '${ENV}' == 'dev' or '${ENV}' == 'qa'          
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Log To Console                                 
[ WARN ] Missing step for name Pause Execution                                
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name "${USE_AWS_PROFILE}" == "not_set"              
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name ${exit_code} > 0                               
[ WARN ] Missing step for name Convert String To Json                         
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Retrieve AWS Secrets                           
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Start Virtual Display                          
[ WARN ] Missing step for name '${SCREEN}' == 'headless'                      
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name '${BROWSER_TYPE}' == 'local'                   
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Maximize Browser Window                        
[ WARN ] Missing step for name Wait Until Location Contains                   
[ WARN ] Missing step for name Begin Session                                  
[ WARN ] Missing step for name Click Element                                  
[ WARN ] Missing step for name Wait Until Element Is Enabled                  
[ WARN ] Missing step for name Input Text                                     
[ WARN ] Missing step for name Wait Until Element Is Enabled                  
[ WARN ] Missing step for name Click Element                                  
[ WARN ] Missing step for name Wait Until Page Contains                       
Forgot Password Request :: Ensure users can request a new password.   | PASS |
------------------------------------------------------------------------------
Tests.Fox.Forgot Password Request :: Validate that users can reque... | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Tests.Fox.Privacy Notice :: Ensure that clients has access to and can view ...
==============================================================================
[ WARN ] Missing step for name Set Selenium Timeout                           
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name '${IN_PIPELINES}' == 'true'                    
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name '${ENV}' not in ${ENVIRONMENTS}                
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name '${ENV}' == 'dev' or '${ENV}' == 'qa'          
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Log To Console                                 
[ WARN ] Missing step for name Pause Execution                                
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name "${USE_AWS_PROFILE}" == "not_set"              
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name ${exit_code} > 0                               
[ WARN ] Missing step for name Convert String To Json                         
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Retrieve AWS Secrets                           
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Start Virtual Display                          
[ WARN ] Missing step for name '${SCREEN}' == 'headless'                      
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name '${BROWSER_TYPE}' == 'local'                   
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Maximize Browser Window                        
[ WARN ] Missing step for name Wait Until Location Contains                   
[ WARN ] Missing step for name Begin Session                                  
[ WARN ] Missing step for name Click Element                                  
[ WARN ] Missing step for name Switch Window                                  
[ WARN ] Missing step for name Wait Until Element Is Enabled                  
[ WARN ] Missing step for name Location Should Contain                        
[ WARN ] Missing step for name Page Should Contain                            
Privacy Notice :: Validate that clients can view the privacy policy.  | PASS |
------------------------------------------------------------------------------
Tests.Fox.Privacy Notice :: Ensure that clients has access to and ... | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Tests.Fox.Sign In :: Validate login functionality.                            
==============================================================================
[ WARN ] Missing step for name Set Selenium Timeout                           
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name '${IN_PIPELINES}' == 'true'                    
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name '${ENV}' not in ${ENVIRONMENTS}                
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name '${ENV}' == 'dev' or '${ENV}' == 'qa'          
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Log To Console                                 
[ WARN ] Missing step for name Pause Execution                                
[ ERROR ] Calling method 'end_keyword' of listener 'qaseio.robotframework.Listener' failed: KeyError: 'NOT RUN'
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Evaluate                                       
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ WARN ] Missing step for name "${USE_AWS_PROFILE}" == "not_set"              
[ WARN ] Missing step for name Run And Return Rc And Output                   
[ ERROR ] Calling method 'end_keyword' of listener 'qaseio.robotframework.Listener' failed: KeyError: 'NOT RUN'
[ WARN ] Missing step for name Fail                                           
[ WARN ] Missing step for name ${exit_code} > 0                               
[ WARN ] Missing step for name Convert String To Json                         
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name Set Global Variable                            
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Retrieve AWS Secrets                           
[ WARN ] Missing step for name Set Log Level                                  
[ WARN ] Missing step for name Start Virtual Display                          
[ WARN ] Missing step for name '${SCREEN}' == 'headless'                      
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name '${BROWSER_TYPE}' == 'local'                   
[ WARN ] Missing step for name Open Browser                                   
[ WARN ] Missing step for name                                                
[ WARN ] Missing step for name Maximize Browser Window                        
[ WARN ] Missing step for name Wait Until Location Contains                   
[ WARN ] Missing step for name Begin Session                                  
[ WARN ] Missing step for name Email                                          
[ WARN ] Missing step for name Password                                       
[ WARN ] Missing step for name Input Text                                     
[ WARN ] Missing step for name Input Password                                 
[ WARN ] Missing step for name Submit Form                                    
[ WARN ] Missing step for name Wait Until Page Contains                       
[ WARN ] Missing step for name Reload Page                                    
[ WARN ] Missing step for name Input Text                                     
[ WARN ] Missing step for name Input Password                                 
[ WARN ] Missing step for name Submit Form                                    
[ WARN ] Missing step for name Wait Until Page Contains                       
[ WARN ] Missing step for name Reload Page                                    
[ WARN ] Missing step for name Input Text                                     
[ WARN ] Missing step for name Input Password                                 
[ WARN ] Missing step for name Submit Form                                    
[ WARN ] Missing step for name Wait Until Page Contains                       
[ WARN ] Missing step for name Reload Page                                    
[ WARN ] Missing step for name Input Text                                     
[ WARN ] Missing step for name Input Password                                 
[ WARN ] Missing step for name Submit Form                                    
[ WARN ] Missing step for name Wait Until Element Is Visible                  
[ WARN ] Missing step for name Wait Until Element Is Enabled                  
Credentials Validation :: Test that combinations of credentials fa... | PASS |
------------------------------------------------------------------------------
Tests.Fox.Sign In :: Validate login functionality.                    | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Tests.Fox                                                             | PASS |
3 tests, 3 passed, 0 failed
==============================================================================
Tests :: This section sets up and tears down all tests.               | PASS |
3 tests, 3 passed, 0 failed
==============================================================================
Output:  /media/veracrypt1/sandbox/tx/rox/assets/results/output.xml
Log:     /media/veracrypt1/sandbox/tx/rox/assets/results/log.html
Report:  /media/veracrypt1/sandbox/tx/rox/assets/results/report.html

Although I'm not aware of a way to suppress the warnings, I do not intend to log those in QASE as those are setup scripts that need to run before any tests can start.

Those steps are specific to our environment.

The following scripts will not execute in your environment and only serves as information as to my setup:

init.robot (base script)

*** Settings ***
Documentation
...                 This section sets up and tears down all tests.

Resource            ../core/resources.resource

Suite Setup         Prep Environment
Suite Teardown      Clear Environment


*** Keywords ***
Prep Environment
    [Documentation]
    ...    Setup environment, getting required information from AWS.

    Clear Files
    Begin Session

Clear Environment
    [Documentation]
    ...    Clear environment of all vaiables as well as close any opened browsers.

    Close All Browsers

Privacy Notice Test Script:

*** Settings ***
Documentation
...    Ensure that clients has access to and can view the privacy policy.

Resource
...    ../../../code/core/resources.resource

Default Tags
...    fox
...    privacy_notice
...    q-2


*** Test Cases ***
Privacy Notice
    [Documentation]
    ...    Validate that clients can view the privacy policy.

    Click Element    data: pq-selector:privacy-notice

    Switch Window    ProQuo AI Privacy Notice
    Wait Until Element Is Enabled    xpath: (//*[contains(text(),"Book a demo")])[1]    ${TIMEOUT}
    Location Should Contain    privacy-notice
    Page Should Contain    CONTACT US

Each test is isolated and only does one thing, nothing is shared across tests aside from the setup script.
They are all coded the same except that the tags and tests are different for each.
@n3r n3r self-assigned this Dec 4, 2022
@stale
Copy link

stale bot commented Mar 4, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Mar 4, 2023
@n3r n3r removed the wontfix This will not be worked on label Mar 5, 2023
@stale
Copy link

stale bot commented Jun 8, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix This will not be worked on label Jun 8, 2023
@stale stale bot closed this as completed Jun 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@n3r n3r

Labels
robotframework wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@n3r @OrrinGradwell