Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System test with robot framework #8481

Merged
merged 101 commits into from Jul 17, 2018
Merged

System test with robot framework #8481

merged 101 commits into from Jul 17, 2018

Commits on May 16, 2018

  1. WIP prototype sysTest 

    Requires:
robotFramework - the system test framework
pyautogui - for controlling keyboard/mouse

To install:
pip install robotframework
pip install pyautogui
    @feerrenrut
    feerrenrut committed May 16, 2018
    Configuration menu
    Copy the full SHA
    f868f5f View commit details
    Browse the repository at this point in the history

  2. WIP include flaky way of exiting 

    Ensure this is run from a cmd prompt. Other shells may cause issues,
for instance, NVDA does not get correct focus when run from cygwin.
    @feerrenrut
    feerrenrut committed May 16, 2018
    Configuration menu
    Copy the full SHA
    e9a2af7 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2018

  1. WIP System testing 

    Remote server implemenation and external abstracted library.
    @feerrenrut
    feerrenrut committed May 17, 2018
    Configuration menu
    Copy the full SHA
    1fb27cb View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2018

  1. Merge branch 'master' into systemTestWithRobot

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    584ea96 View commit details
    Browse the repository at this point in the history

  2. First try at including system tests in appveyor.

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    5c83124 View commit details
    Browse the repository at this point in the history

  3. Disable pyAutoGUI's failsafe check as appeyor moves the mouse???

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    dd9f3e7 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'systemTestWithRobot' into appveyorTests

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    5a75207 View commit details
    Browse the repository at this point in the history

  5. initial.robot: break robot to temporarily bipass freeze.

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    1e6e265 View commit details
    Browse the repository at this point in the history

  6. appveyor.yaml: run robot in powershell so exit code is ignored for now.

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    69f4407 View commit details
    Browse the repository at this point in the history

  7. appveyor.yaml: try again to make system tests return error code 0

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    9bd9760 View commit details
    Browse the repository at this point in the history

  8. appveyor.yaml: try again to make system tests return error code 0

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    488a33e View commit details
    Browse the repository at this point in the history

  9. appveyor.yaml: fix syntax of uplodFile.

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    f275871 View commit details
    Browse the repository at this point in the history

  10. system tests: tell appVeyor they are junit formatted?

    @michaelDCurran
    michaelDCurran committed Jun 7, 2018
    Configuration menu
    Copy the full SHA
    9b889f8 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2018

  1. appVeyor: still upload test results on failed builds.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    c3b4a90 View commit details
    Browse the repository at this point in the history

  2. Upload unitTest reports to appViewer as well.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    8b0e1c2 View commit details
    Browse the repository at this point in the history

  3. Investigate freeze in robot test.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    9270a20 View commit details
    Browse the repository at this point in the history

  4. add some logging to nvdaRobotLib

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    30a1256 View commit details
    Browse the repository at this point in the history

  5. More logging

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    8e02fc4 View commit details
    Browse the repository at this point in the history

  6. more logging again

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    3824887 View commit details
    Browse the repository at this point in the history

  7. sleep before connecting to server.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    e244bcb View commit details
    Browse the repository at this point in the history

  8. more and more logging

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    1c4f7b9 View commit details
    Browse the repository at this point in the history

  9. Fix logging...

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    d80c688 View commit details
    Browse the repository at this point in the history

  10. reimplement quit nvda test inside initial.robot.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    5e130c0 View commit details
    Browse the repository at this point in the history

  11. A try at moving everying to robot language to fix freezes.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    bdb51da View commit details
    Browse the repository at this point in the history

  12. Launch NVDA from robot with py for appveyor.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    1286782 View commit details
    Browse the repository at this point in the history

  13. copy the globalPlugin.

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    a57e95e View commit details
    Browse the repository at this point in the history

  14. Temporarily start rdp for appveyor build

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    5d540e7 View commit details
    Browse the repository at this point in the history

  15. bump

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    8195ee4 View commit details
    Browse the repository at this point in the history

  16. bump 2

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    6e530b6 View commit details
    Browse the repository at this point in the history

  17. bump 3

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    38e696b View commit details
    Browse the repository at this point in the history

  18. bump 4

    @michaelDCurran
    michaelDCurran committed Jun 8, 2018
    Configuration menu
    Copy the full SHA
    2274ea6 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2018

  1. Slightly re-write now with setup and teardown, and insert q for quit … 

    …check and check foreground name is Exit NVDA etc.
    @michaelDCurran
    michaelDCurran committed Jun 11, 2018
    Configuration menu
    Copy the full SHA
    d8cd618 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2018

  1. nvdaRobotLib: use a specific NVDA profile dir.

    @michaelDCurran
    michaelDCurran committed Jun 12, 2018
    Configuration menu
    Copy the full SHA
    d6acae5 View commit details
    Browse the repository at this point in the history

  2. NVDA Spy for robot must use a daemon thread for now to stop further N… 

    …VDAs crashing.
    @michaelDCurran
    michaelDCurran committed Jun 12, 2018
    Configuration menu
    Copy the full SHA
    5c55400 View commit details
    Browse the repository at this point in the history

  3. Robot tests set NVDA not to show the welcome dialog

    @michaelDCurran
    michaelDCurran committed Jun 12, 2018
    Configuration menu
    Copy the full SHA
    06aa06b View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2018

  1. allow asserting on last speech

    @feerrenrut
    feerrenrut committed Jun 14, 2018
    Configuration menu
    Copy the full SHA
    ce75aa0 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/robotOnAppveyor' into systemTest… 

    …WithRobot
    @feerrenrut
    feerrenrut committed Jun 14, 2018
    Configuration menu
    Copy the full SHA
    0c16f1b View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2018

  1. Remove extra tab characters

    @feerrenrut
    feerrenrut committed Jun 19, 2018
    Configuration menu
    Copy the full SHA
    735991a View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2018

  1. Copy in the testSpy globalPlugin for each test.

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    662eabb View commit details
    Browse the repository at this point in the history

  2. Remove sleep 

    The sleep was required because importing the remote library suceeds
regardless of whether it is possible to execute a keyword. We now test
the connection before we load the library.
    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    08c8697 View commit details
    Browse the repository at this point in the history

  3. Some cleanup of connection/disconnection code

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    8981697 View commit details
    Browse the repository at this point in the history

  4. Extract code that can be reused and add better documentation

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    669da0a View commit details
    Browse the repository at this point in the history

  5. cleanup systemTest dir 

    Allow use of different settings files and some cleanup of system test directories
    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    2bc3804 View commit details
    Browse the repository at this point in the history

  6. Assert text on welcome dialog passes

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    6d2032a View commit details
    Browse the repository at this point in the history

  7. Copy nvda log for each test

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    b0477e8 View commit details
    Browse the repository at this point in the history

  8. Try to get systemtest artefacts from appveyor

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    4f9149e View commit details
    Browse the repository at this point in the history

  9. Try manually pushing artifacts

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    36e7ee8 View commit details
    Browse the repository at this point in the history

  10. Try recursing over files and pushing manually

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    e6a6fbf View commit details
    Browse the repository at this point in the history

  11. Fix issue when nvdaProfile folders do not exist

    @feerrenrut
    feerrenrut committed Jun 20, 2018
    Configuration menu
    Copy the full SHA
    4787753 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2018

  1. system tests: don't show usage stats dialog in NVDA

    @michaelDCurran
    michaelDCurran committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    e9f2189 View commit details
    Browse the repository at this point in the history

  2. appveyor: install NVDA on the build system, and run the system tests … 

    …against the installed copy.
    @michaelDCurran
    michaelDCurran committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    5321953 View commit details
    Browse the repository at this point in the history

  3. appveyor: install nvda with start-process and wait up to 3 minutes fo… 

    …r it to complete, failing on timeout.
    @michaelDCurran
    michaelDCurran committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    13b5397 View commit details
    Browse the repository at this point in the history

  4. fix appveyor errors

    @michaelDCurran
    michaelDCurran committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    66b0005 View commit details
    Browse the repository at this point in the history

  5. Temprarily include robotremoteserver in NVDA for system tests. Must f… 

    …igure out a better solution.
    @michaelDCurran
    michaelDCurran committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    6600242 View commit details
    Browse the repository at this point in the history

  6. Breakup up appveyor test code into separate powershell code chunks to… 

    … better show build progress.
    @michaelDCurran
    michaelDCurran committed Jun 21, 2018
    Configuration menu
    Copy the full SHA
    d02fe75 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2018

  1. system tests: use silence synth driver.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    c985677 View commit details
    Browse the repository at this point in the history

  2. Merge master

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    a32f1aa View commit details
    Browse the repository at this point in the history

  3. system test for exit dialog: just sleep instead of varifying dialog.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    5c4d284 View commit details
    Browse the repository at this point in the history

  4. system tests: don't wait for exit dialog, rather just confirm that th… 

    …e NVDA process definitlly exited.
    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    844c73b View commit details
    Browse the repository at this point in the history

  5. appveyor: move NVDA installation to before_test section.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    78432cf View commit details
    Browse the repository at this point in the history

  6. ensure quit system test: exit from the NVDA menu.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    221208e View commit details
    Browse the repository at this point in the history

  7. system tests: try alt tabbing to the exit dialog???

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    7bd259a View commit details
    Browse the repository at this point in the history

  8. More debugging for system tests.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    6a757e1 View commit details
    Browse the repository at this point in the history

  9. IAccessibleHandler: more debugging.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    6edf1e0 View commit details
    Browse the repository at this point in the history

  10. IAccessibleHandler: more debugging.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    167b362 View commit details
    Browse the repository at this point in the history

  11. eventHandler: more debugging.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    27446cf View commit details
    Browse the repository at this point in the history

  12. Up max foreground defer count to 10.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    ce6474a View commit details
    Browse the repository at this point in the history

  13. Up foreground defers to 100 and log them.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    86aeb06 View commit details
    Browse the repository at this point in the history

  14. IAccessibleHandler: drop foreground defers back down and allow welcom… 

    …e dialog in quit from keyboard system test.
    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    8bfbf74 View commit details
    Browse the repository at this point in the history

  15. system tests for quit with keyboard: watch for and close welcome dial… 

    …og first.
    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    8027682 View commit details
    Browse the repository at this point in the history

  16. quit with keyboard system test: wait longer for NVDA to exit.

    @michaelDCurran
    michaelDCurran committed Jun 26, 2018
    Configuration menu
    Copy the full SHA
    2913125 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2018

  1. Restore IAccessibleHandler and eventHandler.

    @michaelDCurran
    michaelDCurran committed Jun 27, 2018
    Configuration menu
    Copy the full SHA
    e6e8188 View commit details
    Browse the repository at this point in the history

  2. Not able to get an index returned from get_index_of_speech

    @feerrenrut
    feerrenrut committed Jun 27, 2018
    Configuration menu
    Copy the full SHA
    9fd13f4 View commit details
    Browse the repository at this point in the history

  3. tests passing

    @feerrenrut
    feerrenrut committed Jun 27, 2018
    Configuration menu
    Copy the full SHA
    52853fc View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2018

  1. move spy lib logic into spy

    @feerrenrut
    feerrenrut committed Jul 1, 2018
    Configuration menu
    Copy the full SHA
    da0df66 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2018

  1. Install systemTestSpy as a package on setup

    @feerrenrut
    feerrenrut committed Jul 2, 2018
    Configuration menu
    Copy the full SHA
    bc869e6 View commit details
    Browse the repository at this point in the history

  2. Cleanup files, add copyright headers

    @feerrenrut
    feerrenrut committed Jul 2, 2018
    Configuration menu
    Copy the full SHA
    f20dc35 View commit details
    Browse the repository at this point in the history

  3. Use _ to prevent imports from leaking as keywords

    @feerrenrut
    feerrenrut committed Jul 2, 2018
    Configuration menu
    Copy the full SHA
    612659d View commit details
    Browse the repository at this point in the history

  4. Move non NVDA specific helpers into a new library

    @feerrenrut
    feerrenrut committed Jul 2, 2018
    Configuration menu
    Copy the full SHA
    5266301 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/installed_systemTestWithRobot' i… 

    …nto systemTestWithRobot
    @feerrenrut
    feerrenrut committed Jul 2, 2018
    Configuration menu
    Copy the full SHA
    e3ae2a1 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2018

  1. Create self contained package for system test spy 

    The system test spy now copies in library dependencies that are not
provided by a NVDA install. This is so that the system tests can be run
on appVeyor against an installed version of NVDA
    @feerrenrut
    feerrenrut committed Jul 3, 2018
    Configuration menu
    Copy the full SHA
    fd67383 View commit details
    Browse the repository at this point in the history

  2. Zip system test output before pushing to artifacts

    @feerrenrut
    feerrenrut committed Jul 3, 2018
    Configuration menu
    Copy the full SHA
    f1be08f View commit details
    Browse the repository at this point in the history

  3. try adding robotlibraries directory to the path

    @feerrenrut
    feerrenrut committed Jul 3, 2018
    Configuration menu
    Copy the full SHA
    da14b4b View commit details
    Browse the repository at this point in the history

  4. Fix case of file name 

    This should resolve the appveyor system test failures
    @feerrenrut
    feerrenrut committed Jul 3, 2018
    Configuration menu
    Copy the full SHA
    6818faa View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2018

  1. further tidy up

    @feerrenrut
    feerrenrut committed Jul 4, 2018
    Configuration menu
    Copy the full SHA
    39f6b07 View commit details
    Browse the repository at this point in the history

  2. Introduce suite name

    @feerrenrut
    feerrenrut committed Jul 4, 2018
    Configuration menu
    Copy the full SHA
    50137bc View commit details
    Browse the repository at this point in the history

  3. fix appveyor system tests 

    testOutput directories need to be created before writting files to them!
    @feerrenrut
    feerrenrut committed Jul 4, 2018
    Configuration menu
    Copy the full SHA
    cbf6c7e View commit details
    Browse the repository at this point in the history

  4. Stop double nvda install log artifact

    @feerrenrut
    feerrenrut committed Jul 4, 2018
    Configuration menu
    Copy the full SHA
    385e8ea View commit details
    Browse the repository at this point in the history

  5. set robot library path

    @feerrenrut
    feerrenrut committed Jul 4, 2018
    Configuration menu
    Copy the full SHA
    df02377 View commit details
    Browse the repository at this point in the history

  6. reduce suite verbosity on appveyor

    @feerrenrut
    feerrenrut committed Jul 4, 2018
    Configuration menu
    Copy the full SHA
    44d94fd View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2018

  1. Revert to running from repo root 

    Running from tests/system/ did not reduce the number of suites mentioned in Robots reports
    @feerrenrut
    feerrenrut committed Jul 5, 2018
    Configuration menu
    Copy the full SHA
    25f1897 View commit details
    Browse the repository at this point in the history

  2. Clean up some of the appveyor code

    @feerrenrut
    feerrenrut committed Jul 5, 2018
    Configuration menu
    Copy the full SHA
    9254b1e View commit details
    Browse the repository at this point in the history

  3. clean up some names and docs for tests

    @feerrenrut
    feerrenrut committed Jul 5, 2018
    Configuration menu
    Copy the full SHA
    edb0e75 View commit details
    Browse the repository at this point in the history

  4. Add scons command to run system tests

    @feerrenrut
    feerrenrut committed Jul 5, 2018
    Configuration menu
    Copy the full SHA
    7bd629e View commit details
    Browse the repository at this point in the history

  5. ensure dir is created for std out/err logs

    @feerrenrut
    feerrenrut committed Jul 5, 2018
    Configuration menu
    Copy the full SHA
    2114664 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2018

  1. Merge remote-tracking branch 'origin/master' into systemTestWithRobot

    @feerrenrut
    feerrenrut committed Jul 6, 2018
    Configuration menu
    Copy the full SHA
    46cfaae View commit details
    Browse the repository at this point in the history

  2. Use dummy speech synth to get speech 

    This captures all NVDA manipulation of speech before it is given to
synthesizers.
    @feerrenrut
    feerrenrut committed Jul 6, 2018
    Configuration menu
    Copy the full SHA
    f8606bc View commit details
    Browse the repository at this point in the history

  3. wait longer before sending input

    @feerrenrut
    feerrenrut committed Jul 6, 2018
    Configuration menu
    Copy the full SHA
    c8b9199 View commit details
    Browse the repository at this point in the history

  4. Stop update dialog from showing on startup

    @feerrenrut
    feerrenrut committed Jul 6, 2018
    Configuration menu
    Copy the full SHA
    0e743ac View commit details
    Browse the repository at this point in the history

  5. simplify the appveyor pip installs

    @feerrenrut
    feerrenrut committed Jul 6, 2018
    Configuration menu
    Copy the full SHA
    21eca3c View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2018

  1. Update What's new

    @michaelDCurran
    michaelDCurran committed Jul 17, 2018
    Configuration menu
    Copy the full SHA
    be440a9 View commit details
    Browse the repository at this point in the history