New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
System test with robot framework #8481
Changes from 95 commits
f868f5f
e9a2af7
1fb27cb
584ea96
5c83124
dd9f3e7
5a75207
1e6e265
69f4407
9bd9760
488a33e
f275871
9b889f8
c3b4a90
8b0e1c2
9270a20
30a1256
8e02fc4
3824887
e244bcb
1c4f7b9
d80c688
5e130c0
bdb51da
1286782
a57e95e
5d540e7
8195ee4
6e530b6
38e696b
2274ea6
d8cd618
d6acae5
5c55400
06aa06b
ce75aa0
0c16f1b
735991a
662eabb
08c8697
8981697
669da0a
2bc3804
6d2032a
b0477e8
4f9149e
36e7ee8
e6a6fbf
4787753
e9f2189
5321953
13b5397
66b0005
6600242
d02fe75
c985677
a32f1aa
5c4d284
844c73b
78432cf
221208e
7bd259a
6a757e1
6edf1e0
167b362
27446cf
ce6474a
86aeb06
8bfbf74
8027682
2913125
e6e8188
9fd13f4
52853fc
da0df66
bc869e6
f20dc35
612659d
5266301
e3ae2a1
fd67383
f1be08f
da14b4b
6818faa
39f6b07
50137bc
cbf6c7e
385e8ea
df02377
44d94fd
25f1897
9254b1e
edb0e75
7bd629e
2114664
46cfaae
f8606bc
c8b9199
0e743ac
21eca3c
be440a9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,7 @@ environment: | |
|
||
init: | ||
- ps: | | ||
iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1')) | ||
if ($env:APPVEYOR_REPO_TAG_NAME -and $env:APPVEYOR_REPO_TAG_NAME.StartsWith("release-")) { | ||
# Strip "release-" prefix. | ||
$version = $env:APPVEYOR_REPO_TAG_NAME.Substring(8) | ||
|
@@ -61,6 +62,7 @@ install: | |
- type ssh_known_hosts >> %userprofile%\.ssh\known_hosts | ||
- cd .. | ||
- git submodule update --init | ||
- py -m pip install robotremoteserver | ||
|
||
build_script: | ||
- ps: | | ||
|
@@ -87,8 +89,8 @@ build_script: | |
Set-AppveyorBuildVariable "sconsArgs" $sconsArgs | ||
- 'echo scons args: %sconsArgs%' | ||
- py scons.py source %sconsArgs% | ||
# We don't need launcher to run tests, so run the tests before launcher. | ||
- py scons.py tests %sconsArgs% | ||
# We don't need launcher to run checkPot, so run the checkPot before launcher. | ||
- py scons.py checkPot %sconsArgs% | ||
# The pot gets built by tests, but we don't actually need it as a build artifact. | ||
- del output\*.pot | ||
- 'echo scons output targets: %sconsOutTargets%' | ||
|
@@ -109,6 +111,58 @@ build_script: | |
- 7z a -tzip -r ..\output\symbols.zip *.dl_ *.ex_ *.pd_ | ||
- cd .. | ||
|
||
before_test: | ||
- py -m pip install robotframework | ||
- py -m pip install robotremoteserver | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a particular reason why this installed twice, also after running submodule update? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Perhaps a mistake, for future reference, the other is on line 65 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah yes, I remember now. For a little while we were packaging robotRemoteServer with the install of NVDA so that it could be used by the addon. Now we package it within the addon. I will remove this line. |
||
- py -m pip install pyautogui | ||
- py -m pip install nose | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can also install them in one run, I'd imagine that could be slightly faster. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes perhaps so, I'm not sure how much time this will save. It might even make sense to convert this to a requirements file in source control. It would make it easier for developers to ready their own systems to run the system tests. |
||
- mkdir testOutput | ||
- mkdir testOutput\unit | ||
- mkdir testOutput\system | ||
- ps: | | ||
$errorCode=0 | ||
$nvdaLauncherFile=".\output\nvda" | ||
if(!$env:release) { | ||
$nvdaLauncherFile+="_snapshot" | ||
} | ||
$nvdaLauncherFile+="_${env:version}.exe" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. a couple of things.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, the tests run on try builds, against an install of NVDA. I can't speak to your second question, it might be best for @michaelDCurran to answer that one. |
||
echo NVDALauncherFile: $NVDALauncherFile | ||
$outputDir=$(resolve-path .\testOutput) | ||
$installerLogFilePath="$outputDir\nvda_install.log" | ||
$installerProcess=start-process -FilePath "$nvdaLauncherFile" -ArgumentList "--install-silent --debug-logging --log-file $installerLogFilePath" -passthru | ||
try { | ||
$installerProcess | wait-process -Timeout 180 | ||
$errorCode=$installerProcess.ExitCode | ||
} catch { | ||
echo NVDA installer process timed out | ||
$errorCode=1 | ||
} | ||
Push-AppveyorArtifact $installerLogFilePath | ||
if($errorCode -ne 0) { $host.SetShouldExit($errorCode) } | ||
|
||
test_script: | ||
- ps: | | ||
$errorCode=0 | ||
$outDir = (Resolve-Path .\testOutput\unit\) | ||
$unitTestsXml = "$outDir\unitTests.xml" | ||
py -m nose --with-xunit --xunit-file="$unitTestsXml" ./tests/unit | ||
if($LastExitCode -ne 0) { $errorCode=$LastExitCode } | ||
Push-AppveyorArtifact $unitTestsXml | ||
$wc = New-Object 'System.Net.WebClient' | ||
$wc.UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", $unitTestsXml) | ||
if($errorCode -ne 0) { $host.SetShouldExit($errorCode) } | ||
- ps: | | ||
$testOutput = (Resolve-Path .\testOutput\) | ||
$systemTestOutput = (Resolve-Path "$testOutput\system") | ||
$testSource = "./tests/system" | ||
py -m robot --loglevel DEBUG -d $systemTestOutput -x systemTests.xml -v whichNVDA:installed -P "$testSource/libraries" "$testSource" | ||
Compress-Archive -Path "$systemTestOutput\*" -DestinationPath "$testOutput\systemTestResult.zip" | ||
Push-AppveyorArtifact "$testOutput\systemTestResult.zip" | ||
if($LastExitCode -ne 0) { $errorCode=$LastExitCode } | ||
$wc = New-Object 'System.Net.WebClient' | ||
$wc.UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path "$systemTestOutput\systemTests.xml")) | ||
if($errorCode -ne 0) { $host.SetShouldExit($errorCode) } | ||
|
||
artifacts: | ||
- path: output\* | ||
- path: output\*\* | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -29,6 +29,11 @@ | |
from logHandler import log | ||
import addonHandler | ||
|
||
import extensionPoints | ||
|
||
# inform those who want to know that NVDA has finished starting up. | ||
postNvdaStartup = extensionPoints.Action() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe that, according to earlier convention discussions we had, this should be post_NVDAStartup. Also note that it might make sense to write NVDA in upper case, as in NVDAObjects, NVDAHelper There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should be added to the changes file. This is useful for addons potentially. |
||
|
||
PUMP_MAX_DELAY = 10 | ||
|
||
#: The thread identifier of the main thread. | ||
|
@@ -470,6 +475,7 @@ def Notify(self): | |
log.debug("initializing updateCheck") | ||
updateCheck.initialize() | ||
log.info("NVDA initialized") | ||
postNvdaStartup.notify() | ||
|
||
log.debug("entering wx application main loop") | ||
app.MainLoop() | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -27,6 +27,10 @@ | |
import speechDictHandler | ||
import characterProcessing | ||
import languageHandler | ||
import extensionPoints | ||
|
||
# inform those who want to know that there is new speech | ||
preSpeech = extensionPoints.Action() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also, this might have to be pre_speech according to the conventions. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the conventions were discussed on PR #7606, though the underscores were dropped from that diff. I would be happy with the underscores and certainly want to keep it consistent, perhaps it should be made official on the wiki so that everyone is on the same page. @josephsl would you be on board with this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what are the performance impacts for this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There may be a minor performance impact here, very little compared to the rest of what happens before speech goes to the synth. Which raises a bigger question about the approach of using an extension point to gather the speech, should we be capturing the behaviour that happens between this line, and the eventual I think we probably should, otherwise there is no way for us to test things like auto language / dialect switching or symbol levels. Perhaps, instead of an extension point for speech, the addon should install and select a dummy synthesizer. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. both could be useful actually. The synth captures the output and the extension point the first call. |
||
|
||
speechMode_off=0 | ||
speechMode_beeps=1 | ||
|
@@ -493,10 +497,11 @@ def speak(speechSequence,symbolLevel=None): | |
""" | ||
if not speechSequence: #Pointless - nothing to speak | ||
return | ||
preSpeech.notify(speechSequence=speechSequence) | ||
import speechViewer | ||
if speechViewer.isActive: | ||
for item in speechSequence: | ||
if isinstance(item,basestring): | ||
if isinstance(item, basestring): | ||
speechViewer.appendText(item) | ||
global beenCanceled, curWordChars | ||
curWordChars=[] | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,11 @@ unitTests = env.SConscript("unit/sconscript", exports=["env"]) | |
env.Depends(unitTests, sourceDir) | ||
env.AlwaysBuild(unitTests) | ||
|
||
systemTests = env.SConscript("system/sconscript", exports=["env"]) | ||
env.Depends(systemTests, sourceDir) | ||
env.AlwaysBuild(systemTests) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why are these always built? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. At the moment, the system tests only run if you explicitly give the option when starting scons. Then we want the tests to run, even if there has been no changes to dependencies. |
||
env.Alias("systemTests", systemTests) | ||
|
||
def checkPotAction(target, source, env): | ||
return checkPot.checkPot(source[0].abspath) | ||
checkPotTarget = env.Command("checkPot", pot, checkPotAction) | ||
|
@@ -31,10 +36,13 @@ env.Alias("checkPot", checkPotTarget) | |
# If specific tests are explicitly specified, only run those. | ||
explicitUnitTests = env.get("unitTests") | ||
explicitCheckPot = "checkPot" in COMMAND_LINE_TARGETS | ||
explicit = explicitUnitTests or explicitCheckPot | ||
explicitSystemTests = "systemTests" in COMMAND_LINE_TARGETS | ||
explicit = explicitUnitTests or explicitCheckPot or explicitSystemTests | ||
tests = [] | ||
if not explicit or explicitUnitTests: | ||
tests.append(unitTests) | ||
if not explicit or explicitCheckPot: | ||
tests.append(checkPotTarget) | ||
if explicit and explicitSystemTests: # only run system tests explicitly | ||
tests.append(systemTests) | ||
env.Alias("tests", tests) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
nvdaProfile/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
# A part of NonVisual Desktop Access (NVDA) | ||
# Copyright (C) 2018 NV Access Limited | ||
# This file may be used under the terms of the GNU General Public License, version 2 or later. | ||
# For more details see: https://www.gnu.org/licenses/gpl-2.0.html | ||
*** Settings *** | ||
Documentation Basic start and exit tests | ||
Default Tags NVDA smoke test | ||
|
||
Library OperatingSystem | ||
Library Process | ||
Library sendKey.py | ||
Library nvdaRobotLib.py | ||
Library helperLib.py | ||
|
||
Test Setup start NVDA standard-dontShowWelcomeDialog.ini | ||
Test Teardown quit NVDA | ||
|
||
Variables variables.py | ||
|
||
*** Test Cases *** | ||
Starts | ||
[Documentation] Ensure that NVDA can start | ||
process should be running nvdaAlias | ||
|
||
Quits from keyboard | ||
[Documentation] Starts NVDA and ensures that it can be quit using the keyboard | ||
[Setup] start NVDA standard-doShowWelcomeDialog.ini | ||
|
||
${Welcome dialog title} = catenate double space Welcome to NVDA dialog | ||
wait for specific speech ${Welcome dialog title} | ||
wait for speech to finish | ||
sleep 1 # the dialog is not always receiving the enter keypress, wait a little longer for it | ||
send key enter | ||
|
||
${Exit NVDA dialog} = catenate double space Exit NVDA dialog | ||
send key insert q | ||
${INDEX} = wait for specific speech ${Exit NVDA dialog} | ||
|
||
wait for speech to finish | ||
${actual speech} = get speech from index until now ${INDEX} | ||
assert strings are equal ${actual speech} ${QUIT_DIALOG_TEXT} | ||
|
||
send key enter | ||
wait for process nvdaAlias timeout=10 sec | ||
process should be stopped nvdaAlias | ||
|
||
Read welcome dialog | ||
[Documentation] Ensure that the welcome dialog can be read in full | ||
[Setup] start NVDA standard-doShowWelcomeDialog.ini | ||
|
||
${Welcome dialog title} = catenate double space Welcome to NVDA dialog | ||
${INDEX} = wait for specific speech ${Welcome dialog title} | ||
wait for speech to finish | ||
${actual speech} = get speech from index until now ${INDEX} | ||
assert strings are equal ${actual speech} ${WELCOME_DIALOG_TEXT} | ||
send key enter |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
WELCOME_DIALOG_TEXT = ( | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This might break if a user runs the system tests with a system language other than English, in which the user default language differs. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it's likely that there will be lots of ways that the tests will not pass on various different systems. To start with, we will be using AppVeyor as the target. Over time, if people are trying to run the tests locally, then the tests will become more tolerant of variations in the underlying system. |
||
"Welcome to NVDA dialog Welcome to NVDA!\n" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why not use a multiline string? |
||
"Most commands for controlling NVDA require you to hold down the NVDA key while pressing other keys.\n" | ||
"By default, the numpad Insert and main Insert keys may both be used as the NVDA key.\n" | ||
"You can also configure NVDA to use the CapsLock as the NVDA key.\n" | ||
"Press NVDA+n at any time to activate the NVDA menu.\n" | ||
"From this menu, you can configure NVDA, get help and access other NVDA functions.\n" | ||
"Options grouping\n" | ||
"Keyboard layout: combo box desktop collapsed Alt+k" | ||
) | ||
QUIT_DIALOG_TEXT = ( | ||
"Exit NVDA dialog\n" | ||
"What would you like to do? combo box Exit collapsed Alt+d" | ||
) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. file missing a new line after ) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
# A part of NonVisual Desktop Access (NVDA) | ||
# Copyright (C) 2018 NV Access Limited | ||
# This file may be used under the terms of the GNU General Public License, version 2 or later. | ||
# For more details see: https://www.gnu.org/licenses/gpl-2.0.html | ||
|
||
"""This file provides general robot library functions for system tests. | ||
This is in contrast with nvdaRobotLib.py which contains helpers related to starting and stopping NVDA for system | ||
tests, or with systemTestSpy which contains methods for extracting information about NVDA's behaviour during system | ||
tests. | ||
""" | ||
from robot.libraries.BuiltIn import BuiltIn | ||
builtIn = BuiltIn() # type: BuiltIn | ||
|
||
def assert_strings_are_equal( actual, expected, ignore_case=False): | ||
try: | ||
builtIn.should_be_equal_as_strings( | ||
actual, | ||
expected, | ||
msg="Actual speech != Expected speech", | ||
ignore_case=ignore_case | ||
) | ||
except AssertionError: | ||
builtIn.log( | ||
"repr of actual vs expected (ignore_case={}):\n{}\nvs\n{}".format( | ||
ignore_case, | ||
repr(actual), | ||
repr(expected) | ||
) | ||
) | ||
raise | ||
|
||
|
||
def catenate_double_space(*args): | ||
return " ".join(args) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having a comment here describing why this is downloaded might help potential future NV access devs if we lose access to the knowledge of why this was originally done.