Skip to content

Commit

Permalink
Make it easier for devs outside of NV Access to utilize the Appveyor …
Browse files Browse the repository at this point in the history
…build process (nvaccess#16221)

Closes nvaccess#16216

Summary of the issue:
Currently, for a developer to make a private Appveyor build of NVDA, requires maintaining different build scripts, in addition to an intricately modified appveyor.yml file, and feeding them to Appveyor on every build.

This PR removes much of the difficulty from private Appveyor builds, by making the build process more configurable in appveyor.yml, without having to modify the build scripts at all to make a basic build work.

In addition, it adds a few sensible state checks to a couple build scripts.

It tries to avoid changing any of the existing logic, except where an if condition could be anded with another to achieve the desired result.

Description of user facing changes
None

Description of development approach
appveyor.yml:
Move the environment section above the init section.
Added a scons_publisher variable, to avoid hard-coding "NV Access" in the scons variable script. Those doing private builds should change it to reflect their (org) name.
Added a feature subsection to the environment section. Included features to control:
building symbols,
Uploading symbols to Mozilla,
building the appx,
crowdinSync, and
package signing.
(For private builds, all of those features should be commented out, except, optionally, the one for building symbols.)
Added logic to check the setting of the buildSymbols feature before building symbols.
appveyor/scripts/buildSymbolStore.ps1: Added a test for feature_buildSymbols.
appveyor/scripts/deployScript.ps1:
Make symbol upload dependent on both the buildSymbols and uploadSymbolsToMozilla features being set in appveyor.yml.
Cause crowdin synchronization to be dependent on its feature from appveyor.yml.
Only deploy to the NV Access server, if it's an nvaccess owned repo.
appveyor/scripts/setSconsArgs.ps1:
Use the scons_publisher variable from appveyor.yml to designate the publisher, instead of hardcoding in the script.
Don't include cert options if not signing based on feature in appveyor.yml.
Respond to the buildAppx feature in appveyor.yml.
appveyor/scripts/pushPackagingInfo.ps1: Change the appveyor message with the exe link to mention a "branch" instead of a "PR", if no PR number is set in the build. It is assumed that most private builds will be for branches, not PRs against their own clone.
appveyor/scripts/decryptFilesForSigning.ps1: Do not attempt to decrypt files if signing feature is not set in appveyor.yml.
appveyor/scripts/logCiTiming.ps1: added a test to exit if the timing record file is not found.
Regarding the last: originally I included a feature to disable CI timing checks. I decided that feature was unnecessary so reverted it, but I kept the change to detect the missing timing file and silently exit rather than failing hard, since it makes sense that something else may cause that file not to exist, and we wouldn't want the build failing over it.
  • Loading branch information
XLTechie authored and Adriani90 committed Mar 13, 2024
1 parent 06b1cbd commit 782eaae
Show file tree
Hide file tree
Showing 8 changed files with 75 additions and 46 deletions.
29 changes: 20 additions & 9 deletions appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,6 @@ branches:
- /try-.*/
- /release-.*/

# scripts that are called at very beginning, before repo cloning
init:
# set the init time for the build, used to calculate the time taken for each stage of the build.
- ps: |
"INIT, $(Get-Date -Format 'o')"| Out-File ../timing.csv -Append
environment:
PY_PYTHON: 3.11-32
secure_authenticode_pass:
Expand All @@ -31,6 +25,20 @@ environment:
crowdinProjectID: 598017
crowdinAuthToken:
secure: E3084gj4JeMZKvZIOLIhqZefuSo/tj7iYPt4yK0geOI/eQgmPvoXt37Xq0KwvXzvZiJny4AsMj1rKMTVxio8EG8KA0YsYYuy+WV1wpFRIn25zGQS+DZ/yycL75SmTWfr
scons_publisher: NV Access
# Comment out any of the feature_* variables to disable the respective build feature.
# They are checked for existence of content, not specific value.
feature_buildSymbols: True
feature_uploadSymbolsToMozilla: True
feature_buildAppx: True
feature_crowdinSync: True
feature_signing: True

# scripts that are called at very beginning, before repo cloning
init:
# set the init time for the build, used to calculate the time taken for each stage of the build.
- ps: |
"INIT, $(Get-Date -Format 'o')"| Out-File ../timing.csv -Append
install:
- ps: |
Expand All @@ -51,9 +59,12 @@ build_script:
- ps: appveyor\scripts\buildSymbolStore.ps1
# The server expects the symbols archive to be structured as ./*.ex_ not ./symbols/*.ex_.
# Change directory to package, as 7z will structure the archive using the relative path.
- cd symbols
- 7z a -tzip -r ..\output\symbols.zip *.dl_ *.ex_ *.pd_
- cd ..
- ps: >-
if ($env:feature_buildSymbols) {
cd symbols
7z a -tzip -r ..\output\symbols.zip *.dl_ *.ex_ *.pd_
cd ..
}
- ps: |
"BUILD_END, $(Get-Date -Format 'o')"| Out-File ../timing.csv -Append
Expand Down
4 changes: 4 additions & 0 deletions appveyor/scripts/buildSymbolStore.ps1
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
$ErrorActionPreference = "Stop";
if (!$env:feature_buildSymbols) {
exit
}

foreach ($syms in
# We don't just include source\*.dll because that would include system dlls.
"source\liblouis.dll", "source\*.pdb",
Expand Down
2 changes: 1 addition & 1 deletion appveyor/scripts/decryptFilesForSigning.ps1
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
if(!$env:APPVEYOR_PULL_REQUEST_NUMBER) {
if(!$env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:feature_signing) {
openssl enc -d -md sha256 -aes-256-cbc -pbkdf2 -salt -pass pass:$env:secure_authenticode_pass -in appveyor\authenticode.pfx.enc -out appveyor\authenticode.pfx
if($LastExitCode -ne 0) {
$errorCode=$LastExitCode
Expand Down
68 changes: 36 additions & 32 deletions appveyor/scripts/deployScript.ps1
Original file line number Diff line number Diff line change
@@ -1,43 +1,47 @@
$ErrorActionPreference = "Stop";
if (!$env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:versionType) {
# Not a try build.
if ($env:APPVEYOR_REPO_BRANCH -eq "beta") {
if ($env:APPVEYOR_REPO_BRANCH -eq "beta" -and $env:feature_crowdinSync) {
# Upload files to Crowdin for translation
py -m pip install --no-warn-script-location requests
py appveyor\crowdinSync.py uploadSourceFile 2 output\nvda.pot 2>&1
}
# Notify our server.
$exe = Get-ChildItem -Name output\*.exe
$hash = (Get-FileHash "output\$exe" -Algorithm SHA1).Hash.ToLower()
$apiVersion = (py -c "import sys; sys.path.append('source'); from addonAPIVersion import CURRENT; print('{}.{}.{}'.format(*CURRENT))")
echo apiversion: $apiVersion
$apiCompatTo = (py -c "import sys; sys.path.append('source'); from addonAPIVersion import BACK_COMPAT_TO; print('{}.{}.{}'.format(*BACK_COMPAT_TO))")
echo apiBackCompatTo: $apiCompatTo
$data = @{
jobId=$env:APPVEYOR_JOB_ID;
commit=$env:APPVEYOR_REPO_COMMIT;
version=$env:version; versionType=$env:versionType;
apiVersion=$apiVersion; apiCompatTo=$apiCompatTo;
avVersion=$env:APPVEYOR_BUILD_VERSION;
branch=$env:APPVEYOR_REPO_BRANCH;
exe=$exe; hash=$hash;
artifacts=$artifacts
# Notify the NV Access server if this is an NV Access build.
if ($env:APPVEYOR_REPO_NAME.StartsWith("nvaccess/")) {
$exe = Get-ChildItem -Name output\*.exe
$hash = (Get-FileHash "output\$exe" -Algorithm SHA1).Hash.ToLower()
$apiVersion = (py -c "import sys; sys.path.append('source'); from addonAPIVersion import CURRENT; print('{}.{}.{}'.format(*CURRENT))")
echo apiversion: $apiVersion
$apiCompatTo = (py -c "import sys; sys.path.append('source'); from addonAPIVersion import BACK_COMPAT_TO; print('{}.{}.{}'.format(*BACK_COMPAT_TO))")
echo apiBackCompatTo: $apiCompatTo
$data = @{
jobId=$env:APPVEYOR_JOB_ID;
commit=$env:APPVEYOR_REPO_COMMIT;
version=$env:version; versionType=$env:versionType;
apiVersion=$apiVersion; apiCompatTo=$apiCompatTo;
avVersion=$env:APPVEYOR_BUILD_VERSION;
branch=$env:APPVEYOR_REPO_BRANCH;
exe=$exe; hash=$hash;
artifacts=$artifacts
}
ConvertTo-Json -InputObject $data -Compress | Out-File -FilePath deploy.json
Push-AppveyorArtifact deploy.json
# Execute the deploy script on the NV Access server via ssh.
# Warning: if the server address is changed,
# The new address must be also included in appveyor\ssh_known_hosts in this repo
# Otherwise ssh will freeze on input!
cat deploy.json | ssh nvaccess@deploy.nvaccess.org nvdaAppveyorHook
}
ConvertTo-Json -InputObject $data -Compress | Out-File -FilePath deploy.json
Push-AppveyorArtifact deploy.json
# Execute the deploy script on the NV Access server via ssh.
# Warning: if the server address is changed,
# The new address must be also included in appveyor\ssh_known_hosts in this repo
# Otherwise ssh will freeze on input!
cat deploy.json | ssh nvaccess@deploy.nvaccess.org nvdaAppveyorHook

# Upload symbols to Mozilla.
py -m pip install --upgrade --no-warn-script-location pip
py -m pip install --no-warn-script-location requests
Try {
py appveyor\mozillaSyms.py
}
Catch {
Add-AppveyorMessage "Unable to upload symbols to Mozilla"
# Upload symbols to Mozilla if feature enabled.
if ($env:feature_buildSymbols -and $env:feature_uploadSymbolsToMozilla) {
py -m pip install --upgrade --no-warn-script-location pip
py -m pip install --no-warn-script-location requests
Try {
py appveyor\mozillaSyms.py
}
Catch {
Add-AppveyorMessage "Unable to upload symbols to Mozilla"
}
}
}
5 changes: 5 additions & 0 deletions appveyor/scripts/logCiTiming.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@
$inputFile = "../timing.csv"
$processedTimesFile = "buildStageTimingWithElapsed.csv"

# Don't run if timing record was not created for some reason
if (!(Test-Path -LiteralPath $inputFile)) {
exit
}

$entries = Import-Csv -Path $inputFile -Header Stage, Time
$lastTime = Get-Date -Date $entries[0].Time

Expand Down
6 changes: 5 additions & 1 deletion appveyor/scripts/pushPackagingInfo.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,9 @@ $appVeyorUrl = "https://ci.appveyor.com"
$exe = Get-ChildItem -Name output\*.exe
if($?){
$exeUrl="$appVeyorUrl/api/buildjobs/$env:APPVEYOR_JOB_ID/artifacts/output/$exe"
Add-AppveyorMessage "Build (for testing PR): $exeUrl"
if ($env:APPVEYOR_PULL_REQUEST_NUMBER -ne $null) {
Add-AppveyorMessage "Build (for testing PR): $exeUrl"
} else {
Add-AppveyorMessage "Build (for testing branch): $exeUrl"
}
}
6 changes: 3 additions & 3 deletions appveyor/scripts/setSconsArgs.ps1
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
$ErrorActionPreference = "Stop";
$sconsOutTargets = "launcher developerGuide changes userGuide keyCommands client moduleList"
if(!$env:APPVEYOR_PULL_REQUEST_NUMBER) {
if(!$env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:feature_buildAppx) {
$sconsOutTargets += " appx"
}
$sconsArgs = "version=$env:version"
Expand All @@ -10,8 +10,8 @@ if ($env:release) {
if ($env:versionType) {
$sconsArgs += " updateVersionType=$env:versionType"
}
$sconsArgs += ' publisher="NV Access"'
if(!$env:APPVEYOR_PULL_REQUEST_NUMBER) {
$sconsArgs += " publisher=`"$env:scons_publisher`""
if (!$env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:feature_signing) {
$sconsArgs += " certFile=appveyor\authenticode.pfx certTimestampServer=http://timestamp.digicert.com"
}
$sconsArgs += " version_build=$env:APPVEYOR_BUILD_NUMBER"
Expand Down
1 change: 1 addition & 0 deletions user_docs/en/changes.t2t
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ What's New in NVDA
Please refer to [the developer guide https://www.nvaccess.org/files/nvda/documentation/developerGuide.html#API] for information on NVDA's API deprecation and removal process.

- Instantiating ``winVersion.WinVersion`` objects with unknown Windows versions above 10.0.22000 such as 10.0.25398 returns "Windows 11 unknown" instead of "Windows 10 unknown" for release name. (#15992, @josephsl)
- Make the AppVeyor build process easier for NVDA forks, by adding configurable variables in appveyor.yml to disable or modify NV Access specific portions of the build scripts. (#16216, @XLTechie)
-

=== Deprecations ===
Expand Down

0 comments on commit 782eaae

Please sign in to comment.