Skip to content

Commit 3eeee42

Browse files
shamil-mubarakshinsergei-pyshnoigithub-actions[bot]459680Actions service account
authored
[Ubuntu ] Sync latest repo changes with ubuntu-new-folder-structure branch (#8679)
* Update folder structure * Adjust packer templates and debug * Fix issues in ubuntu2204 template * Remove Debug steps * Rename template, sync pipelines and adjust resource gen script * fix post-gen path in ubuntu-minimal * fix shebang linter, workflow paths and adjust readme * update preimagedata * [MacOS] Add xcode 15.0.1 (#8604) * Updating readme file for macos-13 version 20231018.14 (#8601) Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> * Updating readme file for macos-13-arm64 version 20231017.1 (#8586) Co-authored-by: Image generation service account <no-reply@microsoft.com> * [MacOS] improve errexit option handling (#8432) * [MacOS] improve errexit option handling * Prevent prompting for override when unzipping * Explicitly allow failing when installing virtualbox * Remove libtcl symlinks * Add debug output * Updating readme file for ubuntu20 version 20231022.1.1 (#8637) Co-authored-by: Image generation service account <no-reply@microsoft.com> * Updating readme file for ubuntu22 version 20231022.1.1 (#8636) Co-authored-by: Image generation service account <no-reply@microsoft.com> * Updating readme file for win19 version 20231023.1.1 (#8644) Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> * Updating readme file for win22 version 20231023.1.1 (#8643) Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> * [macos] simplify "brew_smart_install" helper (#8639) * [macos] homebrew: use hardcoded condition for jq installation we cannot use "jq" if we are asked to install "jq" * [macos] always use "brew install" * [macos] add retries to "get_github_package_download_url" helper * [macos] add retries to chrome install script * [macos] add retries to OpenJDK install script * [macos] add retries to miniconda installer * Update images/macos/provision/core/openjdk.sh Co-authored-by: Vasilii Polikarpov <126792224+vpolikarpov-akvelon@users.noreply.github.com> * fix copy-paste error * Update images/macos/provision/core/openjdk.sh Co-authored-by: Vasilii Polikarpov <126792224+vpolikarpov-akvelon@users.noreply.github.com> * Update images/macos/provision/core/openjdk.sh Co-authored-by: Vasilii Polikarpov <126792224+vpolikarpov-akvelon@users.noreply.github.com> --------- Co-authored-by: Vasilii Polikarpov <126792224+vpolikarpov-akvelon@users.noreply.github.com> * Updating readme file for ubuntu22 version 20231025.1.1 (#8663) Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> * Updating readme file for ubuntu20 version 20231025.1.1 (#8664) Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> * Updating readme file for win22 version 20231024.1.1 (#8668) Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> * Windows Server 2019 (20231025) Image Update (#8667) * Updating readme file for win19 version 20231025.2.1 * Update Windows2019-Readme.md --------- Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Mikhail Koliada <88318005+mikhailkoliada@users.noreply.github.com> Co-authored-by: Actions service account <no-reply@github.com> * [Mac OS] Fix logic for retrying AppleScript execution (#8662) * [Ubuntu] Add PGP validation for swift (#8652) * [Windows] Remove out-of-support vs components (#8678) * [MacOS] Unpin Virtualbox (#8653) * [MacOS] Unpin Virtualbox * Remove obsolete virtualbox kext pester test --------- Co-authored-by: sergei-pyshnoi <121864472+sergei-pyshnoi@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Image generation service account <no-reply@microsoft.com> Co-authored-by: Actions service account <no-reply@github.com> Co-authored-by: Vasilii Polikarpov <126792224+vpolikarpov-akvelon@users.noreply.github.com> Co-authored-by: ilia-shipitsin <125650415+ilia-shipitsin@users.noreply.github.com> Co-authored-by: Mikhail Koliada <88318005+mikhailkoliada@users.noreply.github.com>
1 parent 54a830d commit 3eeee42

18 files changed

+487
-518
lines changed

‎images/macos/macos-13-Readme.md‎

Lines changed: 77 additions & 75 deletions
Large diffs are not rendered by default.

‎images/macos/macos-13-arm64-Readme.md‎

Lines changed: 71 additions & 42 deletions
Large diffs are not rendered by default.

‎images/macos/provision/core/android-toolsets.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ sdkTools="android-sdk-tools.zip"
126126
sdkToolsVersion=$(get_toolset_value '.android."sdk-tools"')
127127
if [ "$sdkToolsVersion" != "null" ]; then
128128
download_with_retries "https://dl.google.com/android/repository/${sdkToolsVersion}" "." $sdkTools
129-
unzip -qq $sdkTools -d ${ANDROID_SDK_ROOT}
129+
unzip -o -qq $sdkTools -d ${ANDROID_SDK_ROOT}
130130
rm -f $sdkTools
131131
fi
132132

‎images/macos/provision/core/chrome.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ echo "Google Chrome version is $FULL_CHROME_VERSION"
1414
# Get Google Chrome versions information
1515
CHROME_PLATFORM="mac-$arch"
1616
CHROME_VERSIONS_URL="https://googlechromelabs.github.io/chrome-for-testing/latest-patch-versions-per-build-with-downloads.json"
17-
CHROME_VERSIONS_JSON=$(curl -fsSL "${CHROME_VERSIONS_URL}")
17+
download_with_retries "$CHROME_VERSIONS_URL" "/tmp" "latest-patch-versions-per-build-with-downloads.json"
18+
CHROME_VERSIONS_JSON=$(cat /tmp/latest-patch-versions-per-build-with-downloads.json)
1819

1920
# Download and unpack the latest release of Chrome Driver
2021
CHROMEDRIVER_VERSION=$(echo "${CHROME_VERSIONS_JSON}" | jq -r '.builds["'"$CHROME_VERSION"'"].version')

‎images/macos/provision/core/commonutils.sh‎

Lines changed: 13 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -13,18 +13,7 @@ done
1313
cask_packages=$(get_toolset_value '.brew.cask_packages[]')
1414
for package in $cask_packages; do
1515
echo "Installing $package..."
16-
if [[ $package == "virtualbox" ]]; then
17-
if ! is_Ventura || ! is_VenturaArm64; then
18-
# VirtualBox 7 crashes
19-
# macOS host: Dropped all kernel extensions. VirtualBox relies fully on the hypervisor and vmnet frameworks provided by Apple now.
20-
vbcask_url="https://raw.githubusercontent.com/Homebrew/homebrew-cask/aa3c55951fc9d687acce43e5c0338f42c1ddff7b/Casks/virtualbox.rb"
21-
download_with_retries $vbcask_url
22-
brew install ./virtualbox.rb
23-
rm ./virtualbox.rb
24-
fi
25-
else
26-
brew install --cask $package
27-
fi
16+
brew install --cask $package
2817
done
2918

3019
# Load "Parallels International GmbH"
@@ -36,18 +25,24 @@ fi
3625
# System Preferences -> Security & Privacy -> General -> Unlock -> Allow -> Not now
3726
if is_Monterey; then
3827
if is_Veertu; then
39-
retry=5
40-
while [ $retry -gt 0 ]; do
28+
for retry in {4..0}; do
29+
echo "Executing AppleScript to change security preferences. Retries left: $retry"
4130
{
31+
set -e
4232
osascript -e 'tell application "System Events" to get application processes where visible is true'
43-
}
44-
osascript $HOME/utils/confirm-identified-developers.scpt $USER_PASSWORD
33+
osascript $HOME/utils/confirm-identified-developers.scpt $USER_PASSWORD
34+
} && break
35+
36+
if [ "$retry" -eq 0 ]; then
37+
echo "Executing AppleScript failed. No retries left"
38+
exit 1
39+
fi
4540

46-
retry=$((retry-1))
47-
echo "retries left "$retry
41+
echo "Executing AppleScript failed. Sleeping for 10 seconds and retrying"
4842
sleep 10
4943
done
5044
else
45+
echo "Executing AppleScript to change security preferences"
5146
osascript $HOME/utils/confirm-identified-developers.scpt $USER_PASSWORD
5247
fi
5348
fi
@@ -82,13 +77,5 @@ bazel
8277
# Install Azure DevOps extension for Azure Command Line Interface
8378
az extension add -n azure-devops
8479

85-
# Workaround https://github.com/actions/runner-images/issues/4931
86-
# by making Tcl/Tk paths the same on macOS 10.15 and macOS 11
87-
if is_Monterey; then
88-
version=$(brew info tcl-tk --json | jq -r '.[].installed[].version')
89-
ln -s /usr/local/Cellar/tcl-tk/$version/lib/libtcl8.6.dylib /usr/local/lib/libtcl8.6.dylib
90-
ln -s /usr/local/Cellar/tcl-tk/$version/lib/libtk8.6.dylib /usr/local/lib/libtk8.6.dylib
91-
fi
92-
9380
# Invoke tests for all basic tools
9481
invoke_tests "BasicTools"

‎images/macos/provision/core/miniconda.sh‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
#!/bin/bash -e -o pipefail
22

3-
MINICONDA_INSTALLER="/tmp/miniconda.sh"
4-
curl -fsSL https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o $MINICONDA_INSTALLER
5-
chmod +x $MINICONDA_INSTALLER
6-
sudo $MINICONDA_INSTALLER -b -p /usr/local/miniconda
3+
source ~/utils/utils.sh
4+
5+
download_with_retries "https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh" "/tmp" "miniconda.sh"
6+
7+
chmod +x /tmp/miniconda.sh
8+
sudo /tmp/miniconda.sh -b -p /usr/local/miniconda
79

810
# Chmod with full permissions recursively to avoid permissions restrictions
911
sudo chmod -R 777 /usr/local/miniconda

‎images/macos/provision/core/openjdk.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,12 @@ installOpenJDK() {
2929
local JAVA_VERSION=$1
3030

3131
# Get link for Java binaries and Java version
32-
assetUrl=$(curl -fsSL "https://api.adoptium.net/v3/assets/latest/${JAVA_VERSION}/hotspot")
32+
download_with_retries "https://api.adoptium.net/v3/assets/latest/${JAVA_VERSION}/hotspot" "/tmp" "openjdk-hotspot.json"
3333

3434
if [[ $arch == "arm64" ]]; then
35-
asset=$(echo ${assetUrl} | jq -r '.[] | select(.binary.os=="mac" and .binary.image_type=="jdk" and .binary.architecture=="aarch64")')
35+
asset=$(jq -r '.[] | select(.binary.os=="mac" and .binary.image_type=="jdk" and .binary.architecture=="aarch64")' /tmp/openjdk-hotspot.json)
3636
else
37-
asset=$(echo ${assetUrl} | jq -r '.[] | select(.binary.os=="mac" and .binary.image_type=="jdk" and .binary.architecture=="x64")')
37+
asset=$(jq -r '.[] | select(.binary.os=="mac" and .binary.image_type=="jdk" and .binary.architecture=="x64")' /tmp/openjdk-hotspot.json)
3838
fi
3939

4040
archivePath=$(echo ${asset} | jq -r '.binary.package.link')

‎images/macos/provision/core/pypy.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ function InstallPyPy
1111
PACKAGE_URL=$1
1212

1313
PACKAGE_TAR_NAME=$(echo $PACKAGE_URL | awk -F/ '{print $NF}')
14-
echo "Downloading tar archive '$PACKAGE_TAR_NAME' - '$PACKAGE_URL'"
14+
echo "Downloading tar archive '$PACKAGE_TAR_NAME'"
1515
PACKAGE_TAR_TEMP_PATH="/tmp/$PACKAGE_TAR_NAME"
1616
download_with_retries $PACKAGE_URL "/tmp" "$PACKAGE_TAR_NAME"
1717

@@ -89,4 +89,4 @@ for toolsetVersion in $toolsetVersions; do
8989
exit 1
9090
fi
9191
InstallPyPy $latestMajorPyPyVersion
92-
done
92+
done

‎images/macos/provision/utils/utils.sh‎

Lines changed: 55 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -15,24 +15,31 @@ download_with_retries() {
1515
local COMMAND="curl $URL -4 -sL -o '$DEST/$NAME' -w '%{http_code}'"
1616
fi
1717

18+
# Save current errexit state and disable it to prevent unexpected exit on error
19+
if echo $SHELLOPTS | grep '\(^\|:\)errexit\(:\|$\)' > /dev/null;
20+
then
21+
local ERR_EXIT_ENABLED=true
22+
else
23+
local ERR_EXIT_ENABLED=false
24+
fi
25+
set +e
26+
1827
echo "Downloading '$URL' to '${DEST}/${NAME}'..."
1928
retries=20
2029
interval=30
2130
while [ $retries -gt 0 ]; do
2231
((retries--))
23-
# Temporary disable exit on error to retry on non-zero exit code
24-
set +e
32+
test "$ERR_EXIT_ENABLED" = true && set +e
2533
http_code=$(eval $COMMAND)
2634
exit_code=$?
35+
test "$ERR_EXIT_ENABLED" = true && set -e
2736
if [ $http_code -eq 200 ] && [ $exit_code -eq 0 ]; then
2837
echo "Download completed"
2938
return 0
3039
else
3140
echo "Error — Either HTTP response code for '$URL' is wrong - '$http_code' or exit code is not 0 - '$exit_code'. Waiting $interval seconds before the next attempt, $retries attempts left"
3241
sleep 30
3342
fi
34-
# Enable exit on error back
35-
set -e
3643
done
3744

3845
echo "Could not download $URL"
@@ -120,98 +127,50 @@ get_brew_os_keyword() {
120127
fi
121128
}
122129

123-
should_build_from_source() {
124-
local tool_name=$1
125-
local os_name=$2
126-
# If one of the parsers aborts with an error,
127-
# we will get an empty variable notification in the logs
128-
set -u
129-
130-
# Geting tool info from brew to find available install methods except build from source
131-
local tool_info=$(brew info --json=v1 $tool_name)
132-
133-
# No need to build from source if a bottle is disabled
134-
local bottle_disabled=$(echo -E $tool_info | jq ".[0].bottle_disabled")
135-
if [[ $bottle_disabled == "true" ]]; then
136-
echo "false"
137-
return
138-
fi
139-
140-
# No need to build from source if a universal bottle is available
141-
local all_bottle=$(echo -E $tool_info | jq ".[0].bottle.stable.files.all")
142-
if [[ "$all_bottle" != "null" ]]; then
143-
echo "false"
144-
return
145-
fi
146-
147-
# No need to build from source if a bottle for current OS is available
148-
local os_bottle=$(echo -E $tool_info | jq ".[0].bottle.stable.files.$os_name")
149-
if [[ "$os_bottle" != "null" ]]; then
150-
echo "false"
151-
return
152-
fi
153-
154-
# Available method wasn't found - should build from source
155-
echo "true"
156-
}
157-
158130
# brew provides package bottles for different macOS versions
159131
# The 'brew install' command will fail if a package bottle does not exist
160132
# Use the '--build-from-source' option to build from source in this case
161133
brew_smart_install() {
162134
local tool_name=$1
163135

164-
local os_name=$(get_brew_os_keyword)
165-
if [[ "$os_name" == "null" ]]; then
166-
echo "$OSTYPE is unknown operating system"
167-
exit 1
168-
fi
136+
echo "Downloading $tool_name..."
169137

170-
local build_from_source=$(should_build_from_source "$tool_name" "$os_name")
171-
if $build_from_source; then
172-
echo "Bottle of the $tool_name for the $os_name was not found. Building $tool_name from source..."
173-
brew install --build-from-source $tool_name
174-
else
175-
echo "Downloading $tool_name..."
138+
# get deps & cache em
176139

177-
# get deps & cache em
140+
failed=true
141+
for i in {1..10}; do
142+
brew deps $tool_name > /tmp/$tool_name && failed=false || sleep 60
143+
[ "$failed" = false ] && break
144+
done
178145

179-
failed=true
180-
for i in {1..10}; do
181-
brew deps $tool_name > /tmp/$tool_name && failed=false || sleep 60
182-
[ "$failed" = false ] && break
183-
done
146+
if [ "$failed" = true ]; then
147+
echo "Failed: brew deps $tool_name"
148+
exit 1;
149+
fi
184150

185-
if [ "$failed" = true ]; then
186-
echo "Failed: brew deps $tool_name"
187-
exit 1;
188-
fi
151+
for dep in $(cat /tmp/$tool_name) $tool_name; do
189152

190-
for dep in $(cat /tmp/$tool_name) $tool_name; do
191-
192-
failed=true
193-
for i in {1..10}; do
194-
brew --cache $dep >/dev/null && failed=false || sleep 60
195-
[ "$failed" = false ] && break
196-
done
197-
198-
if [ "$failed" = true ]; then
199-
echo "Failed: brew --cache $dep"
200-
exit 1;
201-
fi
202-
done
203-
204-
failed=true
205-
for i in {1..10}; do
206-
brew install $tool_name >/dev/null && failed=false || sleep 60
207-
[ "$failed" = false ] && break
208-
done
209-
210-
if [ "$failed" = true ]; then
211-
echo "Failed: brew install $tool_name"
212-
exit 1;
213-
fi
153+
failed=true
154+
for i in {1..10}; do
155+
brew --cache $dep >/dev/null && failed=false || sleep 60
156+
[ "$failed" = false ] && break
157+
done
158+
159+
if [ "$failed" = true ]; then
160+
echo "Failed: brew --cache $dep"
161+
exit 1;
162+
fi
163+
done
164+
165+
failed=true
166+
for i in {1..10}; do
167+
brew install $tool_name >/dev/null && failed=false || sleep 60
168+
[ "$failed" = false ] && break
169+
done
214170

171+
if [ "$failed" = true ]; then
172+
echo "Failed: brew install $tool_name"
173+
exit 1;
215174
fi
216175
}
217176

@@ -240,7 +199,18 @@ get_github_package_download_url() {
240199

241200
[ -n "$API_PAT" ] && authString=(-H "Authorization: token ${API_PAT}")
242201

243-
json=$(curl "${authString[@]}" -fsSL "https://api.github.com/repos/${REPO_ORG}/releases?per_page=${SEARCH_IN_COUNT}")
202+
failed=true
203+
for i in {1..10}; do
204+
curl "${authString[@]}" -fsSL "https://api.github.com/repos/${REPO_ORG}/releases?per_page=${SEARCH_IN_COUNT}" >/tmp/get_github_package_download_url.json && failed=false || sleep 60
205+
[ "$failed" = false ] && break
206+
done
207+
208+
if [ "$failed" = true ]; then
209+
echo "Failed: get_github_package_download_url"
210+
exit 1;
211+
fi
212+
213+
json=$(cat /tmp/get_github_package_download_url.json)
244214

245215
if [[ "$VERSION" == "latest" ]]; then
246216
tagName=$(echo $json | jq -r '.[] | select((.prerelease==false) and (.assets | length > 0)).tag_name' | sort --unique --version-sort | egrep -v ".*-[a-z]" | tail -1)

‎images/macos/tests/BasicTools.Tests.ps1‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,13 @@ Describe "Helm" -Skip:($os.IsMonterey -or $os.IsVentura -or $os.IsVenturaArm64)
6767
}
6868
}
6969

70+
Describe "Tcl/Tk" {
71+
It "libtcl" {
72+
"file /usr/local/lib/libtcl8.6.dylib" | Should -ReturnZeroExitCode
73+
"file /usr/local/lib/libtk8.6.dylib" | Should -ReturnZeroExitCode
74+
}
75+
}
76+
7077
Describe "bazelisk" {
7178
It "bazelisk" {
7279
"bazelisk version" | Should -ReturnZeroExitCode
@@ -164,9 +171,9 @@ Describe "Homebrew" {
164171
}
165172

166173
Describe "Kotlin" {
167-
$kotlinPackages = @("kapt", "kotlin", "kotlinc", "kotlinc-jvm", "kotlin-dce-js")
174+
$kotlinPackages = @("kapt", "kotlin", "kotlinc", "kotlinc-jvm", "kotlin-dce-js")
168175

169-
It "<toolName> is available" -TestCases ($kotlinPackages | ForEach-Object { @{ toolName = $_ } }) {
176+
It "<toolName> is available" -TestCases ($kotlinPackages | ForEach-Object { @{ toolName = $_ } }) {
170177
"$toolName -version" | Should -ReturnZeroExitCode
171178
}
172179
}
@@ -187,4 +194,4 @@ Describe "imagemagick" -Skip:($os.IsVentura -or $os.IsVenturaArm64) {
187194
It "imagemagick" {
188195
"magick -version" | Should -ReturnZeroExitCode
189196
}
190-
}
197+
}

0 commit comments

Comments
 (0)