Updated sdk.py to support local install using ANDROID_HOME (#8969)

* Updated sdk.py to support local install using ANDROID_HOME * build fix * Updated docs * Added verbose support to build.py:check_sdk * Added a build.py:verify_sdk to check that a platform compiler works
defold · May 24, 2024 · a90004d · a90004d
1 parent 3396ece
commit a90004d
Show file tree

Hide file tree

Showing 7 changed files with 205 additions and 61 deletions.
diff --git a/README_ANDROID.md b/README_ANDROID.md
@@ -4,6 +4,25 @@
 
 ### Android SDK/NDK
 
+When building the engine, the build system tries to find the sdk in this order:
+
+* `$DYNAMO_HOME/ext/SDKs/android-sdk` -  This is mainly for the Defold team and their CI
+* `$ANDROID_HOME` - Set this path to point to your android studio sdk installation (e.g. `ANDROID_HOME=~/Library/Android/sdk` is the default installation on macOS)
+
+#### Android Studio
+
+For the easiest build setup, you can install [Android Studio](https://developer.android.com/studio)
+Once installed, you need to also install the NDK.
+Open "Settings" -> "Languages & Frameworks" -> "Android SDK". From the "SDK Tools" tab choose "NDK (Side by side)".
+
+The build system will find the most recent build tools and NDK versions.
+However, the minimum API target numbers are fixed, and are specified our [sdk.py](./build_tools/sdk.py)
+
+#### Version specific builds
+
+This step isn't required if you have Android Studio installed.
+It is mainly for the Defold team and the build server setup.
+
 **Note that the SDK version numbers aren't the same as the Api Level numbers!**
 
 * Download SDK Tools 24.3.4 (or later) from here: [http://developer.android.com/sdk/index.html](http://developer.android.com/sdk/index.html)

diff --git a/README_BUILD.md b/README_BUILD.md
@@ -6,16 +6,28 @@ Defold uses the Python based build system [Waf](https://waf.io/). Most of the in
 
 Make sure you have followed the [setup guide](/README_SETUP.md) before attempting to build the engine. If you do not install all of the required software from the setup guide your attempts to build the engine will likely fail.
 
-## IMPORTANT PREREQUISITE - PACKAGE SDKs!
+## IMPORTANT PREREQUISITE - PLATFORM SDKs!
 
-### Local SDKs
-If you're a contributor, chances are that you have the necessary sdk's and tools already installed (E.g. Windows: Visual Studio+Windows SDK, macOS: Xcode or Linux: clang++)
-If this is the case, then you won't have to run the `install_sdk` build command, and thus would have no requirement for any prepackaged sdks.
+### Using local installation
+
+Our build setup can find the installation the platform SDK's for these platforms
+
+* Windows - Visual Studio + clang++
+* macOS/iOS - XCode
+* Android - Android Studio
+* Linux - clang++
+* Consoles - The vendor specific sdks
+
+In the future, we want to support HTML5 as well, for easier setup for all platforms.
+
+If you have these tools installed, you can skip the `./scripts/build.py install_sdk` step altogether.
 
 Since support for locally installed SDK's is in progress, some platforms still _do_ require the `--install_sdk` step, and thus requires you to have the prepackaged sdk's available.
 
 ### Prepackaged SDKs
 
+This step is currently needed for HTML5.
+
 Due to licensing restrictions **the SDKs are not distributed with Defold**. You need to provide these from a URL accessible by your local machine so that `build.py` and the `install_ext` command can download and unpack them.
 
 __In order to simplify this process we provide scripts to download and package the SDKs__ [Read more about this process here](/scripts/package/README.md).
@@ -72,7 +84,7 @@ $ ./scripts/build.py install_ext --platform=...
 It is important that you provide the `--platform` option to let the `install_ext` command know which platform you intend to build for (the target platform). When the `install_ext` command has finished you will find the external packages and downloaded SDKs in `./tmp/dynamo_home/ext`.
 
 **IMPORTANT!**
-You need to rerun the `install_ext` command whenever you switch target platform, as different packages and SDKs are installed.
+You need to rerun the `install_ext` command for each target platform, as different packages and SDKs are installed.
 
 #### Installing packages
 The `install_ext` command starts by installing external packages, mostly pre-built libraries for each supported platform, found in the `./packages` folder. External packages are things such as Bullet and Protocol Buffers (a.k.a. protobuf).
@@ -88,10 +100,9 @@ This step also installs some Python dependencies:
 
 ### Step 3 - Installing SDKs
 
-The `install_sdk`command will install SDKs (build tools etc) such as the Android SDK when building for Android or the Emscripten SDK for HTML5.
+NOTE: As mentioned above, you may skip this step if your host OS and target OS is in the supported list of platforms that can use the local (host) installations of sdks.
 
-For some platforms, it is not necessary to install the platform SDK.
-For Windows/macOS, it can automatically pick up your local install of Visual Studio and XCode.
+The `install_sdk`command will install SDKs (build tools etc) such as the Android SDK when building for Android or the Emscripten SDK for HTML5.
 
 If you wish to build for any other platform, you will need to install an sdk package where the build system can find it.
 
@@ -101,7 +112,11 @@ Next thing you need to do is to install external packages:
 $ ./scripts/build.py install_sdk --platform=... --package-path=...
 ```
 
-You could also set the package path in an environment variable `DM_PACKAGES_URL`.
+You could also set the package path in an environment variable `DM_PACKAGES_URL`:
+
+```sh
+$ DM_PACKAGES_URL=https://my.url ./scripts/build.py install_sdk --platform=...
+```
 
 ### STEP 4 - Build the engine
 

diff --git a/README_SETUP.md b/README_SETUP.md
@@ -108,9 +108,10 @@ You need the `dos2unix` command line tool to convert line endings of certain sou
 
 <details><summary>Windows...</summary><p>
 
-#### Visual C++ 2019 Community
+#### Visual C++ 2022 Community
 
 [Download](https://visualstudio.microsoft.com/vs/older-downloads/) the Community version or use the Professional or Enterprise version if you have the proper licence. When installing, select the "Desktop Development with C++" workload. There is also an optional 3rd party git client.
+Also, make sure to install the "clang++" compiler as well.
 
 #### Terminal
 
@@ -123,6 +124,8 @@ This terminal has the tool `winget` to install some packages.
 
 #### MSYS/MinGW
 
+This is an alternative to Terminal/git-bash setup.
+
 This will get you a shell that behaves like Linux and is much easier to build Defold through. [Download](https://sourceforge.net/projects/mingw/files/Installer/mingw-get-setup.exe/download) and run the installer and check these packages (binary):
 
 * MingW Base System: `mingw32-base-bin`, 'mingw32-gcc-g++-bin'
@@ -141,6 +144,8 @@ You also need to install `wget`. From the mingw terminal run:
 
 #### Git
 
+This is not needed if you've installed `git-bash`
+
 You need to [download](https://git-scm.com/download/win) a command line version of Git.
 
 During install, select the option to not do any CR/LF conversion.

diff --git a/build_tools/sdk.py b/build_tools/sdk.py
@@ -60,6 +60,9 @@
 # Android
 
 ANDROID_NDK_VERSION='25b'
+ANDROID_BUILD_TOOLS_VERSION = '33.0.1'
+ANDROID_NDK_API_VERSION='19' # Android 4.4
+ANDROID_64_NDK_API_VERSION='21' # Android 5.0
 
 ## **********************************************************************************************
 # Win32
@@ -154,6 +157,42 @@ def get_local_darwin_sdk_version(platform):
     return run.shell_command('xcrun -f --sdk %s --show-sdk-platform-version' % _convert_darwin_platform(platform)).strip()
 
 
+## **********************************************************************************************
+## Android
+
+
+def get_android_local_sdk_path():
+    path = os.environ.get('ANDROID_HOME', None)
+
+    if path is None:
+        if sys.platform == 'darwin':
+            path = os.path.expanduser('~/Library/android/sdk')
+
+    if path and os.path.exists(path):
+        return path
+
+    return None
+
+def _get_latest_version_from_folders(path):
+    dirs = [ x for x in os.listdir(path)]
+    dirs.sort(key=lambda x: tuple(int(token) for token in x.split('.')))
+    return dirs[0]
+
+def get_android_local_ndk_path(platform):
+    sdk_root = get_android_local_sdk_path()
+    ndk_root = os.path.join(sdk_root, 'ndk')
+    version = _get_latest_version_from_folders(ndk_root)
+    return os.path.join(ndk_root, version)
+
+def get_android_local_build_tools_path(platform):
+    path = get_android_local_sdk_path()
+    build_tools_path = os.path.join(path, 'build-tools')
+    version = _get_latest_version_from_folders(build_tools_path)
+    return os.path.join(build_tools_path, version)
+
+def get_android_local_sdk_version(platform):
+    return os.path.basename(get_android_local_build_tools_path(platform))
+
 ## **********************************************************************************************
 
 # ANDROID_HOME
@@ -219,6 +258,11 @@ def get_windows_local_sdk_info(platform):
     if windows_info is not None:
         return windows_info
 
+    if sys.platform != 'win32':
+        # we cannot currently use vswhere.exe on this platform
+        # todo: check using `wine`
+        return None
+
     vswhere_path = '%s/../../scripts/windows/vswhere2/vswhere2.exe' % os.environ.get('DYNAMO_HOME', '.')
     if not os.path.exists(vswhere_path):
         vswhere_path = './scripts/windows/vswhere2/vswhere2.exe'
@@ -337,23 +381,24 @@ def _setup_info_from_windowsinfo(windowsinfo, platform):
 def _get_defold_path(sdkfolder, platform):
     return os.path.join(sdkfolder, defold_info[platform]['pattern'])
 
-def check_defold_sdk(sdkfolder, platform):
+def check_defold_sdk(sdkfolder, platform, verbose=False):
     folders = []
-    print ("check_defold_sdk", sdkfolder, platform)
+    if verbose:
+        log.log("check_defold_sdk: %s %s" % (sdkfolder, platform))
 
     if platform in ('x86_64-macos', 'arm64-macos', 'arm64-ios', 'x86_64-ios'):
         folders.append(_get_defold_path(sdkfolder, 'xcode'))
         folders.append(_get_defold_path(sdkfolder, platform))
 
-    if platform in ('x86_64-win32', 'win32'):
+    elif platform in ('x86_64-win32', 'win32'):
         folders.append(os.path.join(sdkfolder, 'Win32','WindowsKits','10'))
         folders.append(os.path.join(sdkfolder, 'Win32','MicrosoftVisualStudio14.0','VC'))
 
-    if platform in ('armv7-android', 'arm64-android'):
+    elif platform in ('armv7-android', 'arm64-android'):
         folders.append(os.path.join(sdkfolder, "android-ndk-r%s" % ANDROID_NDK_VERSION))
         folders.append(os.path.join(sdkfolder, "android-sdk"))
 
-    if platform in ('x86_64-linux',):
+    elif platform in ('x86_64-linux',):
         folders.append(os.path.join(sdkfolder, "linux"))
 
     if not folders:
@@ -363,20 +408,35 @@ def check_defold_sdk(sdkfolder, platform):
     count = 0
     for f in folders:
         if not os.path.exists(f):
-            log.log("Missing SDK in %s" % f)
+            if verbose:
+                log.log("  Missing SDK in %s" % f)
         else:
             count = count + 1
-    return count == len(folders)
+    result = count == len(folders)
+
+    if verbose:
+        if not result:
+            log.log("  No prepackaged sdk found.")
+        else:
+            log.log("  Found prepackaged sdk folders:")
+            for f in folders:
+                log.log("    %s" % f)
 
-def check_local_sdk(platform):
+    return result
+
+def check_local_sdk(platform, verbose=False):
     if platform in ('x86_64-macos', 'arm64-macos', 'arm64-ios', 'x86_64-ios'):
         xcode_version = get_local_darwin_toolchain_version()
         if not xcode_version:
             return False
-    if platform in ('win32', 'x86_64-win32'):
+    elif platform in ('win32', 'x86_64-win32'):
         info = get_windows_local_sdk_info(platform)
         return info is not None
 
+    elif platform in ('armv7-android', 'arm64-android'):
+        path = get_android_local_sdk_path()
+        return path is not None
+
     return True
 
 
@@ -398,10 +458,20 @@ def _get_defold_sdk_info(sdkfolder, platform):
         info[platform]['version'] = defold_info[platform]['version']
         info[platform]['path'] = _get_defold_path(sdkfolder, platform)
 
-    if platform in ('win32', 'x86_64-win32'):
+    elif platform in ('win32', 'x86_64-win32'):
         windowsinfo = get_windows_packaged_sdk_info(sdkfolder, platform)
         return _setup_info_from_windowsinfo(windowsinfo, platform)
 
+    elif platform in ('armv7-android', 'arm64-android'):
+        info['version'] = ANDROID_BUILD_TOOLS_VERSION
+        info['sdk'] = os.path.join(sdkfolder, "android-sdk")
+        info['ndk'] = os.path.join(sdkfolder, "android-ndk-r%s" % ANDROID_NDK_VERSION)
+        info['build_tools'] = os.path.join(info['sdk'], "build-tools" , ANDROID_BUILD_TOOLS_VERSION)
+        if platform == 'arm64-android':
+            info['api'] = ANDROID_64_NDK_API_VERSION
+        else:
+            info['api'] = ANDROID_NDK_API_VERSION
+
     return info
 
 def _get_local_sdk_info(platform):
@@ -425,25 +495,35 @@ def _get_local_sdk_info(platform):
         info[platform]['version'] = get_local_compiler_version()
         info[platform]['path'] = get_local_compiler_path()
 
-    if platform in ('win32', 'x86_64-win32'):
+    elif platform in ('win32', 'x86_64-win32'):
         windowsinfo = get_windows_local_sdk_info(platform)
         return _setup_info_from_windowsinfo(windowsinfo, platform)
 
+    elif platform in ('armv7-android', 'arm64-android'):
+        info['version'] = get_android_local_sdk_version(platform)
+        info['sdk'] = get_android_local_sdk_path()
+        info['ndk'] = get_android_local_ndk_path(platform)
+        info['build_tools'] = get_android_local_build_tools_path(platform)
+        if platform == 'arm64-android':
+            info['api'] = ANDROID_64_NDK_API_VERSION
+        else:
+            info['api'] = ANDROID_NDK_API_VERSION
+
     return info
 
 # It's only cached for the duration of one build
 cached_platforms = defaultdict(defaultdict)
 
-def get_sdk_info(sdkfolder, platform):
+def get_sdk_info(sdkfolder, platform, verbose=False):
     if platform in cached_platforms:
         return cached_platforms[platform]
 
-    if check_defold_sdk(sdkfolder, platform):
+    if check_defold_sdk(sdkfolder, platform, verbose):
         result = _get_defold_sdk_info(sdkfolder, platform)
         cached_platforms[platform] = result
         return result
 
-    if check_local_sdk(platform):
+    if check_local_sdk(platform, verbose):
         result = _get_local_sdk_info(platform)
         cached_platforms[platform] = result
         return result