diff --git a/assets/contributors.csv b/assets/contributors.csv index 61a086e88e..b469a8a3aa 100644 --- a/assets/contributors.csv +++ b/assets/contributors.csv @@ -85,4 +85,4 @@ Yiyang Fan,Arm,,,, Julien Jayat,Arm,,,, Geremy Cohen,Arm,geremyCohen,geremyinanutshell,, Barbara Corriero,Arm,,,, - +Jun He,Arm,JunHe77,jun-he-91969822,, diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/1_codescanning.md b/content/learning-paths/servers-and-cloud-computing/migrate-ease/1_codescanning.md new file mode 100644 index 0000000000..3c1d358054 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/migrate-ease/1_codescanning.md @@ -0,0 +1,30 @@ +--- +# User change +title: "Why Code Scanning Matters for Arm Migration" + +weight: 2 + +layout: "learningpathall" + +--- + +### Overview of the Common Arm Migration Challenge + +Migrating applications to Arm-based architectures is increasingly common across cloud, data center, and edge environments. Arm-powered servers and instances, such as AWS Graviton processors or Ampere Altra-based systems available on platforms like Google Cloud Platform, Microsoft Azure, Alibaba Cloud and Oracle Cloud Infrastructure (OCI), deliver significant performance-per-watt advantages and compelling cost-efficiency. + +However, porting workloads to new CPU architecture often involves more than just recompiling. While many applications transition smoothly, others contain architecture-specific code or dependencies – developed originally for x86 – that can lead to build failures, runtime errors, or performance degradation on Arm systems. + +Common challenges include detecting: +* Hardcoded x86 SIMD intrinsics +* Inline assembly +* Platform-specific system calls +* Unsupported compiler flags +* Non-portable build scripts or logic + +In large or legacy codebases, these issues are often buried in third-party libraries or auto-generated components, making manual inspection slow and unreliable. + +### Automated Analysis for Portability +To address these challenges proactively, automated static analysis tools play a critical role. Tools specifically designed for portability analysis enable developers to scan local codebases or remote repositories (e.g., GitHub) —and pinpoint architecture-specific constructs before attempting compilation or deployment on Arm. By surfacing portability concerns early in the development cycle, code scanning reduces time-to-first-build and helps prevent complex failures later. + +In this learning path, we will introduce `migrate-ease`, a tool which allows developers to move beyond trial-and-error debugging. It provides clear, actionable insights into potential portability issues, detecting problematic patterns across many mainstream programming languages. These insights, combined with targeted refactoring, accelerate the process of making code portable, maintainable, and production-ready on Arm. + diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/2_migrate_ease_spec.md b/content/learning-paths/servers-and-cloud-computing/migrate-ease/2_migrate_ease_spec.md new file mode 100644 index 0000000000..6447610b07 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/migrate-ease/2_migrate_ease_spec.md @@ -0,0 +1,64 @@ +--- +# User change +title: "Supported Programming Languages and Common Issues Identified" + +weight: 3 + +layout: "learningpathall" + +--- + +### migrate-ease + +This is a fork of [Porting advisor](https://github.com/arm-hpc/porting-advisor), an open source project by the Arm engineering team. Migrate-ease is maintained by the [OpenAnolis](https://github.com/openanolis) Arm Working Group. + + +It is an innovative project designed to analyze codebases specifically for x86_64 architectures and offers tailored suggestions aimed at facilitating the migration process to aarch64. +This tool streamlines the transition, ensuring a smooth and efficient evolution of your software to leverage the benefits of aarch64 architecture. +At present, this tool only supports codebase migration to Linux. It can be run on Arm or non-Arm based machines. The tool does not modify any code, it does not provide API level recommendations, and it does not send any data back to OpenAnolis. + +{{% notice Note %}} +Even though the [migrate-ease](https://github.com/migrate-ease/) project team does the best to identify known incompatibilities, that's still recommend performing appropriate tests on your application before going to Production. +{{% /notice %}} + +### List of Supported Programming Languages + +This tool scans all files in a source tree, regardless of whether they are included by the build system or not. Currently, the tool supports the following languages/dependencies: + +#### C, C++ +- Inline assembly with no corresponding aarch64 inline assembly +- Assembly code with no corresponding aarch64 assembly code +- Use of architecture specific intrinsic +- Use of architecture specific compilation options +- Preprocessor errors that trigger when compiling on aarch64 +- Compiler specific code guarded by compiler specific pre-defined macros +- Missing aarch64 architecture detection in Makefile, Config.guess scripts +- Linking against libraries that are not available on the aarch64 architecture + +#### Go +- Inline assembly with no corresponding aarch64 inline assembly +- Assembly code with no corresponding aarch64 assembly code +- Use of architecture specific intrinsic +- Linking against libraries that are not available on the aarch64 architecture + +#### Python +- Inline assembly with no corresponding aarch64 inline assembly +- Use of architecture specific intrinsic +- Linking against libraries that are not available on the aarch64 architecture +- Use of architecture specific packages + +#### Rust +- Inline assembly with no corresponding aarch64 inline assembly +- Use of architecture specific intrinsic +- Linking against libraries that are not available on the aarch64 architecture + +#### Java +- JAR scanning +- Dependency versions in pom.xml file +- A feature to detect native calls in Java source code +- Compatible version recommendation + +#### Dockerfile +- Use of architecture specific plugin +- The base image that dockfile is based on does not support aarch64 + diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/3_migrate_ease_run.md b/content/learning-paths/servers-and-cloud-computing/migrate-ease/3_migrate_ease_run.md new file mode 100644 index 0000000000..55fdfe39ff --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/migrate-ease/3_migrate_ease_run.md @@ -0,0 +1,101 @@ +--- +# User change +title: "Installing and Running Code Analysis Tools" + +weight: 4 + +layout: "learningpathall" + +--- + +# migrate-ease + +[migrate-ease](https://github.com/migrate-ease/migrate-ease) is an open-source project designed to analyze codebases specifically for x86_64 architectures and offers tailored suggestions aimed at facilitating the migration process to AArch64. This tool streamlines the transition, ensuring a smooth and efficient evolution of your software to leverage the benefits of aarch64 architecture. + +## Pre-requisites +Before you use migrate-ease, certain pre-requesites need to be installed. +{{< tabpane code=true >}} + {{< tab header="Ubuntu 22.04">}} +sudo apt-get install -y python3 python3-pip python3-venv unzip libmagic1 git + {{< /tab >}} + {{< tab header="Debian trixie">}} +sudo apt-get install -y python3 python3-pip python3-venv unzip libmagic1 git + {{< /tab >}} + {{< tab header="Fedora 42">}} +sudo dnf install -y python3 python3-pip unzip git + {{< /tab >}} +{{< /tabpane >}} + +## Install and setup +```bash + +# Check out the project +git clone https://github.com/migrate-ease/migrate-ease + +# Check the project folder +cd migrate-ease + +# Make sure you're at the root directory of migrate-ease +# Enable Python Environment +python3 -m venv .venv +source .venv/bin/activate + +# install python packages dependencies +pip3 install -r requirements.txt +# Setup environment +export PYTHONPATH=`pwd` +``` + +## Usage +### As Python script + +User can scan a project with a scanner type and the result is sent to console by default. +```bash +python3 -m {scanner_name} --arch {arch} {scan_path} +``` +Scan result can be exported as one of txt, csv, json or html. + +To generate a JSON report: +```bash +python3 -m {scanner_name} --output {result_file_name}.json --arch {arch} {scan_path} +``` + +**Parameters** + +`{scanner_name}`: The name of the scanner, which can be one of cpp, docker, go, java, python, rust. + +`{result_file_name}`: The name of the export result file (without the extension). + +`{arch}`: The architecture type, `aarch64` is the default. + +`{scan_path}`: The path to the code that needs to be scanned. + +User can scan a git repository as well by given the URL. +```bash +python3 -m {scanner_name} --output {result_file_name}.json --arch {arch} --git-repo {repo} {clone_path} +``` +In the case of git repository scan, `{clone_path}` is a directory to hold the cloned code. This directory should be empty or to be created. + +There are more parameters for user to control the scan behaviors. To get these information user can get the built-in help +```bash +python3 -m {scanner_name} -h +``` + +### As Web UI +Migrate-ease also provides a Web UI that supports scanning a git repo with cpp, docker, go, java, python and rust scanners in one time. +To start the web server, simply run: +``` +python3 web/server.py +``` + +Once that is successfully done, you can access a web server hosted at http://localhost:8080 + +The web UI looks as following: +![example image alt-text#center](web_ui_index.jpg "Figure 1. Web UI to scan a git repo") + +A git repo URL is required, and you can specify certain branch name to scan. Once the necessary information is filled, user can click "START SCAN" button to proceed project scanning. + +Scanning progress will be shown in the console pane. Once all jobs are done, user will see a web page as following: +![example image alt-text#center](web_ui_result.jpg "Figure 2. Web UI of scan result") + +You can download the result by clicking the "download" icon or view the result by clicking the "eye" icon. diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/4_migrate_ease_analysis.md b/content/learning-paths/servers-and-cloud-computing/migrate-ease/4_migrate_ease_analysis.md new file mode 100644 index 0000000000..a8489fdb7e --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/migrate-ease/4_migrate_ease_analysis.md @@ -0,0 +1,196 @@ +--- +# User change +title: "Interpreting Analysis Results with a Real Example" + +weight: 5 + +layout: "learningpathall" + +--- +### Example from a real-world case +[Protobuf](https://github.com/protocolbuffers/protobuf) is a widely used library for serializing structured data. AArch64 support is introduced in version v3.5.0 released in November 2017. Version v2.5.0 is a popular version without AArch64 support. + +Use migrate-ease to scan protobuf v2.5.0 with JSON result output: +``` +python3 -m cpp --git-repo https://github.com/protocolbuffers/protobuf.git --branch v2.5.0 --output result.json --arch aarch64 protobuf +``` +A json file, `result.json`, will be generated in current directory once the scan is successfully executed. + +### How to read the result +The JSON result is organized as following format: +```output +{ + "arch": "aarch64", + "branch": "v2.5.0", + "commit": null, + "errors": [], + "file_summary": { + "asm": { + "count": 0, + "fileName": "Assembly", + "loc": 0 + }, + "c": { + "count": 93, + "fileName": "C", + "loc": 29031 + }, + "config.guess": { + "count": 0, + "fileName": "Autoconf", + "loc": 0 + }, + "cpp": { + "count": 113, + "fileName": "CPP", + "loc": 79010 + }, + "makefile": { + "count": 4, + "fileName": "Makefile", + "loc": 719 + } + }, + "git_repo": "https://github.com/protocolbuffers/protobuf.git", + "issue_summary": { + "ArchSpecificLibraryIssue": { + "count": 0, + "des": "Use of libraries strongly tied to the processor architecture, which may lead to compatibility issues." + }, + .... + "PreprocessorErrorIssue": { + "count": 15, + "des": "Target platform may enter the #error preprocessing logic" + }, + "SignedCharIssue": { + "count": 0, + "des": "Compatibility issues with signed char type data" + } + }, + "issue_type_config": null, + "issues": [ + { + "checkpoint": null, + "description": "preprocessor error: #error \"We require at least vs2005 for MemoryBarrier\"", + "filename": "/home/my_repo/src/google/protobuf/stubs/atomicops_internals_x86_msvc.h", + "issue_type": { + "des": "Target platform may enter the #error preprocessing logic.", + "type": "PreprocessorErrorIssue" + }, + "lineno": 46, + "snippet": "namespace google {\nnamespace protobuf {\nnamespace internal {\n\ninline Atomic32 NoBarrier_AtomicIncrement(volatile Atomic32* ptr,\n Atomic32 increment) {\n return Barrier_AtomicIncrement(ptr, increment);\n}\n\n#if !(defined(_MSC_VER) && _MSC_VER >= 1400)\n#error \"We require at least vs2005 for MemoryBarrier\"\n#endif\n\ninline Atomic32 Acquire_CompareAndSwap(volatile Atomic32* ptr,\n Atomic32 old_value,\n Atomic32 new_value) {\n return NoBarrier_CompareAndSwap(ptr, old_value, new_value);\n}\n\ninline Atomic32 Release_CompareAndSwap(volatile Atomic32* ptr,\n Atomic32 old_value,\n" + }, + ... + { + "checkpoint": ".*[= \"]+-O2[ \\n\"]+.*", + "description": "\nWhen the compiler optimization option is set to \"-O2\" level or above, the calculation results of the same floating-point multiplication and addition operation on the x86 platform and the ARM64 platform have differences in the 16 decimal places.\n\nReason:\n\n When the compiler optimization option is set to \"-O2\" level or above on the ARM64 platform, the precision of the floating-point multiplication and addition operation (a+=b*c) can only be accurate to 16 decimal places. When configuring the \"-O2\" option, gcc uses the fused instruction fmadd to complete the multiplication and addition operation instead of fadd and fmul.\n\n fmadd regards the multiplication and addition of floating-point numbers as an inseparable operation and does not round the intermediate results, resulting in different calculation results.\n\nImpact on the system:\n\n When the compiler optimization option is set to \"-O2\" level or above, the performance of floating-point multiplication and addition operations is improved, but the accuracy of the operation is affected.\n\nSolution:\n\n Add the compilation option \"-ffp-contract=off\" This optimization can be turned off.\n\n", + "filename": "protobuf/more_tests/Makefile", + "issue_type": { + "des": "Potential compatibility issues related to the compilation build commands", + "type": "BuildCommandIssue" + }, + "lineno": 40, + "snippet": "\t(cd src && make install)\n\n# Verify that headers produce no warnings even under strict settings.\nheader_warning_test.cc: target\n\t( (cd target/include && find google/protobuf -name '*.h') | \\\n\t awk '{print \"#include \\\"\"$$1\"\\\"\"} ' > header_warning_test.cc )\n\nheader_warning_test: header_warning_test.cc\n\t# TODO(kenton): Consider adding -pedantic and -Weffc++. Currently these\n\t# produce tons of extra warnings so we'll need to do some work first.\n\tg++ -Itarget/include -Wall -Werror -Wsign-compare -O2 -c header_warning_test.cc\n", + "target": null + } + ], + "language_type": "cpp", + "march": null, + "output": null, + "progress": true, + "quiet": false, + "remarks": [], + "root_directory": "protobuf", + "source_dirs": [ + "protobuf/src/google/protobuf/compiler/cpp", + ... + ], + "source_files": [ + "protobuf/Makefile.am", + ... + ], + "target_os": "OpenAnolis", + "total_issue_count": 14 +} +``` +The items in result are well self-explained along with their keys. +User needs to check `issue_summary` and `issues` to uncover what potential problems are and suggested solutions. + +The `issue_summary` provides an overview of the types of issues that the current scanner supports, along with the corresponding number of issues. For `cpp` scanner, the available issue types are: + +For each of programming Languages, the available issue types are: +{{< tabpane code=true >}} + {{< tab header="C++, C">}} +Name | Description +------------------------|------------------------------------------------------------------------------------------------------------- +ArchSpecificLibrary | Use of libraries strongly tied to the processor architecture, which may lead to compatibility issues. +AsmSource | Potentially architecture-specific assembly code in the source files that requires manual inspection. +Avx256Intrinsic | Use of AVX256 instructions on the AArch64 architecture lead to compatibility issues. +Avx512Intrinsic | Use of AVX512 instructions on the AArch64 architecture lead to compatibility issues. +BuildCommand | Potential compatibility issues related to the compilation build commands. +CPPLibRecommend | A better-optimized version of this library may be available. +CPPLibVersion | This library version may be incompatible with the AArch64 architecture's compiler. +CPPStdCodes | Compatibility issues or optimization opportunities related to Cpp source and memory order on AArch64. +CompilerSpecific | Code is strongly tied to a compiler version or type, which may lead to compatibility issues. +ConfigGuess | Config.guess file does not contain configurations for AArch64 and may require adaptation. +CrossCompile | Cross-compilation compatibility issues. +DefineOtherArch | Logic in the code that checks for other processor platform types, which may lead to compatibility issues. +HostCpuDetection | Logic for processor platform types in the Makefile, which may lead to platform compatibility issues. +IncompatibleHeaderFile | Incompatible header files. +InlineAsm | Use of inline assembly may lead to AArch64 architecture compatibility issues. +Intrinsic | Use of intrinsics that have compatibility issues with the AArch64 architecture. +NoEquivalentInlineAsm | Use of inline assembly code that does not exist on the AArch64 architecture. +NoEquivalentIntrinsic | Use of intrinsic functions that do not exist on the AArch64 architecture. +OldCrt | Use of an older version of the C runtime library, which may cause compatibility issues or miss optimizations. +Pragma | This #pragma may be incompatible with the AArch64 architecture's compiler. +PreprocessorError | AArch64 architecture may enter the #error preprocessing logic. +SignedChar | Compatibility issues with signed char type data. + {{< /tab >}} + {{< tab header="Java">}} + Name | Description +------------ | -------------------------------------------------------------------------------------------- +JavaJar | Use of JAR package does not support AArch64 architecture. +JavaPom | Pom imports java artifact that does not support AArch64 architecture. +JavaSource | Java source file contains native call that may need modify/rebuild for AArch64 architecture. + {{< /tab >}} + {{< tab header="Python">}} +Name | Description +------------------|----------------------------------------------------------------------------------------- +PythonInlineAsm | Use of inline assembly in the AArch64 architecture may lead to compatibility issues. +PythonIntrinsic | Use of intrinsic functions that have compatibility issues with the AArch64 architecture. +PythonLinkLibrary | Use of libraries that are incompatible with the AArch64 architecture. +PythonPackage | Use of packages that are incompatible with the AArch64 architecture. + {{< /tab >}} + {{< tab header="Go">}} +Name | Description +------------------|----------------------------------------------------------------------------------------------------- +Asm | Potentially architecture-specific assembly code in the source files that requires manual inspection. +GolangInlineAsm | Use of inline assembly may lead to AArch64 architecture compatibility issues. +GolangIntrinsic | Use of intrinsic functions that have compatibility issues with the AArch64 architecture. +GolangLinkLibrary | Use of libraries that are incompatible with the AArch64 architecture. + {{< /tab >}} + {{< tab header="Rust">}} +Name | Description +------------------|------------------------------------------------------------------------------------- +RustInlineAsm | Use of inline assembly in the AArch64 architecture may lead to compatibility issues. +RustIntrinsic | Use of intrinsics that have compatibility issues with the AArch64 architecture. +RustLinkLibrary | Use of libraries that are incompatible with the AArch64 architecture. + {{< /tab >}} + {{< tab header="Dockerfile">}} +Name | Description +-------------------|----------------------------------------------------------------------------------------------------- +ConfigurationInfo | Configuration parameter used in ENV, ARG or LABEL refer an architecture that could not be supported. +Image | A base image is used that might not support AArch64 architecture. +Plugin | A package used in RUN, CMD or ENTRYPOINT does not support AArch64 architecture. + {{< /tab >}} +{{< /tabpane >}} + +The `issues` is a list of those detected issues with details for each of them. +- `checkpoint`: A pattern to identify potential incompatibility. +- `description`: The description of the detected issue. +- `filename`: The file in which issue is detected. +- `issue_type`: The type of issue, including detailed descriptions of the error. +- `lineno`: The line number of the problematic code. +- `snippet`: The block of the problematic code. + +For more information about issue type information perf review [migrate-ease github](https://github.com/migrate-ease/migrate-ease/blob/main/README.md). diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/_index.md b/content/learning-paths/servers-and-cloud-computing/migrate-ease/_index.md new file mode 100644 index 0000000000..2a57721ae6 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/migrate-ease/_index.md @@ -0,0 +1,60 @@ +--- +title: Accelerating Application Migration to Arm Servers Using Code Scanning Tools + +minutes_to_complete: 45 + +who_is_this_for: This is an automated tool introductory for software developers and DevOps engineers who are preparing to migrate existing applications to Arm-based server environments. It introduces a static analysis tool (migrate-ease) that simplifies the process by identifying portability challenges early. + + +learning_objectives: + - Set up an Arm development machine + - Analyze application source code for architecture-specific dependencies + - Understand common migration challenges and solutions + - Use migrate-ease to identify and prioritize portability issues + +prerequisites: + - An [Arm based instance](/learning-paths/servers-and-cloud-computing/csp/) from a cloud service provider. + +author: + - Odin Shen + - Jun He + +### Tags +skilllevels: Introductory +subjects: Libraries +armips: + - Neoverse +operatingsystems: + - Linux +tools_software_languages: + - Coding + - Neon + - SVE + - Go + - Runbook + +further_reading: + - resource: + title: AWS Graviton Getting Started + link: https://github.com/aws/aws-graviton-getting-started + type: documentation + - resource: + title: AWS Graviton Processors + link: https://dev.to/aws-builders/aws-graviton-processors-3nk3 + type: blog + - resource: + title: Arm Cloud Migration Program + link: https://www.arm.com/markets/computing-infrastructure/arm-cloud-migration + type: website + - resource: + title: Migrating applications to Arm servers + link: https://learn.arm.com/learning-paths/servers-and-cloud-computing/migration/ + type: website + + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/migrate-ease/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/migrate-ease/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/web_ui_index.jpg b/content/learning-paths/servers-and-cloud-computing/migrate-ease/web_ui_index.jpg new file mode 100644 index 0000000000..ba77e2d597 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/migrate-ease/web_ui_index.jpg differ diff --git a/content/learning-paths/servers-and-cloud-computing/migrate-ease/web_ui_result.jpg b/content/learning-paths/servers-and-cloud-computing/migrate-ease/web_ui_result.jpg new file mode 100644 index 0000000000..5ff252c01e Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/migrate-ease/web_ui_result.jpg differ