Skip to content

enh: refactor offline scripts and support github-actions#153

Merged
feici02 merged 12 commits intomainfrom
enh/refactor-offline-scripts
Mar 20, 2026
Merged

enh: refactor offline scripts and support github-actions#153
feici02 merged 12 commits intomainfrom
enh/refactor-offline-scripts

Conversation

@jiajingbin
Copy link
Contributor

Description

Issue(s)

  • Close/close/Fix/fix/Resolve/resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

  • Is the user manual updated?
  • Are the test cases passed and automated?
  • Is there no significant decrease in test coverage?

Copilot AI review requested due to automatic review settings March 10, 2026 02:47
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive overhaul of the offline package building infrastructure, moving towards a more automated, robust, and flexible system. By refactoring existing scripts, integrating with GitHub Actions, and enhancing dependency management with uv, the changes streamline the creation of deployment packages for TDengine, TDgpt, and IDMP across diverse operating systems and architectures. The update also includes critical improvements in system configuration, such as automated firewall and SELinux handling, to ensure smoother installations in various environments.

Highlights

  • Refactored Offline Package Build System: The offline package build process has been significantly refactored into a new orchestrator script (build_offline_pkg.sh) and a dedicated GitHub Action, enabling more flexible and automated builds for various environments and deployment types.
  • Enhanced Python Virtual Environment Management: Introduced uv for faster and more robust Python virtual environment creation and package installation, including caching mechanisms for downloaded packages and improved handling of private repositories via GitHub tokens.
  • Improved OS and Architecture Compatibility Checks: Added comprehensive architecture detection and compatibility verification for Docker, Docker Compose, Java, and IDMP packages during installation, preventing mismatches and ensuring correct deployments.
  • Automated Firewall and SELinux Management: The installation scripts now automatically manage system firewalls (firewalld, ufw) and disable SELinux on RPM-based systems to prevent common deployment issues, along with new scripts to set offline/online network configurations.
  • Flexible TDgpt and IDMP Build Options: New parameters allow for specifying TDengine and IDMP versions for requirements download, building all TDgpt model virtual environments, and configuring custom pip and PyTorch mirror URLs.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • .github/actions/build-offline-pkg/action.yml
    • Added a new GitHub Action to orchestrate the building of offline deployment packages, exposing a wide range of configurable inputs.
  • .github/scripts/offline_env/build_offline_pkg.sh
    • Added a new orchestrator script to manage native and Docker-based builds of offline packages.
    • Implemented a container pooling mechanism for Docker builds to reuse existing containers.
    • Introduced a build lock to prevent concurrent builds for the same target output directory.
    • Added logic to inject Nexus repository configurations for both RPM and DEB-based systems.
    • Included functionality to apply default system packages based on the specified deployment type (TSDB, IDMP, TDgpt).
  • .github/scripts/offline_env/build_shells/delivery_tools.sh
    • Added new example commands demonstrating how to use build_offline_pkg.sh for various scenarios, including openEuler with IDMP and TDgpt, and general TSDB/TDgpt/IDMP builds with Nexus mirrors.
  • .github/scripts/offline_env/install.sh
    • Added IDMP_VENV_DIR for configurable IDMP virtual environment paths.
    • Introduced get_system_arch and verify_arch_compatibility functions for architecture validation.
    • Updated install_venv to correctly locate and handle IDMP virtual environments.
    • Enhanced install_binary_tools with chmod +x and backup functionality for existing tools.
    • Integrated architecture compatibility checks into install_docker, install_docker_compose, and install_java functions.
    • Modified Java installation to track java_installed status instead of printing a source command reminder.
    • Improved install_idmp with architecture verification for IDMP packages.
    • Refined install_system_packages to handle empty glob results and improve OS detection for SUSE/openEuler.
    • Added disable_selinux function to automatically disable SELinux on RPM-based systems.
    • Updated the main function to call disable_selinux and provide relevant post-installation reminders.
  • .github/scripts/offline_env/prepare_offline_pkg.sh
    • Added new parameters: TDGPT_ALL, TDENGINE_TSDB_VER, IDMP_VER, GH_TOKEN, PIP_INDEX_URL, PYTORCH_WHL_URL, BPFTRACE_VERSION, TDGPT_BASE_DIR, IDMP_VENV_DIR.
    • Implemented install_uv_cached for efficient uv installation and caching across container runs.
    • Updated show_usage to reflect all new options and examples.
    • Introduced a --deploy-type argument to simplify specifying deployment presets.
    • Added validation to ensure PYTHON_VERSION is provided when Python packages are specified.
    • Centralized logic for pip_index_args and pytorch_index_args to manage Python package mirrors.
    • Improved OS detection for Kylin to extract sub-version and codename more reliably.
    • Enhanced config_yum with robust error handling for setup_env.sh operations.
    • Added architecture checks to download_bpftrace_binary as prebuilt binaries are arch-specific.
    • Improved download_java with more robust caching and re-download logic for corrupt files, including SHA256 verification.
    • Added a jq dependency check for download_docker_compose when fetching the latest version.
    • Included error handling for apt-rdepends and apt-get download in install_system_packages.
    • Refactored build_tdgpt_venvs to download requirements from GitHub tags or use local files, and to build main/extra venvs based on TDGPT_ALL.
    • Added build_idmp_venvs to manage IDMP Python virtual environment creation, including downloading requirements from private GitHub repositories.
    • Updated install_python_packages to integrate uv caching and skip execution if TDgpt/IDMP venvs are handled by dedicated functions.
    • Modified summary to display TDgpt venv build status and post-build notes.
  • .github/scripts/offline_env/set-offline.sh
    • Added logic to stop and disable firewalld and ufw to ensure network access for offline environments.
    • Ensured nf_conntrack kernel modules are loaded for proper connection tracking with iptables.
    • Updated LAN traffic rules to use a broader 192.168.0.0/16 range.
    • Improved iptables rule persistence by distinguishing save paths for DEB and RPM-based systems.
  • .github/scripts/offline_env/set-online.sh
    • Added a new script to reset all iptables rules, clear chains, set default policies to ACCEPT, and persist these changes to restore full network access.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/build-offline-pkg.yml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant refactoring of the offline package build scripts and a new GitHub Action, aiming to improve the robustness, maintainability, and performance of the build system. However, critical security vulnerabilities related to command injection have been identified. Untrusted inputs from GitHub Action inputs and script arguments are directly used in shell commands without proper sanitization or quoting, specifically in action.yml and bash scripts using eval, bash -c, and unquoted variables. Addressing these command injection issues is paramount. Additionally, a few minor issues were noted in one of the example shell scripts.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the offline packaging scripts to support running builds in a standardized way (native or Docker) and adds GitHub Actions integration (workflow + composite action) to build offline deployment artifacts for different target OS/arch combinations.

Changes:

  • Added a GitHub Actions workflow and a composite action to drive offline package builds via workflow_dispatch / workflow_call.
  • Introduced a new build_offline_pkg.sh orchestrator to run prepare_offline_pkg.sh natively or inside a container (with container pooling, caching, and optional Nexus repo injection).
  • Refactored offline env scripts to support TDgpt/IDMP variants, caching uv, architecture checks, and improved install behaviors.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
.github/workflows/build-offline-pkg.yml New workflow to build and upload offline package artifacts via GitHub Actions.
.github/actions/build-offline-pkg/action.yml New composite action wrapping build_offline_pkg.sh with inputs for CI usage.
.github/scripts/offline_env/build_offline_pkg.sh New orchestrator script for native/Docker builds, container pooling, locking, defaults, and Nexus injection.
.github/scripts/offline_env/prepare_offline_pkg.sh Extended build logic for TDgpt/IDMP, uv caching, mirrors, and more robust downloads.
.github/scripts/offline_env/install.sh Adds arch verification for packaged components and SELinux handling; improves binary tools installation.
.github/scripts/offline_env/set-offline.sh Improved offline firewall handling (firewalld/ufw interaction, conntrack, persistence).
.github/scripts/offline_env/set-online.sh New helper to remove iptables restrictions and persist “online” rules.
.github/scripts/offline_env/build_shells/delivery_tools.sh Adds/updates example commands for new orchestrator flows (TSDB/TDgpt/IDMP).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings March 10, 2026 04:14
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

.github/scripts/offline_env/install.sh:632

  • disable_selinux is executed unconditionally on any host that has /etc/selinux/config, which permanently disables SELinux for the entire system. This is a high-impact operational/security change for an install script and may be unacceptable in hardened environments. Consider gating this behind an explicit flag, limiting it to known-problem scenarios (e.g., only when installing TDgpt/IDMP), or switching to a targeted SELinux policy/module approach instead of globally disabling SELinux.
# Disable SELinux permanently on RPM-based distros (CentOS/RHEL/openEuler/Kylin).
# Required for TDgpt/IDMP services: SELinux blocks uwsgi/Python dynamic library loading
# (status=203/EXEC from systemd) even when the binary path is correct.
# Also disables it for the current session via setenforce so no reboot is needed.
function disable_selinux() {
    local selinux_config="/etc/selinux/config"

    # Only applies to RPM-based systems that have SELinux
    if [ ! -f "$selinux_config" ]; then
        return 0
    fi

    # Check if SELinux is enabled
    local current_mode
    current_mode=$(getenforce 2>/dev/null || echo "Disabled")
    if [ "$current_mode" = "Disabled" ]; then
        return 0
    fi

    yellow_echo "Disabling SELinux (current mode: $current_mode)..."

    # Disable for current session immediately (no reboot needed)
    setenforce 0 2>/dev/null || true

    # Persist: set SELINUX=disabled in /etc/selinux/config
    if grep -q '^SELINUX=' "$selinux_config"; then
        sed -i 's/^SELINUX=.*/SELINUX=disabled/' "$selinux_config"
    else
        echo "SELINUX=disabled" >> "$selinux_config"
    fi

    green_echo "SELinux disabled (current session + permanent after reboot)"
}

function main() {
    if [ -f /etc/os-release ]; then
        disable_selinux
        install_venv
        install_binary_tools
        install_docker
        install_docker_compose

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings March 11, 2026 03:03
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Member

@feici02 feici02 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@feici02 feici02 merged commit 847b03d into main Mar 20, 2026
@feici02 feici02 deleted the enh/refactor-offline-scripts branch March 20, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants