Skip to content

fix orion ci from vms #1978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
benjamin-747 merged 1 commit into from
Mar 2, 2026
+143 −297
Merged

fix orion ci from vms #1978

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 47 additions & 44 deletions .github/workflows/orion-client-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ jobs:

# Deploy to legacy orion_vm (root user, backward compatibility)
deploy-legacy:
if: false # Temporarily disabled
if: ${{ github.repository == 'web3infra-foundation/mega' }}
runs-on: ubuntu-latest
needs: build
timeout-minutes: 10
Expand All @@ -81,36 +81,29 @@ jobs:
set -e
echo "==> Listing downloaded artifacts structure:"
find artifacts -type f -ls

echo "==> Preparing deployment files..."
# GitHub Actions preserves directory structure in artifacts
# File structure: artifacts/target/release/orion
# artifacts/orion/runner-config/.env.prod
# artifacts/orion/runner-config/scorpio.toml
# artifacts/orion/runner-config/run.sh

# Move files to artifacts root for deployment
# IMPORTANT: Move config files first (while orion/ directory still exists)
mv artifacts/orion/runner-config/.env.prod artifacts/.env
mv artifacts/orion/runner-config/scorpio.toml artifacts/
mv artifacts/orion/runner-config/run.sh artifacts/
mv artifacts/orion/runner-config/cleanup.sh artifacts/

# Move systemd service file
mv artifacts/orion/systemd/orion-runner.service artifacts/

# Clean up empty orion directory before moving binary
rm -rf artifacts/orion

# Now move the binary (orion/ directory has been removed)
mv artifacts/target/release/orion artifacts/orion
rm -rf artifacts/target

# Set executable permissions
chmod +x artifacts/orion
chmod +x artifacts/run.sh
chmod +x artifacts/cleanup.sh

echo "==> Final artifact structure:"
ls -la artifacts/

Expand All @@ -128,42 +121,52 @@ jobs:
# Stop service to allow binary replacement
systemctl stop orion-runner.service || echo "Service not running"
sleep 2

# Force cleanup if service is stuck
pkill -9 orion || true


# Force unmount FUSE if still mounted
fusermount -uz /workspace/mount 2>/dev/null || true
umount -lf /workspace/mount 2>/dev/null || true

- name: Upload binaries and configs via rsync to orion_vm
uses: burnett01/rsync-deployments@8.0.4
with:
switches: -avz --progress
path: artifacts/
remote_path: /root/orion-runner/
remote_path: /home/orion/orion-runner/
remote_host: ${{ secrets.ORION_DEPLOY_HOST }}
remote_user: root
Comment on lines 135 to 139

@chatgpt-codex-connector chatgpt-codex-connector Bot Mar 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent root rsync from breaking orion-owned runtime files

In deploy-legacy, rsync runs as root with -a (--archive), and rsync --help states archive mode includes -o (--owner, super-user only), so uploaded files in /home/orion/orion-runner/ can end up owned by a non-orion UID; since the service runs as orion and run.sh does chmod +x ./orion, startup can fail with permission errors when ownership does not match.

Useful? React with 👍 / 👎.

remote_key: ${{ secrets.ORION_DEPLOY_SSH_KEY }}

strict_hostkeys_checking: no

- name: Start service on orion_vm
uses: appleboy/ssh-action@v1.0.3
with:
host: ${{ secrets.ORION_DEPLOY_HOST }}
username: root
key: ${{ secrets.ORION_DEPLOY_SSH_KEY }}
command_timeout: 30s
script: |
# Create runtime directories
mkdir -p /data/scorpio/{store,antares/{upper,cl,mnt}}
mkdir -p /workspace/mount

# Set permissions
chmod +x /root/orion-runner/orion
chmod +x /root/orion-runner/run.sh
chmod +x /root/orion-runner/cleanup.sh

# Start service
chmod +x /home/orion/orion-runner/orion
chmod +x /home/orion/orion-runner/run.sh
chmod +x /home/orion/orion-runner/cleanup.sh

# Update systemd service file if changed
if [ -f /home/orion/orion-runner/orion-runner.service ]; then
cp /home/orion/orion-runner/orion-runner.service /etc/systemd/system/
echo "✓ Updated systemd service file"
fi

# Reload and start service asynchronously to avoid SSH timeout
systemctl daemon-reload
systemctl start orion-runner.service

echo "✓ Service start command completed"
systemctl status orion-runner.service --no-pager
nohup systemctl start orion-runner.service > /tmp/orion-start.log 2>&1 &

@chatgpt-codex-connector chatgpt-codex-connector Bot Mar 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Wait for systemd start result before finishing deployment

Both deploy jobs now start orion-runner.service with nohup ... &, which returns success as soon as the command is queued, not when the service has actually started; this means the workflow can report a green deploy even if systemctl start immediately fails (e.g., bad unit file, permission issue, missing config), so production failures become silent.

Useful? React with 👍 / 👎.

echo "✓ Service start command issued (async)"

# Deploy to new GCP VM (orion user, best practice)
deploy-gcp:
Expand All @@ -186,28 +189,27 @@ jobs:
find artifacts -type f -ls

echo "==> Preparing deployment files..."
# Move files to artifacts root for deployment
# IMPORTANT: Move config files first (while orion/ directory still exists)
mv artifacts/orion/runner-config/.env.prod artifacts/.env
mv artifacts/orion/runner-config/scorpio.toml artifacts/
mv artifacts/orion/runner-config/run.sh artifacts/
mv artifacts/orion/runner-config/cleanup.sh artifacts/

# Move systemd service file
mv artifacts/orion/systemd/orion-runner.service artifacts/

# Clean up empty directories before moving binary
rm -rf artifacts/orion

# Now move the binary (orion/ directory has been removed)
mv artifacts/target/release/orion artifacts/orion
rm -rf artifacts/target

# Set executable permissions
chmod +x artifacts/orion
chmod +x artifacts/run.sh
chmod +x artifacts/cleanup.sh

echo "==> Final artifact structure:"
ls -la artifacts/

Expand All @@ -221,14 +223,14 @@ jobs:
# Stop service to allow binary replacement
sudo systemctl stop orion-runner.service || echo "Service not running"
sleep 2

# Force cleanup if service is stuck
sudo pkill -9 orion || true

# Force unmount FUSE if still mounted
fusermount -uz /workspace/mount 2>/dev/null || true
sudo umount -lf /workspace/mount 2>/dev/null || true

- name: Upload binaries and configs via rsync to gcp_vm
uses: burnett01/rsync-deployments@8.0.4
with:
Expand All @@ -238,32 +240,33 @@ jobs:
remote_host: ${{ secrets.ORION_GCP_VM_HOST }}
remote_user: orion
remote_key: ${{ secrets.ORION_GCP_VM_SSH_KEY }}
strict_hostkeys_checking: no

- name: Start service on gcp_vm
uses: appleboy/ssh-action@v1.0.3
with:
host: ${{ secrets.ORION_GCP_VM_HOST }}
username: orion
key: ${{ secrets.ORION_GCP_VM_SSH_KEY }}
command_timeout: 30s
script: |
# Create runtime directories
sudo mkdir -p /data/scorpio/{store,antares/{upper,cl,mnt}}
sudo mkdir -p /workspace/mount
sudo chown -R orion:orion /data/scorpio /workspace/mount

# Set permissions
chmod +x /home/orion/orion-runner/orion
chmod +x /home/orion/orion-runner/run.sh
chmod +x /home/orion/orion-runner/cleanup.sh

# Update systemd service file if changed
if [ -f /home/orion/orion-runner/orion-runner.service ]; then
sudo cp /home/orion/orion-runner/orion-runner.service /etc/systemd/system/
echo "✓ Updated systemd service file"
fi
# Reload systemd and start service

# Reload and start service asynchronously to avoid SSH timeout
sudo systemctl daemon-reload
sudo systemctl start orion-runner.service

echo "✓ Service start command completed"
nohup sudo systemctl start orion-runner.service > /tmp/orion-start.log 2>&1 &
echo "✓ Service start command issued (async)"
Loading
Loading