Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Self-hosted EC2 Runners #3019

Merged
merged 3 commits into from
Apr 25, 2021
Merged

Self-hosted EC2 Runners #3019

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1128777
Better build stage names.
davinchia Apr 22, 2021
6e9774a
Add self-hosted runner for other tests. Clean up naming and steps.
davinchia Apr 22, 2021
e978b14
Run frontend tests.
davinchia Apr 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 54 additions & 2 deletions .github/workflows/gradle.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,36 @@ jobs:
run: ./tools/bin/ci_integration_workflow_launcher.sh
env:
GITHUB_TOKEN: ${{ secrets.SLASH_COMMAND_PAT }}
build:
# In case of self-hosted EC2 errors, remove the `start-build-runner` block.
start-build-runner:
name: Start Build EC2 runner
runs-on: ubuntu-latest
outputs:
label: ${{ steps.start-ec2-runner.outputs.label }}
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Start EC2 runner
id: start-ec2-runner
uses: machulav/ec2-github-runner@v2.1.0
with:
mode: start
github-token: ${{ secrets.SELF_RUNNER_GITHUB_ACCESS_TOKEN }}
ec2-image-id: ami-0b39c2b1b65f75ca8
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are all resources living in the aws dev account.

ec2-instance-type: c5.2xlarge
subnet-id: subnet-01d35f948346c05bd
security-group-id: sg-04399aaf4d51a35f9
build:
# In case of self-hosted EC2 errors, remove the next two lines and uncomment the currently commented out `runs-on` line.
needs: start-build-runner # required to start the main job when the runner is ready
runs-on: ${{ needs.start-build-runner.outputs.label }} # run the job on the newly created runner
# runs-on: ubuntu-latest
name: Build Airbyte
steps:
- name: Checkout Airbyte
uses: actions/checkout@v2
Expand Down Expand Up @@ -128,8 +156,31 @@ jobs:
SLACK_USERNAME: Buildbot
SLACK_TITLE: "Build Success"
SLACK_FOOTER: ""
test_docker:
# In case of self-hosted EC2 errors, remove the `stop-build-runner` block.
stop-build-runner:
name: Stop Build EC2 runner
needs:
- start-build-runner # required to get output from the start-runner job
- build # required to wait when the main job is done
runs-on: ubuntu-latest
if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.SELF_RUNNER_AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.SELF_RUNNER_AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Stop EC2 runner
uses: machulav/ec2-github-runner@v2.1.0
with:
mode: stop
github-token: ${{ secrets.SELF_RUNNER_GITHUB_ACCESS_TOKEN }}
label: ${{ needs.start-build-runner.outputs.label }}
ec2-instance-id: ${{ needs.start-build-runner.outputs.ec2-instance-id }}
acceptance_test:
runs-on: ubuntu-latest
name: Run Acceptance Tests
steps:
- name: Checkout Airbyte
uses: actions/checkout@v2
Expand Down Expand Up @@ -158,6 +209,7 @@ jobs:

launch_e2e_frontend_tests:
runs-on: ubuntu-latest
name: Run Front-end Tests
steps:
- name: Checkout Airbyte
uses: actions/checkout@v2
Expand Down