diff --git a/docs/.gitbook/assets/azure_shell_create_rg.png b/docs/.gitbook/assets/azure_shell_create_rg.png new file mode 100644 index 0000000000000..4c6c64407d2e5 Binary files /dev/null and b/docs/.gitbook/assets/azure_shell_create_rg.png differ diff --git a/docs/.gitbook/assets/azure_shell_create_vm.png b/docs/.gitbook/assets/azure_shell_create_vm.png new file mode 100644 index 0000000000000..ff6d6a4bb5669 Binary files /dev/null and b/docs/.gitbook/assets/azure_shell_create_vm.png differ diff --git a/docs/.gitbook/assets/azure_shell_download_ssh_key.png b/docs/.gitbook/assets/azure_shell_download_ssh_key.png new file mode 100644 index 0000000000000..17c138054107d Binary files /dev/null and b/docs/.gitbook/assets/azure_shell_download_ssh_key.png differ diff --git a/docs/.gitbook/assets/azure_shell_launch.png b/docs/.gitbook/assets/azure_shell_launch.png new file mode 100644 index 0000000000000..0723d5d92030d Binary files /dev/null and b/docs/.gitbook/assets/azure_shell_launch.png differ diff --git a/docs/.gitbook/assets/azure_shell_vm_overview.png b/docs/.gitbook/assets/azure_shell_vm_overview.png new file mode 100644 index 0000000000000..bdae4f34bf541 Binary files /dev/null and b/docs/.gitbook/assets/azure_shell_vm_overview.png differ diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 02a9c0914a8bb..4ba4f5b864aac 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -27,6 +27,7 @@ * [Local Deployment](deploying-airbyte/local-deployment.md) * [On AWS \(EC2\)](deploying-airbyte/on-aws-ec2.md) * [On GCP \(Compute Engine\)](deploying-airbyte/on-gcp-compute-engine.md) + * [On Azure\(VM)](deploying-airbyte/on-azure-vm-cloud-shell.md) * [On Kubernetes \(Alpha\)](deploying-airbyte/on-kubernetes.md) * [On AWS ECS \(Coming Soon\)](deploying-airbyte/on-aws-ecs.md) * [Connectors](integrations/README.md) diff --git a/docs/deploying-airbyte/on-azure-vm-cloud-shell.md b/docs/deploying-airbyte/on-azure-vm-cloud-shell.md new file mode 100644 index 0000000000000..0a16a9707bf0a --- /dev/null +++ b/docs/deploying-airbyte/on-azure-vm-cloud-shell.md @@ -0,0 +1,135 @@ +# On Azure Virtual Machine - Cloud Shell + +{% hint style="info" %} +The instructions have been tested on `Azure VM Linux (ubuntu 18.04)` +{% endhint %} + +## Launch Azure Cloud Shell + +Launch cloud shell by going to [https://shell.azure.com/bash](https://shell.azure.com/bash) + +![](../.gitbook/assets/azure_shell_launch.png) + +## Create a new virtual machine + +#### Create resource group + +```bash +# Inside Azure cloud shell +rgName=airbyte-demo +rgLocation=eastus +az group create --name $rgName --location $rgLocation +``` +![](../.gitbook/assets/azure_shell_create_rg.png) + +#### Create virtual machine + +```bash +# Inside Azure cloud shell +userName=byteuser +vmName=airbyte +dnsName=$(head -3 /dev/urandom | tr -dc a-z | cut -c -16) +publicIp=$(az vm create --resource-group $rgName \ + --name $vmName --image UbuntuLTS \ + --admin-username $userName \ + --public-ip-address-dns-name $dnsName \ + --generate-ssh-keys --query 'publicIpAddress' -o json) +echo $publicIp +``` + +This step will create a virtual machine and add a user account named `byteuser`. The ``--generate-ssh-keys`` option will generate a new ssh key and put it to the default key location (~/.ssh) + +**Note: Copy the ``publicIpAddress`` output, you will need this address later to connect from your workstation.** + +![](../.gitbook/assets/azure_shell_create_vm.png) + +#### Download SSH key + +```bash +# Inside Azure cloud shell +download ~/.ssh/id_rsa +``` +Above command will generate download link and give you pop-up on right bottom side, click on `Click here to download your file.` to download private key. +Note: Save this file, you will need it to connect to your VM in [Connect to Airbyte](#connect-to-airbyte) step. + +![](../.gitbook/assets/azure_shell_download_ssh_key.png) + +#### Connect to virtual machine + +- Connect to virtual machine + +```bash +# Inside Azure cloud shell +ssh $userName@$publicIp +``` + +## Install environment + +- Install `docker` + +```bash +# Inside Azure cloud shell +sudo apt-get update -y +sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg +echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null +sudo apt-get update +sudo apt-get install docker-ce docker-ce-cli -y +sudo usermod -a -G docker $USER +``` + +- Install `docker-compose` + +```bash +# Inside Azure cloud shell +sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose +sudo chmod +x /usr/local/bin/docker-compose +docker-compose --version +``` +- Close the ssh connection to ensure the group modification is taken into account + +```bash +# Inside Azure cloud shell +logout +``` +- Reconnect to virtual machine + +```bash +# Inside Azure cloud shell +ssh $userName@$publicIp +``` + +## Install and Start Airbyte + +```bash +# Inside Azure cloud shell +mkdir airbyte && cd airbyte +wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml} +sudo docker-compose up -d +``` + +## Connect to Airbyte + +{% hint style="danger" %} +For security reasons, we strongly recommend to not expose Airbyte on Internet available ports. Future versions will add support for SSL & Authentication. +{% endhint %} + +{% hint style="info" %} +This part assumes that you have access to a terminal on your workstation +{% endhint %} + +- Create ssh tunnels for port 8000 (the static web server) and port 8001 (the api server) +```bash +# Inside your workstation terminal +# 1. Replace $SSH_KEY with private key path downloaded from earlier steps +# 2. Replace $INSTANCE_IP with publicIpAddress noted from earlier steps +ssh -i $SSH_KEY -L 8000:localhost:8000 -L 8001:localhost:8001 -N -f byteuser@$INSTANCE_IP +``` +- Just visit http://localhost:8000 in your browser and start moving some data! + + + + +## Troubleshooting + +If you encounter any issues, just connect to our [Slack](https://slack.airbyte.io). Our community will help! We also have a [FAQ](../faq/technical-support.md) section in our docs for common problems.