From 17e110f15dc260b7ce4ada6f1334b5c4841e578f Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Thu, 21 Nov 2024 12:17:25 -0500 Subject: [PATCH 01/13] Import from external editor --- .../index.md | 354 ++++++++++++++++++ 1 file changed, 354 insertions(+) create mode 100644 docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md new file mode 100644 index 00000000000..e2b52a16bd5 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -0,0 +1,354 @@ +--- +slug: migrate-from-azure-disk-storage-to-linode-block-storage +title: "Migrate From Azure Disk Storage to Linode Block Storage" +description: "Two to three sentences describing your guide." +og_description: "Optional two to three sentences describing your guide when shared on social media. If omitted, the `description` parameter is used within social links." +authors: ["Linode"] +contributors: ["Linode"] +published: 2024-11-18 +keywords: ['list','of','keywords','and key phrases'] +license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' +external_resources: +- '[Link Title 1](http://www.example.com)' +- '[Link Title 2](http://www.example.net)' +--- + +Introduction + +This guide details the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. While Azure Disk Storage serves a broader range of use cases than Linode Block Storage, this guide focuses on migrating Azure data disks, excluding OS and temporary disks. + +The following topics are covered in this guide: + +Linode Block Storage vs. Azure Disk Storage + +Migration Considerations + +Data Migration Workflow Diagram + +Data Migration Instructions + +Migration Costs + +Also included are instructions for creating two sample data sets, one at 10GB and another at 100GB, to serve as examples for migrating data from Azure Disk Storage to Linode Block Storage. + +## Linode Block Storage vs. Azure Disk Storage + +When you first create an Azure Virtual Machine (VM), Azure Disk Storage creates two managed disks by default: an OS disk and data disk. Like Linode Block Storage volumes, Azure managed disks are also block-level storage volumes used with VMs. Unlike Azure Disk Storage, which fulfills multiple roles, Linode Block Storage is generally used for persistent data, not OS/boot disks or temporary data. These other roles are fulfilled by a Linode compute instance’s bundled disk. Linode’s bundled disk storage is also more suitable for certain applications like high-traffic databases. + +## Migration Considerations +The following are important time, cost, security, and data compression considerations you should keep in mind when migrating your Azure Disk Storage drives to Linode Block Storage. + +### Migration Time Estimates +This guide covers the creation and migration of two test files: one 10 GB file and another 100 GB file. Depending on your specific server and network configuration, the 10 GB file may take over 2 minutes to transfer from Azure to Linode at around 58 MB/s, while the 100 GB can take over 2 ½ hours at 11.95MB/s. + +As discussed later in this guide, you can exclude the -c rsync flag to skip checksum comparisons for larger file sizes. Doing so significantly speeds up the end-to-end data migration time. + +### Migration Egress Costs +The migrations demonstrated in this guide incurred the following costs: + +The 10 GB sample dataset migration incurred $2.22 in storage costs, $0.05 in virtual network costs, and < $0.01 in bandwidth costs. + +After the 100GB sample dataset migration, $3.88 in storage costs, $0.09 in virtual network costs, and < $0.01 in bandwidth costs were incurred. + +### Security and Firewalls + +To securely migrate your files, you should execute rsync over SSH using a public key (.pem file) generated on your Azure VM. Your Azure and Linode firewall settings should also be configured to accept inbound or outbound SSH traffic. + +## Data Compression + +Compressing files to be migrated on the source is generally good practice to shorten data transfer times. The rsync command used in this guide includes the -z flag to enable compression during data transfer, speeding up data transfers and saving network bandwidth. + +## Block Storage Migration Workflow Diagram + +The following workflow diagram outlines the activities covered in this guide. Steps 1 and 3 may not apply to your scenario if you’ve already prepared your Azure Data Disk and Linode Block Storage volumes. + + + +Azure VM disk storage is prepared. + +Test dataset is created on Azure disk storage. + +Linux Block Storage volume created and attached to Linode instance. + +Azure VM public key (.pem file) copied to Linode instance. + +Rsync command issues from Linode instance. + +Test dataset copied to Linode instance. + +## Block Storage Migration Instructions + +### Prerequisites and Assumptions + +This guide assumes that you have both a Linux-based Azure VM instance and destination Linode instance up-and-running. Upon creating your Azure VM, an OS disk and data disk should automatically be attached to your instance. + + + +Ensure that you have already generated a public key (.pem file) for your Azure VM instance, as it will be required to securely run rsync over SSH between your Linode in the Akamai Connected Cloud and your VM in the Microsoft Azure environment. + +This guide assumes the following source filesystem path for your Azure volume: + +/dev/sdc1 mounted to /datadrive + +The destination of your Linode migration is assumed to be: + +/mnt/linode-bs + +Preparing Your Azure Instance with Disk Attached +In the Azure portal, go to your VM and expand the Settings link on the left-hand side. Click on Disks—you should see your OS disk and your Data disk at logical unit number (LUN) 0: + + + +If you SSH into your Azure VM, you can run the lsblk command, pipe the output to grep,and filter on “sd” to verify information about all your available block devices: + + lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd" + +Your results should resemble the following output: + + + +As you can see, sda is the 30 GB OS disk and sdc is the 1 TB data disk at LUN 0. + +Preparing A New Empty Disk +If you’ve just created your Azure VM, the data disk is empty and needs to be prepared. Use the parted and mkfs utility to partition and format the data disk using the XFS filesystem, followed by partprobe to make your OS kernel aware of the new partition and filesystem. + +sudo parted /dev/sdc --script mklabel gpt mkpart xfspart xfs 0% 100% + +sudo mkfs.xfs /dev/sdc1 + +sudo partprobe /dev/sdc1 + +Mounting The Disk +Next, use mkdir to create a directory called /datadrive to mount your filesystem to: + +sudo mkdir /datadrive + +Use mount to mount the /dev/sdc1 partition to your new /datadrive directory: + +sudo mount /dev/sdc1 /datadrive + +To ensure that the drive is unmounted automatically after a reboot, you’ll need to add it to your /etc/fstab file. fstab is a configuration table designed to facilitate mounting and unmounting Linux file systems to a machine. + +First, use the blkid utility to get the universally unique identifier (UUID) of your drive—you’ll need it to add your drive to /etc/fstab. + +sudo blkid + +Your results should resemble the following output: + + + +Take note of the UUID for /dev/sdc1. Next, open the /etc/fstab file in a text editor. Add the following line to the end of the file using the UUID value for the /dev/sdc1 and the mountpoint of /datadrive: + +UUID=4f6fb6f0-38a3-4f9e-a5de-5a3b0ca9bedd /datadrive xfs defaults,nofail 1 2 + +Verifying The Disk +You can now use lsblk again to verify the disk and the mountpoint: + +lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd" + +Your results should resemble the following output: + + + +As you can see, sdc1 is now mounted at /datadrive. + +Creating Sample Data on Your Azure Data Disk +You can now create the test file to migrate from your Azure data disk to your Linode instance. + +Start by creating a 10 GB file using the dd command: + +sudo dd if=/dev/zero of=/datadrive/dummyfile bs=1M count=10240 + +Running this command will create a 10 GB test file filled with zeros at /datadrive/dummyfile. + +bs=1M sets the block size to 1 MB and count=10240 specifies a total number of 10240 blocks to write. With a block size of 1 MB, this creates a file of 10 GB (10240 MB). + +Once the dd command finishes, your results should resemble the following output: + + + +You can verify that your test file was successfully created by navigating to your datadrive directory and typing ls -al: + + + +Preparing you Linode Instance with Attached Block Storage +At this point, your Azure VM and its attached data disk are ready to be migrated. The next step involves preparing your Linode instance and its attached block storage volume to receive the test file. + +Creating Your Linode Block Storage Volume +From your Linode Cloud Manager dashboard, click on Volumes on the left-hand side and the Create Volume button on the top right-hand side. + +Specify a Label, Region, Linode, and Config. Enter 200 GB for the volume’s Size. + + + +Once the volume is created, you’ll see it show up on your Volumes page. Click on the Show Config button. + + + +The Volume Configuration pane will appear on the right-hand side. Copy the values from the pane and paste them into your notepad/notes—you’ll need them later when configuring your new volume. + + + +Configuring Your New Linode Block Storage Volume +The steps to configure your new Linode Block Storage volume are similar to the steps you followed previously in configuring your Azure data disk: you will use mkfs to to create a filesystem on your new volume, create a directory to mount the filesystem to using mkdir, then using mount to mount the new volume. You will then edit your /etc/fstab file to ensure that the volume mounts every time your Linode instance reboots. + +Log in to your Linode instance. You can either SSH directly, or use the LISH console by expanding the menu on the right-hand side and clicking the Launch LISH Console menu item: + + + +After logging in to your Linode instance, run the commands you previously copied to your notepad (mkfs, mkdir, mount): + +mkfs.ext4 "/dev/disk/by-id/scsi-0Linode_Volume_linode-bs” + +mkdir "/mnt/linode-bs" + +mount "/dev/disk/by-id/scsi-0Linode_Volume_linode-bs" "/mnt/linode-bs” + +Your results should resemble the following output: + + + +Open up your /etc/fstab file in a text editor and add the following line to make sure the volume auto-mounts every time your Linode reboots: + +/dev/disk/by-id/scsi-0Linode_Volume_linode-bs /mnt/linode-bs ext4 defaults,noatime,nofail 0 2 + +If you’re using the VI text editor, your changes should look like this: + + + +Save your updated /etc/fstab file. You are now ready to start migrating your Linode Block Storage from Azure Disk Storage. + +### Use Rsync to Migrate Block Storage Data + +You’ll be using rsync to connect to your Azure VM and migrate your block storage data to your Linode Block Storage volume. + +Check Linode and Azure Firewall and Connection Settings +First, make sure the firewall settings on both sides are configured correctly to allow the migration traffic to pass through. + +In Linode, click on Firewalls and make sure that any inbound or outbound rules are set to Accept inbound SSH traffic. + +For example, this is what it would look like in Linode: + + + +Using Rsync To Migrate the 10 GB Test File +NOTE: Be sure to first copy your Azure VM’s public key (.pem file) to your Linode instance. + +Run the rsync command to start migrating the contents of your Azure VM’s /datadrive directory to the /mnt/linode-bs directory on your Linode instance : + +rsync -chavzP --stats -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs + +Here's a detailed explanation of the key flags and parameters used in the rsync command: + +-c: This option tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but -c forces it to compute checksums, which is slower but can be more accurate if you want to be sure that files match exactly. + +-h: Human-readable output, which makes file sizes (like transfer statistics) easier to read by using units like KB and MB, rather than raw byte counts. + +-a: Archive mode. This option enables several options at once (-rlptgoD), which tells rsync to: + +Recursively copy directories (-r). + +Preserve symbolic links (-l). + +Retain file permissions (-p). + +Keep timestamps (-t). + +Preserve group ownership (-g). + +Maintain file ownership (-o). + +Retain device files and special files (-D). + +The result of the -a flag is a complete, near-exact copy of the source directory. + +-v: Verbose mode. This makes rsync output more detailed information about what it is doing, which is helpful for monitoring the progress of a large transfer or troubleshooting. + +-z: Compression. This enables compression during data transfer, which can save bandwidth and speed up the transfer if the network connection is relatively slow. + +-P: This option is a combination of two flags: + +--progress, which displays progress information for each file transfer. + +--partial, which keeps partially transferred files if the transfer is interrupted, allowing it to resume more easily next time. + +--stats: This option provides detailed statistics at the end of the transfer, such as total bytes transferred, transfer speed, and file counts. + +-e "ssh -i /path/to/your_azure_vm_key.pem": This option specifies a remote shell (ssh) with an identity key file for authentication: + +-e lets you specify a remote shell to use (in this case, SSH). + +"ssh -i /path/to/your_azure_vm_key.pem" tells rsync to connect to the remote server using SSH, and the -i option specifies an SSH key (/path/to/your_azure_vm_key.pem) for authentication. + +azureuser@13.93.147.88:/datadrive/: This specifies the source directory you’re syncing from. Here: + +azureuser is the username on the remote server (replace with your username). + +13.93.147.88 is the IP address of the remote server (replace with your IP address . + +/datadrive/ is the path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /datadrive, rather than creating a /datadrive directory in the target. + +/mnt/linode-bs: This is the destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /datadrive contents here. + +Once you enter the rsync command, the syncing process will start and display your progress in real-time: + + + +Transferring the 10 GB test file should take a little over 2 minutes, at around 58 MB/s. Once rsync completes, your results should look like the following output: + + + +Navigate to the /mnt/linode-bs directory and issue an ls -al command to verify that the file has transferred successfully: + + + +Using Rsync To Migrate a 100 GB Test File +Next, you’ll try using rsync on a larger migration: a 100 GB test file. Start by logging in to your Azure VM via SSH and deleting the existing test file: + +rm /datadrive/dummyfile + +You should also remove the test file in the destination directory of your Linode instance. Use weblish to log back into your Linode instance and remove the existing test file in your destination directory: + +rm /mnt/linode-bs/dummyfile + +Back in your Azure VM, run the dd command again—this time, specify the creation of a 100 GB file: + +sudo dd if=/dev/zero of=/datadrive/dummyfile bs=10MB count=10000 + +bs=10M sets the block size to 10 MB and count=10000 specifies a total number of 10000 blocks to write. With a block size of 10 MB, this creates a file of 100 GB. + +NOTE: you should use 10 MB blocks to avoid memory exhaustion issues. Also, the creation process will take longer than before (around 5 minutes) due to the significantly larger test file. + +Once your test file has been created, log back into your Linode instance and run the rsync command again. This time, remove the -c option to speed up the process for the large test file (the -c flag forces rsync to calculate a checksum for all files). + +rsync -havzP --stats -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs + +rsync will take a longer time to receive the incremental file list. Once it has completed the transfer, your results should resemble the following output: + + + +For the 100 GB test file, the rsync data transfer should take approximately 2 ½ hours at 11.95MB/s. + +Verifying Your Rsync Migration +To verify that rsync has synced all the files as expected, re-run the command with the --dry-run –stats flags. + +rsync -havzP --stats --dry-run -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs + +If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A successful rsync transfer should look like the following output (note the number of created, deleted, and transferred files equal 0). + + + +Azure Costs +To give you a cost estimate for using rsync to migrate Azure Disk Storage to Linode Block Storage, here’s a snapshot of Azure Cost Analysis after transferring the initial 10 GB test file: + + + +Here’s a snapshot of Azure Cost Analysis after transferring the 100 GB test file (and a few trial runs): + + + +Linode Costs +Linode offers free, unmetered network transfer for all inbound network transfers. You can view network transfer by clicking on Linodes from the left-hand column, selecting your Linode instance, and clicking on the Network tab. + +Here’s what the Network pane looks like after transferring the initial 10 GB test file: + From a5869f9e17d6a2cbf1fd1e73504355672b5cd28f Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Mon, 2 Dec 2024 10:04:03 -0500 Subject: [PATCH 02/13] Copy edits --- .../index.md | 387 +++++++----------- 1 file changed, 148 insertions(+), 239 deletions(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index e2b52a16bd5..01eb488e956 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -13,342 +13,251 @@ external_resources: - '[Link Title 2](http://www.example.net)' --- -Introduction - -This guide details the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. While Azure Disk Storage serves a broader range of use cases than Linode Block Storage, this guide focuses on migrating Azure data disks, excluding OS and temporary disks. - -The following topics are covered in this guide: - -Linode Block Storage vs. Azure Disk Storage - -Migration Considerations - -Data Migration Workflow Diagram - -Data Migration Instructions - -Migration Costs - -Also included are instructions for creating two sample data sets, one at 10GB and another at 100GB, to serve as examples for migrating data from Azure Disk Storage to Linode Block Storage. +This guide describes the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. This guide does not cover migrating operating system disks from Azure Disk Storage. Instead, it focuses on migrating data disks. ## Linode Block Storage vs. Azure Disk Storage -When you first create an Azure Virtual Machine (VM), Azure Disk Storage creates two managed disks by default: an OS disk and data disk. Like Linode Block Storage volumes, Azure managed disks are also block-level storage volumes used with VMs. Unlike Azure Disk Storage, which fulfills multiple roles, Linode Block Storage is generally used for persistent data, not OS/boot disks or temporary data. These other roles are fulfilled by a Linode compute instance’s bundled disk. Linode’s bundled disk storage is also more suitable for certain applications like high-traffic databases. +When you first create an Azure Virtual Machine, Azure Disk Storage creates a managed disk for the operating system. One or more managed disks can also be created with the Azure Disk Storage service for temporary files and for persistent data disks. + +Like Azure Disk Storage, Linode Block Storage also provides block-level storage volumes used with VMs. Unlike Azure Disk Storage, Linode Block Storage is generally used for persistent data, not OS/boot disks or temporary data. These other roles are fulfilled by a Linode compute instance's bundled disk, which is stored on the same host as the compute instance. Linode's bundled disk storage is also more suitable for applications that feature high disk operations, like high-traffic databases. ## Migration Considerations -The following are important time, cost, security, and data compression considerations you should keep in mind when migrating your Azure Disk Storage drives to Linode Block Storage. + +The following are important time, cost, and security considerations you should keep in mind when migrating your Azure Disk Storage drives to Linode Block Storage. ### Migration Time Estimates -This guide covers the creation and migration of two test files: one 10 GB file and another 100 GB file. Depending on your specific server and network configuration, the 10 GB file may take over 2 minutes to transfer from Azure to Linode at around 58 MB/s, while the 100 GB can take over 2 ½ hours at 11.95MB/s. -As discussed later in this guide, you can exclude the -c rsync flag to skip checksum comparisons for larger file sizes. Doing so significantly speeds up the end-to-end data migration time. +EDITOR: fill in ### Migration Egress Costs -The migrations demonstrated in this guide incurred the following costs: - -The 10 GB sample dataset migration incurred $2.22 in storage costs, $0.05 in virtual network costs, and < $0.01 in bandwidth costs. -After the 100GB sample dataset migration, $3.88 in storage costs, $0.09 in virtual network costs, and < $0.01 in bandwidth costs were incurred. +EDITOR: fill in ### Security and Firewalls -To securely migrate your files, you should execute rsync over SSH using a public key (.pem file) generated on your Azure VM. Your Azure and Linode firewall settings should also be configured to accept inbound or outbound SSH traffic. - -## Data Compression +Files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted. This guide describes how to use [public key authentication](/docs/guides/use-public-key-authentication-with-ssh/) when using rsync. -Compressing files to be migrated on the source is generally good practice to shorten data transfer times. The rsync command used in this guide includes the -z flag to enable compression during data transfer, speeding up data transfers and saving network bandwidth. +Your Azure and Linode firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and Azure virtual machine. ## Block Storage Migration Workflow Diagram -The following workflow diagram outlines the activities covered in this guide. Steps 1 and 3 may not apply to your scenario if you’ve already prepared your Azure Data Disk and Linode Block Storage volumes. - - - -Azure VM disk storage is prepared. - -Test dataset is created on Azure disk storage. - -Linux Block Storage volume created and attached to Linode instance. - -Azure VM public key (.pem file) copied to Linode instance. - -Rsync command issues from Linode instance. - -Test dataset copied to Linode instance. +EDITOR: fill in ## Block Storage Migration Instructions ### Prerequisites and Assumptions -This guide assumes that you have both a Linux-based Azure VM instance and destination Linode instance up-and-running. Upon creating your Azure VM, an OS disk and data disk should automatically be attached to your instance. - - - -Ensure that you have already generated a public key (.pem file) for your Azure VM instance, as it will be required to securely run rsync over SSH between your Linode in the Akamai Connected Cloud and your VM in the Microsoft Azure environment. - -This guide assumes the following source filesystem path for your Azure volume: - -/dev/sdc1 mounted to /datadrive - -The destination of your Linode migration is assumed to be: - -/mnt/linode-bs - -Preparing Your Azure Instance with Disk Attached -In the Azure portal, go to your VM and expand the Settings link on the left-hand side. Click on Disks—you should see your OS disk and your Data disk at logical unit number (LUN) 0: - - - -If you SSH into your Azure VM, you can run the lsblk command, pipe the output to grep,and filter on “sd” to verify information about all your available block devices: - - lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd" - -Your results should resemble the following output: - - - -As you can see, sda is the 30 GB OS disk and sdc is the 1 TB data disk at LUN 0. - -Preparing A New Empty Disk -If you’ve just created your Azure VM, the data disk is empty and needs to be prepared. Use the parted and mkfs utility to partition and format the data disk using the XFS filesystem, followed by partprobe to make your OS kernel aware of the new partition and filesystem. - -sudo parted /dev/sdc --script mklabel gpt mkpart xfspart xfs 0% 100% - -sudo mkfs.xfs /dev/sdc1 - -sudo partprobe /dev/sdc1 - -Mounting The Disk -Next, use mkdir to create a directory called /datadrive to mount your filesystem to: - -sudo mkdir /datadrive - -Use mount to mount the /dev/sdc1 partition to your new /datadrive directory: - -sudo mount /dev/sdc1 /datadrive - -To ensure that the drive is unmounted automatically after a reboot, you’ll need to add it to your /etc/fstab file. fstab is a configuration table designed to facilitate mounting and unmounting Linux file systems to a machine. - -First, use the blkid utility to get the universally unique identifier (UUID) of your drive—you’ll need it to add your drive to /etc/fstab. - -sudo blkid - -Your results should resemble the following output: - - - -Take note of the UUID for /dev/sdc1. Next, open the /etc/fstab file in a text editor. Add the following line to the end of the file using the UUID value for the /dev/sdc1 and the mountpoint of /datadrive: - -UUID=4f6fb6f0-38a3-4f9e-a5de-5a3b0ca9bedd /datadrive xfs defaults,nofail 1 2 - -Verifying The Disk -You can now use lsblk again to verify the disk and the mountpoint: - -lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd" - -Your results should resemble the following output: - - - -As you can see, sdc1 is now mounted at /datadrive. - -Creating Sample Data on Your Azure Data Disk -You can now create the test file to migrate from your Azure data disk to your Linode instance. - -Start by creating a 10 GB file using the dd command: - -sudo dd if=/dev/zero of=/datadrive/dummyfile bs=1M count=10240 - -Running this command will create a 10 GB test file filled with zeros at /datadrive/dummyfile. - -bs=1M sets the block size to 1 MB and count=10240 specifies a total number of 10240 blocks to write. With a block size of 1 MB, this creates a file of 10 GB (10240 MB). - -Once the dd command finishes, your results should resemble the following output: - - - -You can verify that your test file was successfully created by navigating to your datadrive directory and typing ls -al: - - - -Preparing you Linode Instance with Attached Block Storage -At this point, your Azure VM and its attached data disk are ready to be migrated. The next step involves preparing your Linode instance and its attached block storage volume to receive the test file. - -Creating Your Linode Block Storage Volume -From your Linode Cloud Manager dashboard, click on Volumes on the left-hand side and the Create Volume button on the top right-hand side. - -Specify a Label, Region, Linode, and Config. Enter 200 GB for the volume’s Size. - - - -Once the volume is created, you’ll see it show up on your Volumes page. Click on the Show Config button. - - - -The Volume Configuration pane will appear on the right-hand side. Copy the values from the pane and paste them into your notepad/notes—you’ll need them later when configuring your new volume. - - - -Configuring Your New Linode Block Storage Volume -The steps to configure your new Linode Block Storage volume are similar to the steps you followed previously in configuring your Azure data disk: you will use mkfs to to create a filesystem on your new volume, create a directory to mount the filesystem to using mkdir, then using mount to mount the new volume. You will then edit your /etc/fstab file to ensure that the volume mounts every time your Linode instance reboots. - -Log in to your Linode instance. You can either SSH directly, or use the LISH console by expanding the menu on the right-hand side and clicking the Launch LISH Console menu item: - - - -After logging in to your Linode instance, run the commands you previously copied to your notepad (mkfs, mkdir, mount): - -mkfs.ext4 "/dev/disk/by-id/scsi-0Linode_Volume_linode-bs” - -mkdir "/mnt/linode-bs" - -mount "/dev/disk/by-id/scsi-0Linode_Volume_linode-bs" "/mnt/linode-bs” - -Your results should resemble the following output: - - - -Open up your /etc/fstab file in a text editor and add the following line to make sure the volume auto-mounts every time your Linode reboots: - -/dev/disk/by-id/scsi-0Linode_Volume_linode-bs /mnt/linode-bs ext4 defaults,noatime,nofail 0 2 +This guide assumes that you have an Azure virtual machine and an attached data disk on the Azure Disk Storage service. The assumed filesystem path for the Azure data disk is `/datadrive`. The assumed username for the Azure virtual machine is `azureuser`. -If you’re using the VI text editor, your changes should look like this: +### Prepare a Linode Block Storage Volume +1. To transfer data to a Linode Block Storage volume, it must first be attached to a Linode instance. You may create a new Linode instance for the purpose of this migration ([Create a Compute Instance](https://techdocs.akamai.com/cloud-computing/docs/create-a-compute-instance)). Alternatively, you can use an existing Linode instance for the migration. + {{< note >}} + If you create an instance to use for this migration, you may wish to delete it after the migration is complete. Deleting an instance that has an attached volume does not delete the volume. + {{< /note >}} -Save your updated /etc/fstab file. You are now ready to start migrating your Linode Block Storage from Azure Disk Storage. +1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume have a capacity equal to or higher than the source Azure data disk. -### Use Rsync to Migrate Block Storage Data + When creating the volume, the Linode Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/your-volume-name`). -You’ll be using rsync to connect to your Azure VM and migrate your block storage data to your Linode Block Storage volume. +### Configure Firewalls -Check Linode and Azure Firewall and Connection Settings -First, make sure the firewall settings on both sides are configured correctly to allow the migration traffic to pass through. +In this guide, the rsync command is run from a Linode instance and connects to an Azure virtual machine. This means that the Azure VM should accept inbound SSH traffic (port 22). You may also wish to specifically add the IP address of the Linode instance to the allow list for inbound traffic of the Azure VM. -In Linode, click on Firewalls and make sure that any inbound or outbound rules are set to Accept inbound SSH traffic. +Linux distributions (on both Linode instances and Azure virtual machines) can have software firewalls configured inside the instance. The following guides describe some software firewalls that your instances may use: -For example, this is what it would look like in Linode: +- [Configure a Firewall with Firewalld](/docs/guides/introduction-to-firewalld-on-centos/) +- [How to Configure a Firewall with UFW](/docs/guides/configure-firewall-with-ufw/) +- [A Tutorial for Controlling Network Traffic with iptables](/docs/guides/control-network-traffic-with-iptables/) +You may also configure cloud firewalls to control traffic before it arrives at your computing instance. The [Linode Cloud Firewalls](https://techdocs.akamai.com/cloud-computing/docs/cloud-firewall/) product documentation describes how to configure these rules. The [Comparing Cloud Firewalls to Linux firewall software](https://techdocs.akamai.com/cloud-computing/docs/comparing-cloud-firewalls-to-linux-firewall-software) guide further describes the difference between network firewalls and software firewalls. Microsoft Azure's product documentation describes how to configure Azure network firewalls. +### Configure SSH Key Pair -Using Rsync To Migrate the 10 GB Test File -NOTE: Be sure to first copy your Azure VM’s public key (.pem file) to your Linode instance. +This guide uses SSH public key authentication for the rsync connection. You must have a public-private key pair installed on the Linode instance and Azure virtual machine. The [Generate an SSH Key Pair](/docs/guides/use-public-key-authentication-with-ssh/#generate-an-ssh-key-pair) section of the [SSH Public Key Authentication](/docs/guides/use-public-key-authentication-with-ssh/) guide describes how to create a key pair. -Run the rsync command to start migrating the contents of your Azure VM’s /datadrive directory to the /mnt/linode-bs directory on your Linode instance : +This guide assumes the public and private keys are named `id_rsa.pub` and `id_rsa`, but you may wish to choose a different name. -rsync -chavzP --stats -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs +- The *public key* should be uploaded to the Azure virtual machine. It should be appended to a new line of the `authorized_keys` file of the user on the Azure VM (e.g. `/home/azureuser/.ssh/authorized_keys). -Here's a detailed explanation of the key flags and parameters used in the rsync command: +- The *private key* should be uploaded to the Linode instance. It should be uploaded to the `.ssh/` directory of the user on the Linode instance (e.g. `/home/linodeuser/.ssh/`). It should have permissions set to `600`: --c: This option tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but -c forces it to compute checksums, which is slower but can be more accurate if you want to be sure that files match exactly. + ```command {title="SSH session with Linode instance"} + chmod 600 /home/linodeuser/.ssh/id_rsa/ + ``` --h: Human-readable output, which makes file sizes (like transfer statistics) easier to read by using units like KB and MB, rather than raw byte counts. +### Configure Known Hosts --a: Archive mode. This option enables several options at once (-rlptgoD), which tells rsync to: +```command +ssh-keyscan test.rebex.net >> ~/.ssh/known_hosts +``` -Recursively copy directories (-r). +### Initiate the Migration -Preserve symbolic links (-l). +Two recommended practices: -Retain file permissions (-p). +- Running rsync in a persistent process (tmux) -Keep timestamps (-t). +- Sending output and errors to log files -Preserve group ownership (-g). +1. Create a new tmux session. This session is used to initiate the migration, and it will persist between SSH sessions. This is important for large/long-running migrations. This command creates a session with name `block-storage-migration`: -Maintain file ownership (-o). + ```command + tmux new -s block-storage-migration + ``` -Retain device files and special files (-D). + After running this command, the tmux session is immediately activated in your terminal. -The result of the -a flag is a complete, near-exact copy of the source directory. +1. Run the rsync command to start migrating the contents of your Azure VM's /datadrive directory to the /mnt/linode-bs directory on your Linode instance : --v: Verbose mode. This makes rsync output more detailed information about what it is doing, which is helpful for monitoring the progress of a large transfer or troubleshooting. + ```command {title="SSH session with Linode instance (bs-migration tmux session)"} + echo "\n\nInitiating migration $(date)\n---" | tee -a bs-migration-logs.txt bs-migration-errors.txt >/dev/null --z: Compression. This enables compression during data transfer, which can save bandwidth and speed up the transfer if the network connection is relatively slow. + rsync -chavzP --stats -e "ssh -i /home/azureuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt + ``` --P: This option is a combination of two flags: + Here's a detailed explanation of the key flags and parameters used in the rsync command: ---progress, which displays progress information for each file transfer. + - `-c`: Tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but -c forces it to compute checksums, which is slower but can be more accurate if you want to be sure that files match exactly. ---partial, which keeps partially transferred files if the transfer is interrupted, allowing it to resume more easily next time. + - `-h`: Human-readable output, which makes file sizes (like transfer statistics) easier to read by using units like KB and MB, rather than raw byte counts. ---stats: This option provides detailed statistics at the end of the transfer, such as total bytes transferred, transfer speed, and file counts. + - `-a`: Archive mode. This is equivalent to specifying: `-rlptgoD`. The result of the `-a` flag is a complete, near-exact copy of the source directory: --e "ssh -i /path/to/your_azure_vm_key.pem": This option specifies a remote shell (ssh) with an identity key file for authentication: + - `-r`: Recursively copy directories --e lets you specify a remote shell to use (in this case, SSH). + - `-l`: Preserve symbolic links -"ssh -i /path/to/your_azure_vm_key.pem" tells rsync to connect to the remote server using SSH, and the -i option specifies an SSH key (/path/to/your_azure_vm_key.pem) for authentication. + - `-p`: Retain file permissions -azureuser@13.93.147.88:/datadrive/: This specifies the source directory you’re syncing from. Here: + - `-t`: Keep timestamps -azureuser is the username on the remote server (replace with your username). + - `-g`: Preserve group ownership -13.93.147.88 is the IP address of the remote server (replace with your IP address . + - `-o`: Maintain file ownership -/datadrive/ is the path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /datadrive, rather than creating a /datadrive directory in the target. + - `-D`: Retain device files and special files -/mnt/linode-bs: This is the destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /datadrive contents here. + - `-v`: Verbose mode. This makes rsync output more detailed information about what it is doing, which is helpful for monitoring the progress of a large transfer or troubleshooting. -Once you enter the rsync command, the syncing process will start and display your progress in real-time: + - `-z`: Compression. This enables compression during data transfer, which can save bandwidth and speed up the transfer if the network connection is relatively slow. + - `-P`: Combines two other flags: + - `--progress`, which displays progress information for each file transfer. -Transferring the 10 GB test file should take a little over 2 minutes, at around 58 MB/s. Once rsync completes, your results should look like the following output: + - `--partial`, which keeps partially transferred files if the transfer is interrupted, allowing it to resume more easily next time. + - `--stats`: Provides detailed statistics at the end of the transfer, such as total bytes transferred, transfer speed, and file counts. + - `-e "ssh -i /path/to/your_azure_vm_key.pem"`: Specifies a remote shell (SSH) with an identity key file for authentication. -Navigate to the /mnt/linode-bs directory and issue an ls -al command to verify that the file has transferred successfully: + - `azureuser@13.93.147.88:/datadrive/`: This specifies the source directory you're syncing from. Here: + azureuser is the username on the remote server (replace with your username). + 13.93.147.88 is the IP address of the remote server (replace with your IP address . -Using Rsync To Migrate a 100 GB Test File -Next, you’ll try using rsync on a larger migration: a 100 GB test file. Start by logging in to your Azure VM via SSH and deleting the existing test file: + /datadrive/ is the path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /datadrive, rather than creating a /datadrive directory in the target. -rm /datadrive/dummyfile + /mnt/linode-bs: This is the destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /datadrive contents here. -You should also remove the test file in the destination directory of your Linode instance. Use weblish to log back into your Linode instance and remove the existing test file in your destination directory: +### Monitor the Migration -rm /mnt/linode-bs/dummyfile +Because the stdout and stderr streams were redirected to log files, the rsync command will not produce output in the terminal. Follow these steps to inspect and monitor the contents of the logs: -Back in your Azure VM, run the dd command again—this time, specify the creation of a 100 GB file: +1. To avoid interrupting the rsync process, *detach* from the tmux session by entering this sequence of keystrokes: `CTRL-b` followed by `d`. You are returned to the SSH session that created the tmux session: -sudo dd if=/dev/zero of=/datadrive/dummyfile bs=10MB count=10000 + ```output + [detached (from session block-storage-migration)] + ``` -bs=10M sets the block size to 10 MB and count=10000 specifies a total number of 10000 blocks to write. With a block size of 10 MB, this creates a file of 100 GB. +1. Use `tail -f` to inspect the log and error files and monitor any new output from them: -NOTE: you should use 10 MB blocks to avoid memory exhaustion issues. Also, the creation process will take longer than before (around 5 minutes) due to the significantly larger test file. + ```command + tail -f block-storage-migration-logs.txt + ``` -Once your test file has been created, log back into your Linode instance and run the rsync command again. This time, remove the -c option to speed up the process for the large test file (the -c flag forces rsync to calculate a checksum for all files). + ```command + tail -f block-storage-migration-errors.txt + ``` -rsync -havzP --stats -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs + Enter `CTRL-c` to stop `tail`. -rsync will take a longer time to receive the incremental file list. Once it has completed the transfer, your results should resemble the following output: +1. You can re-enter the tmux session with the `tmux attach` command: + ```command + tmux attach -t block-storage-migration + ``` +1. When inside the tmux session, you can end it with the `exit` command: -For the 100 GB test file, the rsync data transfer should take approximately 2 ½ hours at 11.95MB/s. + ```command + exit + ``` -Verifying Your Rsync Migration -To verify that rsync has synced all the files as expected, re-run the command with the --dry-run –stats flags. +1. Once you enter the rsync command, the syncing process will start and display your progress in real-time: -rsync -havzP --stats --dry-run -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs + ```output + receiving incremental file list + ./ + dummyfile + 10.49M 100% 666.67MB/s 0:00:00 (xfr#1, to-chk=0/2) -If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A successful rsync transfer should look like the following output (note the number of created, deleted, and transferred files equal 0). + Number of files: 2 (reg: 1, dir: 1) + Number of created files: 1 (reg: 1) + Number of deleted files: 0 + Number of regular files transferred: 1 + Total file size: 10.49M bytes + Total transferred file size: 10.49M bytes + Literal data: 10.49M bytes + Matched data: 0 bytes + File list size: 84 + File list generation time: 0.004 seconds + File list transfer time: 0.000 seconds + Total bytes sent: 46 + Total bytes received: 475 + sent 46 bytes received 475 bytes 347.33 bytes/sec + total size is 10.49M speedup is 20,126.22 + ``` +### Verify the Migration -Azure Costs -To give you a cost estimate for using rsync to migrate Azure Disk Storage to Linode Block Storage, here’s a snapshot of Azure Cost Analysis after transferring the initial 10 GB test file: +1. Verifying Your Rsync Migration + To verify that rsync has synced all the files as expected, re-run the command with the --dry-run –stats flags. + ```command + rsync -chavzP --stats --dry-run -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs + ``` -Here’s a snapshot of Azure Cost Analysis after transferring the 100 GB test file (and a few trial runs): + ```output + receiving incremental file list + Number of files: 2 (reg: 1, dir: 1) + Number of created files: 0 + Number of deleted files: 0 + Number of regular files transferred: 0 + Total file size: 10.49M bytes + Total transferred file size: 0 bytes + Literal data: 0 bytes + Matched data: 0 bytes + File list size: 84 + File list generation time: 0.003 seconds + File list transfer time: 0.000 seconds + Total bytes sent: 20 + Total bytes received: 95 + sent 20 bytes received 95 bytes 230.00 bytes/sec + total size is 10.49M speedup is 91,180.52 (DRY RUN) + ``` -Linode Costs -Linode offers free, unmetered network transfer for all inbound network transfers. You can view network transfer by clicking on Linodes from the left-hand column, selecting your Linode instance, and clicking on the Network tab. + If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A successful rsync transfer should look like the following output (note the number of created, deleted, and transferred files equal 0). -Here’s what the Network pane looks like after transferring the initial 10 GB test file: +### Cleanup +EDITOR: +close firewall access +revoke keys for transfer \ No newline at end of file From a5bade05f544be69dcd670c6bcacf18c84d983a8 Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 3 Dec 2024 10:42:27 -0500 Subject: [PATCH 03/13] Copy edits --- .../index.md | 174 +++++++++--------- 1 file changed, 86 insertions(+), 88 deletions(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index 01eb488e956..afe814b254f 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -27,27 +27,43 @@ The following are important time, cost, and security considerations you should k ### Migration Time Estimates -EDITOR: fill in +The time it takes to migrate a data disk is a function of the data stored on that disk, which can be substantial for larger migrations. To determine how much data is stored on your disk, run the `df` command from your Azure VM: + +```command {title="SSH session with Azure VM"} +df -h +``` + +Your data disk should appear, and the `Used` column shows how much data is stored on the disk: + +```output +df -h +Filesystem Size Used Avail Use% Mounted on +/dev/sdc1 20G 4.4G 16G 23% /datadrive +``` + +TODO: advise on bandwidth testing ### Migration Egress Costs -EDITOR: fill in +The cost to migrate a data disk is a function of the data stored on that disk, which can be substantial for larger migrations. These costs are incurred as egress fees when the data leaves the Azure platform and are billed by Azure. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk. Please review the Azure documentation for assistance with calculating this amount. + +Inbound traffic sent to your Linode instance and Block Storage volume have no fees incurred on the Linode platform. ### Security and Firewalls -Files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted. This guide describes how to use [public key authentication](/docs/guides/use-public-key-authentication-with-ssh/) when using rsync. +Files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted. Your Azure and Linode firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and Azure virtual machine. ## Block Storage Migration Workflow Diagram -EDITOR: fill in +TODO: fill in diagram ## Block Storage Migration Instructions ### Prerequisites and Assumptions -This guide assumes that you have an Azure virtual machine and an attached data disk on the Azure Disk Storage service. The assumed filesystem path for the Azure data disk is `/datadrive`. The assumed username for the Azure virtual machine is `azureuser`. +This guide assumes that you have an Azure virtual machine and an attached data disk on the Azure Disk Storage service. The assumed filesystem path for the Azure data disk is `/datadrive`, and the username for the Azure virtual machine is `azureuser`. ### Prepare a Linode Block Storage Volume @@ -57,7 +73,7 @@ This guide assumes that you have an Azure virtual machine and an attached data d If you create an instance to use for this migration, you may wish to delete it after the migration is complete. Deleting an instance that has an attached volume does not delete the volume. {{< /note >}} -1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume have a capacity equal to or higher than the source Azure data disk. +1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume should have a capacity equal to or higher than the total data stored on the source Azure disk. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk. When creating the volume, the Linode Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/your-volume-name`). @@ -87,37 +103,47 @@ This guide assumes the public and private keys are named `id_rsa.pub` and `id_rs chmod 600 /home/linodeuser/.ssh/id_rsa/ ``` -### Configure Known Hosts - -```command -ssh-keyscan test.rebex.net >> ~/.ssh/known_hosts -``` - ### Initiate the Migration -Two recommended practices: +These instructions implement two recommended practices: -- Running rsync in a persistent process (tmux) +- Running rsync in a persistent process - Sending output and errors to log files +Migrations can take a long time, so having them run independently of your SSH session is important. This guide uses `tmux` to create a terminal session that persists between SSH connections. By sending output and errors to log files, you can keep a record of any migration failures that may happen. + +1. Install `tmux`: [Installing tmux](https://github.com/tmux/tmux/wiki/Installing#installing-tmux). + 1. Create a new tmux session. This session is used to initiate the migration, and it will persist between SSH sessions. This is important for large/long-running migrations. This command creates a session with name `block-storage-migration`: - ```command + ```command {title="SSH session with Linode instance"} tmux new -s block-storage-migration ``` After running this command, the tmux session is immediately activated in your terminal. -1. Run the rsync command to start migrating the contents of your Azure VM's /datadrive directory to the /mnt/linode-bs directory on your Linode instance : +1. Run the following commands to start migrating the contents of your Azure data disk to your Linode block storage volume: ```command {title="SSH session with Linode instance (bs-migration tmux session)"} echo "\n\nInitiating migration $(date)\n---" | tee -a bs-migration-logs.txt bs-migration-errors.txt >/dev/null - rsync -chavzP --stats -e "ssh -i /home/azureuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt + rsync -chavzP --stats -e "ssh -i /home/linodeuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt ``` - Here's a detailed explanation of the key flags and parameters used in the rsync command: + Be sure to replace these values: + + - `/home/linodeuser/.ssh/id_rsa`: The name and location of the private key + - `azureuser`: The name of the user on the Azure VM + - `AZURE_VM_IP`: The IP address of the Azure VM + - `/datadrive/`: The directory the Azure data disk is mounted under + - `/mnt/linode-block-storage-volume`: The directory your Linode volume is mounted under + + {{< note >}} + You may be prompted to accept the host key of the Azure VM if it is the first time that the Linode has connected to it. + {{< /note >}} + + The first `echo` appends a message to the log files. Here's a detailed explanation of the key flags and parameters used in the `rsync` command: - `-c`: Tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but -c forces it to compute checksums, which is slower but can be more accurate if you want to be sure that files match exactly. @@ -151,17 +177,17 @@ Two recommended practices: - `--stats`: Provides detailed statistics at the end of the transfer, such as total bytes transferred, transfer speed, and file counts. - - `-e "ssh -i /path/to/your_azure_vm_key.pem"`: Specifies a remote shell (SSH) with an identity key file for authentication. + - `-e "ssh -i /home/linodeuser/.ssh/id_rsa"`: Specifies a remote shell (SSH) with an identity key file for authentication. - - `azureuser@13.93.147.88:/datadrive/`: This specifies the source directory you're syncing from. Here: + - `azureuser@AZURE_VM_IP:/datadrive/`: This specifies the source directory you're syncing from: - azureuser is the username on the remote server (replace with your username). + - `azureuser`: The username on the remote server. - 13.93.147.88 is the IP address of the remote server (replace with your IP address . + - `AZURE_VM_IP`: The IP address of the remote server. - /datadrive/ is the path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /datadrive, rather than creating a /datadrive directory in the target. + - `/datadrive/`: The path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /datadrive, rather than creating a /datadrive directory in the target. - /mnt/linode-bs: This is the destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /datadrive contents here. + - `/mnt/linode-block-storage-volume`: The destination directory on the local machine where rsync will copy the files to. In this case, it will create an exact copy of /datadrive contents here. ### Monitor the Migration @@ -175,11 +201,11 @@ Because the stdout and stderr streams were redirected to log files, the rsync co 1. Use `tail -f` to inspect the log and error files and monitor any new output from them: - ```command + ```command {title="SSH session with Linode instance"} tail -f block-storage-migration-logs.txt ``` - ```command + ```command {title="SSH session with Linode instance"} tail -f block-storage-migration-errors.txt ``` @@ -187,77 +213,49 @@ Because the stdout and stderr streams were redirected to log files, the rsync co 1. You can re-enter the tmux session with the `tmux attach` command: - ```command + ```command {title="SSH session with Linode instance"} tmux attach -t block-storage-migration ``` -1. When inside the tmux session, you can end it with the `exit` command: - - ```command - exit - ``` - -1. Once you enter the rsync command, the syncing process will start and display your progress in real-time: - - ```output - receiving incremental file list - ./ - dummyfile - 10.49M 100% 666.67MB/s 0:00:00 (xfr#1, to-chk=0/2) - - Number of files: 2 (reg: 1, dir: 1) - Number of created files: 1 (reg: 1) - Number of deleted files: 0 - Number of regular files transferred: 1 - Total file size: 10.49M bytes - Total transferred file size: 10.49M bytes - Literal data: 10.49M bytes - Matched data: 0 bytes - File list size: 84 - File list generation time: 0.004 seconds - File list transfer time: 0.000 seconds - Total bytes sent: 46 - Total bytes received: 475 - - sent 46 bytes received 475 bytes 347.33 bytes/sec - total size is 10.49M speedup is 20,126.22 - ``` + {{< note >}} + Review the [tmux guide](/docs/guides/persistent-terminal-sessions-with-tmux/) for help with other tmux commands. + {{< /note >}} ### Verify the Migration -1. Verifying Your Rsync Migration +To verify that rsync has synced all the files as expected, re-run the command with the `--dry-run –stats` flags: - To verify that rsync has synced all the files as expected, re-run the command with the --dry-run –stats flags. +```command {title="SSH session with Linode instance"} +rsync -chavzP --stats --dry-run -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs +``` - ```command - rsync -chavzP --stats --dry-run -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs - ``` +If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A previous successful rsync transfer should result in the following output. Note that the number of created, deleted, and transferred files are zero: + +```output +receiving incremental file list + +Number of files: 2 (reg: 1, dir: 1) +Number of created files: 0 +Number of deleted files: 0 +Number of regular files transferred: 0 +Total file size: 10.49M bytes +Total transferred file size: 0 bytes +Literal data: 0 bytes +Matched data: 0 bytes +File list size: 84 +File list generation time: 0.003 seconds +File list transfer time: 0.000 seconds +Total bytes sent: 20 +Total bytes received: 95 + +sent 20 bytes received 95 bytes 230.00 bytes/sec +total size is 10.49M speedup is 91,180.52 (DRY RUN) +``` - ```output - receiving incremental file list - - Number of files: 2 (reg: 1, dir: 1) - Number of created files: 0 - Number of deleted files: 0 - Number of regular files transferred: 0 - Total file size: 10.49M bytes - Total transferred file size: 0 bytes - Literal data: 0 bytes - Matched data: 0 bytes - File list size: 84 - File list generation time: 0.003 seconds - File list transfer time: 0.000 seconds - Total bytes sent: 20 - Total bytes received: 95 - - sent 20 bytes received 95 bytes 230.00 bytes/sec - total size is 10.49M speedup is 91,180.52 (DRY RUN) - ``` +### Cleanup after Migration - If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A successful rsync transfer should look like the following output (note the number of created, deleted, and transferred files equal 0). +After the migration is complete, you may determine that the Azure VM and Linode instance no logner need to communicate. You can close traffic between these servers by doing the following: -### Cleanup +- Remove the firewall access granted in the [Configure Firewalls](#configure-firewalls) section -EDITOR: -close firewall access -revoke keys for transfer \ No newline at end of file +- Revoke the SSH key used for the transfer. This is done by removing the SSH public key that was referenced from the `/home/azureuser/.ssh/authorized_keys` file on the Azure VM. \ No newline at end of file From 9473adad2cfa7dc633c3af0e178e323567ed6bb3 Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 3 Dec 2024 10:44:03 -0500 Subject: [PATCH 04/13] Copy edits --- .../index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index afe814b254f..b43b500b00b 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -226,7 +226,7 @@ Because the stdout and stderr streams were redirected to log files, the rsync co To verify that rsync has synced all the files as expected, re-run the command with the `--dry-run –stats` flags: ```command {title="SSH session with Linode instance"} -rsync -chavzP --stats --dry-run -e "ssh -i /path/to/your_azure_vm_key.pem" azureuser@13.93.147.88:/datadrive/ /mnt/linode-bs +rsync -chavzP --stats --dry-run -e "ssh -i /home/azureuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume ``` If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A previous successful rsync transfer should result in the following output. Note that the number of created, deleted, and transferred files are zero: From 3ab47fd2d479224bd8e6a351f33ea154e1f81648 Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 3 Dec 2024 10:49:58 -0500 Subject: [PATCH 05/13] Copy edits --- .../index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index b43b500b00b..b98931c1274 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -75,7 +75,7 @@ This guide assumes that you have an Azure virtual machine and an attached data d 1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume should have a capacity equal to or higher than the total data stored on the source Azure disk. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk. - When creating the volume, the Linode Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/your-volume-name`). + When creating the volume, the Linode Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/linode-block-storage-volume`). ### Configure Firewalls @@ -131,7 +131,7 @@ Migrations can take a long time, so having them run independently of your SSH se rsync -chavzP --stats -e "ssh -i /home/linodeuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt ``` - Be sure to replace these values: + Be sure to replace these values with the actual values from your Azure VM and Linode instance: - `/home/linodeuser/.ssh/id_rsa`: The name and location of the private key - `azureuser`: The name of the user on the Azure VM From 5eee33ab3fb9744b0f520fe00fd41e1a6cf4f79b Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 3 Dec 2024 10:52:40 -0500 Subject: [PATCH 06/13] Copy edits --- .../index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index b98931c1274..cd789c3cbc7 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -115,7 +115,7 @@ Migrations can take a long time, so having them run independently of your SSH se 1. Install `tmux`: [Installing tmux](https://github.com/tmux/tmux/wiki/Installing#installing-tmux). -1. Create a new tmux session. This session is used to initiate the migration, and it will persist between SSH sessions. This is important for large/long-running migrations. This command creates a session with name `block-storage-migration`: +1. Create a new tmux session named `block-storage-migration`. This session is used to initiate the migration: ```command {title="SSH session with Linode instance"} tmux new -s block-storage-migration From 9a3a336b5eb59742413eff2aea1685f58ee89191 Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 3 Dec 2024 10:53:31 -0500 Subject: [PATCH 07/13] Copy edits --- .../index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index cd789c3cbc7..921c4d474df 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -36,7 +36,6 @@ df -h Your data disk should appear, and the `Used` column shows how much data is stored on the disk: ```output -df -h Filesystem Size Used Avail Use% Mounted on /dev/sdc1 20G 4.4G 16G 23% /datadrive ``` From 5e5b6c66d15b8cac59c361a895de88bc6fde39ec Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 3 Dec 2024 10:55:46 -0500 Subject: [PATCH 08/13] Copy edits --- .../index.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index 921c4d474df..accc425cd7a 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -8,9 +8,6 @@ contributors: ["Linode"] published: 2024-11-18 keywords: ['list','of','keywords','and key phrases'] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' -external_resources: -- '[Link Title 1](http://www.example.com)' -- '[Link Title 2](http://www.example.net)' --- This guide describes the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. This guide does not cover migrating operating system disks from Azure Disk Storage. Instead, it focuses on migrating data disks. From 180955968381419398d4eaf48c0180bca4182e7d Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Wed, 11 Dec 2024 22:04:20 -0500 Subject: [PATCH 09/13] Copy edits, add diagram --- ...azure-block-storage-migration-workflow.svg | 1 + .../index.md | 28 +++++++++++++++---- .../index.md | 0 3 files changed, 24 insertions(+), 5 deletions(-) create mode 100644 docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg create mode 100644 docs/guides/uptime/loadbalancing/configure-failover-for-haproxy-on-akamai/index.md diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg new file mode 100644 index 00000000000..cf2073c0162 --- /dev/null +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index accc425cd7a..e079cbee2be 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -12,6 +12,17 @@ license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' This guide describes the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. This guide does not cover migrating operating system disks from Azure Disk Storage. Instead, it focuses on migrating data disks. +## Block Storage Migration Workflow Diagram + +![Azure to Linode Block Storage Migration Workflow Diagram](azure-block-storage-migration-workflow.svg?diagram-description-id=azure-linode-bs-migration) + +1. rsync command is run on Linode instance and connects to Azure VM. + +1. Azure VM sends data on Azure data disk to Block Storage Volume attached to Linode instance via established rsync connection. + + 1. Egress costs for the migrated data are measured when the data leaves the Azure platform. These costs are billed by Azure. +{#azure-linode-bs-migration .large-diagram} + ## Linode Block Storage vs. Azure Disk Storage When you first create an Azure Virtual Machine, Azure Disk Storage creates a managed disk for the operating system. One or more managed disks can also be created with the Azure Disk Storage service for temporary files and for persistent data disks. @@ -37,7 +48,18 @@ Filesystem Size Used Avail Use% Mounted on /dev/sdc1 20G 4.4G 16G 23% /datadrive ``` -TODO: advise on bandwidth testing +Bandwidth for the transfer can vary according to different factors, including: +- Outbound bandwidth limits for your Azure VM +- Geographic distance between the Azure VM and the Linode instance. +- Disk operation limits + +When planning your migration, you may consider performing a bandwidth test between the two locations first. Then, use the observed bandwidth from the test to calculate the estimated migration time for the data disk. + +Utilities like [iperf](https://en.wikipedia.org/wiki/Iperf) can be useful for performing this type of bandwidth measurement. Alternatively, you could create a test file on the Azure VM, migrate it by following the [instructions](#block-storage-migration-instructions) in this guide, and then view the bandwidth reported by rsync's output. A command that can be used to generate a sample 1GB test file is: + +```command {title="SSH session with Azure VM"} +sudo dd if=/dev/zero of=/datadrive/dummyfile bs=1M count=1024 +``` ### Migration Egress Costs @@ -51,10 +73,6 @@ Files should be migrated over an encrypted connection. Rsync supports using SSH Your Azure and Linode firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and Azure virtual machine. -## Block Storage Migration Workflow Diagram - -TODO: fill in diagram - ## Block Storage Migration Instructions ### Prerequisites and Assumptions diff --git a/docs/guides/uptime/loadbalancing/configure-failover-for-haproxy-on-akamai/index.md b/docs/guides/uptime/loadbalancing/configure-failover-for-haproxy-on-akamai/index.md new file mode 100644 index 00000000000..e69de29bb2d From 3f373206907d76b5c1cebaa9794d67e758822d1e Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Thu, 12 Dec 2024 00:55:03 -0500 Subject: [PATCH 10/13] Update diagram --- ...azure-block-storage-migration-workflow.svg | 180 +++++++++++++++++- 1 file changed, 179 insertions(+), 1 deletion(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg index cf2073c0162..a773bf91145 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/azure-block-storage-migration-workflow.svg @@ -1 +1,179 @@ - \ No newline at end of file + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From 5403d69dd3ba127e22bde2c7c037143629cc4bdf Mon Sep 17 00:00:00 2001 From: John Dutton Date: Thu, 12 Dec 2024 14:38:41 -0500 Subject: [PATCH 11/13] copy edits --- .../index.md | 91 ++++++++++--------- 1 file changed, 47 insertions(+), 44 deletions(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index e079cbee2be..f5726c5ba39 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -3,35 +3,36 @@ slug: migrate-from-azure-disk-storage-to-linode-block-storage title: "Migrate From Azure Disk Storage to Linode Block Storage" description: "Two to three sentences describing your guide." og_description: "Optional two to three sentences describing your guide when shared on social media. If omitted, the `description` parameter is used within social links." -authors: ["Linode"] -contributors: ["Linode"] +authors: ["Leon Yen","Nathan Melehan"] +contributors: ["Leon Yen","Nathan Melehan"] published: 2024-11-18 keywords: ['list','of','keywords','and key phrases'] license: '[CC BY-ND 4.0](https://creativecommons.org/licenses/by-nd/4.0)' --- -This guide describes the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. This guide does not cover migrating operating system disks from Azure Disk Storage. Instead, it focuses on migrating data disks. +This guide describes the process of migrating a single volume from Azure Disk Storage to Linode Block Storage using the rsync file synchronization utility. This guide focuses on migrating data disks from Azure Disk Storage and does not cover migrating operating system disks. ## Block Storage Migration Workflow Diagram ![Azure to Linode Block Storage Migration Workflow Diagram](azure-block-storage-migration-workflow.svg?diagram-description-id=azure-linode-bs-migration) -1. rsync command is run on Linode instance and connects to Azure VM. +1. The rsync command is run from a Linode instance and connects to the Azure virtual machine. -1. Azure VM sends data on Azure data disk to Block Storage Volume attached to Linode instance via established rsync connection. +2. The Azure VM sends data on the Azure data disk to a Block Storage Volume attached to the Linode instance via an established rsync connection. + + 2a. Egress costs for the migrated data are measured when the data leaves the Azure platform. These costs are billed by Azure. - 1. Egress costs for the migrated data are measured when the data leaves the Azure platform. These costs are billed by Azure. {#azure-linode-bs-migration .large-diagram} ## Linode Block Storage vs. Azure Disk Storage -When you first create an Azure Virtual Machine, Azure Disk Storage creates a managed disk for the operating system. One or more managed disks can also be created with the Azure Disk Storage service for temporary files and for persistent data disks. +When an Azure Virtual Machine is first created, Azure Disk Storage creates a managed disk for the operating system. One or more managed disks can also be created with the Azure Disk Storage service for temporary files and for persistent data disks. -Like Azure Disk Storage, Linode Block Storage also provides block-level storage volumes used with VMs. Unlike Azure Disk Storage, Linode Block Storage is generally used for persistent data, not OS/boot disks or temporary data. These other roles are fulfilled by a Linode compute instance's bundled disk, which is stored on the same host as the compute instance. Linode's bundled disk storage is also more suitable for applications that feature high disk operations, like high-traffic databases. +Like Azure Disk Storage, Linode Block Storage also provides block-level storage volumes to be used with virtual machines. Unlike Azure Disk Storage, Linode Block Storage is generally used for persistent data rather than operating system, boot disks, or temporary data. These other roles are fulfilled by a Linode instance's bundled disk, which is stored on the same host as the Compute Instance. Linode's bundled disk storage is also more suitable for applications that feature high disk operations, like high-traffic databases. ## Migration Considerations -The following are important time, cost, and security considerations you should keep in mind when migrating your Azure Disk Storage drives to Linode Block Storage. +The following are important time, cost, and security considerations to keep in mind when migrating your Azure Disk Storage drives to Linode Block Storage. ### Migration Time Estimates @@ -53,9 +54,11 @@ Bandwidth for the transfer can vary according to different factors, including: - Geographic distance between the Azure VM and the Linode instance. - Disk operation limits -When planning your migration, you may consider performing a bandwidth test between the two locations first. Then, use the observed bandwidth from the test to calculate the estimated migration time for the data disk. +When planning your migration, consider performing a bandwidth test between the two locations first. Then, use the observed bandwidth from the test to calculate the estimated migration time for the data disk. + +Utilities like [iperf](https://en.wikipedia.org/wiki/Iperf) can be useful for performing this type of bandwidth measurement. Alternatively, you can create a test file on the Azure VM, migrate it following the [instructions](#block-storage-migration-instructions) in this guide, and then view the bandwidth reported by rsync's output. -Utilities like [iperf](https://en.wikipedia.org/wiki/Iperf) can be useful for performing this type of bandwidth measurement. Alternatively, you could create a test file on the Azure VM, migrate it by following the [instructions](#block-storage-migration-instructions) in this guide, and then view the bandwidth reported by rsync's output. A command that can be used to generate a sample 1GB test file is: +You can use the `dd` command to generate a sample 1GB test file: ```command {title="SSH session with Azure VM"} sudo dd if=/dev/zero of=/datadrive/dummyfile bs=1M count=1024 @@ -63,15 +66,15 @@ sudo dd if=/dev/zero of=/datadrive/dummyfile bs=1M count=1024 ### Migration Egress Costs -The cost to migrate a data disk is a function of the data stored on that disk, which can be substantial for larger migrations. These costs are incurred as egress fees when the data leaves the Azure platform and are billed by Azure. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk. Please review the Azure documentation for assistance with calculating this amount. +The cost to migrate a data disk is a function of the data stored on that disk, which can be substantial for larger migrations. These costs are incurred as egress fees when the data leaves the Azure platform and are billed by Azure. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk, and review [Azure's documentation](https://azure.microsoft.com/en-us/pricing/details/bandwidth/) for assistance with calculating this amount. Inbound traffic sent to your Linode instance and Block Storage volume have no fees incurred on the Linode platform. ### Security and Firewalls -Files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted. +For data security reasons, files should be migrated over an encrypted connection. Rsync supports using SSH as its transport protocol, which is encrypted by default. -Your Azure and Linode firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and Azure virtual machine. +Both your Azure and Linode firewall settings should be configured to allow SSH traffic between the two instances. After the migration is performed, you may wish to close access to SSH between the Linode instance and Azure virtual machine. ## Block Storage Migration Instructions @@ -89,7 +92,7 @@ This guide assumes that you have an Azure virtual machine and an attached data d 1. Follow the [Add volumes](https://techdocs.akamai.com/cloud-computing/docs/manage-block-storage-volumes#add-volumes) product documentation to create and attach a new volume to the Linode instance. This volume should have a capacity equal to or higher than the total data stored on the source Azure disk. Review the [Migration Time Estimates](#migration-time-estimates) section for help with determining how much data is stored on the disk. - When creating the volume, the Linode Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/linode-block-storage-volume`). + When creating the volume, Cloud Manager displays instructions for how to create a filesystem on the new volume and then mount it. Make a note of the filesystem path that it is mounted under (e.g. `/mnt/linode-block-storage-volume`). ### Configure Firewalls @@ -101,17 +104,17 @@ Linux distributions (on both Linode instances and Azure virtual machines) can ha - [How to Configure a Firewall with UFW](/docs/guides/configure-firewall-with-ufw/) - [A Tutorial for Controlling Network Traffic with iptables](/docs/guides/control-network-traffic-with-iptables/) -You may also configure cloud firewalls to control traffic before it arrives at your computing instance. The [Linode Cloud Firewalls](https://techdocs.akamai.com/cloud-computing/docs/cloud-firewall/) product documentation describes how to configure these rules. The [Comparing Cloud Firewalls to Linux firewall software](https://techdocs.akamai.com/cloud-computing/docs/comparing-cloud-firewalls-to-linux-firewall-software) guide further describes the difference between network firewalls and software firewalls. Microsoft Azure's product documentation describes how to configure Azure network firewalls. +You may also configure Cloud Firewalls to control traffic before it arrives at your computing instance. Our [Cloud Firewall](https://techdocs.akamai.com/cloud-computing/docs/cloud-firewall/) product documentation describes how to configure these rules. The [Comparing Cloud Firewalls to Linux firewall software](https://techdocs.akamai.com/cloud-computing/docs/comparing-cloud-firewalls-to-linux-firewall-software) guide further describes the difference between network firewalls and software firewalls. [Microsoft Azure's product documentation](https://learn.microsoft.com/en-us/azure/firewall/overview) describes how to configure Azure network firewalls. ### Configure SSH Key Pair -This guide uses SSH public key authentication for the rsync connection. You must have a public-private key pair installed on the Linode instance and Azure virtual machine. The [Generate an SSH Key Pair](/docs/guides/use-public-key-authentication-with-ssh/#generate-an-ssh-key-pair) section of the [SSH Public Key Authentication](/docs/guides/use-public-key-authentication-with-ssh/) guide describes how to create a key pair. +This guide uses SSH public key authentication for the rsync connection. You must have a public and private key pair installed on the Linode instance and Azure virtual machine. The [Generate an SSH Key Pair](/docs/guides/use-public-key-authentication-with-ssh/#generate-an-ssh-key-pair) section of the [SSH Public Key Authentication](/docs/guides/use-public-key-authentication-with-ssh/) guide describes how to create and use a key pair. -This guide assumes the public and private keys are named `id_rsa.pub` and `id_rsa`, but you may wish to choose a different name. +This guide assumes the public and private keys are named `id_rsa.pub` and `id_rsa`, but your keys may have different names depending on the type of key pair you are using. -- The *public key* should be uploaded to the Azure virtual machine. It should be appended to a new line of the `authorized_keys` file of the user on the Azure VM (e.g. `/home/azureuser/.ssh/authorized_keys). +- The *public key* should be uploaded to the Azure virtual machine. It should be appended to a new line of the `authorized_keys` file of the user on the Azure VM (e.g. `/home/azureuser/.ssh/authorized_keys`). -- The *private key* should be uploaded to the Linode instance. It should be uploaded to the `.ssh/` directory of the user on the Linode instance (e.g. `/home/linodeuser/.ssh/`). It should have permissions set to `600`: +- The *private key* should be located on the Linode instance. It should be uploaded to the `.ssh/` directory of the user on the Linode instance (e.g. `/home/linodeuser/.ssh/`) and have permissions set to `600`: ```command {title="SSH session with Linode instance"} chmod 600 /home/linodeuser/.ssh/id_rsa/ @@ -127,7 +130,9 @@ These instructions implement two recommended practices: Migrations can take a long time, so having them run independently of your SSH session is important. This guide uses `tmux` to create a terminal session that persists between SSH connections. By sending output and errors to log files, you can keep a record of any migration failures that may happen. -1. Install `tmux`: [Installing tmux](https://github.com/tmux/tmux/wiki/Installing#installing-tmux). +Review our [tmux guide](/docs/guides/persistent-terminal-sessions-with-tmux/) for help with other tmux commands. + +1. Install the `tmux` utility on your Linode instance using the official tmux instructions: [Installing tmux](https://github.com/tmux/tmux/wiki/Installing#installing-tmux). 1. Create a new tmux session named `block-storage-migration`. This session is used to initiate the migration: @@ -137,31 +142,33 @@ Migrations can take a long time, so having them run independently of your SSH se After running this command, the tmux session is immediately activated in your terminal. -1. Run the following commands to start migrating the contents of your Azure data disk to your Linode block storage volume: +1. Run the following commands to start migrating the contents of your Azure data disk to your Linode Block Storage Volume: ```command {title="SSH session with Linode instance (bs-migration tmux session)"} echo "\n\nInitiating migration $(date)\n---" | tee -a bs-migration-logs.txt bs-migration-errors.txt >/dev/null - rsync -chavzP --stats -e "ssh -i /home/linodeuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt + rsync -chavzP --stats -e "ssh -i /home/linodeuser/.ssh/id_rsa" {{< placeholder "azureuser" >}}@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume 1>>~/bs-migration-logs.txt 2>>~/bs-migration-errors.txt ``` - Be sure to replace these values with the actual values from your Azure VM and Linode instance: + Replace the following values with the actual values from your Azure VM and Linode instance: - - `/home/linodeuser/.ssh/id_rsa`: The name and location of the private key - - `azureuser`: The name of the user on the Azure VM - - `AZURE_VM_IP`: The IP address of the Azure VM - - `/datadrive/`: The directory the Azure data disk is mounted under - - `/mnt/linode-block-storage-volume`: The directory your Linode volume is mounted under + - `/home/linodeuser/.ssh/id_rsa`: The name and location of the private key on your Linode instance + - `{{< placeholder "azureuser" >}}`: The name of the user on the Azure VM + - `{{< placeholder "AZURE_VM_IP" >}}`: The IP address of the Azure VM + - `/datadrive/`: The directory under which the Azure data disk is mounted + - `/mnt/linode-block-storage-volume`: The directory under which your Linode volume is mounted {{< note >}} You may be prompted to accept the host key of the Azure VM if it is the first time that the Linode has connected to it. {{< /note >}} - The first `echo` appends a message to the log files. Here's a detailed explanation of the key flags and parameters used in the `rsync` command: + **Command breakdown**: + + The first `echo` appends a message to the log files. Below is a detailed explanation of the key flags and parameters used in the `rsync` command: - - `-c`: Tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but -c forces it to compute checksums, which is slower but can be more accurate if you want to be sure that files match exactly. + - `-c`: Tells rsync to use checksum comparison for file differences. Normally, rsync uses file size and modification time to decide if files need to be updated, but `-c` forces it to compute checksums, which is slower but can be more accurate if you want to be sure that files match exactly. - - `-h`: Human-readable output, which makes file sizes (like transfer statistics) easier to read by using units like KB and MB, rather than raw byte counts. + - `-h`: Human-readable output, which makes file sizes like transfer statistics easier to read by using units like KB and MB, rather than raw byte counts. - `-a`: Archive mode. This is equivalent to specifying: `-rlptgoD`. The result of the `-a` flag is a complete, near-exact copy of the source directory: @@ -179,7 +186,7 @@ Migrations can take a long time, so having them run independently of your SSH se - `-D`: Retain device files and special files - - `-v`: Verbose mode. This makes rsync output more detailed information about what it is doing, which is helpful for monitoring the progress of a large transfer or troubleshooting. + - `-v`: Verbose mode. This makes rsync output more detailed information about what it is doing, and can be helpful for monitoring the progress of a large transfer or troubleshooting. - `-z`: Compression. This enables compression during data transfer, which can save bandwidth and speed up the transfer if the network connection is relatively slow. @@ -193,11 +200,11 @@ Migrations can take a long time, so having them run independently of your SSH se - `-e "ssh -i /home/linodeuser/.ssh/id_rsa"`: Specifies a remote shell (SSH) with an identity key file for authentication. - - `azureuser@AZURE_VM_IP:/datadrive/`: This specifies the source directory you're syncing from: + - `{{< placeholder "azureuser" >}}@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/`: This specifies the source directory you're syncing from: - - `azureuser`: The username on the remote server. + - `{{< placeholder "azureuser" >}}`: The username on the remote server. - - `AZURE_VM_IP`: The IP address of the remote server. + - `{{< placeholder "AZURE_VM_IP" >}}`: The IP address of the remote server. - `/datadrive/`: The path on the remote server that you want to sync. The trailing slash (/) means rsync will copy the contents of /datadrive, rather than creating a /datadrive directory in the target. @@ -207,7 +214,7 @@ Migrations can take a long time, so having them run independently of your SSH se Because the stdout and stderr streams were redirected to log files, the rsync command will not produce output in the terminal. Follow these steps to inspect and monitor the contents of the logs: -1. To avoid interrupting the rsync process, *detach* from the tmux session by entering this sequence of keystrokes: `CTRL-b` followed by `d`. You are returned to the SSH session that created the tmux session: +1. To avoid interrupting the rsync process, *detach* from the tmux session by entering this sequence of keystrokes: Ctrl + B followed by D. You are returned to the SSH session that created the tmux session: ```output [detached (from session block-storage-migration)] @@ -223,7 +230,7 @@ Because the stdout and stderr streams were redirected to log files, the rsync co tail -f block-storage-migration-errors.txt ``` - Enter `CTRL-c` to stop `tail`. + Enter Ctrl + C to stop `tail`. 1. You can re-enter the tmux session with the `tmux attach` command: @@ -231,16 +238,12 @@ Because the stdout and stderr streams were redirected to log files, the rsync co tmux attach -t block-storage-migration ``` - {{< note >}} - Review the [tmux guide](/docs/guides/persistent-terminal-sessions-with-tmux/) for help with other tmux commands. - {{< /note >}} - ### Verify the Migration -To verify that rsync has synced all the files as expected, re-run the command with the `--dry-run –stats` flags: +To verify that rsync has synced all the files as expected, re-run the `rsync` command with the `--dry-run –stats` flags, replacing the same values as before: ```command {title="SSH session with Linode instance"} -rsync -chavzP --stats --dry-run -e "ssh -i /home/azureuser/.ssh/id_rsa" azureuser@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume +rsync -chavzP --stats --dry-run -e "ssh -i /home/azureuser/.ssh/id_rsa" {{< placeholder "azureuser" >}}@{{< placeholder "AZURE_VM_IP" >}}:/datadrive/ /mnt/linode-block-storage-volume ``` If the output displays files yet to be transferred, then rsync did not fully replicate the files in the destination directory. A previous successful rsync transfer should result in the following output. Note that the number of created, deleted, and transferred files are zero: From bf7fd1365229f02acb65befd5b9eb7ff183f0f2b Mon Sep 17 00:00:00 2001 From: John Dutton Date: Thu, 12 Dec 2024 14:47:32 -0500 Subject: [PATCH 12/13] typo fix --- .../index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md index f5726c5ba39..d578ed9cbe9 100644 --- a/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md +++ b/docs/guides/platform/migrate-to-linode/migrate-from-azure-disk-storage-to-linode-block-storage/index.md @@ -271,7 +271,7 @@ total size is 10.49M speedup is 91,180.52 (DRY RUN) ### Cleanup after Migration -After the migration is complete, you may determine that the Azure VM and Linode instance no logner need to communicate. You can close traffic between these servers by doing the following: +After the migration is complete, you may determine that the Azure VM and Linode instance no longer need to communicate. You can close traffic between these servers by doing the following: - Remove the firewall access granted in the [Configure Firewalls](#configure-firewalls) section From d2365ad4d213e91fde4b6ed69dd790d8751e6599 Mon Sep 17 00:00:00 2001 From: Nathan Melehan Date: Tue, 17 Dec 2024 13:45:01 -0500 Subject: [PATCH 13/13] Remove failover guide from PR --- .../configure-failover-for-haproxy-on-akamai/index.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 docs/guides/uptime/loadbalancing/configure-failover-for-haproxy-on-akamai/index.md diff --git a/docs/guides/uptime/loadbalancing/configure-failover-for-haproxy-on-akamai/index.md b/docs/guides/uptime/loadbalancing/configure-failover-for-haproxy-on-akamai/index.md deleted file mode 100644 index e69de29bb2d..00000000000